Discovering private APIs with Charles.app
Nowadays, so many companies have apps, and they have to get their data from somewhere. This has opened a whole new world of (quasi-)open data.
By using a proxy to intercept the request made by these mobile apps, you get access to often incredible private APIs.
I've built some great stuff this way, from Pug-o-matic to my Rap Genius ruby gem to my latest project, an availability finder for British Airways air mile redemptions.
In this article, I'll explain how to delve into how your favorite app works so you can use its data in new and novel ways.
Installing Charles.app
Most of my readers will be Mac users, and on OS X, I've found Charles.app to be the best tool for the job. It's also available for Windows and Linux.
Click here to download Charles. On a Mac, you'll just need to open the .DMG and drag it into your Applications directory.
Setting up your iPhone
Getting your iPhone ready to use Charles takes just two steps:
- Install the SSL certificate
- Set up the proxy
Most iOS apps will connect to an API with SSL enabled (i.e. over HTTPS). This will stop you from being able to read the data.
Fortunately, there's a way around this. Charles can act as the middleman, presenting its own certificate, reading the data, and then passing it onto the server behind the app you're using. To do this, you'll need to ask iOS to trust Charles's certificate.
Just go to http://charlesproxy.com/charles.crt and click "Install" a couple of times.
An important security advisory: You should think about removing this certificate between uses - otherwise, someone could intercept your requests when you're not expecting it, and you'd be none the wiser. To remove the certificate, go into Settings, then General, then Profiles.
Now, you can set up Charles as a proxy for all your device's HTTP requests. Your iOS device and computer should be connected to the same network to do this.
First, let's get your computer's IP address. You can do this in System Preferences on Mac - go to "Network", choose your interface, and it'll say just under the status.
On your iOS device, open Settings.app, go to WiFi, then click the arrow or "i" icon next to your network's name. Scroll down, set "HTTP proxy" to "Manual", then enter your IP under "Server", and "8888" as the port.
Intercepting requests
Time to open Charles.app! As an example, I'm going to show you how I discovered the API of the (Rap) Genius app. You can download it here.
The left hand side of Charles.app shows domains that have received requests. There'll already be a fair bit there, as Charles automatically configures your Mac to pass its own network requests through it. Open the Genius app, and you'll see a new entry added.
Right click it, and click "SSL Proxying" to tell Charles to spoof the SSL certificate so you can read HTTPS-secured API calls. Expand the tree for the relevant domain (you might have to go through a few, depending on the app), and have a look through.
A really good idea now is to have a click around the app, particularly doing the sorts of things you'd like to do via an API - in the case of the Genius app, I wanted to be able to programatically load annotations for songs, so I looked at a few songs and their annotations.
Once you've done that, it's a good idea to save your session in Charles. This will save you doing the same thing more than once, and it's especially given that Charles, until you've bought a copy, will close after half an hour. Click "File", then "Save As...", and put the .chls file somewhere. You can re-open this any time from the "File" menu.
Understanding the API
In my case, I've opened The Weekend's "Live For" song in the app on my phone. In Charles, I can see that there's been a request.
The "Overview" and "Request" tabs at the top are a good place to start. Under here, you'll obviously care about the URL the application hit.
Depending on how much effort the app creator has gone to to lock down the API, it's worth keeping the "User-Agent", "Accept" and "Content-Type" headers, amongst other things. There might be odd kinds of authentication, access tokens or signing routines which demand trial and error.
You might need to send these with your requests for things to work as expected. Some experimentation is likely to be required!
Once you've worked out how the requests work, you'll want to look at the data that comes back. Go to the "Response" tab, and then most likely you'll want to choose "JSON Text" at the bottom (most APIs these days respond with JSON).
Fundamentally, all you need to replicate the API is to (a) understand how to make a request and (b) get to grips with the JSON format. With this, you can build a basic library, as I've done with APIs from American Express to British Airways to Rap Genius.
A few closing points
When you're working with private APIs, there are a few things you should consider.
They can change at any time
Don't take these things for granted, or use them in production apps - an API without any kind of public documentation or commitment to versioning can change any time without notice, breaking your software.
These sorts of things are safe to use for side projects, but not for much more than that. On the bright side, private APIs are way better than screen scraping!
In an ideal situation, you'd write some sort of automated testing suite to run automatically every now and then to watch for changes in the API.
The company behind the API might not be happy
I've not experienced this personally, but it's quite conceivable that the app developer might not want you to use their private API - I've expected complaints from American Express and Avios, but fortunately haven't got them.
The key here is probably to be flexible - don't get yourself into any legal battles you can't afford. If you get a cease and desist, you should seriously consider taking down what you've built. It probably isn't worth it.
Make your code open source
Whenever I build things using APIs I've found, I open source the code (or at least write an in-depth blog post about it).
This has two benefits: it lets other make use of useful data or utilities, and it's a great way of building your profile and job prospects.
comments powered by Disqus