Recently I’ve faced the problem of populating different statistics from iTunesConnect portal. Mostly I was intrested in getting full downloading statistics. The iTunesConnect portal provides user with ability of viewing and downloading weekly sales reports only for last three monthes and daily reports only for last month which means the oldest stat is simply getting lost. The solution is obvious: we need to create some mechanism which would populate data and save it to database everyday. After implementing main routine we can enjoy with making different decorations such as visualizing and analytics. Besides, we can easily save current application states which can be useful if some of apps are in state of Wating for review. 
Thinking about the problem, I’ve been researching if there some compete solutions exists, but unsusccessfully, looks like nobody cares about it. There are lot of native iOS apps that allow to observe you app right from the device. Easy? Agrre, but for some reasons I’m not happy with that solution. First of all because of local data storage. I prefer to store data centralized and to have full control. Besides, I don’t like the restrictions of third-party applications which you can’t remove. Also these apps can’t provide sales reports data ingerity because in order to provide it you should run the app everyday – in iOS no background jobs are permitted.

Well, let it be. Let’s produce our automatic iTunesConnect scraper. Scraping is the process of collecting data from public source such as web site. Different search engines bots (like google bots) which do indexing of all world wide web can be represented as an example of scraper. Our scraper will index only one site – http://itunesconnect.apple.com.

Looking ahead, the result can be presented as follows (the image is clickable):

There is no public itunesconnect API due to some political reasons so we’ll have to reproduce all user’s actions perfomed in web browser, such as authorization, clicking certain links, saving downloaded reports.

In fact, how it can be implemented? There are two ways of doing this. If the site is based on classic HTTP GET/POST it can be achived by analyzing web pages content. But there are less classic sites today, moreover itunesconnect doesn’t belong to them. Nowadays javascript and ajax rule. Besides, here comes HTML5 epoch.

It’s very hard to analyze all the ajax mechanics – learning a dozens miles of javascirpt code is not for nervous people. In this situation http sniffing can be much more helpful. Let’s take our favorite sniffer tool and see what browser transfers to server and what server is responding when user do certain actions. All that’s left to do is reproduce this process programmatically. iTunesConnect has modern web2.0 interface and that’s why it doesn’t make sense to analyze its javascript part. From other side http sniffing is not useful for 100% because the portal uses HTTPS (secured) protocol and all data is being encrypted. All we can see in sniffer tool is some binary garbage. What can be done? The solution was found by me in one of my less favorite browser FireFox. There is a beautiful plugin which calls temperdata.

I don’t want to describe in details the entire source, I hope it’s well enough commented. You can download PHP iTunes scraper here (this time I decided to upload sources to google repository)

For applications states and reports storage we need to create two tables:

CREATE TABLE  `apps`.`itc_apps` (
  `Title` varchar(255) NOT NULL DEFAULT '',
  `SKU` varchar(255) NOT NULL DEFAULT '',
  `BundleId` varchar(255) NOT NULL DEFAULT '',
  `AppleId` varchar(64) NOT NULL DEFAULT '',
  `AppType` varchar(64) NOT NULL DEFAULT '',
  `DefaultLanguage` varchar(64) NOT NULL DEFAULT '',
  `AppstoreLink` varchar(255) NOT NULL DEFAULT '',
  `cver_version` varchar(16) DEFAULT NULL,
  `cver_status` varchar(32) DEFAULT NULL,
  `cver_status_color` varchar(16) DEFAULT NULL,
  `cver_date_created` int(11) DEFAULT NULL,
  `cver_date_released` int(11) DEFAULT NULL,
  `nver_version` varchar(16) DEFAULT NULL,
  `nver_status` varchar(32) DEFAULT NULL,
  `nver_status_color` varchar(16) DEFAULT NULL,
  `nver_date_created` int(11) DEFAULT NULL,
  `nver_date_released` int(11) DEFAULT NULL,
  `creation_date` int(11) NOT NULL DEFAULT '0',
  `update_date` int(11) NOT NULL DEFAULT '0',
  PRIMARY KEY (`Title`,`BundleId`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8

CREATE TABLE  `apps`.`itc_sales` (
  `Provider` varchar(255) NOT NULL DEFAULT '',
  `ProviderCountry` varchar(255) NOT NULL DEFAULT '',
  `SKU` varchar(255) NOT NULL DEFAULT '',
  `Developer` varchar(255) NOT NULL DEFAULT '',
  `Title` varchar(255) NOT NULL DEFAULT '',
  `Version` varchar(255) NOT NULL DEFAULT '',
  `ProductTypeIdentifier` varchar(255) NOT NULL DEFAULT '',
  `Units` int(11) NOT NULL DEFAULT '0',
  `DeveloperProceeds` float NOT NULL DEFAULT '0',
  `BeginDate` int(11) NOT NULL DEFAULT '0',
  `EndDate` int(11) NOT NULL DEFAULT '0',
  `CustomerCurrency` varchar(255) NOT NULL DEFAULT '',
  `CountryCode` varchar(255) NOT NULL DEFAULT '',
  `CurrencyOfProceeds` varchar(255) NOT NULL DEFAULT '',
  `AppleIdentifier` varchar(255) NOT NULL DEFAULT '',
  `CustomerPrice` float NOT NULL DEFAULT '0',
  `PromoCode` varchar(255) DEFAULT NULL,
  `ParentIdentifier` varchar(255) DEFAULT NULL,
  `Subscription` varchar(255) DEFAULT NULL,
  `Period` varchar(255) DEFAULT NULL,
  `creation_date` int(11) NOT NULL DEFAULT '0',
  `update_date` int(11) NOT NULL DEFAULT '0'
) ENGINE=MyISAM DEFAULT CHARSET=utf8

itc_scraper.php script performs data collecting. In order to use it you need to configure config.php file. There you can specify such parameters as database credentials, iTunes logins list, the list of applications you’d like to monitor. The script also creates on the disk agregated data snapshot. The information is stored in `meta` folder in form of serialized php-objects so there is no need to establish db-connection each time the script is accessed by user browser.  Неrе is an example of how I designed the data visualization. This script uses only data from `meta` folder.

P.S. Now I’m thinking about composing simple iOS-application which will do server request and display recevied data. There is no need to create some complecated parse routines in Objective-C, we can receive complete html code and display it in standard UIWebView component. We can also improve this idea e.g. by implementing the concept of repositories. For instance, we could keep the stat from different sets of iTunes logins. That could be useful, e.g. if you have your personal iTunes account and account of your employer – I wouldn’t like to show my data to some sales guys))