cURLing Android Market stats in my website

cURLing Android Market stats in my website

Last week I thought it would be nice to collect some statistics about my apps in the Android Market. Seeing Websites like androlib.com and androidpit.de I thought it shouldn’t be a problem. However, apart from strazzere.com I haven’t found much useful information. Since I’m only interested in the stats of my own apps I took a deeper look on Google’s Developer Console for the Android Market.

I played a bit with Firebug and learned that the Developer Console is a GWT application, that JSON is used to get the app descriptions from the server, and that the GWT stuff is horrible to reverse engineer. Luckily I found a post that shows how to get the stats from the Android Marketplace with PHP/cURL. It didn’t worked for me at first but after toying around a bit it works now for me. However, I still have no clue what the JSON stuff I get from the GWT server means and I’m only guessing the most important values. It will likely break when I change anything in my developer account.

Below is the PHP script I use to fetch the data from the developer console. 99% is copied from Craige Thomas (I have absolutely no clue about PHP or cURL!). I only added the part that guesses the position of the values and the caching. The script is used to produce the output of the widget on the right.

<?php
if ((!file_exists('market_stats.txt')) || (time()-filemtime('market_stats.txt')>60)) {
 
	//do google authorization
 
	$data = array('accountType' => 'GOOGLE',
	'Email' => 'YOUR GOOGLE LOGIN'
	'Passwd' => 'YOUR PASSWORD',
	'source'=>'',
	'service'=>'androiddeveloper');
 
	$ch = curl_init();
	curl_setopt($ch, CURLOPT_URL, "https://www.google.com/accounts/ClientLogin");
	curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
	curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
	curl_setopt($ch, CURLOPT_POST, true);
	curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
	curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
 
	$output = curl_exec($ch);
 
	$info = curl_getinfo($ch);
	curl_close($ch);
 
	//grab the AUTH token for later
 
	$auth = '';
	if($info['http_code'] == 200) {
		preg_match('/Auth=(.*)/', $output, $matches);
 
		if(isset($matches[1])) {
			$auth = $matches[1];
		}
	}
 
	//login to Android Market
	//this results in a 302
	//I think this is necessary for a cookie to be set
 
	$ch = curl_init ("http://market.android.com/publish?auth=$auth");
	curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookies.txt');
	curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
	$output = curl_exec($ch);
 
	//go to the Developer Console
	$ch = curl_init ("http://market.android.com/publish/Home");
	curl_setopt($ch, CURLOPT_COOKIEFILE, $ckfile);
	curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
	curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookies.txt');
	$output = curl_exec($ch);
 
	//grab the JSON data
	$perm = "746E1BE622B08CBF950F619C16FCFF1E";
	$headers = array(
		"Host: market.android.com",
		"User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; de; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2",
		"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
		"Accept-Language: de-de,de;q=0.8,en-us;q=0.5,en;q=0.3",
		"Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7",
		"Keep-Alive: 115",
		"Connection: keep-alive",
		"Content-Type: text/x-gwt-rpc; charset=utf-8",
		"X-GWT-Permutation: $perm",
		"X-GWT-Module-Base: http://market.android.com/publish/gwt/",
		"Referer: http://market.android.com/publish/gwt/$perm.cache.html");
 
	//not sure what x-gwt-permutation means, I think it may have to do with which version of GWT they serve based on your browser
 
//Change here?
	$postdata = "5|0|4|http://market.android.com/publish/gwt/|14E1D06A04411C8FE46E62317C1AF191|com.google.wireless.android.vending.developer.shared.AppEditorService|getFullAssetInfosForUser|1|2|3|4|0|";
 
	$ch = curl_init ("http://market.android.com/publish/editapp");
	curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
	curl_setopt($ch, CURLOPT_POST, 1);
	curl_setopt($ch, CURLOPT_POSTFIELDS, $postdata);
	curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookies.txt');
	curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
 
	$output = curl_exec($ch);
 
	$output = substr($output,4);
 
	$json = json_decode($output);
	$csv = explode(',',$output);
 
	$apps = array();
 
	$index = 0;
	$app_count = 0;
	for($i = 0; $i < sizeof($csv); ++$i) {
		if (is_array($json[$i])) {
			$innerArray=$json[$i];
			break;
		}
		if (strpos($csv[$i], ".") !== false) {
			if ($index==1) $apps[$app_count][comments]=$csv[$i];
			else if ($index==2) $apps[$app_count][rating]=$csv[$i];
			else if ($index==4) $apps[$app_count][installs]=$csv[$i];
			else if ($index==6) $apps[$app_count][total]=$csv[$i];
			$index++;
			if ($index==7) {
				$index=0;
				$app_count++;
			}
		}
	}
	for($i = 4; $i < sizeof($innerArray); ++$i) {
		if ((substr($innerArray[$i],-1)=='k') && (substr($innerArray[$i-2],0,17)=='GetImage?imageId=')){
			$app_count--;
			$apps[$app_count][icon]="http://market.android.com/publish/".$innerArray[$i-2];
			$apps[$app_count][name]=$innerArray[$i-3];
			$apps[$app_count][size]=$innerArray[$i];
			$apps[$app_count][package]=$innerArray[$i-1];
		}
	}
 
	$Handle = fopen('market_stats.txt', 'w');
	fwrite($Handle, '<table width="100%">');
	for($i = 0; $i < sizeof($apps); $i++) {
		fwrite($Handle, "<tr><td>");
		fwrite($Handle, '<img width="48" height="48" src="'.$apps[$i][icon].'"/>');
		fwrite($Handle, "</td><td>");
		fwrite($Handle, '<a href="http://www.androidpit.de/android/de/de/market/apps/app/'.$apps[$i][package].'">');
		fwrite($Handle, $apps[$i][name].'</a><br/>');
		fwrite($Handle, 'Total installs '.round($apps[$i][total]));
		fwrite($Handle, "</td></tr>");
	}
	fwrite($Handle, '</table>');
	fclose($Handle);
 
	$Handle = fopen('market_stats_history.txt', 'a');
	for($i = 0; $i < sizeof($apps); $i++) {
		fwrite($Handle, $apps[$i][package].', ');
		fwrite($Handle, round($apps[$i][total]).', ');
		fwrite($Handle, time());
		fwrite($Handle, "\n");
	}
	fclose($Handle);
}
$readHandle = fopen('market_stats.txt', 'r');
echo fread($readHandle, filesize('market_stats.txt'));
fclose($readHandle);
?>

10 Responses

  1. Unfortunately there is a problem with the code. Once in a while Google changes “something”. Afterwards, my script does not work anymore. The suspicious line is
    $postdata = “5|0|4|http://market.android.com/publish/gwt/|8962DA36A8613A353E193A7B0C216D42|com.google.wireless.android.vending.developer.shared.AppEditorService|getFullAssetInfosForUser|1|2|3|4|0|”;

    There is a magic number included and I have no idea how I can get this number using PHP. I use the Firefox plug-in Firebug to look at the “editapp” request. Then I just copy the post data and it works again. If you know how to fix it I wouldn’t mind to learn about it :-)

  2. Great script, working for me.

    I’m going to modify it to timestamp each value and then have it do the math to return just the # of downloads that day vs. total downloads.

    If you’re interested in my changes, let me know.

  3. Great script, working for me.

    I’m going to modify it to timestamp each value and then have it do the math to return just the # of downloads that day vs. total downloads.

    If you’re interested in my changes, let me know.

  4. Hi Michelle & Brad,

    I would be interested in any improvement :-) The problem with the code is the line below “//Change here?”. I have to update the magic number (in the code it is “14E1D06A04411C8FE46E62317C1AF191″) every now and then because something changed in the developer console. Unfortunately I have no idea where to get this number from.

    Niels

  5. i tried above script but not able to execute it sucesffuly does google has changed their API?

    Thanks,