Android & iOS screenshot fetcher

November 2012

Here's a python script that I started a long time ago to fetch screenshots of the top ranking Android and iPhone app shots from the App stores. I had a vision that I would make a public site that lists the screen shots and design patterns of the best mobile apps, a listing that could provide inspiration and learning points for mobile developers and designers.

Fast forward months and I realize that I haven't had the time to start this project. Been busy with customer projects. However, some nice people have actually created sites with lists of mobile screenshots. Take a look of these nice sites:

These sites are good starting points for people looking for design inspiration. Be sure also to read:

Anyway, I'm still publishing my screenshot fetcher script, and the montage of shots I just created of the top Android and iOS apps. Enjoy.

Top free apps in Finland, November 2012

iPhone top 50

3.5MB

iPad top 50

2.0MB

Android top 24

1.7MB

Fetching

Both Google Play and App Store provide pages that list the top ranking free and paid apps in the store. The script fetches these listing pages and then scrapes the individual app pages for the screenshots. App Store provides both iPhone and iPad screenshots separately, Google Play does not differentiate phone vs tablet shots.

The script uses urllib2 from the standard python library to fetch the HTML pages from the app stores. lxml is used for the scraping of HTML of the page. lxml is a wrapper lib to the great C library libxml2. A few years ago I tested some python xml libs and found lxml to be fastest. Been using that since.

Having been exposed to XPath queries earlier, I'm using XPath to find the particular DOM elements from the HTML. lxml also provides lxml.cssselect module, which provides CSS alike selectors for finding elements. To find out what elements to find for in the pages, I used the web inspector in Chrome.

Google Play seems to be able to resize the screenshots upon fetch, see the height parameter in the script. To get the original size of the shot, I'm using a large height, 2000 pixels.

The script

The script takes a single argument "android" or "ios" to determine which app store to fetch shots from. The downloaded shots will appear under a sub folder "android", "iphone" or "ipad".

Here's the complete python script.

#!/usr/bin/python
# -*- coding: utf-8 -*-

# screenshot-fetcher.py: fetch android/ios app screenshots from app stores
#
# Author: Tomi.Mickelsson@iki.fi

import os, sys
from lxml.html import parse
from urllib2 import urlopen

def fetch_ios_top_list(free_apps=True):
    " Fetch list of top ios apps "

    if free_apps:
        url = "http://www.apple.com/itunes/charts/free-apps/"
    else:
        url = "http://www.apple.com/itunes/charts/paid-apps/"

    doc = parse(url).getroot()
    grid = doc.get_element_by_id("grid")
    return [x.find("a").get("href") for x in grid.iter("li")]

def fetch_ios_app_shots(rank, url):
    " Fetch all screenshots of this iphone/ipad app "

    print "fetch app page", url

    doc = parse(urlopen(url)).getroot()

    # get title
    title = doc.xpath("//div[@id='title']//h1")[0].text
    print "TITLE", title

    # get iphone shots
    iphoneimg = doc.xpath("//div[contains(@class, 'iphone-screen-shots')]//img")
    for i, img in enumerate(iphoneimg):
        url = img.get("src")
#         if "landscape" in img.get("class"):
#             fname = "L-"+fname
        save_img(url, "iphone", rank, title, i)

    # get ipad shots
    ipadimg = doc.xpath("//div[contains(@class, 'ipad-screen-shots')]//img")
    for i, img in enumerate(ipadimg):
        url = img.get("src")
        save_img(url, "ipad", rank, title, i)


def fetch_android_top_list(free_apps=True):
    " Fetch list of app urls of top android apps "

    if free_apps:
        url = "https://play.google.com/store/apps/collection/topselling_free"
    else:
        url = "https://play.google.com/store/apps/collection/topselling_paid"

    # 2nd page:
    # https://play.google.com/store/apps/collection/topselling_free?start=24&num=24

    doc = parse(urlopen(url)).getroot()
    li = doc.xpath("//ul[contains(@class, 'snippet-list')]//li//a[contains(@class, 'thumbnail')]")
    return [x.get("href") for x in li]


def fetch_android_app_shots(rank, url):
    " Fetch all screenshots of this iphone/ipad app "

    if not url.startswith("http"):
        url = "https://play.google.com/" + url

    doc = parse(urlopen(url)).getroot()

    # get title
    title = doc.xpath("//h1[contains(@class, 'doc-banner-title')]")
    if title:
        title = title[0].text
    print "TITLE", title

    # get shots
    li = doc.xpath("//img[contains(@class, 'doc-screenshot-img')]")
    imglist = [x.get("src") for x in li]

    for i, url in enumerate(imglist):
        url = url.replace("h230", "h2000") # height of image
        save_img(url, "android", rank, title, i)


def save_img(url, dir, rank, title, imgindex):
    " Fetch an image from the url to a file "

    fname = "%s/%03d.%s_%d.jpg" % (dir, rank, title[:20], imgindex)
    fname = fname.replace(" ", "_")
    print "  ",fname

    f = urlopen(url)
    data = f.read()
    f.close()

    dest = file(fname, "wb")
    dest.write(data)
    dest.close()


def main():
    ostype = sys.argv[-1] # last arg

    # create dirs for downloaded screenshots
    dirs = ["android", "iphone", "ipad"]
    for d in dirs:
        if not os.path.exists(d):
            os.mkdir(d)

    free_apps = True

    if ostype == "android":
        # fetch android shots
        urllist = fetch_android_top_list(free_apps)
        print "android app count:", len(urllist)

        for i, url in enumerate(urllist):
            fetch_android_app_shots(i+1, url)
    elif ostype == "ios":
        # fetch ios shots
        urllist = fetch_ios_top_list(free_apps)[:50]
        print "ios app count:", len(urllist)

        for i, url in enumerate(urllist):
            fetch_ios_app_shots(i+1, url)
    else:
        print "unknown os"


if __name__ == '__main__':
    main()

Remember to install the lxml library:

sudo easy_install lxml

ImageMagick montage

ImageMagick is a wonderful package of command line tools for versatile manipulation of images. For the creation of the montage of screenshots, I'm using the tool named montage:

montage -label %f -tile 5x -title 'iPhone top 50 apps - Nov 2012' \
  -pointsize 11 -font Helvetica -geometry 160x240+4+8 \
  -background '#ddddff'  * ../iphone50.jpg

This single invocation of montage generates the whole image including sorting shots based on name, labeling shots, having a layout of 5 columns, and resizing shots to 160x240. Very powerful.

To make a smaller final image (to save bandwidth), I only include the 50 of the top 100 iOS apps. Likewise, I resize the images to smaller size because I want to save bandwidth of my site. You can leave the -geometry option out to have the full size shots in your own montage.

Back