Mapping a whole market when the API stops at 60 results

Some research questions sound like an afternoon of googling: “who are all the automation companies in India?”. Then you meet the API limits.

Limit 1: 60 results, then nothing

A Places text search returns at most 60 results, however many exist. Ask for “home automation in Mumbai” and you get 60. There are far more.

The fix is a quadtree. Search a square of the map. If it returns results, it may be hiding more, so split it into four smaller squares and search each. Keep splitting until the squares are small: about 1.4 km in Mumbai and 3–6 km elsewhere. Results stream to a file as they arrive, so a stopped run resumes where it left off.

def sweep(cell, term):
    hits = search(term, bounds=cell)      # ≤ 60 place IDs
    save(hits)
    if hits and cell.size_km > MIN_KM[cell.region]:
        for child in cell.split4():
            sweep(child, term)

Limit 2: details cost money

Asking for names, websites and phone numbers on every request would have blown the budget. So the sweep asks for place IDs only, the cheapest kind of request. Only later does fetch.py buy details. It picks the smallest set of paid searches that covers the most IDs we want (a greedy set cover), then looks up the leftovers one by one. There’s a dry-run plan, a per-tier request counter and a hard budget cap.

Limit 3: India is big

Running 13 keywords over every square in India would cost far too much. So I ran Mumbai first, thoroughly, and checked which keywords found firms no other keyword found. Six of the thirteen found almost everything, and those six ran nationally. (“Smart home” turned out to return mostly electronics shops.) An exclusion list of 66 circles kept the sweep out of neighbouring countries, with a country check on every result as a backstop.

Cleaning: most results aren’t what you asked for

“Automation” returns office-copier dealers, RPA software firms, training institutes and real-estate agents. Classification uses name rules, Google’s own categories and keyword scoring of each firm’s own website. Anything ambiguous goes to a “possible matches” sheet instead of being silently dropped. City names written in Marathi, Hindi, Tamil and other scripts are mapped to English, duplicates are merged, and tracking parameters are stripped from website links.

Result

• Mumbai: 7,210 map squares searched, 5,911 unique places, 1,176 firms listed (192 of them in home automation) plus 787 possible matches.
• India: 36,211 squares, 78,624 unique places, and 11,007 firms listed outside Mumbai.

The research itself stays internal, and Google’s terms limit what you can do with the stored data anyway. The method is the reusable part: when an API caps your results, split the space and sweep it cheaply before you pay for detail.