The shared token that switched off the website’s leads

On 15 Oct, leads from AHA’s website stopped reaching Zoho CRM for about 30 minutes. The cause was a key shared by about nine systems, the BI I was building among them.

Two kinds of token

Zoho’s API uses OAuth, which works with two kinds of token. (I covered the basic swap in an earlier post on lead capture.)

• A refresh token is the long-lived key. It can’t read anything by itself; its one job is to be swapped for an access token.
• An access token is what every API call carries. It lasts an hour, and then you swap again.

The catch is a limit that rarely bites when each system has its own key: Zoho keeps only about 10 live access tokens per refresh token. Mint one more and the oldest is cancelled, whoever happens to be using it.

One key, many systems

Here, one refresh token sat behind the website’s lead server, the BI’s pullers and about seven other systems. Each minted access tokens on its own schedule, and none of them knew about the others.

So any of them could knock out another. A fresh token for one system could cancel a token another was still using, and that system’s next call to Zoho would be refused. When the system that lost its token was the website’s lead server, enquiries stopped reaching the CRM. On 15 Oct that lasted about 30 minutes.

The fix

The fix on 16 Oct, built with Claude Code, had four parts.

Its own refresh token. The BI got a second refresh token from the same Zoho app, with the same permissions. That gives it its own pool of access tokens: nothing the BI mints can cancel the website’s token, and nothing on the old key can cancel the BI’s.

Leave the old key alone. The original refresh token stays live for every system still using it. Revoking it to tidy up would have broken all of them at once.

One access token an hour. The first puller of the hour mints a token and caches it with its expiry. Every other puller reuses it until two minutes before it runs out. The cache also records which refresh token made it, so a key change can’t leave an old token in use.

Retry once on a refused token. A cached token can be cancelled before it expires, so the code has to notice. When Zoho refuses a token, the call gets one fresh token and one retry. If another puller has renewed it in the meantime, it uses that one instead of minting again.

# Simplified: one access token an hour, shared by every puller.
def access_token():
    c = read_cache()                    # token, expiry, which key minted it
    if c and c.key == MY_KEY and c.seconds_left > 120:
        return c.token
    token = mint(MY_KEY)                # the only place a token is created
    write_cache(token)
    return token

def zoho_get(path):
    r = call(path, access_token())
    if r.token_refused:                 # expired or cancelled
        r = call(path, renew_once())    # one retry, never a loop
    return r

Where it stands

The BI’s pullers now share one access token an hour, minted from a refresh token of their own. The other systems still share the original key. Moving each onto its own is the same fix, one system at a time.

What I’d tell anyone wiring up OAuth

1. One refresh token per system. Each gets its own pool of access tokens, and its own failures.
2. Shared keys couple outages. Two systems on one key can break each other with no bug in either.
3. Mint rarely, cache with a margin. An access token lasts an hour, so use it for the hour.
4. Expect cancellation. A cached token can die early. Retry once with a fresh one, then fail loudly.
5. Don’t revoke a shared key to fix it. Move yourself off it and leave the rest running.