InventoryAllocation

Caching Access Tokens in Redis with Proactive Refresh

Run a channel-manager sync across eight worker processes and each one holds its own in-memory token, so a single OTA credential gets refreshed eight times an hour instead of once — burning through a per-client token-issuance quota and, on providers that rotate the refresh token per use, occasionally invalidating a session another worker is still holding. This page shows how to hoist the token into Redis so the whole pool shares one cached credential, refresh it proactively before it expires, and guard the refresh with a distributed lock so exactly one worker ever calls the token endpoint. It is the multi-process counterpart to the in-process single-flight approach covered across OAuth2 token refresh strategies.

The reader here is running horizontally: a fleet of async workers, or several containers behind a scheduler, all pushing rate and availability for the same portfolio to booking_com and expedia. The goal is one token issuance per expiry cycle for the entire fleet, a fresh token always warm in the cache before anyone needs it, and no worker ever stalling because another is mid-refresh.

Prerequisites & environment

The cache sits between the token endpoint and every worker’s HTTP client, so pin these so the lock and TTL semantics are identical everywhere:

One decision to make up front: who refreshes. This page uses opportunistic refresh — the first worker to notice the token is near expiry takes the lock and refreshes — because it needs no extra process. If you already run a scheduler, a dedicated refresher cron is a clean alternative; the lock and cache-key design below are identical either way.

A shared Redis token cache with a distributed refresh lock across a worker pool Four worker processes each read a shared access token from a single Redis key on every request. When the cached token nears expiry within the skew margin, the workers race for a distributed lock implemented as a SET NX with a short TTL. One worker acquires the lock, calls the OTA token endpoint once, writes the new token back to the shared key, and releases the lock. The other three workers, finding the lock held, briefly wait and then re-read the freshly written token, so the whole fleet performs one token issuance per expiry cycle instead of one per worker. One shared token, one refresh per cycle across the fleet Workers read from Redis; a distributed lock elects the single worker that refreshes. worker 1 worker 2 (lock holder) worker 3 worker 4 Redis oauth:booking_com:token envelope + TTL oauth:booking_com:lock SET NX PX 5000 read token acquire lock OTA token endpoint POST /oauth/token called once per cycle refresh write new token → cache Proactive: refresh fires while the cached token is still valid, so no request ever waits on an expired token. token still valid — refresh triggered inside skew margin (e.g. last 90s) hard expiry (never reached in steady state)
Figure 1: Every worker reads one shared token from Redis; a distributed lock elects the single worker that refreshes it, proactively, before expiry.

Step-by-step implementation

Four steps: serialize the token as a validated envelope with a stored expiry, read it from Redis on the fast path, elect a single refresher with a SET NX lock, and release the lock safely with a check-and-delete so a slow refresh can’t wipe another worker’s lock.

Step 1 — Serialize a token envelope with an absolute expiry

Persist the token as JSON carrying an absolute expires_at epoch timestamp, and mirror that lifetime as the Redis key’s own TTL so a crashed fleet never leaves a permanently stale token behind.

python
import time
from pydantic import BaseModel

class CachedToken(BaseModel):
    access_token: str
    refresh_token: str | None = None
    expires_at: float          # absolute epoch seconds, shared across processes
    channel: str

    @classmethod
    def from_response(cls, body: dict, channel: str) -> "CachedToken":
        return cls(
            access_token=body["access_token"],
            refresh_token=body.get("refresh_token"),
            expires_at=time.time() + float(body["expires_in"]),
            channel=channel,
        )

    def needs_refresh(self, skew_seconds: float = 90.0) -> bool:
        # proactive: true while the token is still valid but inside the skew margin
        return time.time() >= (self.expires_at - skew_seconds)

The expiry is stored as an absolute epoch timestamp rather than a monotonic deadline precisely because it crosses process and host boundaries — every worker must agree on when the shared token dies, and time.time() is the only clock they can compare, so the workers’ hosts must run NTP.

Step 2 — Read from the cache on the fast path

Every request first reads the shared key. A hit that is not yet inside the skew margin returns immediately with no lock and no network call — the overwhelming majority of requests take this path.

python
import redis.asyncio as redis
import structlog

log = structlog.get_logger()
TOKEN_KEY = "oauth:{channel}:token"

async def read_cached_token(r: redis.Redis, channel: str) -> CachedToken | None:
    raw = await r.get(TOKEN_KEY.format(channel=channel))
    if raw is None:
        return None
    try:
        return CachedToken.model_validate_json(raw)
    except ValueError:
        # corrupt/partial entry — treat as a miss, force a clean refresh
        log.warning("token_cache_corrupt", channel=channel)
        return None

Validating the JSON on read rather than trusting the blob means a truncated or half-written cache entry — from a killed worker mid-SET — degrades to a cache miss and a clean refresh, instead of poisoning every worker with an unparseable bearer string.

Step 3 — Elect one refresher with a SET NX lock

When a worker sees needs_refresh(), it attempts to acquire a short-lived lock with SET key value NX PX. Only the worker that wins does the HTTP refresh; the losers skip it and re-read, because the winner will have written a fresh token within a second or two.

python
import uuid

LOCK_KEY = "oauth:{channel}:lock"

async def refresh_if_elected(r: redis.Redis, client, channel: str, creds: dict) -> None:
    token_val = str(uuid.uuid4())   # our unique fence token for safe release
    got = await r.set(LOCK_KEY.format(channel=channel), token_val,
                      nx=True, px=5000)   # 5s lock TTL > worst-case refresh time
    if not got:
        log.info("refresh_skipped_lock_held", channel=channel)
        return   # another worker is refreshing; caller will re-read the cache
    try:
        resp = await client.post(creds["token_url"], data=creds["body"])
        resp.raise_for_status()
        token = CachedToken.from_response(resp.json(), channel)
        ttl = max(int(token.expires_at - time.time()), 1)
        await r.set(TOKEN_KEY.format(channel=channel),
                    token.model_dump_json(), ex=ttl)
        log.info("token_refreshed", channel=channel, ttl=ttl)
    finally:
        await _release_lock(r, LOCK_KEY.format(channel=channel), token_val)

The lock TTL (px=5000) is a deadlock guard, not a timing knob: if the elected worker crashes mid-refresh, the lock auto-expires and another worker can take over — but it must comfortably exceed the worst-case token-endpoint latency, or a slow refresh loses its lock while still running.

Step 4 — Release the lock with a check-and-delete

Never delete the lock key blindly. If a worker’s refresh outran the lock TTL, the key it deletes might belong to a different worker that has since acquired the lock. A tiny Lua script makes the compare-and-delete atomic.

python
_RELEASE = """
if redis.call('get', KEYS[1]) == ARGV[1] then
  return redis.call('del', KEYS[1])
else
  return 0
end
"""

async def _release_lock(r: redis.Redis, key: str, token_val: str) -> None:
    # only delete the lock if we still own it (value matches our fence token)
    freed = await r.eval(_RELEASE, 1, key, token_val)
    if not freed:
        log.warning("lock_already_expired_or_stolen", key=key)

Making release a check-and-delete under one Lua script closes the classic distributed-lock race: without the fence-token check, a worker whose refresh ran long would release a lock now held by someone else, letting two workers refresh at once — the exact duplication this whole design exists to prevent, which parallels the atomic refill-and-consume used in distributing token buckets across async workers.

Gotchas & production notes

Verification snippet

Prove the fleet-level invariant: with many workers concurrently seeing a token inside the skew margin, the token endpoint is called exactly once. A fakeredis instance and a call-counting client are enough.

python
import asyncio
import fakeredis.aioredis

async def test_only_one_worker_refreshes() -> None:
    r = fakeredis.aioredis.FakeRedis()
    calls = {"n": 0}

    class CountingClient:
        async def post(self, url, data):
            calls["n"] += 1
            await asyncio.sleep(0.05)   # simulate token endpoint latency
            class Resp:
                def raise_for_status(self): ...
                def json(self): return {"access_token": "abc", "expires_in": 3600}
            return Resp()

    creds = {"token_url": "https://ota/oauth/token", "body": {}}
    client = CountingClient()
    # 20 workers all try to refresh the same channel at once
    await asyncio.gather(*(
        refresh_if_elected(r, client, "booking_com", creds) for _ in range(20)
    ))
    cached = await read_cached_token(r, "booking_com")
    assert calls["n"] == 1                 # one issuance for the whole fleet
    assert cached is not None and cached.access_token == "abc"

asyncio.run(test_only_one_worker_refreshes())

A green run demonstrates the payoff of the distributed lock: twenty workers that all decided the token needed refreshing produced a single call to the OTA’s token endpoint, and every one of them ends up reading the same freshly cached credential — one issuance per cycle for the whole fleet, no matter how wide it scales.

← Back to OAuth2 Token Refresh Strategies