InventoryAllocation

Deduplicating Polled Inventory Deltas

Polling re-reads the whole availability snapshot every cycle, but most of that snapshot is identical to last time — a property might change two dates an hour while you poll five hundred rows a minute. If every polled row becomes a downstream write, you burn OTA quota re-pushing prices nobody changed and drown the reconciliation logs in noise. The fix is to emit only what actually changed: fingerprint each row, diff the current snapshot against the last-seen one, and forward only the cells that moved. This guide implements that content-hash deduplication, the efficiency layer in front of async polling for inventory updates that turns a full snapshot into a minimal change set.

The core idea is a per-row SHA-256 fingerprint over the fields that define state. An unchanged row hashes to the same value it did last cycle, so a set difference against the previous snapshot’s fingerprints yields exactly the changed rows — and an unchanged poll cycle yields an empty set and zero writes.

Prerequisites and environment

The fingerprint must cover exactly the fields that constitute state — available_units, min_stay, stop_sell, and any restriction that a downstream write would carry — and must exclude volatile metadata like fetched_at, or every row looks changed every cycle.

Fingerprint diff between the current and last-seen snapshot emits only changed rows The current polled snapshot on the left has three rows, each reduced to a SHA-256 fingerprint over its state fields. The last-seen snapshot on the right also has fingerprints from the previous cycle. A diff compares them per triplet: row one is unchanged so its fingerprint matches and it is dropped, row two changed from three units to zero so its fingerprint differs and it is emitted, row three is unchanged and dropped. Only the single changed row flows downstream as a write, and the last-seen store is updated to the current fingerprints. Diff by fingerprint: emit only what moved Same state → same SHA-256 → dropped. Changed state → new hash → one write. current poll snapshot DLX_KING · BAR · 5u → a1f… STD_TWIN · BAR · 0u → c7d… STE_OCEAN · NONREF · 2u → e3b… last-seen fingerprints DLX_KING · BAR → a1f… (match) STD_TWIN · BAR → 9b2… (differs) STE_OCEAN · NONREF → e3b… (match) Polars fingerprint diff join per triplet · keep hash mismatches 1 changed row → downstream write STD_TWIN · BAR · now 0u (stop-sell) · 2 rows dropped
Figure 1: Only the row whose fingerprint changed since last cycle is emitted; an unchanged poll produces an empty change set and no writes.

Step-by-step implementation

Step 1 — Fingerprint each row over its state fields only

Reduce every polled row to a stable SHA-256 hash of the fields that define its state. Serialize those fields in a fixed order so the hash is reproducible, and exclude any volatile metadata.

python
import hashlib

STATE_FIELDS = ("available_units", "min_stay", "stop_sell", "closed_to_arrival")

def fingerprint(row: dict) -> str:
    # Fixed field order → reproducible hash. Volatile fields (fetched_at) excluded
    # so an unchanged row hashes identically across cycles.
    basis = "|".join(str(row[f]) for f in STATE_FIELDS)
    return hashlib.sha256(basis.encode()).hexdigest()

def triplet_key(row: dict) -> str:
    return f"{row['property_id']}:{row['room_type_code']}:{row['rate_plan_code']}:{row['stay_date']}"

Hashing only STATE_FIELDS in a fixed order is the load-bearing choice: include fetched_at or reorder the fields and every row’s fingerprint changes every cycle, which defeats the dedup entirely and re-emits the whole snapshot as spurious deltas.

Step 2 — Diff the current snapshot against last-seen with Polars

Build the current snapshot as a Polars frame with a fingerprint column, join it against the last-seen fingerprints keyed by triplet, and keep only rows where the hash differs (or the triplet is new). This is one vectorized pass over the whole snapshot.

python
import polars as pl

def compute_deltas(current: pl.DataFrame, last_seen: pl.DataFrame) -> pl.DataFrame:
    # current  : columns [triplet, fingerprint, ...state...]
    # last_seen: columns [triplet, fingerprint]
    joined = current.join(last_seen, on="triplet", how="left", suffix="_prev")
    # A row is a real delta when the hash changed OR the triplet is brand new (null prev).
    return joined.filter(
        (pl.col("fingerprint") != pl.col("fingerprint_prev"))
        | pl.col("fingerprint_prev").is_null()
    ).drop("fingerprint_prev")

Using a left join with a null check means a triplet appearing for the first time (no previous fingerprint) is correctly treated as a change and emitted, rather than being silently dropped because it has nothing to compare against.

Step 3 — Persist the new fingerprints atomically after emit

Once the deltas are handed downstream, update the last-seen store to the current fingerprints so the next cycle diffs against this one. Do it after a successful emit, not before, or a crash mid-cycle would advance the baseline past deltas that were never forwarded.

python
import redis.asyncio as redis
import structlog

log = structlog.get_logger()

async def persist_fingerprints(r: redis.Redis, property_id: str,
                               current: pl.DataFrame) -> None:
    mapping = dict(zip(current["triplet"].to_list(),
                       current["fingerprint"].to_list()))
    # HSET the whole property's fingerprints in one round trip.
    await r.hset(f"fp:{property_id}", mapping=mapping)
    log.info("fingerprints.persisted", property_id=property_id, rows=len(mapping))

Persisting after the downstream emit (not before) preserves at-least-once semantics for the change set: if the process dies between diff and persist, the next cycle re-detects the same deltas rather than losing them, and the idempotent downstream apply absorbs the duplicate.

Step 4 — Short-circuit an entirely unchanged cycle

The common case is that nothing changed. Detect an empty delta set and skip all downstream work, logging the no-op so an operator can see the poller is alive but idle rather than stalled.

python
async def run_cycle(r: redis.Redis, property_id: str,
                    current: pl.DataFrame, last_seen: pl.DataFrame) -> int:
    deltas = compute_deltas(current, last_seen)
    if deltas.height == 0:
        log.info("poll.no_change", property_id=property_id)  # alive, nothing to do
        return 0
    for row in deltas.iter_rows(named=True):
        await forward_delta(row)          # to the stream / apply path
    await persist_fingerprints(r, property_id, current)
    log.info("poll.deltas_emitted", property_id=property_id, count=deltas.height)
    return deltas.height

Emitting a poll.no_change event on an empty cycle distinguishes a healthy idle poller from a hung one — without it, “no downstream writes” is ambiguous between “nothing changed” and “the poller died,” and only one of those should page someone.

Gotchas and production notes

Verification snippet

Confirm the two behaviours that define correct dedup — an unchanged snapshot emits nothing, and a single changed cell emits exactly one delta.

python
import polars as pl

def _snap(units_twin: int) -> pl.DataFrame:
    rows = [
        {"triplet": "LON-STJ-01:DLX_KING:BAR:2026-08-14",  "available_units": 5,
         "min_stay": 1, "stop_sell": False, "closed_to_arrival": False},
        {"triplet": "LON-STJ-01:STD_TWIN:BAR:2026-08-14", "available_units": units_twin,
         "min_stay": 1, "stop_sell": units_twin == 0, "closed_to_arrival": False},
    ]
    return pl.DataFrame(rows).with_columns(
        pl.struct(["available_units", "min_stay", "stop_sell", "closed_to_arrival"])
          .map_elements(lambda s: fingerprint(dict(s)), return_dtype=pl.Utf8)
          .alias("fingerprint"))

def test_unchanged_snapshot_emits_nothing():
    snap = _snap(3)
    last_seen = snap.select("triplet", "fingerprint")
    assert compute_deltas(snap, last_seen).height == 0        # identical → no deltas

def test_single_change_emits_one_delta():
    prev = _snap(3).select("triplet", "fingerprint")
    now = _snap(0)                                            # STD_TWIN 3 → 0 (stop-sell)
    deltas = compute_deltas(now, prev)
    assert deltas.height == 1
    assert deltas["triplet"][0].startswith("LON-STJ-01:STD_TWIN")

test_unchanged_snapshot_emits_nothing()
test_single_change_emits_one_delta()

The zero-delta assertion is the one that pays for the whole approach: it proves an identical poll cycle produces no downstream writes at all, so quota and reconciliation noise scale with the rate of change, not the size of the snapshot. The single-delta case confirms a real change — a stop-sell — is still caught and emitted alone.

← Back to Async Polling for Inventory Updates