InventoryAllocation

Handling Out-of-Order Webhook Events in Inventory Sync

Webhooks arrive in the order the network and your worker pool happen to deliver them, not the order the events actually happened. A sender fires “3 rooms left” then “sold out” a second later, but a retry, a slow TLS handshake, or two concurrent workers can flip them — and if the stale “3 rooms left” lands last, you have just re-opened a sold-out date for booking. Solving this needs an ordering rule that depends on the event’s content, not its arrival time. This guide implements last-write-wins with a per-triplet version guard, the reordering discipline that sits underneath channel manager webhook integration once signatures are verified and payloads are enqueued.

The unit of ordering is the saleable triplet (property_id, room_type_code, rate_plan_code). Events for different triplets are independent and need no ordering relative to each other; events for the same triplet must apply newest-first, and anything older than what has already been applied must be discarded rather than written.

Prerequisites and environment

This logic runs in the apply stage, after a webhook has passed signature verification and been normalized. Pin these:

The version can be a sender sequence (source_seq) or an event timestamp in epoch milliseconds. A sequence is preferable when the sender guarantees it is strictly increasing per triplet; a timestamp is the fallback, with the caveat that two events in the same millisecond need a tiebreaker. Both plug into the same guard.

A version guard applies newer events and discards stale ones per triplet Three webhook events for the same property, room type, and rate plan triplet arrive out of order: version 1010 with three units, then version 1008 with five units arriving late, then version 1012 with zero units meaning sold out. A per-triplet high-water mark starts at zero. Version 1010 is greater than the mark so it is applied and the mark advances to 1010. Version 1008 is not greater than 1010 so it is discarded as stale. Version 1012 is greater than 1010 so it is applied, the mark advances to 1012, and the date is correctly sold out. The late stale event never re-opens the sold-out date. Version guard: newer applies, stale is discarded Ordering by content, not arrival — a late "5 units" never re-opens a sold-out date. arrival order (out of order) v1010 · 3 units arrives 1st v1008 · 5 units arrives 2nd (late) v1012 · sold out arrives 3rd Version guard apply iff version > high-water mark applied state (per triplet high-water mark) v1010 applied mark 0 → 1010 · 3 units v1008 discarded 1008 ≤ 1010 · stale v1012 applied mark 1010 → 1012 · sold out ✓ date stays sold out stale event never re-opened it
Figure 1: The high-water mark makes ordering a property of the event's version, so a late-arriving stale update is discarded instead of overwriting fresher state.

Step-by-step implementation

Step 1 — Carry a monotonic version on the normalized event

The event model must expose the version and the triplet it orders within. Keep the model frozen so a version cannot be mutated after normalization.

python
from datetime import date
from pydantic import BaseModel, ConfigDict, Field

class InventoryWebhookEvent(BaseModel):
    model_config = ConfigDict(frozen=True)

    property_id: str = Field(pattern=r"^[A-Z]{3}-[A-Z]{3}-\d{2}$")  # "LON-STJ-01"
    room_type_code: str = Field(pattern=r"^[A-Z0-9_]+$")            # "DLX_KING"
    rate_plan_code: str = Field(pattern=r"^[A-Z0-9_]+$")            # "BAR"
    stay_date: date
    available_units: int = Field(ge=0)
    version: int = Field(ge=0)     # monotonic per triplet: source_seq or epoch-ms

    def triplet(self) -> str:
        # The ordering scope — one high-water mark per saleable unit.
        return f"{self.property_id}:{self.room_type_code}:{self.rate_plan_code}"

Scoping the version to the triplet rather than to the whole property is deliberate: two different room types changing concurrently are independent, so a global sequence would force false ordering between them and needlessly discard valid concurrent updates.

Step 2 — Guard the apply with an atomic compare-and-set

The high-water mark must be updated atomically, or two workers reading the same current value will both decide their event is newer and both write. A Redis Lua script does the compare-and-set in one round trip; the script advances the mark only if the incoming version is strictly greater and reports whether the caller should apply.

python
import redis.asyncio as redis

# Atomically: set mark = version and return 1 iff version > current mark, else 0.
_CAS = """
local cur = tonumber(redis.call('GET', KEYS[1]) or '-1')
local ver = tonumber(ARGV[1])
if ver > cur then
  redis.call('SET', KEYS[1], ver)
  return 1
end
return 0
"""

class OrderingGate:
    def __init__(self, r: redis.Redis):
        self.r = r
        self._cas = r.register_script(_CAS)

    async def should_apply(self, event: InventoryWebhookEvent) -> bool:
        key = f"invmark:{event.triplet()}"
        won = await self._cas(keys=[key], args=[event.version])
        return bool(won)

Doing the comparison and the write inside one Lua script makes the check-and-claim a single atomic operation, so during a consumer rebalance two workers briefly handling the same triplet cannot both pass the guard — exactly the concurrency hazard a read-then-write in Python would open.

Step 3 — Apply on win, discard-and-log on loss

Wire the gate into the apply path. A winning event is written to the channel; a losing (stale) event is discarded with a structured log so the suppression is observable and auditable, not silent.

python
import structlog
import httpx

log = structlog.get_logger()

async def apply_ordered(event: InventoryWebhookEvent, gate: OrderingGate,
                        client: httpx.AsyncClient) -> str:
    if not await gate.should_apply(event):
        # Stale: a newer version already won this triplet. Drop it, but record it.
        log.info("webhook.discard_stale", triplet=event.triplet(),
                 version=event.version, units=event.available_units)
        return "discarded"

    resp = await client.put(
        f"/inventory/{event.property_id}/availability",
        json=event.model_dump(mode="json"),
    )
    resp.raise_for_status()
    log.info("webhook.applied", triplet=event.triplet(), version=event.version)
    return "applied"

Logging the discard with the triplet and version turns “the update did not take effect” from an unexplained ghost into a greppable record showing exactly which older version lost to which newer one — indispensable when a revenue manager asks why a webhook “did nothing.”

Step 4 — Give timestamp-based versions a deterministic tiebreaker

If your version is an event timestamp rather than a strict sequence, two events in the same millisecond compare equal and the guard’s strict > discards the second — which may be the one you wanted. Fold a deterministic tiebreaker into a composite version so equal timestamps resolve consistently across workers.

python
def composite_version(epoch_ms: int, available_units: int) -> int:
    # Pack ms into the high bits and a stable tiebreaker into the low bits.
    # When two events share a millisecond, the lower unit count (more restrictive)
    # wins deterministically — a stop-sell beats a same-instant "still available".
    tiebreak = 9999 - min(available_units, 9999)
    return epoch_ms * 10_000 + tiebreak

Encoding “more restrictive wins” into the tiebreaker is a safety bias: when the true order is genuinely ambiguous, resolving toward the lower availability errs on the side of not overselling, which is the cheaper mistake for a hotel to make.

Gotchas and production notes

Verification snippet

Confirm the two behaviours that define correct ordering — a newer version applies and advances the mark, and a stale version is discarded without writing.

python
import pytest
from datetime import date

def _ev(version: int, units: int) -> InventoryWebhookEvent:
    return InventoryWebhookEvent(
        property_id="LON-STJ-01", room_type_code="DLX_KING", rate_plan_code="BAR",
        stay_date=date(2026, 8, 14), available_units=units, version=version,
    )

@pytest.mark.asyncio
async def test_stale_event_is_discarded(gate):        # gate seeded at mark=1010
    assert await gate.should_apply(_ev(1010, 3)) is True    # advances 0 → 1010
    assert await gate.should_apply(_ev(1008, 5)) is False   # 1008 ≤ 1010 → stale
    assert await gate.should_apply(_ev(1012, 0)) is True    # 1012 > 1010 → applies

def test_composite_tiebreak_prefers_restriction():
    sold_out = composite_version(1_700_000_000_000, 0)
    still_open = composite_version(1_700_000_000_000, 5)   # same millisecond
    assert sold_out > still_open                            # stop-sell wins the tie

The three-call sequence is the important assertion: it reproduces the exact out-of-order arrival from the diagram — apply, discard, apply — and proves the sold-out event at v1012 sticks while the late “5 units” at v1008 is dropped. In production, also assert that each discard emits exactly one webhook.discard_stale log so no suppression is invisible.

← Back to Channel Manager Webhook Integration