Building Parity Drift Alerts with Prometheus

Parity drift is a duration problem, not an instant one: a channel that dips 3% below the authoritative rate for one scrape is noise, while the same 3% held for twenty minutes across peak-season dates is an incident. Prometheus is built for exactly this distinction — its for clause fires an alert only when a condition holds for a sustained window — which makes it the natural place to turn the deviation signal from parity monitoring and alerting into pages that respect the on-call’s attention. This page builds the Python exporter that publishes parity deviation as Prometheus metrics and the alerting rules that page on sustained drift while staying silent on blips.

The result is two moving parts: a prometheus_client exporter that maps each observed parity deviation onto a labelled gauge and a histogram, and a set of Alertmanager rules with for durations, severity labels, deduplication, and routing that send criticals to paging and warnings to Slack. The design goal throughout is that a single bad scrape never pages anyone and a real markdown always does.

Prerequisites & environment

The exporter runs as a sidecar to the parity scorer, scraped by Prometheus on a fixed interval. Pin these versions so metric names, label cardinality, and rule evaluation behave identically across environments:

Python 3.11+ — for the exporter process and modern type hints.
prometheus_client 0.20+ — provides Gauge, Histogram, and the start_http_server scrape endpoint used below.
Prometheus 2.50+ and Alertmanager 0.27+ — the alerting rules use for, keep_firing_for, and label-based routing that assume a recent server.
The parity deviation signal — normalized, scored deviation_pct per cell from computing parity deviation scores across channels; the exporter publishes this, it does not compute it.
structlog 24.x — the exporter logs every metric update as a key=value event so a missing series can be traced to a silent scorer rather than a broken exporter.
A bounded label set — a plan for label cardinality (see the first gotcha); publishing one time series per stay_date will overwhelm Prometheus, so the exporter aggregates before it exports.

Figure 1: the alerting path — the exporter publishes deviation as a gauge and histogram, Prometheus scrapes and evaluates a for-duration rule, and Alertmanager deduplicates and routes criticals to paging and warnings to Slack.

Step-by-step implementation

The exporter defines a small, bounded set of metrics, updates them each scoring cycle, and serves them for Prometheus to scrape. The rules then encode the duration and severity logic. Build them in that order.

Step 1 — Define bounded gauges and a deviation histogram

Define a gauge for the current per-channel worst deviation and a histogram for the distribution of absolute deviations. Keep labels to low-cardinality dimensions — property_id, channel, and direction — and never label by stay_date or room_type_code, which would create millions of series.

python

from prometheus_client import Gauge, Histogram

# Worst (most negative) current deviation per channel — a gauge, because it is a
# point-in-time level Prometheus samples, not an accumulating counter.
PARITY_DEVIATION = Gauge(
    "parity_deviation_pct",
    "Current worst signed parity deviation percent per channel",
    ["property_id", "channel", "direction"],   # direction = undercut | overcharge
)

# Distribution of absolute deviation magnitudes, for percentile alerting and dashboards.
PARITY_DEVIATION_HIST = Histogram(
    "parity_deviation_abs_pct",
    "Distribution of absolute parity deviation percent",
    ["property_id", "channel"],
    buckets=(0.5, 1.0, 2.0, 3.0, 5.0, 8.0, 13.0),   # tuned around the tolerance band
)

Choosing histogram buckets clustered around the tolerance band (the 2.0–5.0 region) rather than Prometheus’s linear defaults is what lets an alert rule ask “what fraction of cells breached 5%” with useful resolution — default buckets would smear the signal across ranges nobody alerts on.

Step 2 — Update the metrics each scoring cycle

After each scoring pass, push the per-channel worst deviation to the gauge and observe every cell’s absolute deviation into the histogram. Set the gauge explicitly rather than incrementing, because a gauge represents the current level, and observe each cell so the histogram reflects the true distribution.

python

import polars as pl
import structlog

log = structlog.get_logger()

def export_cycle(scored: pl.DataFrame, property_id: str) -> None:
    # scored has: channel, deviation_pct (signed), abs_deviation_pct
    worst = (scored.group_by("channel")
                   .agg(pl.col("deviation_pct").min().alias("worst_dev")))
    for row in worst.iter_rows(named=True):
        direction = "undercut" if row["worst_dev"] < 0 else "overcharge"
        PARITY_DEVIATION.labels(property_id, row["channel"], direction).set(row["worst_dev"])
        log.info("parity_gauge_set", property_id=property_id,
                 channel=row["channel"], worst_dev=row["worst_dev"])
    for row in scored.iter_rows(named=True):
        PARITY_DEVIATION_HIST.labels(property_id, row["channel"]).observe(
            abs(row["deviation_pct"] or 0.0)
        )

Reporting the minimum signed deviation as the per-channel gauge value is deliberate: the most negative number is the worst undercut, which is the parity-clause exposure that matters, so the gauge tracks the single number a rule should alert on rather than an average that a few overcharges could mask.

Step 3 — Serve the metrics endpoint

Expose the metrics on an HTTP port for Prometheus to scrape. Run the server once at process start and let the scoring loop update the metric objects in place; prometheus_client handles the concurrency between the scrape handler and your updates.

python

from prometheus_client import start_http_server
import time

def run_exporter(port: int = 9188) -> None:
    start_http_server(port)                 # serves /metrics on all interfaces
    log.info("parity_exporter_started", port=port)
    while True:
        scored = load_latest_scores()       # your scorer output for this cycle
        export_cycle(scored, property_id="LON-STJ-01")
        time.sleep(60)                       # align with the Prometheus scrape_interval

Aligning the update loop’s sleep with the Prometheus scrape_interval avoids a subtle aliasing bug: if the exporter refreshes far faster than Prometheus scrapes, brief sub-scrape breaches vanish between samples, and if it refreshes far slower, the gauge goes stale and a rule fires on old data — matching the two cadences keeps every real breach visible to exactly one scrape.

Step 4 — Write the alerting rules with a `for` duration

The rule is where blip suppression lives. Fire only when the deviation gauge breaches the band and holds for a sustained window via for, add keep_firing_for so a flapping recovery does not resolve-and-refire, and attach a severity label that Alertmanager routes on.

yaml

groups:
  - name: parity_drift
    rules:
      - alert: ParityUndercutSustained
        # undercut worse than 2% held for 15 minutes — a real markdown, not a cache blip
        expr: parity_deviation_pct{direction="undercut"} < -2.0
        for: 15m
        keep_firing_for: 5m
        labels:
          severity: warning
          team: revenue
        annotations:
          summary: "{{ $labels.channel }} undercutting parity on {{ $labels.property_id }}"
          description: "Sustained undercut of {{ $value | printf \"%.2f\" }}% for 15m."

      - alert: ParityUndercutCritical
        # a large, visible undercut escalates faster
        expr: parity_deviation_pct{direction="undercut"} < -5.0
        for: 5m
        keep_firing_for: 5m
        labels:
          severity: critical
          team: revenue
        annotations:
          summary: "Severe parity breach on {{ $labels.channel }}"
          description: "Undercut of {{ $value | printf \"%.2f\" }}% held 5m — page on-call."

Setting keep_firing_for: 5m on top of the for clause is the non-obvious detail that stops alert flapping: without it, a deviation that oscillates around the threshold resolves and re-fires repeatedly, spamming the on-call, whereas keeping the alert firing for a grace period after it recovers collapses that oscillation into one continuous incident.

Step 5 — Deduplicate and route in Alertmanager

Group alerts so one channel-wide breach is a single notification, and route by the severity label so criticals page and warnings go to Slack. Grouping on property_id and channel is what turns hundreds of per-cell breaches into one actionable incident.

yaml

route:
  receiver: slack_revenue
  group_by: ['alertname', 'property_id', 'channel']   # one incident per channel breach
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: [severity="critical"]
      receiver: pagerduty_oncall
      continue: false                       # criticals page and do NOT also spam Slack

receivers:
  - name: slack_revenue
    slack_configs:
      - channel: '#parity-alerts'
        send_resolved: true
  - name: pagerduty_oncall
    pagerduty_configs:
      - routing_key: '<integration-key>'

Grouping on alertname, property_id, and channel rather than leaving alerts ungrouped is what makes a season-wide misconfiguration arrive as a single notification naming the channel, instead of one Slack message per breaching date cell — the same deduplication principle the parity monitoring loop applies with its cell-identity dedup key.

Gotchas & production notes

Label cardinality is the failure mode that kills the Prometheus server, not the alert. Every unique label combination is a separate time series; labelling by stay_date, room_type_code, and rate_plan_code on a portfolio produces millions of series and OOMs Prometheus. Aggregate to (property_id, channel, direction) in the exporter — the worst-offender identity lives in the alert annotation and the scorer’s logs, not in a metric label.
A gauge that stops updating looks healthy, not broken. If the exporter crashes, its last-set gauge value persists at whatever it was and no rule fires — a silent monitoring outage. Add an up-based deadman rule (absent(parity_deviation_pct) or scrape up == 0) so a dead exporter pages, exactly as monitoring-health is treated separately from parity in the parent guide.
for duration must exceed one scrape interval, or it never fires. A for: 15m with a scrape_interval of 15m gives Prometheus a single sample to evaluate and the alert may never satisfy the sustained condition. Keep for at several multiples of the scrape interval so the duration is measured across real samples.
Histogram observations accumulate for the process lifetime. The _bucket, _sum, and _count series only grow, so alert on a rate() or increase() over a window rather than the raw counter — rate(parity_deviation_abs_pct_bucket[10m]) answers “breaching now”, while the raw value answers “since the exporter last restarted”, which is rarely what you want.

Verification snippet

Confirm the exporter publishes the series a rule depends on and that the gauge reports the worst undercut, not an average. Use prometheus_client’s in-process registry to assert on metric values without standing up a full Prometheus.

python

from prometheus_client import REGISTRY
import polars as pl

def test_gauge_reports_worst_undercut() -> None:
    scored = pl.DataFrame({
        "channel": ["booking_com", "booking_com", "expedia"],
        "deviation_pct": [-1.0, -4.0, 2.0],       # booking worst is -4.0, not the -1.0 mean
        "abs_deviation_pct": [1.0, 4.0, 2.0],
    })
    export_cycle(scored, property_id="LON-STJ-01")
    booking = REGISTRY.get_sample_value(
        "parity_deviation_pct",
        {"property_id": "LON-STJ-01", "channel": "booking_com", "direction": "undercut"},
    )
    assert booking == -4.0                          # the worst undercut, not an average

def test_histogram_counts_every_observation() -> None:
    count = REGISTRY.get_sample_value(
        "parity_deviation_abs_pct_count",
        {"property_id": "LON-STJ-01", "channel": "booking_com"},
    )
    assert count == 2.0                             # both booking_com cells observed

test_gauge_reports_worst_undercut()
test_histogram_counts_every_observation()

The first assertion is the one that matters: it proves the gauge surfaces the single worst undercut a channel is running rather than an average that a mild overcharge could soften, which is precisely what the ParityUndercutCritical rule keys on. Run it in CI so a refactor that accidentally reverts the gauge to a mean is caught before it silences a real page. The metrics themselves then flow into the same dashboards as the rest of revenue KPI observability.

Parity monitoring & alerting — the parent workflow: the sampling, scoring, and dedup logic these metrics and rules operationalize.
Computing parity deviation scores across channels — the Polars scorer that produces the deviation_pct this exporter publishes.
Revenue KPI observability — where the parity gauges sit alongside ADR, RevPAR, and delivery-latency panels.
Handling OTA API rate limits — pacing the shop reads that feed these metrics inside the request budget.
PMS & channel manager architecture foundations — the structured-logging and alert-threshold conventions this exporter follows.

← Back to Parity Monitoring & Alerting