InventoryAllocation

Sanitizing PCI Cardholder Data in Sync Logs

A card number written to a log file is a card number stored — and the moment it lands in a shared log aggregator, that aggregator, its backups, and everyone with query access are inside PCI DSS scope. Reservation payloads flowing through a PMS-to-channel-manager pipeline routinely carry a card_number, a cvv, or a cardholder name, and a single unguarded log.info(payload) copies all three into permanent storage. This page builds the structlog processor that redacts cardholder data in-process, before any byte reaches the sink, as the card-data control that complements the compliance and audit logging trail. The reader is the engineer who owns the logging configuration and needs redaction that cannot be bypassed.

The design commits to one principle: redaction happens in the process that emits the event, not downstream. Scrubbing at the aggregator is scrubbing after the raw PAN already traversed the network and was buffered on disk somewhere in between. A structlog processor sits in the emission pipeline itself, so every event from every logger in the process is filtered before serialization — there is no code path that logs and skips the redactor.

Prerequisites & environment

The processor is installed once in the logging configuration and applies to every structured event the process emits. Pin these:

The one hard constraint: the redactor must be the last-but-one processor, running after your own key-value binding but before the renderer that serializes to the sink. Place it after the renderer and it sees an already-formatted string it cannot reliably parse; place it too early and later processors can re-introduce raw fields. Credentials for the sink itself follow the pipeline’s security and authentication boundaries.

Redaction processor position in the structlog chain A log event with a raw payload — card_number 4111 1111 1111 1111, cvv 123, cardholder_name, and a note field containing a leaked PAN — enters a structlog processor chain. It passes through context binding, then a redaction processor that applies two rules: a sensitive-key allowlist that replaces card_number, cvv and cardholder_name with REDACTED, and a value-level PAN regex that catches the card number leaked into the free-text note. The cleaned event, with all card data replaced, then reaches the JSON renderer and the log sink. A dashed arrow shows the raw PAN never reaching the sink. Redaction runs inside the logging pipeline, before the sink A key allowlist plus a value-level PAN regex strip card data in-process — the sink never sees a raw PAN. Raw event card_number 4111… cvv 123 · cardholder note: "…4111 1111…" Context binding property_id, stage… Redaction processor key allowlist → [REDACTED] PAN regex → [REDACTED-PAN] Renderer → sink no card data present raw PAN blocked at the processor — never reaches the sink
Figure 1: the redaction processor sits between context binding and the renderer, applying a key allowlist and a value-level PAN regex so cardholder data is stripped in-process and the raw PAN never reaches the sink.

Step-by-step implementation

Step 1 — Define the sensitive-key set and PAN pattern

Two rules cover the two ways card data enters a log: a field literally named for it, and a card number hiding inside free text. Encode both as module-level constants so they compile once and are auditable in one place.

python
import re

# Redacted unconditionally by key name — the expected, structured case.
SENSITIVE_KEYS = frozenset({
    "card_number", "pan", "cvv", "cvc", "card_cvv",
    "cardholder_name", "card_expiry", "track_data",
})

# 13–19 digit card numbers, allowing spaces or hyphens between digit groups.
PAN_PATTERN = re.compile(r"\b(?:\d[ -]?){13,19}\b")

REDACTED = "[REDACTED]"
REDACTED_PAN = "[REDACTED-PAN]"

Compiling PAN_PATTERN at module scope matters because the redactor runs on every log event in the process: recompiling the regex per call would put avoidable CPU on the hot logging path of a high-throughput sync worker.

Step 2 — Write the recursive redaction function

A payload is rarely flat — card data hides in nested dicts and lists (a payment sub-object, a list of guests). Redact recursively so a PAN two levels deep is caught, applying the key rule and the value rule at every level.

python
def redact_value(value):
    if isinstance(value, str):
        return PAN_PATTERN.sub(REDACTED_PAN, value)
    if isinstance(value, dict):
        return {k: (REDACTED if k in SENSITIVE_KEYS else redact_value(v))
                for k, v in value.items()}
    if isinstance(value, list):
        return [redact_value(item) for item in value]
    return value  # ints/floats/None pass through untouched

Applying the key check (k in SENSITIVE_KEYS) before recursing into the value means a field named card_number is redacted whole even if its value is an unusual shape — a list of digits, say — that the PAN regex alone would miss, so the two rules cover each other’s blind spots.

Step 3 — Wrap it as a structlog processor

A structlog processor is a callable (logger, method_name, event_dict) -> event_dict. Wrap the recursive redactor so it filters the entire event dict, and insert it into the processor chain just before the renderer.

python
import structlog

def redact_processor(logger, method_name, event_dict: dict) -> dict:
    # Runs on every event; returns a cleaned dict the renderer then serializes.
    return redact_value(event_dict)

structlog.configure(
    processors=[
        structlog.contextvars.merge_contextvars,
        structlog.processors.add_log_level,
        structlog.processors.TimeStamper(fmt="iso"),
        redact_processor,                      # last transform before rendering
        structlog.processors.JSONRenderer(),
    ]
)

Placing redact_processor immediately before JSONRenderer guarantees it sees the fully-assembled event dict — every bound context variable and every keyword passed to the log call — but still operates on structured data rather than a rendered string, so nothing a caller logs can slip past it.

Step 4 — Prove the processor is installed, not just available

A redactor that exists but is not wired into the active configuration is worse than none, because it creates false confidence. Add a startup self-check that logs a known-sensitive value and asserts the emitted record is clean, failing fast if the chain is misconfigured.

python
from structlog.testing import capture_logs

def assert_redaction_active() -> None:
    log = structlog.get_logger()
    with capture_logs() as captured:
        log.info("card_check", card_number="4111111111111111",
                 note="fallback pan 4111 1111 1111 1111")
    event = captured[0]
    assert event["card_number"] == REDACTED, "key redaction not active"
    assert "4111" not in event["note"], "value redaction not active"

Running assert_redaction_active() at process startup turns a silent misconfiguration — a redactor removed from the chain during a refactor — into an immediate boot failure, rather than a leak discovered months later during a PCI assessment.

Gotchas & production notes

Verification snippet

Confirm the redactor catches card data through both doors — the named field and the free-text leak — and survives nesting.

python
def test_key_and_value_redaction():
    dirty = {
        "event": "payment_attached",
        "card_number": "4111111111111111",       # caught by key
        "payment": {"cvv": "123",                # caught by nested key
                    "note": "guest gave 4111 1111 1111 1111 by phone"},  # caught by regex
        "amount": 540.00,                        # untouched
    }
    clean = redact_value(dirty)
    assert clean["card_number"] == REDACTED
    assert clean["payment"]["cvv"] == REDACTED
    assert "4111" not in clean["payment"]["note"]
    assert clean["amount"] == 540.00             # non-sensitive numeric preserved

test_key_and_value_redaction()
assert_redaction_active()

The nested payment.note assertion is the one that catches the real-world leak: a PAN dictated over the phone and dropped into a free-text note two levels deep, which key-based redaction alone would miss and only the recursive value pass removes. In production, run assert_redaction_active() on every boot and fail the deploy if it raises, so the control is verified continuously rather than assumed.

← Back to Compliance & Audit Logging for Sync Pipelines