Kafka reliability and event-time correctness#

Level: Expert

You will: reason about delivery semantics, event time, contracts,

deduplication, retries, and bounded failure exercises. Before you start: the Integration runbooks and Docker.

Services: Kafka and Schema Registry for broker checks; the logical chaos

path is offline.

FraudTwin’s Kafka publisher is intentionally explicit about contract versions, partition keys, idempotence, retries, acknowledgements, and pacing. The chaos harness adds deterministic logical-message faults around either boundary:

For the optional broker-backed path, install the client and start the local profile:

!pip install confluent-kafka

docker compose --profile streaming up -d kafka schema-registry streaming-topics

The publication tutorials then list broker topics, list Schema Registry subjects, and publish a contract-backed record. The cells use short timeouts; when the broker is unavailable they continue with the deterministic chaos path.

from pathlib import Path

from fraudtwin import generate
from fraudtwin.config import load_config
from fraudtwin.kafka import publication_records
from fraudtwin.kafka_chaos import KafkaChaosConfig, simulate_delivery

written_run = generate(
    load_config(Path("configs/minimal.yaml")),
    write=True,
    output_dir=Path("runs"),
)
data = written_run.load_data()
records = publication_records(data.behavior, written_run.run_id)
result = simulate_delivery(
    records,
    KafkaChaosConfig(
        boundary="producer",
        seed=17,
        drop_probability=0.02,
        duplicate_probability=0.03,
        retry_probability=0.05,
        max_delay_seconds=30,
        reorder_window=100,
        partition_count=3,
    ),
)
print(result.manifest)

Supported logical failures include:

Fault

What it teaches

drop/outage

loss handling and reconciliation

duplicate/retry

idempotent consumers and stable event IDs

delay/late delivery

event-time windows and watermarks

reordering

separating business time from arrival order

partition skew

hot partitions and consumer imbalance

schema change

compatibility checks and reader defaults

event_id, business keys, and payload bytes are preserved. A chaos manifest reports input/output fingerprints, attempts, transport IDs, topic/partition counts, and deduplicated totals. This simulates Kafka message semantics, not physical network packets; use Docker/Linux tc/netem separately when testing socket-level failures.

Event-flow semantics#

The observable publication path is:

source run → contract validation → topic/partition assignment → producer acknowledgement → consumer processing → deduplication → event-time projection

The six observable topics use payment_id as the partition key. Ordering is guaranteed only within a topic and key; arrival order is not business event time. Producers use idempotence and acknowledgements, but process restarts can still produce at-least-once delivery. Consumers must deduplicate by stable event_id (or the documented business identity for a subject) before applying side effects.

Concern

FraudTwin guidance

Offsets

Commit only after validation and idempotent projection

Retries

Preserve event identity and increment transport attempt metadata

Dead-letter records

Retain the original payload, error, topic, partition, and offset

Event time

Use event_time for windows and ingested_at for lag

Watermarks

Advance only according to the chosen lateness policy

Schema changes

Validate reader defaults and full-transitive compatibility before send

Replay

Reprocess a bounded source interval without rewriting source truth

Reliability report#

The deterministic recovery summary is generated from bounded tutorial data; it is evidence for the logical fault model, not a packet-loss benchmark.

Kafka recovery counts for sent, acknowledged, lost, duplicated, and recovered messages

Figure: logical-message recovery counts generated by the Kafka chaos workflow; it does not represent physical network loss.

Every chaos or broker-backed exercise should report the following counts in one table:

Counter

Definition

Sent

Input logical records offered to the boundary

Acknowledged

Records accepted by the producer boundary

Dropped/lost

Records intentionally not delivered

Retried

Additional attempts for the same logical record

Duplicated

Additional delivered envelopes for an identity

Late

Delivered after the configured event-time lateness policy

Reordered

Records whose delivery order differs from source order

Deduplicated

Duplicate envelopes removed before projection

Lag

Processing or delivery time minus event time

Always include input/output fingerprints and topic/partition counts. A lower duplicate rate after deduplication is not evidence that the producer was exactly-once; it only shows that the consumer projection was idempotent.

MLOps checklist#

  • verify topic and schema versions before deployment;

  • set explicit acknowledgements, idempotence, retry, and timeout policies;

  • monitor consumer lag, rebalance frequency, duplicate rate, invalid records, late-event rate, and dead-letter volume;

  • test an outage with drop, delay, and buffer-and-flush behavior;

  • replay a bounded interval and reconcile stable event IDs;

  • document the watermark and acceptable lateness policy;

  • keep broker credentials and registry credentials outside configuration files;

  • test schema evolution before registering a producer or consumer revision.

Researchers can run the same logic with simulate_delivery and no broker; the offline result is suitable for deterministic tests and teaching, but it is not a substitute for socket-level failure testing or broker capacity testing.

For broker configuration, security, capacity, and production operations, use the Confluent Kafka Python documentation and Schema Registry documentation.

Next#

Use Spark Structured Streaming for bounded downstream processing, or return to Data-quality incidents for offline fault repair.