Audit quality faults and build observable projections#
Goal. Work through a bounded, reproducible example and inspect the evidence before connecting an external service.
Prerequisites. Base FraudTwin install. Optional extras and Docker commands are clearly marked.
Produces. Tables, fingerprints, manifests, and verification output.
Source size. The default cells generate approximately 1,000 logical payments; increase duration and population together for a 10,000-payment run.
Offline path. All marked offline cells run without Docker or network services. Service cells are optional and explicitly marked in notebook metadata.
Cleanup. Outputs are written under a temporary directory; remove any local run directory if you changed the output location.
!pip install pyiceberg[s3fs]
!pip install prometheus-client requests
Requirement already satisfied: pyiceberg[s3fs] in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (0.12.0)
Requirement already satisfied: mmh3>=4.0.0 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from pyiceberg[s3fs]) (5.3.0)
Requirement already satisfied: requests>=2.20.0 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from pyiceberg[s3fs]) (2.34.2)
Requirement already satisfied: click>=7.1.1 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from pyiceberg[s3fs]) (8.5.0)
Requirement already satisfied: rich>=10.11.0 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from pyiceberg[s3fs]) (15.0.0)
Requirement already satisfied: strictyaml>=1.7.0 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from pyiceberg[s3fs]) (1.7.3)
Requirement already satisfied: pydantic!=2.12.0,!=2.12.1,!=2.4.0,!=2.4.1,<3.0,>=2.0 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from pyiceberg[s3fs]) (2.13.5)
Requirement already satisfied: fsspec>=2023.1.0 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from pyiceberg[s3fs]) (2026.9.0)
Requirement already satisfied: pyparsing>=3.1.0 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from pyiceberg[s3fs]) (3.3.3)
Requirement already satisfied: tenacity>=8.2.3 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from pyiceberg[s3fs]) (9.1.4)
Requirement already satisfied: pyroaring>=1.0.0 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from pyiceberg[s3fs]) (1.1.0)
Requirement already satisfied: cachetools>=5.5 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from pyiceberg[s3fs]) (7.2.0)
Requirement already satisfied: zstandard>=0.13.0 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from pyiceberg[s3fs]) (0.25.0)
Requirement already satisfied: s3fs>=2023.1.0 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from pyiceberg[s3fs]) (2026.9.0)
Requirement already satisfied: annotated-types>=0.6.0 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from pydantic!=2.12.0,!=2.12.1,!=2.4.0,!=2.4.1,<3.0,>=2.0->pyiceberg[s3fs]) (0.8.0)
Requirement already satisfied: pydantic-core==2.46.5 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from pydantic!=2.12.0,!=2.12.1,!=2.4.0,!=2.4.1,<3.0,>=2.0->pyiceberg[s3fs]) (2.46.5)
Requirement already satisfied: typing-extensions>=4.14.1 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from pydantic!=2.12.0,!=2.12.1,!=2.4.0,!=2.4.1,<3.0,>=2.0->pyiceberg[s3fs]) (4.16.0)
Requirement already satisfied: typing-inspection>=0.4.2 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from pydantic!=2.12.0,!=2.12.1,!=2.4.0,!=2.4.1,<3.0,>=2.0->pyiceberg[s3fs]) (0.4.4)
Requirement already satisfied: charset_normalizer<4,>=2 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from requests>=2.20.0->pyiceberg[s3fs]) (3.5.1)
Requirement already satisfied: idna<4,>=2.5 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from requests>=2.20.0->pyiceberg[s3fs]) (3.19)
Requirement already satisfied: urllib3<3,>=1.26 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from requests>=2.20.0->pyiceberg[s3fs]) (2.7.0)
Requirement already satisfied: certifi>=2023.5.7 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from requests>=2.20.0->pyiceberg[s3fs]) (2026.7.22)
Requirement already satisfied: markdown-it-py>=2.2.0 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from rich>=10.11.0->pyiceberg[s3fs]) (3.0.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from rich>=10.11.0->pyiceberg[s3fs]) (2.21.0)
Requirement already satisfied: mdurl~=0.1 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from markdown-it-py>=2.2.0->rich>=10.11.0->pyiceberg[s3fs]) (0.1.2)
Requirement already satisfied: aiobotocore<4.0.0,>=2.19.0 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from s3fs>=2023.1.0->pyiceberg[s3fs]) (3.9.1)
Requirement already satisfied: aiohttp!=4.0.0a0,!=4.0.0a1,>=3.9.0 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from s3fs>=2023.1.0->pyiceberg[s3fs]) (3.14.3)
Requirement already satisfied: aioitertools<1.0.0,>=0.5.1 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from aiobotocore<4.0.0,>=2.19.0->s3fs>=2023.1.0->pyiceberg[s3fs]) (0.13.0)
Requirement already satisfied: botocore<1.43.76,>=1.43.66 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from aiobotocore<4.0.0,>=2.19.0->s3fs>=2023.1.0->pyiceberg[s3fs]) (1.43.75)
Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from aiobotocore<4.0.0,>=2.19.0->s3fs>=2023.1.0->pyiceberg[s3fs]) (1.1.0)
Requirement already satisfied: multidict<7.0.0,>=6.0.0 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from aiobotocore<4.0.0,>=2.19.0->s3fs>=2023.1.0->pyiceberg[s3fs]) (6.9.1)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from aiobotocore<4.0.0,>=2.19.0->s3fs>=2023.1.0->pyiceberg[s3fs]) (2.9.0.post0)
Requirement already satisfied: wrapt<3.0.0,>=1.10.10 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from aiobotocore<4.0.0,>=2.19.0->s3fs>=2023.1.0->pyiceberg[s3fs]) (2.4.1)
Requirement already satisfied: aiohappyeyeballs>=2.5.0 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1,>=3.9.0->s3fs>=2023.1.0->pyiceberg[s3fs]) (2.7.1)
Requirement already satisfied: aiosignal>=1.4.0 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1,>=3.9.0->s3fs>=2023.1.0->pyiceberg[s3fs]) (1.4.0)
Requirement already satisfied: attrs>=17.3.0 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1,>=3.9.0->s3fs>=2023.1.0->pyiceberg[s3fs]) (26.1.0)
Requirement already satisfied: frozenlist>=1.1.1 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1,>=3.9.0->s3fs>=2023.1.0->pyiceberg[s3fs]) (1.8.0)
Requirement already satisfied: propcache>=0.2.0 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1,>=3.9.0->s3fs>=2023.1.0->pyiceberg[s3fs]) (0.5.4)
Requirement already satisfied: yarl<2.0,>=1.17.0 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from aiohttp!=4.0.0a0,!=4.0.0a1,>=3.9.0->s3fs>=2023.1.0->pyiceberg[s3fs]) (1.25.1)
Requirement already satisfied: six>=1.5 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from python-dateutil<3.0.0,>=2.1->aiobotocore<4.0.0,>=2.19.0->s3fs>=2023.1.0->pyiceberg[s3fs]) (1.17.0)
Requirement already satisfied: prometheus-client in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (0.26.0)
Requirement already satisfied: requests in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (2.34.2)
Requirement already satisfied: charset_normalizer<4,>=2 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from requests) (3.5.1)
Requirement already satisfied: idna<4,>=2.5 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from requests) (3.19)
Requirement already satisfied: urllib3<3,>=1.26 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from requests) (2.7.0)
Requirement already satisfied: certifi>=2023.5.7 in /home/emc/Projects/test/.venv/lib/python3.14/site-packages (from requests) (2026.7.22)
Optional lakehouse and observability setup#
The first cell installs the clients. Start the local catalog and metrics stack only for service-backed verification; operational projections and SLO calculations remain offline.
docker compose --profile lakehouse --profile observability up -d minio minio-init iceberg-rest prometheus grafana
Stop with docker compose --profile lakehouse --profile observability down; unavailable services use the local fallback.
Set up a deterministic source run
import json
from pathlib import Path
import polars as pl
from fraudtwin.config import load_config
from fraudtwin.generation import generate
root = next(
(p for p in (Path.cwd(), *Path.cwd().parents) if (p / "configs" / "minimal.yaml").exists()),
Path.cwd(),
)
base = load_config(root / "configs" / "minimal.yaml")
# Scale the population so the bounded example produces about 1,000 payments.
population = base.population.model_copy(
update={
"customers": 200,
"accounts": 300,
"cards": 240,
"devices": 240,
"pix_keys": 160,
"merchants": 60,
}
)
simulation = base.simulation.model_copy(update={"duration_days": 10})
fraud = base.fraud.model_copy(update={"enabled": True, "target_rate": 0.05})
config = base.model_copy(
update={"population": population, "simulation": simulation, "fraud": fraud}
)
data = generate(config, write=False)
run_id = data.run_id
payments = pl.DataFrame([item.model_dump(mode="json") for item in data.behavior.payments])
print({"run_id": run_id, "payments": len(payments), "events": len(data.behavior.payment_events)})
{'run_id': 'RUN-2a3ad02ee370aeb8', 'payments': 1092, 'events': 4699}
Inspect schema, grain, and counts
from fraudtwin.lakehouse import build_bronze_records, silver_rows
bronze = build_bronze_records(data.entities, data.behavior, data.manifest, include_oracle=False)
print({"bronze": len(bronze), "subjects": sorted({r.subject for r in bronze})})
{'bronze': 9386, 'subjects': ['accounts', 'behavior_profiles', 'cards', 'customer-dispute', 'customers', 'devices', 'fraud-alert', 'fraud-case', 'fraud-case-confirmation', 'fraud-label', 'institutions', 'ledger_entries', 'merchants', 'payment-event', 'payments', 'pix_keys']}
Run the core operation
display(pl.DataFrame([r.as_row() for r in bronze[:10]]))
shape: (10, 16)
| run_id | source | subject | table_name | event_family | record_id | record_key | contract_fingerprint | payload_json | raw_payload_b64 | headers_json | event_time | received_at | kafka_topic | kafka_partition | kafka_offset |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| str | str | str | str | str | str | str | str | str | str | str | str | str | null | null | null |
| "RUN-2a3ad02ee370aeb8" | "parquet" | "payment-event" | "payment_events" | "payment-event" | "EVT-F01-000001-000001" | "PAY-F01-000001-000001" | "e7f8a63404fb5037bd6e83333550eb… | "{"account_id":"ACC-000123","af… | "KkVWVC1GMDEtMDAwMDAxLTAwMDAwMS… | "{"fraudtwin-contract-fingerpri… | "2026-01-01T00:00:00+00:00" | "2026-01-01T00:00:00+00:00" | null | null | null |
| "RUN-2a3ad02ee370aeb8" | "parquet" | "payment-event" | "payment_events" | "payment-event" | "EVT-F02-000002-000001" | "PAY-F02-000002-000001" | "e7f8a63404fb5037bd6e83333550eb… | "{"account_id":"ACC-000142","af… | "KkVWVC1GMDItMDAwMDAyLTAwMDAwMS… | "{"fraudtwin-contract-fingerpri… | "2026-01-01T00:00:00+00:00" | "2026-01-01T00:00:00+00:00" | null | null | null |
| "RUN-2a3ad02ee370aeb8" | "parquet" | "payment-event" | "payment_events" | "payment-event" | "EVT-F03-000003-SIGNAL-01" | "PAY-F03-000003-000001" | "e7f8a63404fb5037bd6e83333550eb… | "{"account_id":"ACC-000021","af… | "MEVWVC1GMDMtMDAwMDAzLVNJR05BTC… | "{"fraudtwin-contract-fingerpri… | "2026-01-01T00:00:00+00:00" | "2026-01-01T00:00:00+00:00" | null | null | null |
| "RUN-2a3ad02ee370aeb8" | "parquet" | "payment-event" | "payment_events" | "payment-event" | "EVT-F04-000004-000001" | "PAY-F04-000004-000001" | "e7f8a63404fb5037bd6e83333550eb… | "{"account_id":"ACC-000021","af… | "KkVWVC1GMDQtMDAwMDA0LTAwMDAwMR… | "{"fraudtwin-contract-fingerpri… | "2026-01-01T00:00:00+00:00" | "2026-01-01T00:00:00+00:00" | null | null | null |
| "RUN-2a3ad02ee370aeb8" | "parquet" | "payment-event" | "payment_events" | "payment-event" | "EVT-F05-000005-000001" | "PAY-F05-000005-000001" | "e7f8a63404fb5037bd6e83333550eb… | "{"account_id":"ACC-000023","af… | "KkVWVC1GMDUtMDAwMDA1LTAwMDAwMS… | "{"fraudtwin-contract-fingerpri… | "2026-01-01T00:00:00+00:00" | "2026-01-01T00:00:00+00:00" | null | null | null |
| "RUN-2a3ad02ee370aeb8" | "parquet" | "payment-event" | "payment_events" | "payment-event" | "EVT-HN-F01-000001-000001" | "PAY-HN-F01-000001-000001" | "e7f8a63404fb5037bd6e83333550eb… | "{"account_id":"ACC-000186","af… | "MEVWVC1ITi1GMDEtMDAwMDAxLTAwMD… | "{"fraudtwin-contract-fingerpri… | "2026-01-01T00:00:00+00:00" | "2026-01-01T00:00:00+00:00" | null | null | null |
| "RUN-2a3ad02ee370aeb8" | "parquet" | "payment-event" | "payment_events" | "payment-event" | "EVT-HN-F02-000002-000001" | "PAY-HN-F02-000002-000001" | "e7f8a63404fb5037bd6e83333550eb… | "{"account_id":"ACC-000294","af… | "MEVWVC1ITi1GMDItMDAwMDAyLTAwMD… | "{"fraudtwin-contract-fingerpri… | "2026-01-01T00:00:00+00:00" | "2026-01-01T00:00:00+00:00" | null | null | null |
| "RUN-2a3ad02ee370aeb8" | "parquet" | "payment-event" | "payment_events" | "payment-event" | "EVT-HN-F03-000003-SIGNAL-01" | "PAY-HN-F03-000003-000001" | "e7f8a63404fb5037bd6e83333550eb… | "{"account_id":"ACC-000021","af… | "NkVWVC1ITi1GMDMtMDAwMDAzLVNJR0… | "{"fraudtwin-contract-fingerpri… | "2026-01-01T00:00:00+00:00" | "2026-01-01T00:00:00+00:00" | null | null | null |
| "RUN-2a3ad02ee370aeb8" | "parquet" | "payment-event" | "payment_events" | "payment-event" | "EVT-HN-F04-000004-000001" | "PAY-HN-F04-000004-000001" | "e7f8a63404fb5037bd6e83333550eb… | "{"account_id":"ACC-000021","af… | "MEVWVC1ITi1GMDQtMDAwMDA0LTAwMD… | "{"fraudtwin-contract-fingerpri… | "2026-01-01T00:00:00+00:00" | "2026-01-01T00:00:00+00:00" | null | null | null |
| "RUN-2a3ad02ee370aeb8" | "parquet" | "payment-event" | "payment_events" | "payment-event" | "EVT-HN-F05-000005-000001" | "PAY-HN-F05-000005-000001" | "e7f8a63404fb5037bd6e83333550eb… | "{"account_id":"ACC-000144","af… | "MEVWVC1ITi1GMDUtMDAwMDA1LTAwMD… | "{"fraudtwin-contract-fingerpri… | "2026-01-01T00:00:00+00:00" | "2026-01-01T00:00:00+00:00" | null | null | null |
Measure and interpret the result
silver = silver_rows(bronze)
print({"silver": len(silver), "duplicates_removed": len(bronze) - len(silver)})
{'silver': 9386, 'duplicates_removed': 0}
Exercise a parameter or failure mode
quality = {
"null_payloads": sum(not row["payload_json"] for row in silver),
"unique_ids": len({row["record_id"] for row in silver}),
}
print(quality)
{'null_payloads': 0, 'unique_ids': 9386}
Write a compact artifact and fingerprint
assert quality["null_payloads"] == 0
print("observable projection verified; oracle tables remain excluded")
observable projection verified; oracle tables remain excluded
Verify invariants and clean up
# A compact inspection is more useful than printing an entire run.
sample_columns = [
c for c in ("payment_id", "amount", "initiated_at", "payer_account_id") if c in payments.columns
]
sample_rows = payments.select(sample_columns).head(8).to_dicts()
print(f"Sample payments ({len(sample_rows)} of {payments.height} rows):")
for row in sample_rows:
print(
f" - {row.get('payment_id')}: amount={row.get('amount')}, "
f"initiated_at={row.get('initiated_at')}, payer={row.get('payer_account_id')}"
)
nulls = {name: count for name, count in payments.null_count().to_dicts()[0].items() if count}
print("\nData quality summary:")
print(f" rows: {payments.height}")
print(f" columns: {payments.width}")
if not nulls:
print(" nulls: none")
else:
print(" columns with nulls:")
for name, count in sorted(nulls.items()):
print(f" - {name}: {count}")
Sample payments (8 of 1092 rows):
- PAY-00000001: amount=70.7, initiated_at=2026-01-03T16:25:00Z, payer=ACC-000123
- PAY-00000002: amount=25.52, initiated_at=2026-01-05T11:37:00Z, payer=ACC-000174
- PAY-00000003: amount=18.37, initiated_at=2026-01-05T11:21:00Z, payer=ACC-000174
- PAY-00000004: amount=5.54, initiated_at=2026-01-02T22:30:00Z, payer=ACC-000003
- PAY-00000005: amount=13.71, initiated_at=2026-01-02T09:41:00Z, payer=ACC-000029
- PAY-00000006: amount=70.06, initiated_at=2026-01-04T10:40:00Z, payer=ACC-000179
- PAY-00000007: amount=42.73, initiated_at=2026-01-02T18:04:00Z, payer=ACC-000247
- PAY-00000008: amount=36.42, initiated_at=2026-01-05T09:27:00Z, payer=ACC-000255
Data quality summary:
rows: 1092
columns: 15
columns with nulls:
- card_id: 565
- merchant_id: 565
- payee_account_id: 86
- payee_institution_id: 86
- payee_pix_key_id: 857
- payer_institution_id: 86
- payer_pix_key_id: 857
Optional service integration
summary = {
"run_id": run_id,
"payments": len(data.behavior.payments),
"payment_events": len(data.behavior.payment_events),
"fraud_records": len(data.behavior.fraud_records),
}
assert summary["payments"] == len(payments)
assert summary["payments"] > 0
print(json.dumps(summary, indent=2, default=str))
{
"run_id": "RUN-2a3ad02ee370aeb8",
"payments": 1092,
"payment_events": 4699,
"fraud_records": 51
}
Review the expected outcome
import os
try:
from pyiceberg.catalog import load_catalog
catalog = load_catalog(
"fraudtwin",
type="rest",
uri=os.getenv("FRAUDTWIN_ICEBERG_CATALOG_URI", "http://localhost:8181"),
)
import requests
namespaces = catalog.list_namespaces()
prometheus = requests.get("http://localhost:9090/-/ready", timeout=3)
print({"connected": True, "namespaces": namespaces, "prometheus": prometheus.status_code})
except Exception as exc:
print({"connected": False, "offline_fallback": True, "reason": type(exc).__name__})
{'connected': True, 'namespaces': [], 'prometheus': 200}
Record the generated shape and tutorial contract.#
summary = {
"payments": len(data.behavior.payments),
"events": len(data.behavior.payment_events),
}
print(summary)
assert summary["payments"] >= 0
{'payments': 1092, 'events': 4699}
assert len(bronze) > 0
print({"offline_fallback": True, "bronze_records": len(bronze)})
{'offline_fallback': True, 'bronze_records': 9386}