Model lifecycle: from simulation to service#

Level: Intermediate

You will: move from a generated dataset to a tracked model artifact and a

validated local scoring hand-off. Before you start: ML evaluation.

Services: None for local artifacts; MLflow and FastAPI are optional.

This guide is the production-shaped path through FraudTwin. It keeps the generator, feature construction, evaluation, artifact, and serving boundaries explicit:

simulation run → PIT dataset → train → time-held-out evaluation
             → artifact/MLflow → HTTP or Kafka scoring → monitoring

1. Generate and inspect the source#

Use a bounded 1,000–10,000-payment run for local development. Record the run manifest, configuration hash, schema versions, payment/event counts, and the fraud scenario mix before training.

2. Build leakage-safe data#

CONFIG=configs/benchmarks/camouflage-v1.yaml
RUN_ID=RUN-...
RUNS_DIR=./runs

fraudtwin ml build-dataset "$CONFIG" \
  --run-id "$RUN_ID" --output-dir "$RUNS_DIR"

Inspect prediction_time, source_available_at, feature_available_at, and label_available_at. A feature must be available at prediction time; an unresolved label must not silently become a negative label.

3. Train and choose a model#

DATASET=./runs/$RUN_ID/ml/dataset.parquet
EVALUATIONS_DIR=./runs/evaluations

fraudtwin ml train "$DATASET" \
  --config configs/ml-baselines.yaml --output-dir "$EVALUATIONS_DIR"

Compare a deterministic heuristic with Logistic Regression, LightGBM, XGBoost, or CatBoost on identical temporal splits. Prefer the simplest model that meets the PR-AUC, recall-at-FPR, calibration, latency, and interpretability goals.

4. Promote an artifact#

An evaluation directory contains predictions, metrics, an evaluation manifest, and optional models/*.joblib artifacts. The manifest records the feature allowlist, preprocessing policy, source run, seed, and package versions. When a tracking URI is configured, write_evaluation also logs the run to MLflow.

5. Serve and monitor#

The reference FastAPI service loads one artifact per process and validates typed point-in-time requests. Keep offline and online feature code identical, emit model/version metadata with every score, and monitor input drift, score drift, latency, errors, and delayed-label quality.

Next#

Follow Production serving for the local HTTP boundary, then review Compatibility before a package upgrade.