fraudtwin.ml#

Point-in-time datasets, replay, and machine-learning evaluation.

Status: Stable

Classes#

`fraudtwin.ml.BacktestResult`	Fold rows, metrics, and the immutable result manifest.
`fraudtwin.ml.BaselineEvaluationConfig`	Strict, analysis-only configuration for M19.
`fraudtwin.ml.BenchmarkPack`	Versioned frozen evaluation definition.
`fraudtwin.ml.BenchmarkPackWindows`	Fixed train, validation, test, and optional stress windows.
`fraudtwin.ml.BenchmarkWindow`	One immutable half-open benchmark interval.
`fraudtwin.ml.DatasetBuilder`	alias of `PointInTimeDatasetBuilder`
`fraudtwin.ml.DriftConfig`	Immutable policy controlling a drift comparison.
`fraudtwin.ml.DriftMetric`	One measured drift value and its decision threshold.
`fraudtwin.ml.DriftReport`	Reproducible comparison of two row windows.
`fraudtwin.ml.EvaluationResult`	EvaluationResult(predictions: tuple[dict[str, typing.Any], ...], metrics: tuple[dict[str, typing.Any], ...], manifest: dict[str, typing.Any], model_artifacts: dict[str, bytes] \| None = None)
`fraudtwin.ml.FoldSpec`	FoldSpec(fold_id: str, train_from: datetime.datetime, train_to: datetime.datetime, validation_from: datetime.datetime \| None, validation_to: datetime.datetime \| None, test_from: datetime.datetime, test_to: datetime.datetime, stress_from: datetime.datetime \| None = None, stress_to: datetime.datetime \| None = None)
`fraudtwin.ml.PointInTimeDataset`	Stable rows and the manifest that describes their construction.
`fraudtwin.ml.PointInTimeDatasetBuilder`	Build historical features and labels using only available source data.
`fraudtwin.ml.PredictionAdapter`	Small file/library adapter for external point-in-time predictions.
`fraudtwin.ml.PredictionRecord`	One external prediction at a point in time.

Functions#

`fraudtwin.ml.build_dataset`	Convenience API for deterministic in-memory M9 construction.
`fraudtwin.ml.build_point_in_time_dataset`	Convenience API for deterministic in-memory M9 construction.
`fraudtwin.ml.compare_performance`	Compare matching evaluation metrics under one threshold policy.
`fraudtwin.ml.compare_windows`	Compare two row windows using deterministic distribution metrics.
`fraudtwin.ml.evaluate_predictions`	Evaluate external scores against a leakage-safe dataset.
`fraudtwin.ml.heuristic_predictions`	Score PIT rows with the dependency-free deterministic baseline.
`fraudtwin.ml.load_baseline_config`	Load and validate a baseline-model YAML policy.
`fraudtwin.ml.load_benchmark_pack`	Load and validate one versioned benchmark-pack YAML file.
`fraudtwin.ml.load_generated_run`	Load one existing generated run without regenerating unrelated records.
`fraudtwin.ml.load_model_artifact`	Load and validate a persisted baseline artifact once for reuse.
`fraudtwin.ml.load_predictions`	Load strict Parquet or JSONL predictions.
`fraudtwin.ml.run_backtest`	Build reproducible rolling folds from one existing generated history.
`fraudtwin.ml.run_model_backtest`	Run selected M19 model adapters over the existing M10 temporal folds.
`fraudtwin.ml.score_loaded_model`	Score rows with an already-loaded baseline artifact.
`fraudtwin.ml.score_model_artifact`	Score feature rows with a persisted FraudTwin baseline artifact.
`fraudtwin.ml.train_baselines`	Train configured deterministic baselines on train rows only.
`fraudtwin.ml.write_backtest`	Persist fold rows, metrics, and the append-only backtest manifest.
`fraudtwin.ml.write_evaluation`	Persist predictions, metrics, model artifacts, and the evaluation manifest.
`fraudtwin.ml.write_point_in_time_dataset`	Write the fixed-schema M9 Parquet table and its JSON manifest.
`fraudtwin.ml.write_predictions`	Write canonical external predictions as a typed Parquet file.

Constants and protocols#

Name	Reference
`ALL_MODEL_NAMES`	`fraudtwin.ml.ALL_MODEL_NAMES`
`BACKTEST_ROW_SCHEMA`	`fraudtwin.ml.BACKTEST_ROW_SCHEMA`
`DATASET_SCHEMA`	`fraudtwin.ml.DATASET_SCHEMA`
`FOLD_METRIC_SCHEMA`	`fraudtwin.ml.FOLD_METRIC_SCHEMA`
`METRIC_NAMES`	`fraudtwin.ml.METRIC_NAMES`
`MODEL_FEATURES`	`fraudtwin.ml.MODEL_FEATURES`
`MODEL_NAMES`	`fraudtwin.ml.MODEL_NAMES`
`PIT_DATASET_SCHEMA`	`fraudtwin.ml.PIT_DATASET_SCHEMA`
`PREDICTION_SCHEMA`	`fraudtwin.ml.PREDICTION_SCHEMA`

Detailed API#

Small, local point-in-time dataset and backtesting tools.

fraudtwin.ml.DatasetBuilder[source]: alias of PointInTimeDatasetBuilder

class fraudtwin.ml.PointInTimeDataset(rows, manifest)[source][source]

Bases: object

Stable rows and the manifest that describes their construction.

Parameters:

rows (tuple[dict[str, Any], ...])
manifest (DatasetManifest)

property frame: DataFrame: Return the dataset as a frame with the stable output schema.

class fraudtwin.ml.PointInTimeDatasetBuilder(config, entities, behavior, source_manifest=None)[source][source]

Bases: object

Build historical features and labels using only available source data.

Parameters:

config (SimulationRunConfig)
entities (EntityDataset)
behavior (BehaviorDataset)
source_manifest (RunManifest | None)

build(prediction_times=None)[source][source]

Build rows in stable payment-ID order.

By default each payment is scored when its initial source event becomes available. Callers may provide a deterministic payment-to-prediction mapping for snapshot or delayed-prediction use cases.

Return type:: PointInTimeDataset
Parameters:: prediction_times (Mapping[str, datetime] | None)

build_rows(prediction_times=None)[source][source]

Build every eligible PIT row without applying M9 split boundaries.

M10 uses this shared row construction to assign several independent rolling folds while retaining the exact M9 availability logic.

Return type:: tuple[dict[str, Any], ...]
Parameters:: prediction_times (Mapping[str, datetime] | None)

fraudtwin.ml.build_point_in_time_dataset(config, entities, behavior, source_manifest=None, prediction_times=None)[source][source]

Convenience API for deterministic in-memory M9 construction.

Return type:

PointInTimeDataset

Parameters:

config (SimulationRunConfig)
entities (EntityDataset)
behavior (BehaviorDataset)
source_manifest (RunManifest | None)
prediction_times (Mapping[str, datetime] | None)

fraudtwin.ml.build_dataset(config, entities, behavior, source_manifest=None, prediction_times=None)[source]

Convenience API for deterministic in-memory M9 construction.

Return type:

PointInTimeDataset

Parameters:

config (SimulationRunConfig)
entities (EntityDataset)
behavior (BehaviorDataset)
source_manifest (RunManifest | None)
prediction_times (Mapping[str, datetime] | None)

fraudtwin.ml.load_generated_run(run_dir, *, allow_missing_delivery=False)[source][source]

Load one existing generated run without regenerating unrelated records.

Return type:

tuple[EntityDataset, BehaviorDataset, RunManifest]

Parameters:

run_dir (Path)
allow_missing_delivery (bool)

fraudtwin.ml.write_point_in_time_dataset(dataset, output_path, manifest_path=None)[source][source]

Write the fixed-schema M9 Parquet table and its JSON manifest.

Return type:

tuple[Path, Path]

Parameters:

dataset (PointInTimeDataset)
output_path (Path)
manifest_path (Path | None)

class fraudtwin.ml.DriftConfig(**data)[source][source]

Bases: BaseModel

Immutable policy controlling a drift comparison.

reference_name and comparison_name identify the windows in the report. Numeric PSI bins are fitted from the reference values only. Thresholds are absolute metric values; an alert is raised when a metric is greater than or equal to its threshold.

Parameters:

reference_name (str)
comparison_name (str)
numeric_bins (int)
psi_threshold (float)
wasserstein_threshold (float)
js_threshold (float)
prevalence_threshold (float)
performance_threshold (float)
minimum_samples (int)
smoothing (float)
fields (tuple[str, ...] | None)
label_policy (str)

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'frozen': True}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class fraudtwin.ml.DriftMetric(**data)[source][source]

Bases: BaseModel

One measured drift value and its decision threshold.

Parameters:

field (str)
field_type (Literal['numeric', 'categorical', 'prevalence', 'quality', 'performance'])
method (str)
reference_value (float | None)
comparison_value (float | None)
threshold (float | None)
alerted (bool)
reference_count (int)
comparison_count (int)
metadata (dict[str, Any])

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'frozen': True}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class fraudtwin.ml.DriftReport(**data)[source][source]

Bases: BaseModel

Reproducible comparison of two row windows.

Parameters:

report_version (str)
reference_window (str)
comparison_window (str)
reference_count (int)
comparison_count (int)
reference_fingerprint (str)
comparison_fingerprint (str)
label_policy (str)
metrics (tuple[DriftMetric, ...])
performance_metrics (tuple[DriftMetric, ...])
manifest (dict[str, Any])

property alerts: tuple[DriftMetric, ...]: Return metrics that exceeded their configured thresholds.

property fingerprint: str: Return a stable fingerprint for policy, inputs, and measurements.

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'frozen': True}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

fraudtwin.ml.compare_windows(reference, comparison, config=None)[source][source]

Compare two row windows using deterministic distribution metrics.

Parameters:

reference (Iterable[Mapping[str, Any]]) – Baseline rows. Numeric bins are fitted from this window.
comparison (Iterable[Mapping[str, Any]]) – New rows to evaluate against the baseline.
config (DriftConfig | None) – Thresholds, window names, and fields. Defaults to DriftConfig.

Return type:

DriftReport

Returns:

A report containing feature, missingness, prevalence, and duplicate metrics plus input fingerprints.

Raises:

ValueError – If either input is empty or configured fields are absent.

fraudtwin.ml.compare_performance(reference, comparison, *, config=None, label_policy=None)[source][source]

Compare matching evaluation metrics under one threshold policy.

Parameters:

reference (Mapping[str, float]) – Metric name to value mapping for the baseline window.
comparison (Mapping[str, float]) – Metric name to value mapping for the new window.
config (DriftConfig | None) – Provides performance_threshold and sample policy.
label_policy (str | None) – Optional explicit policy; it must match the config when both are supplied.

Return type:

tuple[DriftMetric, ...]

Returns:

One relative absolute-change metric for every common metric.

Raises:

ValueError – If metric keys differ or label policies conflict.

class fraudtwin.ml.BaselineEvaluationConfig(**data)[source][source]

Bases: BaseModel

Strict, analysis-only configuration for M19.

Parameters:

models (tuple[str, ...])
fixed_fpr (float)
fixed_recall (float)
ranking_k_fraction (float)
label_policy (Literal['exclude_unresolved'])
random_seed (int | None)
deterministic_presets (dict[str, Any])
tracking_uri (str | None)

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'frozen': True}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class fraudtwin.ml.EvaluationResult(predictions, metrics, manifest, model_artifacts=None)[source][source]

Bases: object

Parameters:

predictions (tuple[dict[str, Any], ...])
metrics (tuple[dict[str, Any], ...])
manifest (dict[str, Any])
model_artifacts (dict[str, bytes] | None)

class fraudtwin.ml.PredictionRecord(**data)[source][source]

Bases: BaseModel

One external prediction at a point in time.

Parameters:

event_id (str | None)
payment_id (str | None)
customer_id (str | None)
account_id (str | None)
prediction_timestamp (datetime)
fraud_score (float)
predicted_class (Literal['FRAUD', 'LEGITIMATE'] | None)

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'frozen': True}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

property score: float

Backward-compatible alias for fraud_score.

fraud_score is the canonical serialized field. The short alias is intentionally read-only so existing scoring notebooks remain concise without creating a second source of truth in prediction artifacts.

class fraudtwin.ml.PredictionAdapter[source][source]

Bases: object

Small file/library adapter for external point-in-time predictions.

fraudtwin.ml.evaluate_predictions(rows, predictions, config, *, model_id='external', source_run_dir=None)[source][source]

Evaluate external scores against a leakage-safe dataset.

Parameters:

rows (Sequence[Mapping[str, Any]]) – PIT rows containing split and label columns.
predictions (Sequence[PredictionRecord]) – Scores keyed by the row’s event, payment, customer, or account ID.
config (BaselineEvaluationConfig) – Threshold and metric policy.
model_id (str) – Stable name recorded in metrics and lineage.
source_run_dir (Path | None) – Optional run directory used to enrich lineage metadata.

Return type:

EvaluationResult

Returns:

Predictions, partition/segment metrics, and a reproducibility manifest.

Raises:

ValueError – If predictions do not resolve one-to-one to PIT rows.

fraudtwin.ml.heuristic_predictions(rows)[source][source]

Score PIT rows with the dependency-free deterministic baseline.

Return type:: tuple[PredictionRecord, ...]
Parameters:: rows (Sequence[Mapping[str, Any]])

fraudtwin.ml.load_baseline_config(path)[source][source]

Load and validate a baseline-model YAML policy.

Parameters:

path (Path) – YAML file containing model names, thresholds, and tracking options.

Return type:

BaselineEvaluationConfig

Returns:

An immutable BaselineEvaluationConfig.

Raises:

FileNotFoundError – If path does not exist.
ValueError – If the YAML root or any option is invalid.

fraudtwin.ml.load_model_artifact(path)[source][source]

Load and validate a persisted baseline artifact once for reuse.

Return type:: dict[str, Any]
Parameters:: path (Path)

fraudtwin.ml.load_predictions(path)[source][source]

Load strict Parquet or JSONL predictions.

Return type:: tuple[PredictionRecord, ...]
Parameters:: path (Path)

fraudtwin.ml.train_baselines(rows, config, *, source_run_dir=None)[source][source]

Train configured deterministic baselines on train rows only.

Parameters:

rows (Sequence[Mapping[str, Any]]) – PIT rows with train, validation, and test splits.
config (BaselineEvaluationConfig) – Model list and evaluation policy.
source_run_dir (Path | None) – Optional source run used for lineage metadata.

Return type:

EvaluationResult

Returns:

Evaluation results and in-memory model artifacts for optional models.

Raises:

ValueError – If train/validation partitions or both classes are missing.
RuntimeError – If an optional model dependency is unavailable.

fraudtwin.ml.write_evaluation(result, output_dir)[source][source]

Persist predictions, metrics, model artifacts, and the evaluation manifest.

Parameters:

result (EvaluationResult) – Output returned by train_baselines() or evaluate_predictions().
output_dir (Path) – New directory for the immutable evaluation artifacts.

Return type:

tuple[Path, Path, Path]

Returns:

Paths to predictions, metrics, and manifest files.

Raises:

FileExistsError – If output_dir already exists.
RuntimeError – If MLflow tracking is requested but not installed.

fraudtwin.ml.write_predictions(predictions, path)[source][source]

Write canonical external predictions as a typed Parquet file.

Parameters:

predictions (Iterable[PredictionRecord]) – Point-in-time scores keyed by exactly one target ID.
path (Path) – Destination path; parent directories are created as needed.

Return type:

Path

Returns:

The destination path.

fraudtwin.ml.score_model_artifact(path, rows)[source][source]

Score feature rows with a persisted FraudTwin baseline artifact.

Parameters:

path (Path) – .joblib artifact written under an evaluation models directory.
rows (Sequence[Mapping[str, Any]]) – PIT-shaped feature mappings. Missing numeric values use the same deterministic zero policy used during training.

Return type:

tuple[float, ...]

Returns:

Fraud probabilities in input order.

Raises:

FileNotFoundError – If the artifact path is missing.
RuntimeError – If joblib is not installed.
ValueError – If the artifact does not contain a compatible model.

fraudtwin.ml.score_loaded_model(artifact, rows)[source][source]

Score rows with an already-loaded baseline artifact.

Return type:

tuple[float, ...]

Parameters:

artifact (Mapping[str, Any])
rows (Sequence[Mapping[str, Any]])

class fraudtwin.ml.BenchmarkPack(**data)[source][source]

Bases: BaseModel

Versioned frozen evaluation definition.

Parameters:

id (str)
version (str)
source_seed (int)
source_config_hash (str)
windows (BenchmarkPackWindows)
regimes (tuple[FraudRegimeConfig, ...])
label_policy (Literal['exclude', 'include_unresolved'])
scenario_parameters (dict[str, object])
metric_definitions (list[str])
label_maturity_gap_seconds (int)

model_config: ClassVar[ConfigDict] = {'extra': 'forbid'}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class fraudtwin.ml.BenchmarkPackWindows(**data)[source][source]

Bases: BaseModel

Fixed train, validation, test, and optional stress windows.

Parameters:

train (BenchmarkWindow)
validation (BenchmarkWindow | None)
test (BenchmarkWindow)
stress (BenchmarkWindow | None)

model_config: ClassVar[ConfigDict] = {'extra': 'forbid'}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class fraudtwin.ml.BenchmarkWindow(**data)[source][source]

Bases: BaseModel

One immutable half-open benchmark interval.

Parameters:

from_time (datetime)
to (datetime)

model_config: ClassVar[ConfigDict] = {'extra': 'forbid'}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class fraudtwin.ml.BacktestResult(fold_rows, fold_metrics, manifest)[source][source]

Bases: object

Fold rows, metrics, and the immutable result manifest.

Parameters:

fold_rows (tuple[dict[str, Any], ...])
fold_metrics (tuple[dict[str, object], ...])
manifest (BacktestManifest)

class fraudtwin.ml.FoldSpec(fold_id, train_from, train_to, validation_from, validation_to, test_from, test_to, stress_from=None, stress_to=None)[source][source]

Bases: object

Parameters:

fold_id (str)
train_from (datetime)
train_to (datetime)
validation_from (datetime | None)
validation_to (datetime | None)
test_from (datetime)
test_to (datetime)
stress_from (datetime | None)
stress_to (datetime | None)

fraudtwin.ml.load_benchmark_pack(path)[source][source]

Load and validate one versioned benchmark-pack YAML file.

Return type:: BenchmarkPack
Parameters:: path (Path)

fraudtwin.ml.run_backtest(config, entities, behavior, source_manifest, *, benchmark_pack=None)[source][source]

Build reproducible rolling folds from one existing generated history.

Return type:

BacktestResult

Parameters:

config (SimulationRunConfig)
entities (EntityDataset)
behavior (BehaviorDataset)
source_manifest (RunManifest)
benchmark_pack (BenchmarkPack | None)

fraudtwin.ml.run_model_backtest(config, entities, behavior, source_manifest, *, models, benchmark_pack=None)[source][source]

Run selected M19 model adapters over the existing M10 temporal folds.

Return type:

BacktestResult

Parameters:

config (SimulationRunConfig)
entities (EntityDataset)
behavior (BehaviorDataset)
source_manifest (RunManifest)
models (tuple[str, ...])
benchmark_pack (BenchmarkPack | None)

fraudtwin.ml.write_backtest(result, output_dir)[source][source]

Persist fold rows, metrics, and the append-only backtest manifest.

Return type:

tuple[Path, Path, Path]

Parameters:

result (BacktestResult)
output_dir (Path)