fraudtwin.ml.backtest#

Replay, folds, backtesting, and metrics.

Status: Stable

Classes#

fraudtwin.ml.backtest.BacktestResult

Fold rows, metrics, and the immutable result manifest.

fraudtwin.ml.backtest.BenchmarkPack

Versioned frozen evaluation definition.

fraudtwin.ml.backtest.BenchmarkPackWindows

Fixed train, validation, test, and optional stress windows.

fraudtwin.ml.backtest.BenchmarkWindow

One immutable half-open benchmark interval.

fraudtwin.ml.backtest.FoldSpec

FoldSpec(fold_id: str, train_from: datetime.datetime, train_to: datetime.datetime, validation_from: datetime.datetime | None, validation_to: datetime.datetime | None, test_from: datetime.datetime, test_to: datetime.datetime, stress_from: datetime.datetime | None = None, stress_to: datetime.datetime | None = None)

Functions#

fraudtwin.ml.backtest.load_benchmark_pack

Load and validate one versioned benchmark-pack YAML file.

fraudtwin.ml.backtest.run_backtest

Build reproducible rolling folds from one existing generated history.

fraudtwin.ml.backtest.run_model_backtest

Run selected M19 model adapters over the existing M10 temporal folds.

fraudtwin.ml.backtest.write_backtest

Persist fold rows, metrics, and the append-only backtest manifest.

Constants and protocols#

Name

Reference

BACKTEST_ROW_SCHEMA

fraudtwin.ml.backtest.BACKTEST_ROW_SCHEMA

FOLD_METRIC_SCHEMA

fraudtwin.ml.backtest.FOLD_METRIC_SCHEMA

METRIC_NAMES

fraudtwin.ml.backtest.METRIC_NAMES

Detailed API#

Leakage-safe rolling backtests over one generated FraudTwin history.

class fraudtwin.ml.backtest.BenchmarkPack(**data)[source][source]

Bases: BaseModel

Versioned frozen evaluation definition.

Parameters:

  • id (Annotated[str, MinLen(min_length=1)])

  • version (Annotated[str, MinLen(min_length=1)])

  • source_seed (Annotated[int, Ge(ge=0)])

  • source_config_hash (Annotated[str, MinLen(min_length=1)])

  • windows (BenchmarkPackWindows)

  • regimes (tuple[FraudRegimeConfig, ...])

  • label_policy (Literal['exclude', 'include_unresolved'])

  • scenario_parameters (dict[str, object])

  • metric_definitions (list[str])

  • label_maturity_gap_seconds (Annotated[int, Ge(ge=1)])

model_config: ClassVar[ConfigDict] = {'extra': 'forbid'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class fraudtwin.ml.backtest.BenchmarkPackWindows(**data)[source][source]

Bases: BaseModel

Fixed train, validation, test, and optional stress windows.

Parameters:

  • train (BenchmarkWindow)

  • validation (BenchmarkWindow | None)

  • test (BenchmarkWindow)

  • stress (BenchmarkWindow | None)

model_config: ClassVar[ConfigDict] = {'extra': 'forbid'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class fraudtwin.ml.backtest.BenchmarkWindow(**data)[source][source]

Bases: BaseModel

One immutable half-open benchmark interval.

Parameters:

  • from_time (datetime)

  • to (datetime)

model_config: ClassVar[ConfigDict] = {'extra': 'forbid'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class fraudtwin.ml.backtest.BacktestResult(fold_rows, fold_metrics, manifest)[source][source]

Bases: object

Fold rows, metrics, and the immutable result manifest.

Parameters:

  • fold_rows (tuple[dict[str, Any], ...])

  • fold_metrics (tuple[dict[str, object], ...])

  • manifest (BacktestManifest)

class fraudtwin.ml.backtest.FoldSpec(fold_id, train_from, train_to, validation_from, validation_to, test_from, test_to, stress_from=None, stress_to=None)[source][source]

Bases: object

Parameters:

  • fold_id (str)

  • train_from (datetime)

  • train_to (datetime)

  • validation_from (datetime | None)

  • validation_to (datetime | None)

  • test_from (datetime)

  • test_to (datetime)

  • stress_from (datetime | None)

  • stress_to (datetime | None)

fraudtwin.ml.backtest.load_benchmark_pack(path)[source][source]

Load and validate one versioned benchmark-pack YAML file.

Return type:

BenchmarkPack

Parameters:

path (Path)

fraudtwin.ml.backtest.run_backtest(config, entities, behavior, source_manifest, *, benchmark_pack=None)[source][source]

Build reproducible rolling folds from one existing generated history.

Return type:

BacktestResult

Parameters:

  • config (SimulationRunConfig)

  • entities (EntityDataset)

  • behavior (BehaviorDataset)

  • source_manifest (RunManifest)

  • benchmark_pack (BenchmarkPack | None)

fraudtwin.ml.backtest.run_model_backtest(config, entities, behavior, source_manifest, *, models, benchmark_pack=None)[source][source]

Run selected M19 model adapters over the existing M10 temporal folds.

Return type:

BacktestResult

Parameters:

  • config (SimulationRunConfig)

  • entities (EntityDataset)

  • behavior (BehaviorDataset)

  • source_manifest (RunManifest)

  • models (tuple[str, ...])

  • benchmark_pack (BenchmarkPack | None)

fraudtwin.ml.backtest.write_backtest(result, output_dir)[source][source]

Persist fold rows, metrics, and the append-only backtest manifest.

Return type:

tuple[Path, Path, Path]

Parameters:

  • result (BacktestResult)

  • output_dir (Path)