fraudtwin.benchmark#

Benchmark packs and reproducible suite execution.

Status: Experimental

Classes#

fraudtwin.benchmark.BenchmarkModelRunner

Framework-neutral adapter implemented by optional model environments.

fraudtwin.benchmark.BenchmarkRequest

Strict command-level benchmark request.

fraudtwin.benchmark.BenchmarkResult

Paths and stable metadata emitted by one benchmark invocation.

fraudtwin.benchmark.BenchmarkRunnerInput

Read-only paths and metadata supplied to an external model runner.

fraudtwin.benchmark.PublicBenchmarkPack

Immutable, distributable M21 benchmark definition.

fraudtwin.benchmark.RunnerMetadata

Framework and input lineage declared by an external model runner.

Functions#

fraudtwin.benchmark.build_suite_config

Build and validate one standard suite without mutating global config.

fraudtwin.benchmark.list_public_packs

Return all bundled public packs in stable identity order.

fraudtwin.benchmark.load_public_pack

Resolve an exact or unambiguous major/minor public-pack reference.

fraudtwin.benchmark.run_benchmark

Generate and evaluate one or every standard benchmark suite.

fraudtwin.benchmark.run_public_benchmark

Run one immutable bundled M21 pack and verify its frozen outputs.

fraudtwin.benchmark.verify_public_benchmark

Verify an existing public benchmark artifact without regenerating it.

Constants and protocols#

Name

Reference

STANDARD_SUITES

fraudtwin.benchmark.STANDARD_SUITES

SUITE_DEFINITION_VERSION

fraudtwin.benchmark.SUITE_DEFINITION_VERSION

Detailed API#

Reproducible fraud stress benchmark orchestration.

The benchmark layer owns suite composition and reporting. It deliberately delegates generation to the existing deterministic engines and model scoring to the M19 prediction contract.

class fraudtwin.benchmark.BenchmarkModelRunner(*args, **kwargs)[source][source]

Bases: Protocol

Framework-neutral adapter implemented by optional model environments.

class fraudtwin.benchmark.BenchmarkRequest(**data)[source][source]

Bases: BaseModel

Strict command-level benchmark request.

Parameters:

  • suite (Literal['baseline', 'temporal', 'boundary', 'camouflage', 'graph', 'observability', 'calibrated', 'mixed', 'all'])

  • difficulty (int)

  • seed (int)

  • output_dir (Path)

  • models (tuple[str, ...])

  • runners (tuple[str, ...])

  • calibration_profile (Path | None)

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class fraudtwin.benchmark.BenchmarkResult(benchmark_id, root, manifest_path, results_path, descriptors_path, results)[source][source]

Bases: object

Paths and stable metadata emitted by one benchmark invocation.

Parameters:

  • benchmark_id (str)

  • root (Path)

  • manifest_path (Path)

  • results_path (Path)

  • descriptors_path (Path)

  • results (tuple[dict[str, Any], ...])

class fraudtwin.benchmark.BenchmarkRunnerInput(suite, difficulty, dataset_path, run_dir, manifest)[source][source]

Bases: object

Read-only paths and metadata supplied to an external model runner.

Parameters:

  • suite (str)

  • difficulty (int)

  • dataset_path (Path)

  • run_dir (Path)

  • manifest (dict[str, Any])

class fraudtwin.benchmark.PublicBenchmarkPack(**data)[source][source]

Bases: BaseModel

Immutable, distributable M21 benchmark definition.

Parameters:

  • id (Annotated[str, _PydanticGeneralMetadata(pattern='^FT-B0[1-8]-[A-Z]+$')])

  • version (Annotated[str, _PydanticGeneralMetadata(pattern='^(?:0|[1-9]\\d*)\\.(?:0|[1-9]\\d*)\\.(?:0|[1-9]\\d*)$')])

  • suite (Literal['baseline', 'temporal', 'boundary', 'camouflage', 'graph', 'observability', 'calibrated', 'mixed'])

  • generator_compatibility (Annotated[str, _PydanticGeneralMetadata(pattern='^>=\\d+\\.\\d+\\.\\d+,<\\d+\\.\\d+\\.\\d+$')])

  • seed (Annotated[int, Ge(ge=0)])

  • seed_tree_version (Annotated[str, MinLen(min_length=1)])

  • difficulty (Annotated[int, Ge(ge=1), Le(le=10)])

  • simulation_start (datetime)

  • simulation_end (datetime)

  • split_boundaries (dict[str, datetime])

  • scenario_definitions (dict[str, Any])

  • stress_parameters (dict[str, Any])

  • label_observation_policy (dict[str, Any])

  • calibration (dict[str, Any] | None)

  • metric_definitions (dict[str, Any])

  • resolved_configuration_hash (Annotated[str, MinLen(min_length=1)])

  • expected_descriptors (dict[str, Any])

  • expected_fingerprints (dict[str, str])

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class fraudtwin.benchmark.RunnerMetadata(**data)[source][source]

Bases: BaseModel

Framework and input lineage declared by an external model runner.

Parameters:

  • model_id (str)

  • framework (str)

  • framework_version (str)

  • required_inputs (tuple[str, ...])

  • parameters (dict[str, Any])

  • artifact_checksums (dict[str, str])

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'frozen': True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

fraudtwin.benchmark.build_suite_config(suite, *, difficulty, seed, calibration_profile=None)[source][source]

Build and validate one standard suite without mutating global config.

Return type:

SimulationRunConfig

Parameters:

  • suite (str)

  • difficulty (int)

  • seed (int)

  • calibration_profile (Path | None)

fraudtwin.benchmark.list_public_packs()[source][source]

Return all bundled public packs in stable identity order.

Return type:

tuple[PublicBenchmarkPack, ...]

fraudtwin.benchmark.load_public_pack(reference)[source][source]

Resolve an exact or unambiguous major/minor public-pack reference.

Return type:

PublicBenchmarkPack

Parameters:

reference (str)

fraudtwin.benchmark.run_public_benchmark(reference, *, output_dir=Path('runs/benchmarks'), models=('deterministic_heuristic',), runners=())[source][source]

Run one immutable bundled M21 pack and verify its frozen outputs.

Return type:

BenchmarkResult

Parameters:

  • reference (str)

  • output_dir (Path)

  • models (tuple[str, ...])

  • runners (tuple[str, ...])

fraudtwin.benchmark.verify_public_benchmark(run_dir, *, reference=None)[source][source]

Verify an existing public benchmark artifact without regenerating it.

Return type:

dict[str, Any]

Parameters:

  • run_dir (Path)

  • reference (str | None)

fraudtwin.benchmark.run_benchmark(request)[source][source]

Generate and evaluate one or every standard benchmark suite.

Return type:

BenchmarkResult

Parameters:

request (BenchmarkRequest)