fraudtwin.quality_benchmark#

Generator-quality benchmark contracts.

Status: Experimental

Classes#

`fraudtwin.quality_benchmark.QualityAdapterMetadata`	Identity and capabilities advertised by an external generator.
`fraudtwin.quality_benchmark.QualityAdapterRequest`	Read-only public workload delivered to an external adapter.
`fraudtwin.quality_benchmark.QualityArtifactBundle`	Normalized external output consumed by the quality scorer.
`fraudtwin.quality_benchmark.QualityBenchmarkProfile`	Immutable public workload and protocol selection.
`fraudtwin.quality_benchmark.QualityBenchmarkRequest`	Programmatic equivalent of the quality-benchmark CLI command.
`fraudtwin.quality_benchmark.QualityBenchmarkResult`	Immutable result and artifact location for a quality benchmark run.
`fraudtwin.quality_benchmark.QualityCandidateReport`	Normalized quality dimensions for one generator candidate.
`fraudtwin.quality_benchmark.QualityCapability`	Declared dimensions supplied by a native or external candidate.
`fraudtwin.quality_benchmark.QualityGeneratorAdapter`	Protocol implemented by an external generator benchmark adapter.
`fraudtwin.quality_benchmark.QualityMetric`	One independently interpretable quality result.

Functions#

`fraudtwin.quality_benchmark.load_quality_profile`	Load one bundled immutable M22 profile or a YAML profile path.
`fraudtwin.quality_benchmark.report_run`	Create an invariant report for one existing native generated run.
`fraudtwin.quality_benchmark.run_quality_benchmark`	Run the native or external M22 quality protocol.

Detailed API#

Generator quality benchmarking.

The quality benchmark is deliberately separate from the M20 model benchmark. It evaluates the generator and its artifacts, while keeping correctness, fidelity, difficulty, scalability, and engineering observations independent. External implementations can either be loaded through a small adapter protocol or provide the same normalized artifact bundle on disk.

class fraudtwin.quality_benchmark.QualityAdapterRequest(**data)[source][source]

Bases: BaseModel

Read-only public workload delivered to an external adapter.

Parameters:

profile_id (str)
profile_fingerprint (str)
public_pack (str)
public_definition (dict[str, Any])
output_dir (Path)

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'frozen': True}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class fraudtwin.quality_benchmark.QualityArtifactBundle(**data)[source][source]

Bases: BaseModel

Normalized external output consumed by the quality scorer.

Parameters:

candidate_id (str)
candidate_version (str)
manifest_path (Path)
artifact_paths (dict[str, Path])
logical_fingerprints (dict[str, str])
capabilities (QualityCapability)
generation_seconds (Annotated[float | None, Ge(ge=0)])

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'frozen': True}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class fraudtwin.quality_benchmark.QualityBenchmarkProfile(**data)[source][source]

Bases: BaseModel

Immutable public workload and protocol selection.

Parameters:

profile_id (Annotated[str, _PydanticGeneralMetadata(pattern='^standard-v1(?:-(?:dev|medium|large|xlarge|billion))?$')])
profile_version (str)
public_packs (tuple[str, ...])
scale_size (str)
protocol_version (str)

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'frozen': True}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class fraudtwin.quality_benchmark.QualityBenchmarkRequest(**data)[source][source]

Bases: BaseModel

Programmatic equivalent of the quality-benchmark CLI command.

Parameters:

profile (str)
output_dir (Path)
adapter (str | None)
bundle (Path | None)
scale_manifest (Path | None)

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'frozen': True}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class fraudtwin.quality_benchmark.QualityBenchmarkResult(**data)[source][source]

Bases: BaseModel

Immutable result and artifact location for a quality benchmark run.

Parameters:

report_id (str)
profile (QualityBenchmarkProfile)
report_path (Path)
candidate (QualityCandidateReport)

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'frozen': True}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class fraudtwin.quality_benchmark.QualityCandidateReport(**data)[source][source]

Bases: BaseModel

Normalized quality dimensions for one generator candidate.

Parameters:

candidate_id (str)
candidate_version (str)
capabilities (QualityCapability)
correctness (tuple[QualityMetric, ...])
fidelity (tuple[QualityMetric, ...])
difficulty (tuple[QualityMetric, ...])
scalability (tuple[QualityMetric, ...])
engineering_performance (tuple[QualityMetric, ...])
reproducibility (tuple[QualityMetric, ...])
packs (tuple[dict[str, Any], ...])

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'frozen': True}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class fraudtwin.quality_benchmark.QualityCapability(**data)[source][source]

Bases: BaseModel

Declared dimensions supplied by a native or external candidate.

Parameters:

financial_invariants (bool)
temporal_invariants (bool)
pit_validation (bool)
scenario_coverage (bool)
ledger_reconciliation (bool)
reproducibility (bool)
statistical_fidelity (bool)
temporal_fidelity (bool)
graph_fidelity (bool)
difficulty (bool)
scalability (bool)
engineering_performance (bool)

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'frozen': True}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class fraudtwin.quality_benchmark.QualityGeneratorAdapter(*args, **kwargs)[source][source]

Bases: Protocol

Protocol implemented by an external generator benchmark adapter.

class fraudtwin.quality_benchmark.QualityMetric(**data)[source][source]

Bases: BaseModel

One independently interpretable quality result.

Parameters:

name (str)
status (str)
score (Annotated[float | None, Ge(ge=0), Le(le=1)])
details (dict[str, Any])

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'frozen': True}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

class fraudtwin.quality_benchmark.QualityAdapterMetadata(**data)[source][source]

Bases: BaseModel

Identity and capabilities advertised by an external generator.

Parameters:

candidate_id (Annotated[str, MinLen(min_length=1)])
candidate_version (str)
framework (str)
deterministic (bool)
capabilities (QualityCapability)
parameters (dict[str, Any])

model_config: ClassVar[ConfigDict] = {'extra': 'forbid', 'frozen': True}: Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

fraudtwin.quality_benchmark.load_quality_profile(reference='standard-v1')[source][source]

Load one bundled immutable M22 profile or a YAML profile path.

Return type:: QualityBenchmarkProfile
Parameters:: reference (str | Path)

fraudtwin.quality_benchmark.report_run(run_id, *, runs_dir=Path('runs'), output_dir=Path('runs/quality-reports'))[source][source]

Create an invariant report for one existing native generated run.

Return type:

Path

Parameters:

run_id (str)
runs_dir (Path)
output_dir (Path)

fraudtwin.quality_benchmark.run_quality_benchmark(profile='standard-v1', *, output_dir=Path('runs/quality-benchmarks'), adapter=None, bundle=None, scale_manifest=None)[source][source]

Run the native or external M22 quality protocol.

Return type:

QualityBenchmarkResult

Parameters:

profile (str | Path | QualityBenchmarkProfile | QualityBenchmarkRequest)
output_dir (Path)
adapter (str | None)
bundle (Path | None)
scale_manifest (Path | None)