Nightly Self-Verification

Configurations:

• Baseline: Full data — name, entity type, date of birth, nationality. Tests the system under ideal conditions.
• Without entity type: Tests resilience when type is unknown — a common real-world scenario when screening from unstructured data.
• Without tertiary data: Tests name-only matching strength. The baseline for any system that receives a name without biographical context.
• Production mode: Re-evaluates results with production DOB rescue thresholds. Tests whether entities that narrowly miss the threshold in baseline would be recovered by the DOB rescue pass used in live screening.
• Without ML: Disables the Machine Learning override entirely. Isolates the heuristic pipeline performance — the score floor without ML assistance.
• Cross-source validation: Screens each source's entities against all other sources simultaneously. Tests whether a designation on one list is detectable via a different list's entry for the same actor.

Mutation Levels:

• M0 — Unmodified: Original name as it appears on the list. Must achieve complete recall. Any miss at M0 is a critical failure.
• M1 — One mutation: Random character swap, omission, insertion, replacement, diacritic change, or transliteration. Models a single transcription error or variant spelling.
• M2 — Two mutations: Two mutations applied simultaneously. Tests the degradation curve — how quickly recall falls as input quality degrades.
• M3 — Name reorder: Token order is reversed or first and last tokens are swapped. Tests resilience to name component reordering — a common variant when names cross cultural conventions (given name first vs. family name first).
• M4 — Partial name: Only the first or last token is kept, simulating partial name entry. Tests whether the system can still surface a candidate when only a fragment of the full name is provided.

Every mutation is deterministically seeded. Every false negative is traceable to the exact mutation that caused it.

• True Positive Rate (Recall): Proportion of entities that find themselves as the top result. The primary metric. M0 recall must be 100%.
• Mean Reciprocal Rank: At which position does the entity appear in its own top-5 results? A rank-1 result is ideal. Rank 2 or lower flags a concern.
• Score Distribution: Mean, median, and percentiles across all sessions. Detects drift — a gradual score decline that precedes recall failures.
• False Negative Analysis: Every missed entity logged with exact mutation applied and the actual top-1 result returned. Makes failure modes visible and debuggable.
• Wilson Confidence Intervals: Statistical bounds on the true positive rate at 95% confidence. Prevents overstating reliability on small source sizes.

Results are exported as a structured XLSX report after each nightly cycle.