How to prevent evaluation-data leakage in candidate-ranking tests

An evaluation is only useful when its test cases remain unseen while the ranking method is chosen. Data leakage occurs when information from the evaluation set influences preprocessing, feature selection, prompt tuning, rubric changes or threshold selection. It can inflate the estimate and make the method look stronger than it is on new candidate material.

This guide adapts the train/test discipline from the scikit-learn common-pitfalls guide to candidate-ranking tests. It is an educational method with synthetic examples, not a report of Talent Summoner implementation or performance. The public Candidate Ranking workflow describes reviewing supplied CVs; it does not publish a private evaluation set or benchmark.

Freeze the question and split first

Write the evaluation question before inspecting results. State the role brief version, ranking rubric, unit being ranked, success labels and the action that a result may inform. Separate development material from a final holdout set. Split by the unit that could repeat: a candidate, CV, role, or near-duplicate document. A random row split can still leak when several records describe the same person or role.

Keep the holdout set sealed until the ranking method, preprocessing and decision threshold are frozen. A test set used repeatedly to choose settings becomes another training signal. If a reviewer opens the holdout to resolve an ambiguous label, record the change and move that case back to development; do not quietly keep it as independent evidence.

Fit transformations on development data only

Scikit-learn's guidance is direct: split before preprocessing, call fit only on the training data, and apply the learned transformation to the test data with transform. A pipeline helps keep those operations together. The same rule applies to ranking tests:

• Build vocabularies, normalizers, feature selectors, thresholds and rubric weights from development cases only.
• Apply the frozen transformation to holdout cases without recalculating statistics from them.
• Keep holdout labels, reviewer adjudication and holdout error categories out of any feature or prompt-selection step. Development labels are allowed inputs when they are part of the method being trained.
• Do not use the holdout's candidate order to rewrite the role brief or decide what counts as a match.

This prevents leakage from both obvious fields, such as a final reviewer label, and indirect fields, such as a token frequency calculated over every CV in the corpus.

A fixed synthetic example

The following example is intentionally small and fictional. It demonstrates a fixed split and a transformation fit on development rows. It does not produce a Talent Summoner score.

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline

texts = [
    "owned a production data migration",       # positive
    "supported a reporting dashboard",         # negative
    "designed an event driven API",             # positive
    "attended an API workshop",                # negative
    "maintained an on call rotation",          # positive
    "read about on call practice",             # negative
]
labels = [1, 0, 1, 0, 1, 0]

dev_text, test_text, dev_y, test_y = train_test_split(
    texts, labels, test_size=0.33, random_state=7, stratify=labels
)
model = make_pipeline(TfidfVectorizer(), LogisticRegression(max_iter=1000))
model.fit(dev_text, dev_y)
print(model.score(test_text, test_y))

The vectorizer learns its vocabulary inside the pipeline from dev_text. The holdout is transformed only when the frozen model scores it. For a ranking evaluation, replace the classifier with a job-related rubric or ordering procedure, preserve the same split, and inspect evidence citations rather than treating the numeric score as hiring quality.

Check for leakage paths

Before accepting a result, ask where each input came from and when it became available:

Retain split IDs, transformation version, rubric or prompt version, code revision, random seed, eligible counts and exclusion reasons. A passing run is evidence about that frozen design; it is not proof of fairness, accuracy, job performance or generalisation to every role.

Report uncertainty and human decisions

State the denominator for every measure: candidate-level agreement, criterion-level evidence, top-k overlap or invalid-output rate. Keep development and holdout counts separate. Explain missing labels, ties, abstentions and reviewer disagreement. A lower-ranked profile can be correct when the evidence is unknown; a plausible top rank can still contain an unsupported claim.

Use a human owner for ambiguous criteria and a stop rule for privacy exposure, non-job-related proxies, inaccessible review or an irreproducible split. Preserve the original output, correction and rerun result. Never present a synthetic fixture result as a customer outcome or a production benchmark.

Next step: write the evaluation contract, seal a synthetic holdout and run the Candidate Ranking workflow only within the documented review boundary.