Talent SummonerCandidate Ranking QA Checklist
A practical candidate ranking QA checklist for panels: verify the role brief, candidate packet, evidence trail and review questions before calibration.
Candidate ranking QA is the check that a panel's ranking packet is fit to discuss before people anchor on the order. It asks whether the role brief, candidate inputs, evidence labels and explanations are complete enough for a comparable review. It is a readiness gate, not a second score, a fairness guarantee or a replacement for panel judgement.
For a panel calibrating scores at the planning stage, the useful outcome is a short list of confirmed inputs, visible gaps and questions that a reviewer can answer. Run this checklist before the panel meeting. Keep the QA record tied to one role and one candidate packet so a later change can be traced.
Define what passing means
Start with one decision question, such as: "Which candidates should receive the next human review for this role?" Record the decision owner, panel date, candidate-set boundary and the version of the approved role brief. Do not let the ranking output redefine the question.
The brief should state the work the person must do and separate must-haves, nice-to-haves and genuine operating boundaries. For each criterion, write the evidence that would support it, what remains unknown and who can verify it. Avoid a general requirement such as "strong culture fit" unless the team has translated it into job-related behaviour that every reviewer can inspect.
Agree the labels before opening the results. A small set such as supported, adjacent, unknown and contradicted is useful when the definitions are written down. These labels describe the evidence for a criterion; they do not describe a person's value. An unknown is a question to route, not a failed score.
Check the input packet
Review the packet before examining the ranking. Use the following checks as a gate, not as a reason to silently repair a candidate record.
| QA check | Question to answer | Passing signal |
|---|---|---|
| Brief integrity | Is the role brief approved, versioned and owned? | One current brief is named, with its approval context and decision owner. |
| Criterion clarity | Can another reviewer tell what each criterion means? | Each item has a class, evidence standard and permitted equivalent route. |
| Candidate boundary | Is it clear which records are included and why? | The packet has a dated candidate list and no unexplained additions or omissions. |
| File and identity check | Can the reviewer open each source and distinguish records? | Files or links are readable, de-duplicated and tied to a candidate identifier. |
| Provenance check | Can the reviewer locate the statement behind a material claim? | The source file, passage or link is recorded where available. |
| Access and sensitivity | Is the packet shared only with people who need it? | The owner follows the organisation's handling process and removes unnecessary personal detail. |
If a check fails, mark the packet hold and name the smallest correction. A missing source is not a reason to infer a negative result. If a duplicate is found, keep the original records and record which one is the source of truth. If the brief changed after candidates were collected, create a new version and decide whether the packet needs to be rerun.
Inspect the ranking as a review aid
Now sample the actual output across the packet: include candidates near the top, candidates near a progression boundary and at least one record whose evidence is incomplete. This is a packet-readiness sample, not a synthetic edge-case test. The purpose is to see whether a panel can interpret the output without guessing what the system or operator meant.
Check each sampled result against these questions:
- Does the explanation point to a specific role criterion, rather than a broad impression or a repeated keyword?
- Are must-haves and nice-to-haves visibly separate, so a preference cannot quietly compensate for an unresolved requirement?
- Is missing evidence shown as unknown, with a person responsible for checking it?
- Are contradictory statements or unclear ownership visible instead of being blended into a confident summary?
- Can a reviewer move from the explanation back to the supplied source?
- Is any changed order explained by the approved brief and candidate evidence, rather than by an unrecorded rule change?
Do not treat a high position as proof of capability, authorship, availability or likely performance. Do not treat a low position as a final rejection. The QA result should say what the panel can inspect and what it still needs to ask.
Turn QA findings into panel questions
Convert every material gap into one neutral question before the meeting. This reduces improvised challenges and keeps comparable candidates on a comparable review path.
| QA finding | Panel question | Next owner |
|---|---|---|
| A requirement is mentioned but no source passage is attached. | Which source should support this criterion, and what is the narrowest question the candidate should answer? | Evidence owner |
| A strong preference appears to drive the order while a must-have is unknown. | Is the must-have unresolved, or is there reliable evidence that it is not met? | Role owner |
| The record describes a team result without the person's contribution. | What did this candidate own, decide or change, and how will comparable candidates be asked? | Interview owner |
| Two records appear to describe the same person or work. | Which record is authoritative, and has the duplicate been retained in the audit note? | Packet owner |
| Reviewers could read one criterion in two ways. | What observable work would make the interpretation the same for every reviewer? | Decision owner |
Write the expected answer format beside each question. For example, ask for a source, a bounded example or a defined verification step. Do not ask a panel to resolve an input problem by taking a faster vote.
Sign off the packet before calibration
Give the packet one status and record the reason:
- Ready for panel: required inputs are present, material unknowns have owners and questions are prepared.
- Hold for evidence: a missing or conflicting source could change the next step.
- Fix the brief: a criterion is ambiguous, duplicated or not expressed as job-related work.
- Verify handling: access, privacy or another operating boundary needs the responsible owner before review.
The sign-off should include the QA owner, date, brief version, candidate-set identifier, findings, next actions and stop condition. A stop condition might be "do not compare progression scores until the must-have source is checked". Keep the original packet and the corrected version linked. QA is complete when the panel knows which questions it can answer, not when every unknown has been forced into a score.
Copy-ready QA record
This record is copyable on the page for use in a hiring document; no separate download is provided.
Role and brief version:
Decision owner:
QA owner and date:
Candidate-set boundary:
[ ] Approved brief and criterion definitions are recorded
[ ] Must-haves, nice-to-haves and operating boundaries are separated
[ ] Candidate files or links are readable and de-duplicated
[ ] Material evidence has a source or an explicit unknown label
[ ] Sampled explanations map to the approved criteria
[ ] Conflicts, missing evidence and ownership gaps have next actions
[ ] Panel questions use a comparable prompt and named owner
[ ] Status, stop condition and next review point are recordedWhere Talent Summoner fits
Talent Summoner is our product. Its current Candidate Ranking tool, checked on 3 September 2026, accepts a pasted or uploaded job description and up to 50 CVs in PDF, DOCX, MD or TXT format, requires no account and returns a ranked shortlist with plain-English reasoning and shareable reports. Use the report as an input to this QA gate. Our product does not determine the panel's criteria, verify every source or make the final hiring decision.
When the candidate packet is incomplete and an approved discovery pass is needed, candidate sourcing, checked on 3 September 2026, starts from a role brief and searches LinkedIn, GitHub and other public professional sources across 200M+ profiles. Its ranked results are discovery material to verify, not proof of ownership, current availability or interest. Keep the source route clear when adding discovered candidates to a panel packet.
What does candidate ranking QA check?
It checks whether the role brief, candidate packet, evidence trail, ranking explanations and panel questions are ready for a comparable human review. It does not certify the ranking or predict performance.
Is QA the same as panel calibration?
No. QA checks that the review packet is interpretable before calibration starts. Calibration is the later discussion about how reviewers apply the agreed criteria to evidence.
What should a panel do with missing evidence?
Keep it as unknown, identify the authorised source or comparable question, assign an owner and record the stop condition. Do not convert silence into a rejection score.
Can Talent Summoner complete the panel QA automatically?
No. Our product can rank supplied CVs against a job description and provide plain-English reasoning, but the team still checks sources, prepares comparable questions and owns the decision.
Before the next panel, copy the QA record, freeze the brief and candidate boundary, sample the explanations and write one owned question for each material gap. Then use candidate ranking for supplied CVs or candidate sourcing when an approved discovery pass is needed.
