ATS Evaluation Checklist: 30 Questions to Ask in the Demo (and the Answers That Are Red Flags)
An ATS evaluation checklist for the demo: 30 questions in six blocks, what to ask the vendor to show, a four-state score per question and the red-flag answers.
An ATS evaluation checklist is a fixed list of questions you put to every vendor in the demo, with a score per answer, so the third demo is judged on the same sheet as the first. The one below has 30 questions in six blocks - workflow, communication, agency and multi-client, data and migration, AI explainability, commercial - and for each question it names what to ask the vendor to show on screen and the answer that should worry you. Score every question as supported, configurable, roadmap or missing, weight the blocks, and the shortlist sorts itself.
I run ScalingPPL, a recruitment agency, and I built Pickr, an AI recruiting platform, so I have sat on both sides of these demos. The checklist works for an in-house team as well.
Why another ATS evaluation checklist?
Because the ones on the first page of results are either gated or stop where the hard questions start. I opened five on 2026-10-14. Greenhouse's AI in ATS evaluation checklist (dated 9 June 2026) is an eBook behind a contact form; the page promises "what to demand on explainability and accountability" but shows none of the questions, and it covers AI only. Recruitee's evaluation checklist links to a Google Doc scoring vendors "from must-have features to usability and integrations"; it has a checkbox for whether the vendor disclosed its data storage region and a 0-2 score for "AI screening / matching recommendations", and nothing on client separation, export at contract end or how an AI score is explained. Compono's 2026 agency ATS guide lists nine criteria in prose and names "strict data segregation between competing clients" as a gap in generic software, without a question for the vendor. HR Morning's 14 ATS vendor questions (updated 8 September 2026) do ask how transparent AI rankings are, and still skip hosting region, client portals and export at contract end. Paycor's ATS demo checklist was last updated 14 May 2018.
The gap is consistent: nobody asks who sees what across clients, how the data leaves the system when you leave, or how an AI score can be explained to the candidate it was about, and only Recruitee's doc asks where the data sits, as a checkbox. Those are four of the six blocks below.
How do you score a vendor demo?
Four states per question, scored before the vendor's next sentence:
| State | Points | What it means |
|---|---|---|
| Supported | 3 | Shown on screen, in the demo, in the standard product |
| Configurable | 2 | Possible with settings or an admin, shown or convincingly described, no custom work |
| Roadmap | 1 | "Coming next quarter", "in beta", "we are building it" |
| Missing | 0 | Not possible, or the answer changed the subject |
Three rules make the score mean something. Send the vendor two of your real job briefs and ten anonymised candidate records a week before, and ask for the demo to run on them rather than on the seeded demo company. Ask them to show, not to tell; a described feature scores configurable at best. Write the score down during the call, because after the third demo the impressions merge.
Roadmap scores one point rather than zero on purpose: it tells you the vendor has heard the question before. It is still not a feature you can rely on.
The 30 questions, block by block
Block 1: Workflow (questions 1-5)
| # | Question | Ask them to show | Red-flag answer |
|---|---|---|---|
| 1 | Can we set up our stages, per job, in the demo, now? | A pipeline built live from one of your job briefs, with a stage renamed and one removed | "Our best-practice stages work for most customers" |
| 2 | What happens automatically when a candidate changes stage? | A stage change that triggers a candidate email, a hiring-manager notification and a task, with the trigger rules on screen | Automations that live in a separate add-on or a partner tool |
| 3 | How does a scorecard get filled in, and what happens when it is not? | An interviewer's view of a scorecard, and the reminder that goes out when it is overdue | "Hiring managers love our interface", with no reminder mechanism |
| 4 | Where do we see which stage is slow? | Time in stage per job on a report, filtered to one job and one month | A single overall time-to-hire number |
| 5 | How do duplicates and repeat applicants get handled? | The same person applying twice to two jobs, and what the recruiter sees | "That rarely happens" |
Question 1 is where the vendor's "fully configurable" meets your real process. From our work at ScalingPPL, the stage set is rarely the standard one, and a system that makes the recruiter adapt to it is one the team works around within a month.
Block 2: Communication (questions 6-10)
Candidates leave processes over silence. In the Talent Board 2023 candidate experience research, as reported by SHRM (9 April 2024; 240,000 candidates, 150 organisations), the top reason candidates withdrew was poor communication or never receiving feedback, and 36 percent of candidates (US data) had not heard back one to two months after applying. This block tests whether the system makes a reply the default.
| # | Question | Ask them to show | Red-flag answer |
|---|---|---|---|
| 6 | Where do candidate replies land? | A reply by email arriving on the candidate record, linked to the job, not in a mailbox | "Recruiters forward emails into the system" |
| 7 | Which channels are in the same inbox? | Email, plus whichever of WhatsApp or LinkedIn your candidates actually use, in one thread view | A channel that works through a browser plugin someone has to keep alive |
| 8 | Can a candidate see their own status? | A candidate-side page showing the stage, with document upload and consent in one place | "They can email us" |
| 9 | What does the hiring manager or client get, and how? | A shortlist delivered to a person who does not work in the system all day, with their feedback landing on the record | A PDF export and an email thread |
| 10 | What is sent without a human pressing send? | The list of automatic messages, and the approval step for anything drafted by AI | "Everything is automatic", or no list exists |
Question 10 is the one to push on. A drafted reply that a recruiter approves is a time saver; a sent reply nobody saw is a complaint waiting for a client.
Block 3: Agency and multi-client (questions 11-15)
Skip this block if you hire for one employer. For an agency it decides whether the system was built for you or adapted for you.
| # | Question | Ask them to show | Red-flag answer |
|---|---|---|---|
| 11 | Is a client a first-class object, with its own pipelines, or a tag on a job? | Two clients side by side, each with its own jobs, stages and contacts | A "department" field relabelled "client" |
| 12 | Can one candidate sit in two clients' pipelines at once, with separate states? | The same candidate at "interview" for client A and "rejected" for client B, both histories intact | One status per candidate |
| 13 | What can a client user see, and what can they never see? | A client login that shows only that client's shortlist, with another client's candidates provably absent | "You control that with permissions", shown nowhere |
| 14 | Can a client give feedback without emailing? | A client rating and commenting on a shortlist, with the comment appearing on the candidate record | Feedback by reply-all |
| 15 | Can we report per client, and send the report? | Pipeline, time in stage and placements for one client, as a report the client receives | A global dashboard with no client filter |
Question 12 is the one most in-house ATSs fail, because a system built for one employer models a candidate as having one application state. For an agency that separation is the job. The companion piece on choosing recruitment agency software makes the longer argument.
Block 4: Data and migration (questions 16-20)
Two legal facts frame this block, and this is not legal advice. Under GDPR Article 20(1) a candidate can ask for their data "in a structured, commonly used and machine-readable format", and under Article 28(3)(g) your processor must, at your choice, delete or return all personal data when the service ends. Both depend on the system being able to export. Ask to see it.
| # | Question | Ask them to show | Red-flag answer |
|---|---|---|---|
| 16 | Where is candidate data stored and processed, including backups and the AI provider? | A named region in writing, and the sub-processor list | "We are on AWS" with no region, or EU residency as a higher tier |
| 17 | How does our existing data come in? | An import from your current ATS or a CSV, with stage history and notes intact, not only names | "Our services team handles migration", with no demo and no timeline |
| 18 | What does a full export look like at contract end? | The export itself: candidates, jobs, applications, stage history, notes and attachments, in a documented format | "You can export a candidate list to CSV" |
| 19 | Is there an API, and what does it cover? | Public API documentation, with candidates, jobs and applications readable and writable | API access "on the Enterprise plan" or by request |
| 20 | How is a deletion confirmed, and what survives it? | A deleted candidate searched for in notes and attachments afterwards, and the deletion record | "Archived" offered as deleted |
If your current system is Bullhorn, the migration guide walks through what to export and in what order; the questions here are the same for any source system.
Block 5: AI explainability (questions 21-25)
AI used "for the recruitment or selection of natural persons, in particular to place targeted job advertisements, to analyse and filter job applications, and to evaluate candidates" is listed as high-risk in Annex III point 4(a) of the EU AI Act (Regulation (EU) 2024/1689). Article 14 requires that the people overseeing such a system can interpret its output and decide not to use it, override it or reverse it; Article 26(11) requires the deployer, which is you, to inform the people subject to it. The Digital Omnibus on AI, Regulation (EU) 2026/1744, was published in the Official Journal on 24 July 2026, entered into force on 27 July 2026, and moved the application date of the Annex III obligations to 2 December 2027 (Cooley, 3 August 2026). Any ATS you sign for now will be in use on that date.
Greenhouse's AI recruiting page puts the buyer's rule in one line: "Avoid systems that use black-box composite scoring to rank candidates." Ashby's AI page says its AI "never 'ranks' or gives numerical ratings to applicants". Two different answers to the same question, both defensible. A score with no reasoning is not.
| # | Question | Ask them to show | Red-flag answer |
|---|---|---|---|
| 21 | For one candidate, why this score? | The reasoning next to the score: which requirements are met, partly met and missing, in words a client could read | A number, a colour, or "the model weighs forty factors" |
| 22 | What does the AI never do on its own? | The written rule: no rejection, no stage move, no message without a named person confirming | "You can switch automation on for that" |
| 23 | Can a recruiter overrule a score, and is the overrule recorded? | A score overridden with a reason, visible later on the record | Overrides that silently replace the score |
| 24 | What leaves the system when the AI runs, and where does it go? | The data sent to the model provider, what is redacted first, the provider's region, and the no-training clause | "The AI is secure" |
| 25 | How are candidates told, and how can they ask for an explanation? | The candidate-facing notice, and the path for a candidate to request the reasoning | "That is your responsibility as the employer", with nothing in the product to support it |
Question 25 becomes your obligation on 2 December 2027. The vendor cannot carry it for you, but the product decides whether you can discharge it.
Block 6: Commercial (questions 26-30)
Ask these as questions; the price sheet answers none of them.
| # | Question | Ask them to show | Red-flag answer |
|---|---|---|---|
| 26 | What is the contract term, when does it renew, and what is the notice period? | The clause in the order form, with renewal and notice written as dates | "Standard annual terms", with the clause arriving after the verbal agreement |
| 27 | What counts as a seat: recruiter, hiring manager, interviewer, client user, read-only? | The seat definition in writing, and what a person who only fills in a scorecard needs | A definition that depends on who you ask |
| 28 | What can change the price during the term and at renewal? | The clauses on seat true-ups, indexation, plan changes and feature retirement | "We have never raised prices", unwritten |
| 29 | What is in the price and what is an add-on? | A list: implementation, support tier, API, SSO, AI features, integrations, each marked included or extra | AI or API "available", with no column saying on what terms |
| 30 | Who do you have of our size and type, and what went wrong after their go-live? | A reference call with a customer like you, and a straight answer to the second half | A logo wall instead of a call |
None of these ask the vendor to be cheap. They ask the vendor to be specific, which is the better predictor of how the renewal conversation will go.
What about GDPR and compliance?
That is a seventh block, and it is already written: the nine questions every vendor should answer on GDPR-compliant recruiting software cover storage location, the data processing agreement, sub-processors, AI prompts, retention, erasure, lawful basis, consent and audit logs, each with the good and the bad answer side by side. I have not repeated them here. Score that block on the same four states and carry it into the sheet below as block 7, with the highest weight if you operate in the EU or the UK. Questions 16 and 24 overlap with it on purpose; a vendor who answers them differently in the two conversations has told you something.
The weighted scoring sheet
Each block has five questions, so the block maximum is 15 points; the GDPR block has nine, so 27. Multiply each block score by its weight, add the blocks up, and divide by the weighted maximum. The weights below are a starting point, not a rule; set them before the first demo and do not move them afterwards.
| Block | Questions | Max points | Weight, agency | Weight, in-house | Vendor A | Vendor B | Vendor C |
|---|---|---|---|---|---|---|---|
| 1 Workflow | 1-5 | 15 | 2 | 3 | |||
| 2 Communication | 6-10 | 15 | 2 | 2 | |||
| 3 Agency and multi-client | 11-15 | 15 | 3 | 0 | |||
| 4 Data and migration | 16-20 | 15 | 2 | 2 | |||
| 5 AI explainability | 21-25 | 15 | 2 | 2 | |||
| 6 Commercial | 26-30 | 15 | 1 | 1 | |||
| 7 GDPR (the nine questions) | nine | 27 | 3 | 3 | |||
| Weighted total |
Worked example with agency weights: a vendor scoring 13, 11, 9, 8, 6 and 12 on blocks 1 to 6 and 20 on block 7 has a weighted total of 26 + 22 + 27 + 16 + 12 + 12 + 60 = 175, against a weighted maximum of 30 + 30 + 45 + 30 + 30 + 15 + 81 = 261, which is 67 percent. The useful output is not the percentage but the block that dragged it down: here AI explainability at 6 of 15, the block whose obligations land on you in December 2027.
Two must-pass rules sit outside the arithmetic. A "missing" on question 18 (full export at contract end) or question 22 (what the AI never does on its own) ends the evaluation for that vendor regardless of the total, for the same reason a must-pass flag exists on an interview scorecard: some gaps cannot be offset by strengths elsewhere.
Which answers are red flags?
The tables list the answer per question. Six patterns repeat across every block:
- The subject change. You asked about export at contract end; the answer is about how easy import is. Score the question you asked.
- "On the Enterprise plan." Data residency, API access, SSO and audit logs as tier features tell you what the default looks like for everyone else.
- "Our customers love it." Sentiment in place of a screen. Ask for the screen.
- The partner tool. A capability that exists through an integration partner is a second contract, a second data processing agreement and a second renewal.
- "That is your responsibility." Correct for lawful basis and for informing candidates, but the product still has to make the responsibility possible to discharge.
- The number without the reason. A match score or ranking with no visible reasoning. Under Article 14 the person overseeing the system has to be able to interpret its output; a bare number cannot be interpreted, only obeyed.
Where Pickr stands on these 30 questions
I built Pickr, so score it on the same sheet and discount this section. Pickr runs one pipeline per client mandate with all mandates visible side by side, and clients review shortlists and leave feedback in a client portal, with the feedback landing on the candidate record (block 3). Email, WhatsApp and LinkedIn conversations land in one inbox linked to the candidate and the job, replies are drafted and a person approves them before anything is sent, and every candidate has a page showing where their application stands (block 2). Every match score comes with written reasoning tied to the job requirements, candidates are informed before AI matching is used, no hiring decision is fully automated, and names and emails are redacted from AI prompts by default (block 5). Hiring history imports read-only from Ashby, Greenhouse, Lever and Bullhorn, or from a CSV, with stage history included, and candidate data has data stored in Europe with a data processing agreement included (block 4). Pricing on request (block 6): ask us questions 26 to 30 like anyone else.
Where others are stronger: Greenhouse states on its AI page that it undergoes monthly third-party bias audits and holds ISO 42001:2023 certification for AI governance, and, from our work at ScalingPPL, its approval chains and integration marketplace go deeper than Pickr's; Ashby's position of never scoring candidates numerically is the right answer for a team that wants no ranking at all. Where Pickr's pages do not answer a question, such as the shape of a full export at contract end, ask in the demo and score what you see. For the vendors worth putting on the sheet, start with the recruiting software comparison for 2026; for the numbers on your own process before any demo, the free recruiting audit reads your current pipeline and shows where the hours and the candidates go.
Sources
- Greenhouse, The AI in ATS evaluation checklist (9 June 2026): https://www.greenhouse.com/guidance/ai-in-ats-evaluation-checklist (accessed 2026-10-14)
- Greenhouse, AI recruiting: https://www.greenhouse.com/uk/ai-recruiting (accessed 2026-10-14)
- Tellent Recruitee, Applicant tracking system evaluation checklist: https://recruitee.com/resources/ats-evaluation-checklist (accessed 2026-10-14)
- Compono, The complete guide to choosing an ATS for recruitment agencies in 2026 (updated 30 September 2026): https://www.compono.com/articles/ats-recruitment-agencies-guide-2026 (accessed 2026-10-14)
- HR Morning, 14 ATS vendor questions to ask before you buy (updated 8 September 2026): https://www.hrmorning.com/articles/ats-vendor-questions/ (accessed 2026-10-14)
- Paycor, ATS demo checklist (updated 14 May 2018): https://paycor.com/resource-center/ats-demo-checklist (accessed 2026-10-14)
- Cooley, Digital AI Omnibus delays key deadlines, introduces new rules (3 August 2026): https://cdp.cooley.com/digital-ai-omnibus-delays-key-deadlines-introduces-new-rules/ (accessed 2026-10-14)
- EU AI Act, Regulation (EU) 2024/1689, Article 14 Human oversight: https://artificialintelligenceact.eu/article/14/ (accessed 2026-10-14)
- EU AI Act, Regulation (EU) 2024/1689, Article 26 Obligations of deployers of high-risk AI systems: https://artificialintelligenceact.eu/article/26/ (accessed 2026-10-14)
- EU AI Act, Regulation (EU) 2024/1689, Annex III High-risk AI systems: https://artificialintelligenceact.eu/annex/3/ (accessed 2026-10-14)
- GDPR, Regulation (EU) 2016/679, Article 20 Right to data portability: https://gdpr-info.eu/art-20-gdpr/ (accessed 2026-10-14)
- GDPR, Regulation (EU) 2016/679, Article 28 Processor: https://gdpr-info.eu/art-28-gdpr/ (accessed 2026-10-14)
- Ashby, AI at Ashby: https://ashbyhq.com/ai (accessed 2026-10-14)
- SHRM, Roy Maurer, Candidate experience research from Talent Board (9 April 2024), reporting the Talent Board 2023 CandE benchmark: https://www.shrm.org/in/topics-tools/news/talent-acquisition/candidate-experience-talent-board-research-candes (accessed 2026-10-14)
Frequently Asked Questions
What should an ATS evaluation checklist cover?
Six blocks: workflow, communication, agency and multi-client handling, data and migration, AI explainability, and commercial terms, plus a separate compliance block. Each question should name what the vendor has to show on screen, not describe, and should be scored on the same scale for every vendor you see.
How do you score an ATS vendor demo?
Score every question as supported (shown in the standard product), configurable (possible with settings, no custom work), roadmap (promised, not shipped) or missing. Weight the blocks for your situation before the first demo, write the scores down during the call, and treat a few questions as must-pass regardless of the total.
Which questions should a recruitment agency add to a generic ATS checklist?
Whether a client is a first-class object with its own pipelines, whether one candidate can hold different states in two clients' pipelines at once, what a client user can and cannot see, whether clients can give feedback inside the system, and whether reporting works per client. Most generic checklists skip all five.
What are the red-flag answers in an ATS demo?
A change of subject, a capability that only exists on a higher tier or through a partner tool, sentiment offered instead of a screen, a match score with no visible reasoning, and 'that is your responsibility' with nothing in the product to support the responsibility. Score the question you asked, not the answer you were given.
Does the EU AI Act affect which ATS I choose?
Yes, if the system evaluates or filters candidates with AI. That use is listed as high-risk in Annex III of the AI Act, and the Digital Omnibus moved the application date of those obligations to 2 December 2027, inside the term of any contract signed now. Ask how scores are explained, overridden and disclosed. This is not legal advice.
Written by Andreas Amann, Founder of Pickr and of ScalingPPL
Read more
Add Pickr as a preferred source on Google (opens in a new tab)