I Ran a Study -- Part 2 of 2
Study Results Analysis: Qualified Candidates Rejected When Screener Prompted to List Candidate Cons
kiki<k8@k8port.io> v1.2 October 2026 :icons: font :imagesdir: /blog_images
_This is part 2 of 2. Part 1 covered background information including context and reasoning for running this experiment. Part 2 covers experiment results and analysis.
View the preregistration documents on OSF (Open Science Framework), which establishes the hypotheses and documents my experiment’s design. It was registered prior to collecting any data, and exists to reduce bias and manipulation of experiment results.
My hypothesis was proved mostly wrong by the 1,800 screening calls run against the frontier model Claude Sonnet 5.5. How it was wrong is actually more interesting than if my hypotheses had been correct.
Briefly stated, layering increasing rigor onto an LLM résumé screener, including a weaknesses assesment, scoring rubrics, and an ATS-style evaluation, did not cause the screener to reject any qualified candidate.
Curiously, the single instruction that caused a 15% increase in rejection rate of qualified candidates prompted the screener to list reasons a candidate should not advance before making a qualified recommendation. More curiously , all rejections cut the same person: the nontraditional candidate. More on this below, as the study is broken down into finer detail.
The Question
Here’s a little background for this posting if you missed the first part: I have been on the receiving end of AI screening rejections since the beginning of 2025. At first, I was clueless as to why I was not receiving responses to my applications. After a few weeks of automated rejection notices or simply radio silence, it became apparent that the system is broken. It is designed to keep out frauds and phonies but unwittingly promotes the opposite. Here’s why: if one does not check off every box in a job description, the candidate is summarily rejected. This encourages applicants to tailor resumes to job descriptions rather than to state a summary of their most important qualifications. I wondered if it could be shown, that the more a system is asked to scrutinize, the more likely it will find reason to reject, even when a candidate is clearly qualified to perform a job.
I registered my hypothesis with Open Science Foundation (OSF) before collecting any data:
"We hypothesize that the more detailed screening instructions become, the higher the rejection rate of clearly qualified candidates."
This registration is public and timestamped at the following link: OSF registration. All peripheral documents relevant to the study are described below, including a job description, candidate profiles, five screening conditions, labels, and an analysis plan. All were registered with the hypothesis before making any API calls.
Full disclosure (in case I haven’t mentioned it enough times yet): I am a job seeker, and I have a stake in the answer. That is why I preregistered my study beforehand: because is harder to look for grievances when the expectation is explicity recorded.
Why Measure This
LLM-based résumé reviews are now standard and widely used by employers to vet job candidates. They are the accepted filtering mechanism of candidates prior to the interview phase. That these systems behave differently depending on the instructions given is no longer disputed—what is less clear is how candidates are filtered from the applicant pool and if those filters succumb to bias or other systematic errors.
What’s Not In Scope
Not considered as part of my study are the following hot-button issues:
-
Legacy keyword-matching ATS software, which doesn’t reason and can’t manufacture anything
-
Any specific company’s hiring pipeline (and their unknown prompts and models used)
-
Demographic bias, a real and studied problem that deserves its own design and a review board, not just a solo project.
Names and backgrounds were held demographically constant across all profiles so that the cultural dimension could not leak into the result. This is a study of a single class of system—an LLM given a résumé, a job description, and an instruction under a single reasonable configuration.
Study documents & their design
I designed and drafted the following documents specifically for this study:
| Job Description |
A pseudo Senior Software Engineer job posting for a hypothetical payments platform team in Plano, Texas, with seven required and six preferred qualifications. It was AI-drafted to my specification and checked against live DFW postings to make sure it read as typical. Find it in the github repo titled |
| 12 candidate profiles (resumes) |
Fictional profiles templated uniformly with same sections, of comparable length and density, and using same register so that the only difference between them was a person’s actually experience. Each labeled according to degree of capability before running the study. Used the following definition for clearly capable:
|
|
As a sidebar, it is my belief that a competent human reviewer reads a résumé and assesses the person as a whole (in entirety). For instance, they may be impressed by the accomplishments listed and decide not to assess capability with a 1-to-1 mapping of job requirements to career experience. The capability assessment, therefore, would be less based on "checking every box" and more on holistic judgment and knowledge of people. Humanity often falls prey to the trap of reducing one another to 2-dimensional checklists and breaking one another down to a list of attributes, assessing each other’s worth by first picking each other apart. It’s like trying to remember your grandmother’s face by looking for women with similar hair color or stature. The essesnce of what marks us distinct is something more abstract. This abstract impression should not be disregarded as it is essential to human uniqueness. |
For the purpose of this study, candidates were passed through if they met 4 out of 5 of the major critiria.
By this metric, 4 of the candidates were labeled as CLEARLY_CAPABLE, 4 were labelled BORDERLINE_CAPABLE, and 4 were labelled NOT_CLEARLY_CAPABLE. These labels were not revealed to the screener at any point during the study and labels were attached to results after the fact and for the purpose of analysis.
Career Paths of the CLEARLY_CAPABLE
The 4 profiles labelled as CLEARLY_CAPABLE were designed to qualify via different career paths. These paths are defined as follows:
| C02 |
Conventional path. Computer Science (CS) degree, 8 years in payments, Java/Kafka/AWS, mirrors posting nearly line for line. |
| C04 |
No degree path. Apprenticeship, 9 years, 6 in payments infrastructure. Qualifies under posting’s "equivalent practical experience" clause. |
| C07 |
The adjacent route. Data platform engineer for 8 years, Python (not Java), healthcare and analytics (not payments), runs Kafka at 40M events a day and owns Terraform and observability. Technically overqualified; conventionally shaped wrong. |
| C10 |
The nonlinear path. Prior career in actuarial analysis, took family leave in 2023—24, seven years of backend work in credit services. |
Screening Conditions
Each "condition" is a different screening instruction appended to the same job description and résumé. Exact text can be found in the registration.
Condition |
Instruction |
1 |
Is there sufficient evidence this candidate could perform the role? ADVANCE or REJECT. |
2 |
Identify and list the candidate’s weaknesses, then decide ADVANCE or REJECT. |
3 |
Build a scoring rubric from the requirements, score the candidate, then decide ADVANCE or REJECT. |
4 |
List the reasons this candidate should not advance, then decide ADVANCE or REJECT. |
5 |
Full ATS-style evaluation: strict requirements, competency scoring, keyword matching, gap and risk detection, threshold assessment. Then decide ADVANCE or REJECT. |
The Study
The study was run using a total of 12 candidates. Each candidate was evaluated under each of the 5 conditions outlined in the Conditions Summary table above, running each 30 times per condition which results in a total of 1,800 API calls.
12 candidates x 5 conditions x 30 runs = 1,800 calls
Each of the 1,800 API calls used a fresh model instance which had no memory of or relationship to any other API call in the study. Each condition required a response from the screening model of either FINAL: ADVANCE to continue the candidate or FINAL: REJECT to eliminate the candidate from consideration. This is the number measured by the study.
The testing harness contains code for the screener, a parser, and associated document materials. The github repo named llm_job_seeker_study contains the complete code and every raw response from each of the 1800 API calls in the study.
Post-Registration Changes
Four changes were made to the repo post registration to OSF. These are briefly described here and also carefully logged in the git commit history.
-
The first candidate profile had a different markdown header level from the other profiles that followed. Consequentially, it was skipped on the study’s first run. I adjusted the markdown level of the header to match the other candidates and reran the study to capture the 150 additional cells (profile text unchanged).
-
The parser was stricter than the model, which resulted in some valid responses failing to parse. Responses that ended in
FINAL: ADVANCE(asciidoc bold) failed to parse. I loosened the parser to tolerate asciidoc and re-scored all 1,800 responses under the single new rule. -
Condition 5 produced long scoring tables, and six responses hit the token ceiling before reaching the final line. I raised the token limit and reran those cells. This mattered because the truncated responses were all strong candidates midway to receiving an ADVANCE determination, and dropping them would have biased condition 5 toward rejection.
-
The system prompt, designed as a constant wrapper that tells the model it is a screener and specifies the output format, was written post-registration. It is identical across all conditions and is in the repo.
None of these changes affected the terms of the study. Each should have been caught during piloting (test run of the study prior to official registration). My own pilot run did not catch any of them, which I can attribute to being my first study and therefore not having awareness of all potential failures beforehand.
Results Analysis
1,800 responses were parsed in all, with 0 being unparseable after the fixes above were implemented.
| Condition | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|
CLEARLY_CAPABLE |
0.0% |
0.8% |
0.0% |
15.0% |
0.0% |
BORDERLINE_CAPABLE |
98.3% |
100% |
100% |
100% |
100% |
NOT_CLEARLY_CAPABLE |
100% |
100% |
100% |
100% |
100% |
95% Wilson CI, n=120 per cell
The main preregistered comparison sits along the top row. Three of the four "rigorous" conditions did nothing to affect the screener. Weakness-hunting, scoring rubrics, and the full ATS-style evaluation advanced clearly-capable candidates every time, exactly as the bare question did. Condition 4 is the only one that moved the rejection rate, and it moved it from zero to 15% (15% rejection rate increase for a single candidate only), with a confidence interval of roughly 10% to 22%.
| Condition | 1 | 2 | 3 | 4 | 5 |
|---|---|---|---|---|---|
C02 (conventional) |
0 |
0 |
0 |
0 |
0 |
C04 (no degree) |
0 |
0 |
0 |
0 |
0 |
C07 (adjacent route) |
0 |
1 |
0 |
18 |
0 |
C10 (nonlinear path) |
0 |
0 |
0 |
0 |
0 |
Screener Responses
Candidate C07, the Data Platform Engineer, was rejected 18 out of 30 times under condition 4 and only once for all other conditions combined (120 total runs). The candidate without a degree and the candidate with a career gap were not rejected.
The screener cited candidate weaknesses the same across multiple conditions and when prompted to provide them.
Condition 2, which prompted the model to identify and list candidate weaknesses, resulted in it listing candidate 7’s weaknesses yet determinting to advance her anyway:
…lack of payments experience are genuine concerns but manageable for a strong candidate with clear technical depth. The missing mentorship signal is a notable omission against an explicit requirement.
On balance, the candidate meets most required qualifications and nearly all preferred ones technically. The concerns are worth probing in an interview but do not warrant rejection.
FINAL: ADVANCE
Condition 3, which prompts the model to design a scoring rubric and assess the candidate against it, scored her and noted that the Java gap was explicitly covered by the posting’s "or comparable" language.
Condition 5, to apply a full ATS treatment, resulted in a score of roughly 88 out of 100 and called the domain gap "a hiring-team conversation, not a disqualifier".
Condition 4, which asks the model to list the reasons this candidate should not advance and then decide ADVANCE or REJECT, the same facts became a case against:
…Despite strong preferred-qualification matches (Kafka, Terraform, EKS/Kubernetes, Prometheus/Grafana, CI/CD, PostgreSQL), the core role requirements — backend service engineering for payments, Java or strong equivalent, and mentorship — are insufficiently demonstrated.
FINAL: REJECT
12 runs still advanced her under the same instruction as observed in run 21 (among others):
…despite gaps, the candidate meets most required qualifications (8 years experience, Python, REST APIs, SQL, AWS, CI/CD, cloud) and nearly all preferred technical qualifications (Kafka, Docker/Kubernetes, Terraform, Prometheus/Grafana). The profile is strong technically, and the gaps are relatively minor in the context of the full requirement set.
FINAL: ADVANCE
Neither the inputs nor the instructions changed between these two responses for Candidate 07. The model sampled a different path through the same argument and landed on opposite sides.
Thoughts & Analysis
My hypothesis stated that increasing layers of scrutiny can unwittingly disqualify qualified candidates when run againsts a large dataset of candidates who have applied to a job listing. Study results do not agree with this assertion. The model can be prompted to hunt for weaknesses, score competencies, and run a full ATS gauntlet and with increasing layers of rigor applied without disqualifying a clearly-qualified candidate.
What actually moved the model to disqualify a clearly capable candidate was not how much it was asked to evaluate someone but what it was asked to do with what it found. Condition 2 and condition 4 surface the same facts about C07. However, Condition 2 frames weaknesses as inventory while condition 4 frame them as an argument. The model acted on the profile with the most weaknesses listed, and disqualified it because it had the most material arguing against and not because it was actually weak. Among the 5 candidates labelled as CLEARLY_CAPABLE, Candidate 07 is arguably the strongest Engineer among the 5 considered as such for the purposes of the study.
It appears to have chosen the candidate whose profile offered the most material for an argument against hiring them. The other, more conventional candidates did not provide as many arguments meaning there was less to bring up when listing their weaknesses.
Another observation is that models do not retain context once contexts become hyper-theoretical and more than once removed from the original instruction. For instance, the screener holds the context throughout the study that it is determining which candidates amongst the pool demonstrate sufficient skill and experience for the Job description. Once the prompt layers on a meta-level discussion instigated by a list of candidate weaknesses, the model’s focus appears to drift away from the original evaluative task and to consider the candidate entirely based on the list of weaknesses produced.
Previous context that passes candidates with similar and overlapping skill transfer, or that there are at least four acceptable career paths that a candidate can follow and qualify as capable are temporarily forgotten and dropped completely from decision-making once the model is prompted to choose to advance or reject after listing the candidate’s weaknesses. In fact, condition 4 was intentionally written to not provide explicit direction-- that the model should consider the new list of weaknesses when making its determination. It is heavily implied that it should and is reasonable if the model assumes this. However, once contradiction is introduced (an instruction that conflics with a previous context), the model appears to prioritize the most recent instruction over the previously established context rather than asking clarifying questions. Still, it also only allows the list of weaknesses to supersede previous instruction for a single candidate and only on 15% of all of this candidate’s runs, which suggests a gap or weakness with the model, or a place where it can fall through rather than of where is consistently performs. To reject a candidate’s candidacy based on sheer number of tick marks that are seemingly racked up against the candidate indicates that previous tasks that passed the candidate on the same profile are not being considered or even "held in mind". It seems like a stack overflow issue for contextual memory location, similar to a person multiplying 2 triple digit number in one’s own head and forgetting one of the places to carry the one forward, et.al..
I cannot conclude from this study that "adjacent-route candidates" as a class are more vulnerable to instructions that prompt them to look for reasons against passing a candidate. With a study containing one adjacent-route candidate, I can say that this study and run under this model results in it being the only condition where a qualified person was rejected based on making a case against.
The study results indicate that the model’s decision-making is highly sensitive to how weaknesses are framed and that it may prioritize the most recent instruction over previously established context. It also suggests that this sensitivity exists only around candidates whose profiles already buck some established norm or expectation and only about 15% of the time.
Candidates with Borderline Capabilities
The middle row of Table 2 shows candidates labelled as having borderline capabilities. These are people who match some of the requirements and might be worth interviewing, especially when the candidate pool is thin. These candidates were rejected 98 to 100% of the time by the screeners in this study and under every condition, including bare sufficiency. This indicates that the screeners themselves have no concept of "middle ground"… they have no gray areas.
I did not preregister any claim about this, so am reporting it as an observation and not a finding. What I can say is that the model’s selection of candidates for "clearly capable" matched the pre-labelled candidates in this study four for four, for four of five conditions. That there was no distinguishing of borderline candidates could simply be that there was no reason for a 3rd division amongst the candidates, as the screener was asked to determine whether to pass or reject only. In hindsight, labels could have been kept to "capable" or "not capable" only, but a third category was used in case the screener showed less consistency choosing which candidates to pass than it did. It holds for a tool that has little capacity for layering context and nuance, that candidates do not fall into any gray areas from the model’s perspective and demonstrates the limited conceptual ability of the model. While it does not prove so, nor was the study designed to test this statement, models when used as screeners are holding candidates to very rigorous conditions, expecting 100% line for line matching of requirements to experience unless explicitly instructed to accept a lower bar. If you are a borderline candidate for a job listing, these results suggest that you will not pass initial screeenings and will be held to matching all or nearly all of the job requirements. It is sad because this is a very narrow way to measure human ability and worth.
|
Things that Lend me Pause
|
What to run next
If I could wave a wand and run an unlimited number of studies as follow up, I would run with the following variations (each run dedicated to a single modification from the original study):
-
Adjacent-route profiles to gather more data on whether C07 becomes a pattern - this is the result where the candidate with the non-traditional path of career specialization in an adjacent area was treated differently from the other candidates
-
2 runs — the first using the original model followed by another run using a newer model
-
Condition 4 phrasing changed to something more neutral to see how much wording led to presupposition
-
Addidtional conditions, including::
-
One that is pool-relative
-
One that ranks the candidates
-
One that advances the top 20% of candidates (which changes the unit of analysis, but is closer to what real pipelines do)
-
Materials
-
Preregistration: link: https://osf.io/832yx
-
Code, materials, & raw responses link: https://www.github.com/k8port/llm_job_screener_study
-
Job description & candidate profiles can be found in the repo under
harness/
I began this project because I wanted to know whether I could run a study on an AI model from my laptop. I can, by the way, albeit on a pretty small scale.
The study I designed produced a result I would not have predicted and one that I cannot explain with irrefutable certainty. It indicates, at the very least, a potential pitfall in the reasoning of ATS screeners, and pokes holes in the "pro" argument for ATS screeners as valuable tools in screening job candidates.
While the study itself is imperfect, one might argue that while ATS screeners reduce hiring times and enhance productivity, over time their use leads to increasing systemic inequality and missed opportunities for both employers and candidates. It stands to reason that when candidates are not selected dynamically, the result is a less dynamic workforce.
The results support the idea that careful evaluation and potential recalibration of automated screening tools is in order, or consideration of whether they should be used to measure human worth. A differently-designed workflow or mechanism might filter candidates without flattening the employee landscape nearly as much and still save time and resources during the hiring process. We should keep an open mind and stay aware of how we use agentic tools in hiring and beyond. Rationale and logic are but one way to measure value but not the entire picture and we should be cautious about crowning it "king".