Video Interviews with AI Scoring: 6 Factors That Determine Accuracy

Most teams evaluating AI scoring in video interviews open with the question "Is it accurate?", which sounds like the responsible thing to ask, but this assumes something that is not true: namely, that the process it would replace is accurate to begin with. The more useful question is "accurate compared to what?", because the answer changes the entire evaluation.

Consider what that comparison looks like in most hiring teams. Ratings from unstructured interviews show inter-rater reliability of around 0.27, which means two interviewers watching the same candidate answer the same questions will often reach different conclusions, and both will leave the conversation feeling certain they were right. Neither of them is lying about their confidence. They simply had no shared definition of what a strong answer looked like, so each one filled that gap with instinct.

Rather than measuring the candidate, an unstructured process ends up measuring the interviewer. Without written criteria, ratings mostly capture the habits, blind spots, and social chemistry of whoever happened to be on the call that afternoon, which is why the same candidate can be a clear yes for one reviewer and a soft no for another. When teams introduce anchored scorecards instead, reliability climbs from roughly 0.37 to 0.67, and that jump is the tell. Most of the inconsistency was not about the interviewers' quality. It was about the absence of a process for them to follow.

So the benchmark for AI scoring was never a perfect assessor, and it was never meant to be. Instead, it is a rushed, inconsistent, thinly documented human judgment that most HR teams already suspect is the weakest link in their funnel, but have never had the time or the volume tolerance to fix properly.

That is the bar. And once you understand how a scoring pipeline is built, you can work out fairly quickly whether a given tool clears it, or whether it simply reproduces the same guesswork faster and with a more confident number attached.

Adoption Ran Ahead of Understanding

AI moved into recruiting faster than almost any other HR function. According to SHRM's State of AI in HR 2026 report, which surveyed 1,908 HR professionals, 39% of HR teams had adopted AI in recruiting by early 2026, with another 46% expecting to adopt before the year closed.

Recruiting is now the single most common AI use case inside HR, ahead of HR technology, learning and development, and employee experience.

But adoption is uneven. SHRM puts adoption at 60% among companies with 5,000+ employees, versus 33% among companies with fewer than 100 employees. Smaller teams are often the ones with the worst screening bottleneck and the least capacity to evaluate vendors carefully.

Meanwhile, the regulatory floor has risen. Colorado's SB 205 took effect in February 2026 and requires impact assessments for high-risk AI systems, including employment tools. The EU AI Act's high-risk obligations for hiring and promotion decisions apply from August 2026. NYC's Local Law 144 has required annual independent bias audits since 2023.

So the stakes changed. Two years ago, a bad scoring model cost you a few mis-hires. Now it can cost you a compliance finding.

The 6 Factors That Determine Whether the Scores Hold Up

1. Structure, not the model

This is the part people skip past on the way to the demo. The accuracy gain comes mostly from structuring the interview, not from the AI reading it.

Interviews built on predefined questions and anchored scoring carry roughly double the predictive validity of conversational ones, and the wider body of meta-analytic work has placed structured formats at the top of the selection-method rankings for decades. None of that gain requires a model. It requires a set of questions and a rubric.

Part of why this works is unglamorous. Every candidate gets the same prompts in the same order, with the same prep time and the same limit on how long they can talk, which removes the variable that ruins most interview data: that every candidate was effectively sitting a different test. Scoring only becomes meaningful once the answers are comparable in the first place.

Which leads to the conclusion that if your interviews are unstructured today and you buy a scoring tool tomorrow, most of the improvement you see will come from the structure the tool forces you to adopt, not from the scoring itself.

Takeaway: Before you evaluate vendors, ask whether your interviews are structured at all. Fixing that is the bigger win, and it costs nothing.

2. Rubric specificity

"Communication skills, score 1 to 5" produces noise. "Explains a technical concept to a non-technical stakeholder without using internal jargon" produces a signal.

This is also the step that must happen first, before any candidate records anything. The competencies, the questions that surface them, and the descriptions of what a 1, a 3, and a 5 look like are all written up front because everything downstream is measured against them. A scoring engine pointed at a vague rubric will not fail. Instead, it will produce vague scores with impressive consistency, which is harder to catch.

The difference shows up the moment two people try to use it. A vague rubric sends everyone back to instinct, which is where you started. A specific one gives them something to argue about in writing, which is where consistency comes from.

There is a simple test for this. Put three reviewers in a room, give them the same recorded answer, and ask them to score it against your rubric without discussing it first. If they land far apart, the rubric is the problem, and no model will patch over it.

Takeaway: Write behavioral anchors for every competency. If your own team cannot agree on what a 4 looks like, a scoring engine will not either.

3. How the vendor's accuracy claim was tested

Two questions tend to break an accuracy claim open.

The first is which roles the validation covered. Validity does not transfer cleanly across job families. A scoring engine tuned on high-volume customer service interviews will underperform on senior engineering interviews, because the answers are shaped differently and the signal sits in different places.

The second is what the number is being compared against. VidCruiter, for example, reports reliability above 0.95 against expert human reviewers on standardized scales. A figure like that needs a reference point, and the reference point is trained humans, who agree with each other in the 0.6 to 0.8 range in published studies. So a tool scoring 0.7 against trained reviewers is at parity, not falling short, which is a very different story from the one the raw number tells.

Even at parity, though, decisions happen at the margin. Nobody agonizes over candidate 2 or candidate 40. The arguments happen in the narrow band around the cutoff, and a strong overall correlation can still reshuffle that band.

Takeaway: Ask which job families the validation covered, and ask for agreement rates at the shortlist cutoff rather than across the full pool.

4. Speech and communication variation

Almost every scoring engine works from a transcript rather than the video itself, which means the recording is converted to text before anything is evaluated. That detail matters more than it sounds, because whatever the transcription gets wrong does not stay contained. It travels into every score built on top of it.

Accents, cultural communication styles, and neurodivergent speech patterns can all pull scores down for candidates who are perfectly qualified, partly because of these factors and partly because of how the model reads the text afterward. This is the most documented failure mode in automated video assessment, and it is the one regulators are watching most closely.

It is also the one you are least likely to notice from the inside. A candidate who was marked down for how they sound does not appear in any dashboard. They appear as an absence, and absences do not raise flags.

Takeaway: Ask for the most recent bias audit and read the impact ratios by subgroup. If a vendor cannot produce one, that answers the question for you.