Why two judges score the same photo differently

The short answer

Two judges disagree because they are not running the same process. A person has taste and habits, an AI persona is tuned to pay attention to particular things, and neither is a ruler. Different results from different judges are the expected outcome, not a sign that one of them got it wrong.

What you can do about it is compare the results properly, which mostly means working out what each one was actually responding to.

Disagreement is the normal case for any rater

This is not unique to rating sites. Wherever people assess the same thing, the question of agreement comes up, and measuring it is standard practice. As McHugh put it in a review of interrater reliability in Biochemia Medica, the question of agreement arises because of variability among human observers, who may experience and interpret the same thing differently.

So the right starting assumption is that two judges will differ, and the useful question is how and why. A single judge is one reading, and a second reading will almost never be identical.

Taste and scale habits

A human judge answers in her own words, in the medium she offers, and what she notices first is hers. Some lead with what works, some with what does not. When a judge gives a number at all, it is on her own scale, and a score that is generous for one judge can be an ordinary one for another.

That is why the same photo can come back sounding very different from two people without either being unfair. Reading a judge’s profile and intro clip before you order tells you a lot about how she tends to deliver, which is covered in how to read a judge profile.

Different settings on the AI side

The AI side differs for a more mechanical reason. An AI judge is tuned: a tone, intensity dials, and up to four focus areas that decide what the write-up spends its words on. Two personas given the same photo are therefore being asked slightly different questions.

The AI scan’s own total is a summary of six separate readings, and the six rating axes are not weighted equally, so a scan and a persona with a narrower focus can reasonably land in different places on the same photo.

Scores versus words

A lot of what a judge gives you is not a number at all. A written, audio or video verdict is a set of observations, and two people can arrive at similar conclusions while sounding nothing alike. Comparing the words usually tells you more than comparing a figure.

Pay attention to emphasis. The thing a judge mentions first, and the thing she skips, tell you what she values. Two judges who agree on substance and differ on emphasis have not really disagreed.

When the gap is large

A big gap is worth a short checklist before it is worth a worry. Were both given the same photo, at the same crop? Did both answer in the same medium, given that a live call can respond to you and a recording cannot? Were the AI settings similar, or did one persona have a narrow focus?

If the answers are all yes and the gap remains, treat both as valid readings and look for what they share. A third view is only worth getting if the decision rides on it, and how to get a second opinion covers how to do that without overspending.

One thing disagreement is not: a problem with the order. Refunds and cancellations are about delivery, such as a judge who never delivers or declines, and the refunds and cancellations guide covers those rules. A verdict you disagree with is a verdict, not a fault.

Why disagreement is useful

A judge who always matched you would tell you nothing you did not already think. The disagreement is where the information is, because it shows which parts of a result are robust and which are one rater’s taste.

McHugh’s review also makes a point that applies here: raters find finer distinctions harder to agree on. Two judges will usually agree that a photo is well lit or poorly lit, and differ much more on subtle matters of taste. Read the agreement as the firm part of the result and the disagreement as the soft part.

How to compare two results

First check that the inputs match: same photo, same crop. Then compare what each result talks about before you compare any number. If one stresses lighting and the other stresses proportions, they did not disagree, they answered different questions.

Then look for what both mention independently, which is the sturdier signal, and treat anything only one mentions as a taste. What your score means covers reading a result without over-reading it. If you are comparing on the leaderboard, remember it ranks posted AI scans, so it is a comparison between scans rather than between judges.

Photo choice matters here too. If you resend a slightly different photo to the second judge, you have changed the input as well as the rater, and you cannot tell which one caused the gap. Send the identical file both times.