All research

RESEARCH

Human Behavior research

Peer-reviewed work from the team, and the parts of it that changed how the product is built. Everything below is reported at the level of detail we would want if we were reading it — the objective, the constraints, the tables, and the ablations that did not go the way we expected.

Overview

Human Behavior is a product-analytics and session-replay platform whose agents grade things: replay states, error groups, funnel drops, conversation quality. Every one of those is a machine judging an interaction that a person could also judge, which makes the reliability of LLM judges an infrastructure question for us rather than an academic one.

Our most recent published contribution, MELISSA, was accepted at NeurIPS 2025. It asks a narrow question with a broad answer: when an LLM grades a twenty-minute conversation and disagrees with a human expert, what exactly is it getting wrong? The finding is that it is usually not the ranking. It is the scale. A judge that reliably distinguishes a good technical question from a bad one can still be off by nearly a full point on a five-point Likert scale, in the same direction, every time.

That reframes the fix. Instead of a larger model, a fine-tuning run, or a specialized architecture, the correction is a convex least-squares problem over a handful of parameters fitted against human labels. It reduces error by more than half, and it makes GPT-4o-mini competitive with models that cost an order of magnitude more per evaluation.

Publications

NeurIPS 2025

MELISSA: Multi-level Evaluation with LLM-based Integrated Self-Scrutiny and Auditing

Amirhossein Afsharrad1,2*, Sri Jaladi1,3*, Nima Yazdani2, Ali Ansari1,2, Seyed Shahabeddin Mousavi1,2, Sanjay Lall1

1 Stanford University · 2 micro1 · 3 Human Behavior · * Equal contribution

Abstract

As AI systems increasingly conduct complex multi-turn interactions, reliable evaluation becomes critical yet challenging. Current LLM-as-judge approaches suffer from severe biases and struggle with lengthy conversations, while monolithic evaluation misses quality variations across dialogue segments. We present MELISSA, a framework that hierarchically decomposes conversations and learns bias corrections from human judgments, requiring no model fine-tuning. Evaluating 100 AI-conducted technical interviews with expert annotations reveals surprising insights: bias correction alone reduces error by over 50%, indicating LLMs struggle with scale calibration rather than quality discrimination; properly aligned GPT-4o-mini outperforms unaligned Claude 3.5, enabling order-of-magnitude cost reductions; and optional audit mechanisms show mixed results — while potentially providing confidence signals, they often introduce unnecessary edits that degrade performance when models already produce well-calibrated evaluations, highlighting the importance of empirical validation over intuitive design. These findings demonstrate that simple calibration transforms weak models into reliable judges, while reinforcing that high-quality human data remains essential for automated evaluation systems.

The MELISSA framework

MELISSA takes a conversation C with T turns and decomposes it into L hierarchical levels, each holding its own set of evaluation units. Level 1 is individual turns, intermediate levels are topical sections or conversation phases, and level L is the complete transcript. Each criterion is scored at every level, and the levels are combined by a learned weighting.

Relevance weighting

Not every unit bears on every criterion. A greeting exchange is irrelevant to technical question quality and entirely relevant to conversational quality, so each unit gets a relevance score — binary to filter, continuous to weight.

rc,u = frel(u, c) ∈ [0, 1] or {0, 1}
[1]

Multi-pass evaluation

Every relevant unit is scored N times independently. The repeats buy two things: averaging reduces variance, and the spread across trials is itself a usable confidence signal.

sinitc,ℓ,u,n = feval(u, c, ℓ)   ∀ u ∈ Uℓ, n ∈ {1, …, N}
[2]
sauditc,ℓ,u,n = faudit(u, sinitc,ℓ,u,n, c)
[3]

Equation 3 is optional and, per Finding 5, off by default.

Hierarchical aggregation

Level scores are relevance-weighted means over the units at that level. The final score is a convex combination of level scores plus a scalar bias, optionally rescaled when the model and human scales differ.

s̄c,ℓ = Σu ∈ Uℓrc,u · 1NNΣn = 1sc,ℓ,u,nΣu ∈ Uℓrc,u
[4]
Sc = αc · [LΣℓ = 1wc,ℓ · s̄c,ℓ] + bc
[5]

Algorithm

Six stages: decompose, assess relevance, evaluate N times, optionally audit, compute level scores, aggregate. Nothing in the loop requires fine-tuning, a reward model, or specialized infrastructure — it is standard API calls plus one convex solve.

Algorithm 1 · MELISSA evaluation pipelinesimplified
Require: conversation C, criteria C, parameters (N, L, audit flag)
Ensure: final scores {S_c} for all criteria
1Decompose C into L hierarchical levels {U_1, …, U_L}
2for each criterion c ∈ C do
3Assess relevance r_{c,u} for all units ▸ Eq. 1
4Perform N independent evaluations of relevant units ▸ Eq. 2
5if audit enabled then review each score ▸ Eq. 3
6Compute relevance-weighted level scores s̄_{c,ℓ} ▸ Eq. 4
7if human scores available then
8Learn w*_c, b*_c via constrained least squares ▸ Eq. 6
9else use uniform weights w_{c,ℓ} = 1/L, b_c = 0
10Aggregate final score ▸ Eq. 5
11end for
12return {S_c : c ∈ C}

Parameter learning

Given m conversations with human scores, the level weights and bias are fitted by minimizing squared error subject to the weights forming a probability distribution over levels.

minw ∈ ℝᴸ, b ∈ ℝmΣi = 1[y(i)c − αc · LΣℓ = 1wc,ℓ · s̄(i)c,ℓ − bc]2subject to LΣℓ = 1wc,ℓ = 1,  wc,ℓ ≥ 0  ∀ ℓ ∈ {1, …, L}
[6]

Quadratic objective, linear constraints — a convex program solved with an off-the-shelf solver. With L = 3 and two criteria that is eight free parameters in total. The bias-only configuration has a closed form: the optimal bias is just the mean gap between the human scores and the unweighted mean of the level scores.

b*c = 1mmΣi = 1[y(i)c − 1LLΣℓ = 1s̄(i)c,ℓ]
[9]

Learned parameters

ModelCriterionw · turnw · sectionw · holisticbias bMAE
GPT-4o-miniTQQ0.8540.0000.146+1.11350.4045
GPT-4o-miniHLI0.4940.3490.158+0.55350.4514
GPT-4oTQQ0.2080.0000.791+0.37440.4068
GPT-4oHLI0.1010.1030.796+0.20090.3940
Claude 3.5TQQ0.9390.0610.000+1.09020.4249
Claude 3.5HLI0.7650.0000.234+0.58260.3541
Claude 3.7TQQ0.7800.0580.162+0.81260.3977
Claude 3.7HLI0.7220.0740.204+0.42410.3810
Claude 4TQQ0.7150.2310.054+1.20080.4540
Claude 4HLI0.7850.0490.166+0.75310.4192

Table 3 (WA+B rows). Weights live on the probability simplex, so the three level weights sum to one. Every learned bias is positive: the judges score below the humans and the scalar pulls them back up. Section weight collapses to or near zero in seven of ten rows.

Where the bias comes from

ModelPre-audit TQQPre-audit HLIPost-audit TQQPost-audit HLI
GPT-4o-mini-0.5909-0.1307-0.9664-0.3537
GPT-4o-0.4551+0.0265-0.5679-0.0336
Claude 3.5-0.4667+0.1111-0.8657-0.3823
Claude 3.7-0.4220+0.2031-0.7225-0.1282
Claude 4-0.8607-0.1834-1.3080-0.5163
Average-0.5591+0.0053-0.8861-0.2828

Table 6. Net error is the LLM score minus the human score for unaligned models. Negative means the judge scored lower than the humans. Technical question quality is underestimated by roughly three times as much as conversational quality, which is why it needs the larger bias.

Evaluation metrics

Training and evaluation deliberately use different objectives. Squared error during fitting keeps calibration honest across the whole range; a thresholded absolute error during reporting reflects what a half-point of disagreement actually costs on a five-point Likert scale.

Mean absolute error

MAE = 1mmΣi = 1|yi − ŷi|
[7]

Threshold absolute error

TAEτ(y, ŷ) = max(0, |y − ŷ| − τ)
[8]

With τ = 0.5, a prediction of 3.4 against a ground truth of 3 takes zero loss, because both round to the same integer and scores are rounded for display and decision-making anyway. Predictions are real-valued — they come from averaging N trials and taking a weighted combination across levels — while a single expert's label is an integer. Averaging k experts puts the ground truth on the grid { j/k } for integers j between k and 5k.

Dataset

100 AI-conducted technical interviews with expert human annotations, covering Python, PL/SQL, and data analytics.

100

Interviews

Conducted by Zara, an industry AI interviewer

20+ min

Typical length

Long enough that decomposition is necessary

3

Independent raters

Per interview, with confidence scores

2 × 1–5

Criteria and scale

Question quality and conversational quality

Three experts scored each interview on both criteria on a 1–5 scale, each score carrying a 1–3 confidence rating so aggregation could be quality-aware. After cleaning, the final ground truth averages the remaining high-confidence evaluations. Raters worked from guidelines written to mirror the model prompts, and were told to grade the interviewer rather than the candidate, to complete the whole interview before scoring, and to use the full scale instead of clustering around the middle.

These are real interactions rather than scripted dialogues, so they carry follow-up questions, clarification requests, partial answers, and the ordinary mess of a technical discussion. That matters for the framework being tested: a short conversation can be judged holistically, and only a long multi-topic one forces the question of how to combine local and global assessments.

Results

Five judges — GPT-4o, GPT-4o-mini, Claude 3.5, Claude 3.7, Claude 4 — across four alignment configurations, with L = 3, N = 5, α = 1, and relevance held at 1 because nearly every segment of a technical interview bears on both criteria.

AlignmentClaude 3.5Claude 3.7Claude 4GPT-4oGPT-4o-mini
NANo alignmentUniform weights w = 1/L, bias b = 0.
0.9550.8361.4390.5321.092
WAWeight alignmentOptimized level weights, bias held at 0.
0.8670.8221.2850.5001.040
BBias onlyUniform weights, optimized scalar bias.
0.4240.4580.5700.4150.559
WA+BWeight + biasBoth level weights and bias optimized.
0.4250.3980.4540.4070.405

Table 1. Lower is better. Cell tint scales with error inside the selected criterion and metric; the best configuration per model is set in bold. All numbers are post-audit, MSE-trained, on the same 100 interviews.

Findings

1

Bias correction dominates weight optimization

On technical question quality, Claude 3.5 goes from 0.955 MAE unaligned to 0.424 with a bias term alone — a 56% reduction — while optimizing weights alone only reaches 0.867. The pattern holds for every model. The judges can tell a good question from a bad one; they cannot place it on the humans' scale.

2

Alignment makes small models viable

GPT-4o-mini with weights and bias reaches 0.405 MAE on technical question quality, beating unaligned Claude 3.5 at 0.955 and landing within 0.002 of aligned GPT-4o. The framework closes a capability gap that would otherwise be paid for per token.

3

The bias is positive everywhere

Every post-audit optimized bias is non-negative, from 0.0 to 1.2. Technical question quality needs an average of 0.89, conversational quality only 0.28. With no human data to fit against, the paper recommends a default of b ≈ 0.5 rather than zero.

4

The middle level often carries no information

Section-level weights frequently optimize to zero when a strong model's context window already exceeds the interview. For that regime L = 2 would suffice — but the result is specific to the setting, and longer conversations or weaker models should keep the intermediate level.

5

Self-audit is counterproductive for strong models

Average degradation from the audit stage is 15–20% MAE and 20–30% TAE without alignment; Claude 4 goes from 0.513 TAE to 0.946. Prompting a model to review its score induces edits regardless of whether the score was wrong.

6

Train on MSE, report TAE

Fitting directly on the threshold metric never beat fitting on squared error, even when measured by the threshold metric. The smooth gradient and the stricter penalty across the whole score range produce better calibration than optimizing the lenient objective.

Audit ablation

The audit stage asks the judge to review and potentially revise its own score. It was designed to add confidence. Measured against human labels, it mostly adds edits.

ModelTQQ (MAE)TQQ (TAE)HLI (MAE)HLI (TAE)
GPT-4o-mini-0.35-0.33-0.15-0.08
GPT-4o-0.08-0.07-0.020.00
Claude 3.5-0.32-0.27-0.14-0.04
Claude 3.7-0.21-0.180.00+0.01
Claude 4-0.44-0.43-0.18-0.14
Δ = pre-audit − post-audit performanceaudit hurtaudit helped

Figure 1(b). Without alignment, asking the judge to review its own score degrades almost every cell — Claude 4 loses 0.44 MAE on technical question quality. Once weights and bias are learned, the same audit is close to a no-op, which is why MELISSA ships it as an optional stage.

The interesting part is the right-hand column. Once weights and bias are learned, pre- and post-audit performance converge to within 5% MAE — the fitted parameters absorb the noise the audit introduces. That is a reasonable argument for keeping the stage available for genuinely weak models or conversations that strain the context window, and a decisive one against running it by default.

Complexity

Cost is dominated by the number of evaluation units times the number of trials. The interactive figures below use the paper's accounting.

Evaluation units

52

Relevance calls

52

Evaluation calls

260

Total LLM calls

312

Appendix C.2. Evaluation cost is O(N · Σ|Uℓ|), which simplifies to O(NT + NL) when level 1 holds the turns and higher levels hold constant-size units. The audit stage doubles it. The weight-fitting step is separate and costs O(mL²) for m training conversations — with m = 100 and L = 3 that is a nine-parameter least-squares solve, not a training run.

Choosing L

The number of levels is a free parameter, and the right value depends on conversation length, model capability, and how much topical structure there is to exploit.

L = 310 ≤ T < 50 turns
  1. 1Level 1 — individual turns
  2. 2Level 2 — topical sections or phases
  3. 3Level 3 — complete conversation

Suitable for. Technical interviews (the configuration used in the paper), tutoring sessions, multi-topic discussions.

Trade-off. Balances granularity against cost; captures local and global patterns.

Appendix G. Adjust from the baseline: subtract a level for GPT-4-class models or a single-topic conversation, add one for weaker models or highly structured content.

Prompts

Reproducibility for a prompting framework means publishing the prompts.

appendix-f/tqq.prompt
34 lines
You are evaluating the quality of an interviewer's technical questions.
Evaluate how well-formed and well-phrased the interviewer's questions
are, and how effectively they assess a candidate's qualifications and
skill level for the given role.

Consider the following aspects:
- Relevance to the stated role and required skills
- Clarity of wording and specificity
- Technical appropriateness and depth
- Whether they test real, practical skills vs memorized trivia
- Progression and follow-up quality

Scoring Scale (1-5):

5 (Excellent): Precisely targeted questions that are perfectly clear
and probe real skills at appropriate depth. Questions are highly
relevant to the role. Hard to suggest improvements.

4 (Good): Well-targeted questions with clear wording and good balance
of theory and practice. Questions assess meaningful skills. Minor
improvements possible.

3 (Okay): On-topic and clear but somewhat generic. Tests basic skills.
Functional but unremarkable. You could definitely improve several
things.

2 (Poor): Tangentially related to role, unclear wording, focuses on
trivia, technically shallow or slightly off. Many obvious
improvements needed.

1 (Very Poor): Off-topic or confusing questions that fail to assess
relevant skills. May include technical errors.

Provide your evaluation as a single integer from 1 to 5.

Appendix F, verbatim. Level-specific context is appended to the base prompt at evaluation time, and the same rubric text is what the three human raters worked from — the alignment between the two is deliberate.

Notation

Appendix A, for reading the equations above without scrolling back.

CComplete conversation with T turns
TTotal number of turns in the conversation
LNumber of hierarchical levels
ℓLevel index, ℓ ∈ {1, …, L}
U_ℓSet of units at level ℓ
u_{ℓ,j}The j-th unit at level ℓ
NIndependent evaluation trials per unit
cA specific criterion drawn from the criteria set
s^init_{c,ℓ,u,n}Initial score for criterion c, level ℓ, unit u, trial n
s^audit_{c,ℓ,u,n}Audited score, when the audit stage is enabled
r_{c,u}Relevance weight for unit u on criterion c
s̄_{c,ℓ}Relevance-weighted mean score for criterion c at level ℓ
w_cLevel weight vector for criterion c, on the simplex
b_cLearned bias term for criterion c
α_cScale alignment factor (1 when the scales match)
S_cFinal aggregated score for criterion c
y^(i)_cHuman ground-truth score for sample i
mNumber of training samples
f_evalLLM evaluation function
f_relRelevance assessment function
f_auditAudit function (optional)

Applied research

What the paper changed about how we build. These are engineering positions we hold because of a measured result, not because they sound right.

Calibrated grading for replay analysis

render-replays

The Path B state grader scores rendered replay states with an LLM. MELISSA's first finding applies directly: the grader is good at ordering states and bad at picking an absolute number, so the useful correction is a learned scalar against human labels, not a bigger model.

Hierarchical decomposition of sessions

trace-compiler

A session is a multi-turn interaction with the same structure the paper decomposes: individual events at the bottom, semantic segments in the middle, the whole session at the top. The deterministic rrweb → semantic trace pass is what makes level-2 units exist at all.

Grouping without a judge in the loop

ingestion pipeline

Error fingerprinting groups exceptions deterministically before any model sees them. Where the paper warns that judges miscalibrate scale, the cheapest defence is to keep the parts that can be computed exactly out of the model's hands.

Small models, properly aligned

hb-agents

Finding 2 is the economics of the whole agents layer: an aligned small model matched an unaligned frontier one. Every agent path that runs per-session rather than per-request is built on that result.

Audit stages are opt-in

hb-agents

Self-review is off by default in our own scoring paths. The paper measured what self-correction literature predicts — prompting a capable model to re-check a well-calibrated score mostly buys unnecessary edits.

There is a limitation worth stating plainly, and the paper states it: the performance ceiling of any learning-based evaluation system is bounded by the quality of the human data it is fitted against. Automating evaluation does not reduce the need for human judgment. It concentrates it.

Cite this work

If MELISSA is useful to you, the citation is below. If you are working on evaluation, calibration, or agent reliability and want to compare notes, we would like to hear from you.

melissa.bib
@inproceedings{afsharrad2025melissa,
  title     = {{MELISSA}: Multi-level Evaluation with {LLM}-based
               Integrated Self-Scrutiny and Auditing},
  author    = {Afsharrad, Amirhossein and Jaladi, Sri and
               Yazdani, Nima and Ansari, Ali and
               Mousavi, Seyed Shahabeddin and Lall, Sanjay},
  booktitle = {Advances in Neural Information Processing Systems
               (NeurIPS)},
  year      = {2025},
  note      = {Equal contribution: Afsharrad and Jaladi},
}

Correspondence to sjaladi@stanford.edu. For anything about the product side of this work, email founders@humanbehavior.co.