Decision Models for Voice AI: What a 48-hour Hackathon on 586 Real Call Turns Taught Us

TL;DR

Our in-house decision models sit on the quality-against-speed frontier for both text and audio questions, answering a turn's questions in 0.05 to about 1 s. Only one external model scored higher on text, and it took 8 s.

From 14 to 16 October 2026, two engineering teams and one agent-built entry trained small models for the closed-choice decisions a Voice AI Agent makes on every turn. We scored all of them on one benchmark: 586 decision points from 368 real calls, 43 questions, 14,573 gold answers.

Three things to read off the plots: 

1. On text, NITI-v2 is 0.048 below Gemini 3.8 Flash and answers in 0.10 s against 8.0 s. 
2. On audio, every point on the frontier is ours, though PARAM's lead over Gemini 4 Argon is only 0.003. 
3. And the only external model inside the agent's 0.8 s reply budget, Jev 1.13, scores below all three of our entries on audio and below NITI and VG-Decision on text.

Two caveats shape how far these numbers go: We released the gold answers on day two for error analysis, so final runs were not blind. And our latency figures come from our own GPUs while external models were timed end to end through APIs. Both are covered in the limitations section.

‍

Why decisions

Every lead turn forces the agent to settle a dozen closed questions before it speaks, and all of them share a 0.8 s median reply budget.

Take the first reply on an automotive call. The agent asks, "Am I speaking with [name]?" The lead says, "Yes, but I'm driving." Before replying, the agent has to decide who answered (the lead), what they want (a call later), whether there is an objection (timing), what their situation is (driving), how the line sounds (choppy) and what to do next (book a callback). Nobody said "call me later". The model has to infer it.

Today many of these decisions go to large general-purpose language models. That path has three costs:

1. The model writes free text, and our code has to parse a decision out of it. Parsing can fail.
2. One round trip takes seconds. On our benchmark, answering one turn's questions took Gemma 4 31B 6.2 s, Gemini 3.8 Flash 8.0 s and GPT-6 Luna 25 s.
3. Free text gives code no per-option probability, so it cannot tell a sure answer from a guess and route accordingly.

The hackathon asked whether small models we train ourselves can make these decisions as well, in a fraction of the time.

What a decision model is

A decision model takes the call state and a set of typed, closed-choice questions, and returns one answer per question with a probability for every option.

The architecture is a shared encoder that reads the turn once, plus one small head per question. The benchmark asks 16 to 31 questions of every turn, and all of them are answered in a single forward pass. Adding a question adds a head, not a request.

The top path is how most of these decisions run today. The bottom path is where every hackathon entry is built.

A request carries the state and the questions:

The answer is a choice with its distribution:

Three properties follow from this design:

• Answers are always valid. The model can only pick from the options it was given, so there is no parser to fail.
• Latency barely grows with question count. One pass answers every question on the turn.
• The distribution is a routing signal. A peaked distribution means code acts on the answer directly. A flat one means the turn goes to a larger model or a person.

The setup

Every entry answered the same questions on the same turns, and only the input changed, so the leaderboard measures what each signal is worth as well as which team trained better.

Both teams used AI assistants in their own work. VG-Decision differed in who drove the build: an architect agent planned, split and reviewed the work, and parallel agents handled data, training and evaluation. Ved facilitated and built the benchmark with a five-person annotation team.

Timeline

1. Wed 14 Oct, 10:30. Kickoff: the brief, the question list, and a 50-turn sample set with a scoring portal. V0 models train overnight.
2. Thu 15 Oct, morning. The benchmark design and baseline scores are shared.
3. Thu 15 Oct, 13:30. The full benchmark goes live, with gold answers and a portal that scores a run in seconds. Both teams rework their data strategies and ask for a second overnight training run.
4. Fri 16 Oct, morning. Final submissions land.
5. Fri 16 Oct, 10:30. Each team presents its model, scores and next steps.

Rules

• Submit as often as you like: a CSV with one row per turn and one column per question.
• Every run stays on the board, and the leaderboard rescores all stored runs whenever the gold changes.
• Teams report median latency per turn, so every entry sits on a quality-against-speed chart.

Building the benchmark

We built the benchmark before any team wrote training code: 586 decision points from 368 real calls placed between 1 and 11 October 2026.

Sampling the calls

1. Start from live traffic. Every live campaign, in Hindi, Hinglish and regional languages, on 8 kHz phone lines.
2. Rare cases first. Do-not-call requests, voicemails, objections and regional languages, then a random and a stratified sample. No campaign contributes more than about 6% of units.
3. Check every recording. Drop calls with no transcript, duplicates and low-information calls over a cap. Align each turn to the stereo recording.
4. Keep only what an annotator heard. A turn from a call survives only if an annotator listened and confirmed that the turn's audio clip, from start to end, matched the turn transcript.

We planned 625 calls and 1,150 units. 368 calls and 586 units (51%) passed step 4.

What one decision point holds

Each unit is one turn, judged only on what came before it. Nothing after the target turn is shown. A unit carries:

• Call details: industry, campaign, agent language, whose turn it is.
• The call so far as stereo audio (lead and agent on separate channels) plus the transcript.
• The target turn as mono audio and text.
• On agent turns only, six candidate replies to rank: the real reply, three alternatives, one that misses the point and one that makes a claim it should not. Two of the alternatives are written to sound plausible and be wrong.

The set has 428 lead turns and 158 agent turns.

The questions

We drafted 62 closed-choice questions and kept 43. Three allow multiple answers. Each is asked only where it applies, so a turn gets 16 to 31. By input, 30 can be answered from the words, 8 only by listening and 5 need both.

Most of the 19 dropped questions asked for entities such as loan amount or monthly income, which appeared too rarely in the sample to score fairly.

Where the gold comes from

Each question has a planned gold source. Where sources disagree, the human answer wins.

A sample was labelled twice to measure annotator agreement, and a share of the AI labels was audited by hand. Lead names and numbers never left the labelling tool.

Scoring

Every question is scored with macro-F1, so a model that always gives the most common answer scores 0.267 overall. That is the floor.

Macro-F1 matters here because most answers sit in one class. Emotion intensity is "low" on most turns and the line is usually "good". Take 100 turns with 80 low, 15 medium and 5 high. Always guessing "low" gets 0.80 accuracy but per-class F1 of 0.89, 0 and 0, so macro-F1 is 0.30.

Four headline scores

• Overall: plain mean of macro-F1 over all 43 questions.
• Overall (weighted): the same 43, weighted as below.
• Text Questions Score: 35 questions (30 from the words plus 5 that need both), weighted.
• Audio Questions Score: 13 questions (8 by ear only plus the same 5), weighted.

A question's weight scales with difficulty and is discounted by label reliability:

Difficulty d is the mean of three normalised terms: size of the choice space, rarity of the answers and context needed. Reliability r is annotator agreement for human gold, pass-to-pass agreement for AI gold and 0.95 for platform data. At equal reliability, the hardest question counts four times as much as the easiest.

Rules for the trickier questions

• Classes absent from the gold do not count. F1 is averaged over answers present in the gold, so guessing an unused class costs recall on the true answer instead of adding a free zero.
• Multi-choice questions take a list. Objections, emotions and acceptable replies take zero or more options, scored as the mean F1 over each option the gold uses. An empty answer is valid; most turns raise no objection.
• Speed counts. Each run reports median wall-clock time per turn, with every question answered in one request.

Baselines

Gemini 3.8 Flash reading the transcript was the strongest external model on text questions, and Gemini 4 Argon the strongest on audio questions. Both took more than 8 s per turn.

We ran seven external configurations zero-shot through OpenRouter and kept adding models through day two. Each baseline got one request per turn carrying all of that turn's questions, with no examples, no JSON mode and the provider's default temperature. A missing or invalid answer was left empty and counted as wrong.

What each baseline type received:

• Text models: call details, transcript so far, target turn text and candidate replies in the prompt. Replied with one JSON object.
• Audio models: two audio files (target turn and stereo call so far) in place of the transcript.
• Audio and text: both of the above.
• Decision model API (Jev 1.13): the call as a state object and typed questions, with multi-choice questions asked as one yes-or-no per option.

Every baseline is plotted against our models in the two frontier plots in the TL;DR, with its text score, audio score and answer time per turn.

The zero-shot setup is deliberately plain. Few-shot prompting, JSON mode or tuned temperature would likely lift the external scores, so read these as a floor for what those models can do.

What each entry built

The three entries submitted 16 runs in about 40 hours, and each gained between 0.15 and 0.16 overall from first to best run.

NITI: fine-tuning a large open-weight model

NITI benchmarked two open-weight bases zero-shot on text questions: d3 27B scored 0.657 and Dflash 0.577. The team then trained LoRA adapters of about 0.1B parameters on calls we already own, with the 27B base frozen.

NITI-v1 reached 0.703 on text questions and 0.645 overall at about 0.10 s per turn on our GPUs. Because the adapter sits on the same base, serving cost does not change.

PARAM: starting from the audio

PARAM built the audio-first model. Midway, the team added the transcript as a second input to learn how the two signals combine. Audio Questions Score moved from 0.377 (v0) to 0.367 (v1) to 0.462 (v2), at about 1 s per turn. Overall, PARAM-v2 scored 0.599.

The 0.462 audio score is the highest on the board, 0.003 above Gemini 4 Argon at 0.459. Both are single runs, and we treat per-question differences under about 0.02 as noise, so the defensible claim is that a model trained in two days matched the best external audio model we tested.

VG-Decision: a model built by agents

Vikas spent the first day setting context for an architect agent, which then split the work across parallel agents. The result is a compact open-weight model of about 4B parameters with a decision head, trained on a single H100 in two stages:

1. Generalist stage: about 28,500 labelled items from seven public datasets.
2. Our calls: 20,437 call turns, text only, with silver labels from a model ensemble, mixed 50/50 with the public data. The last version added class balancing.

It runs text-only at 0.05 s per turn, or with the lead's last 30 s of audio at 0.10 s. The workflow shipped three versions and six scored runs in about 13 hours, moving from 0.46 to 0.61 overall.

Results: quality against speed

On text questions our models hold the fast end of the frontier, and on audio questions they hold all of it. Both frontier plots are in the TL;DR.

On text, VG-Decision v2 (0.671 at 0.05 s) and NITI-v2 (0.704 at 0.10 s) are on the frontier. Gemini 3.8 Flash is the only model ahead of NITI, at 0.752, and takes 8.0 s.

On audio, the frontier runs from NITI-v1 (0.431 at 0.10 s) to PARAM-v2 (0.462 at 1.08 s). Gemini 4 Argon scores 0.459 in 10.8 s, so PARAM's place at the top rests on a 0.003 lead that is within noise. Every point on the audio frontier still comes from a model trained in under two days.

Team-led entries beat the agent-built one on every headline score: NITI over VG-Decision overall (0.645 against 0.614) and on text (0.704 against 0.671), and PARAM over VG-Decision on audio (0.462 against 0.421).

The latency gap is one to two orders of magnitude, but treat the exact ratio with care. Our times are self-reported on our own GPUs. External times include network and provider queueing.

What we learned about audio

Running Gemini 3.8 Flash three ways on the same turns (reading the transcript, hearing the audio, and both) gave us a controlled view of what audio adds and what it costs.

1. Listening adds signal the transcript lacks

On questions labelled by ear, the same model scored higher hearing the call than reading it.

The largest gains are on who is speaking and what surrounds them. A decision model needs its own access to the audio, beyond the words our speech recogniser outputs. PARAM's audio-first model is our starting point.

2. Adding audio to the transcript lowered the overall score

Given both inputs, Gemini 3.8 Flash scored 0.663 overall, below its own text-only 0.677. Adding audio helped where both signals carry information and hurt on judgements about content.

Our read is that a combined model should train on a mix of text-only and audio-and-text turns, so it learns which channel to trust per question. That is why PARAM added text midway.

3. Full-call audio context is the expensive part

Sending the whole call's audio with every turn doubled Gemini 3.8 Flash's answer time from 8.0 s to 16.2 s, and it still scored below reading the transcript. NITI answers the same turns in 0.10 s.

To stay inside a sub-second reply, the practical design is the current turn's audio plus the text history, or caching what the model has already heard.

Limitations

These results support a direction for our decision-model work. They do not yet support a ranking claim against external models.

• Final runs were not blind. We released the gold answers on day two so teams could do error analysis. Every best run on the leaderboard came after that release, so some of the first-to-best gain may be fitting to the test set.
• External baselines are zero-shot and untuned. No examples, no JSON mode, default temperature, one run each. Tuned prompts would likely score higher.
• Single runs. Per-question differences under about 0.02 are within noise. That covers PARAM's 0.003 lead on audio.
• Latency is not like for like. Team times are self-reported on our GPUs. External times include network round trips and provider queueing.
• Half the gold is AI-labelled. 51% of gold answers come from two agreeing AI passes, used mainly for text questions. We audited a share by hand, but text-reading models may still benefit from labels produced by reading.
• Audio is understated by design. Most text-question gold was labelled from the transcript. The cleanest audio evidence is the eight by-ear questions.
• One benchmark, one company's traffic. 586 units from 368 calls across our live campaigns, in Hindi, Hinglish and regional languages on 8 kHz lines. Results may not carry over to other languages, channels or call types.
• No cost comparison. We report latency and treat it as a partial proxy for cost. It is not a direct measure: API pricing is set per token by providers, and self-hosted cost depends on throughput and GPU utilisation as well as per-turn latency.

What comes next

The benchmark is now the standing test for every future decision model: a model ships only when it beats what runs in production on the same real calls.

Changes for the next round

• Keep a final split fully hidden. Release one split for error analysis and hold another back to decide the ranking.
• Label every audio question by ear. Any question meant to need listening gets gold from an annotator who listened.
• Add conflict items. Turns where the words and the voice disagree ("theek hai", fine, said in an irritated voice) are the only clean test of whether a model is listening.
• Measure recovery from transcription errors. Annotators flagged about 11% of turns as having major transcript errors. Labelling those by ear would show whether an audio model recovers what the transcript lost.
• Score turn-taking causally. Give a model audio only up to a moment, then score both what it decides and when it commits, against timestamps measured from the two channels.

From one benchmark to five

We plan to grow the same stereo calls into a full AudioLLM benchmark by adding label layers, without collecting new datasets. Each stage asks a harder question than the last.

Some stages label the calls once and score any model against the same labels. Others reuse the calls as prompts and need fresh ratings or live runs for each model.

The next version of each decision model is already training.