How we measure

Every stack answers real phone calls placed by Roark’s simulation engine, the same engine our customers use, over the public phone network. The agent is the same in every stack: one prompt, one set of mocked tools, one scenario. So is the caller: a Roark persona following a fixed script, an impatient caller who interrupts and corrects. Only the models behind the agent change, so a difference between two stacks comes from their models.

Every call is recorded. Judge models read each transcript for the outcome and the conversation, and the recording gives the timings.

A full sweep takes each industry’s call 5 times with a cascade, once from each of the five callers, and 20 times with a speech-to-speech model, four times from each caller. A cascade’s figures also draw on its parts, each heard in dozens of other stacks; a speech-to-speech model has only its own calls, so it takes more of them.

Roark is on both sides of every call. It plays the caller, so it keeps the exact text its caller said on each call, including every name, date and number it gave. It runs the agent under test, so it sees what that agent’s transcriber heard and what its language model wrote for its voice to say. And it records the line, so it has the audio the voice produced. The transcript and voice figures come from lining those up after both sides are normalised: a transcriber is measured by what it heard against what the caller said, a voice by what Roark’s own transcriber hears in its audio against what the model wrote. Nobody transcribes these calls by hand; the caller speaks live, and its words differ from call to call.

The transcriber’s ranked figure is its semantic word error rate, as defined in Pipecat’s STT benchmark: the caller’s words and the transcript are lined up, a difference inside a code or number is decided by its characters, and a judge keeps only the other differences an agent would respond differently to. A plural, a filler or a split word is forgiven; a changed name, number, code, date or word counts. The plain word error rate, every difference counted and no judge involved, is shown beside it as the audit trail and is not scored. Three things differ from that benchmark, so the two sets of figures are not interchangeable: the reference here is the caller’s exact script, not a transcript of clean audio; the alignment is Roark’s own and the judge may be a different model; and the calls are live telephony with noise, accents and interruptions, not curated clips.

Each figure has a known margin. The caller’s words are spoken by a synthetic voice, so a word it mispronounced counts against the transcriber; the agent’s audio is heard by Roark’s transcriber, so a word it misheard counts against the voice. Every transcriber hears the same caller voice under the same conditions, and every voice is heard by the same transcriber, so the figures compare models on the same calls through the same voice and the same ear. That does not make those slips cancel: a voice can mispronounce a word one transcriber copes with and another does not, and a transcriber can mishear one voice and not another, so a model’s figure carries them. On top of that sits the voice’s own variation from one call to the next, which the volume of calls averages out: each transcriber is measured across about seventy stacks and over a thousand calls. The rates will not match the figures a vendor publishes on a curated test set; what holds is the comparison between models, which faced the same calls, the same references and the same rules.

Transcribers, language models and voices are run in every combination with each other, beside the speech-to-speech models. A model’s own figures pool every stack it runs in, adjusted for its partners in each one, so every model is compared as if it ran with the same partners. A model is ranked ahead of another only where it is ahead stack for stack, on the stacks they share; leads inside that noise are called a tie.

That measures a model where it actually works: inside a running agent, on live phone audio, through interruptions, corrections and real turn-taking, with hundreds of calls behind each figure. A test that feeds a model clean audio files or one prompt at a time never sees any of that.

Each section ranks on one score. Every figure scores 0 to 100 between two published points: full marks where doing better makes no difference to the caller, zero where the call breaks down. Each then counts at its weight. The weights follow one order in every section: whether the caller got what they rang for, then whether the agent got it right, then speed.

A stack, cascade or speech-to-speech, is scored on the outcome and the reply first, then on what each of its components got right on its own calls: the words its transcriber got wrong in a way that changes the meaning, whether its voice read codes and amounts back right, how human it sounded, and whether its tools were handled. A component that leads its section lifts every stack that carries it, by the same recipe for every stack. A figure a stack cannot carry, such as a semantic word error rate for a speech-to-speech model with no transcript, is credited at the field median, so the stack is neither flattered nor punished for it.

On a full sweep each cascade's figure is 25% what its own calls measured and 75% what every other stack shows about its transcriber, language model and voice, fitted across the whole sweep. The split is the same for every stack, so a lucky run does not lead and a stack's score does not move with how noisy its calls were.

That order comes from real-world voice agent calls, where getting the conversation right predicts how a call ends more strongly than speed does. The latency points come from the same calls: replying faster than about 2 seconds made no measurable difference, and calls went measurably worse once the agent took more than about 3 seconds to reply. The weights themselves are our judgement, published so they can be checked and argued with.

Task completion scores between 80% and 100% because that is where the field sits, 88% to 95% on this suite, and because the difference is real: in the benchmark’s own judged calls, a call with a misheard turn is resolved 13 points less often than one without, about the same cost as a reply in the slower half of calls rather than the faster. A few points of completion have to count for as much as they do on the line.

A board weighs the figures its sweep collected, with the weights above rescaled over them in the same proportions. A figure outside the recipe, such as barge-in handling, is shown but not scored.

Price is a switch on the score, not part of it. Switched on, the section’s cost takes 25% of the score and every other weight keeps its proportion. Prices span a hundredfold, so price is scored on a log scale, where each doubling costs the same number of points:

• Stacks and speech-to-speech: cost per task, full marks at $0.05, zero at $5.00.
• Language model: cost per call, full marks at $0.020, zero at $2.000.
• Speech-to-Text: cost per call, full marks at $0.005, zero at $0.500.
• Text-to-Speech: cost per call, full marks at $0.020, zero at $2.000.

A cascade rests on its own 20 or so calls, so one call moves a rate by several points. Every transcriber, language model and voice, though, runs in dozens of stacks, so what each one adds to a figure is known far more precisely. On a full sweep a cascade’s figure is therefore 25% what its own calls measured and 75% what every other stack shows about its three parts, fitted across the whole sweep. The split is the same for every stack: a lucky run does not lead, and a stack’s score does not move with how noisy its calls were. A speech-to-speech model has no parts to borrow from, so it is scored on its own 80 calls. Every figure shown on its own is the raw measurement.

In one industry a stack has only a handful of calls, too few to say how its pairing does. There a cascade’s figure is what its transcriber, language model and voice show across every stack in that industry’s calls, a few hundred calls behind each part, and its figures shown on their own are that prediction too. A speech-to-speech model’s figure is its own calls in the industry, so its margin is far wider. How a particular pairing beats or misses its parts shows only on the main board.

Each score carries a 95% margin of error, built from its figures’ intervals. A rank is the best place a stack could hold once every margin is allowed for: one more than the stacks whose whole interval sits above its own. Stacks the calls cannot tell apart share a rank, listed by the low end of their margin, and the worst place a stack could hold is a hover away. A stack with a figure that published no interval shows no margin and no rank spread, and counts for nobody else’s, so nothing is claimed about it.

In an industry, two cascades are told apart on the difference between them. Their scores come from the same fit, and two stacks that share a transcriber and a language model differ only by their voices, so the margin of that difference is far narrower than their two margins taken separately. Its margin comes from how far each cascade’s score on its own calls sits from what its parts predict. A pair that includes a speech-to-speech model is judged on their two margins.

The best overall pick, on the main board and in each industry, is the stack whose score is highest at the low end of its margin of error, so a few lucky calls cannot carry a stack past one whose score rests on hundreds. Stacks that share a rank are listed the same way, so the first row is the pick.

Numeric metrics report the median with a 95% interval; pass rates report a 95% Wilson score interval. N is the number of scored calls behind each figure.

A model isn’t an agent. Each target runs behind the same reference agent, differing only in the model behind it, with vendor-recommended settings. Every agent on the board runs on Pipecat with the same Smart Turn end-of-turn model, so turn timing is held equal across stacks. The exceptions are the models that detect turns themselves: the speech-to-speech models, whose bridge has no turn-taking logic of its own, and a transcriber run with its own endpointing, marked as such in its row. No vendor pays to be included or sees results before publication.

Every score links to the calls behind it: transcripts and audio. Publishing the audio is what makes the numbers checkable, and shareable.