This week at Roark

The public voice-agent leaderboard now runs every stack through four full industries, scores tool use alongside speech, and ranks on a weighted quality score with a price switch.

What changed:

• Four scenarios, five callers each. Reschedule a healthcare appointment, triage an urgent home repair, file an auto claim, and troubleshoot an internet outage. The same five callers (US quiet, Spanish-accented on a street, Indian English in a café, flustered Southern with a crying baby, fast and interrupting under construction) run every scenario, so industries and voices stay comparable.
• Tool use is a first-class figure. Every stack is graded on whether its agent actually called the right tools, including hallucinated calls and skipped ones. Boards for LLM and speech-to-speech stacks now show tool-call completion and tool hallucination.
• Voice consistency and same-speaker checks. We flag lines where an agent drifts off its usual voice or the wrong speaker lands on the agent track, and the board scores on both.
• Semantic word error rate for transcribers. The raw rate is still shown, but ranking uses a judged rate that forgives differences a listener would read the same way.
• Weighted Roark score with a price switch. Pick quality-only or price-adjusted, filter by industry or caller, and open any model or stack on its own page beside the board.

👉 Browse the live benchmarks

Run several simulation plans together and read them as one report. Keep the whole thing in config as code with scheduled delivery when the group finishes.

What's new:

• Run groups. Group existing runs, or start several plans at once from the dashboard. One combined report with an executive summary, numbered issues, strengths, metric aggregates, and a PDF export.
• Groups in config as code. kind: simulationGroup collects plans by name, schedules them on the same cadence block as plans, and announces each run to any mix of email recipients and up to 10 Slack channels (PDF in the thread).
• Schedule any plan from config. A schedule block on kind: simulationPlan runs it on cadence. The dashboard view goes read-only when the schedule is config-owned.
• Full REST surface. POST /v1/simulation/run-group to start a group, GET to read its full report document, regenerate on demand.

👉 See the simulation groups guide

A collector in kind: collector can now carry each metric's per-collector settings in the file, and you can re-score past calls using exactly those settings over the API.

What you can do:

• Pin settings in YAML. For each metric a collector runs, declare verifiedProperties, excludedPhrases, voicemailScript or screeningBrief inline. The file is the source of truth: an apply resets anything drifted in the app, and misspellings fail validation.
• Re-score past calls with a policy. POST /v1/metric/collection-jobs takes an optional policyId. The job runs each metric at the version and settings that collector runs it at live, so a re-collection against an updated Property Verification setup checks only the properties you now care about.
• Choose which properties a collector verifies. Property Verification's "Properties to verify" is now a first-class collector setting, visible everywhere the collector is edited.

👉 Read the collector config reference

Dashboards now take a filter that applies to every widget, plus 2 to 3 comparison groups (say, agent A vs agent B) that split every widget on the dashboard at once.

Highlights:

• Dashboard-level filters. Set them once, every widget respects them.
• Compare mode. Each group becomes a coloured series on every chart, sharing the same axis instead of sitting in separate cards. Setup is guided: the first filter opens group A, the next populates group B, and the popover closes when every group is filled.
• Lives in the URL. Share a link to a specific comparison without saving it onto the dashboard.

A batch of upgrades for custom metric authors.

• Reference a single property in a prompt. {{properties.partner_outcome}} resolves to that one value and shows up as its own line in Data Sources Used. Mix it with {{properties}} or use it on its own; the Studio picker now lists every custom property on the project.
• Grade per agent reply or per line. Custom LLM metrics can now score each agent response or each transcript line, and failed checks land on the exact turns the judge cited.
• Archive and restore. Hide a custom metric from the library, pickers, collectors and dashboards, and restore it later. The dialog previews everything the archive touches before you confirm.
• Longer prompt cap. Custom metric prompts now accept up to 20,000 characters.