What’s new at Roark

Alerts got a full rebuild around a first-principles model: WHEN [trigger] THEN [action]. Metric thresholds and platform events both flow through the same unified dialog, and every fire is now traceable end to end.

What's new:

• Event triggers: subscribe an alert to the full platform event catalog (Calls, Evaluations, Simulations, Metrics, Chats, Issues) with human labels and multi-select.
• Webhook actions: add one or more HTTPS URLs to any threshold or event alert. Deliveries go through the same pipeline as customer webhooks, with SSRF protection.
• Alert detail page: every alert now has its own page with config, a "Recent fires" timeline showing per-action outcomes (Slack N/M, webhook status, issue opened), and a "Recent deliveries" list with expandable payloads and responses, filterable by status.
• Slack channel picker on trend alerts, plus batched notification-channel loading so the alerts list stays fast.

If you've attached a knowledge base to your project, you can now measure whether your agent is actually using it correctly, with no per-metric setup and no hand-built variant. Add the Knowledge Base Grounding package to any run, policy, or dashboard, and you get three metrics from one LLM pass.

The three metrics:

• Knowledge Base Grounding (1 to 5): how faithful the agent's factual claims and offers are to your KB. Ships with a >= 3 system threshold.
• Ungrounded Claim (per turn): every agent turn that stated or offered something the KB doesn't support, classified as CONTRADICTS (the KB says otherwise) vs NOT_IN_KB (the KB is silent).
• Missed Answer (per turn): the inverse. Flags when the caller asked something the KB could answer but the agent deflected, said "I don't know", or answered wrongly.

Scope it at run time: in Studio Evaluate mode, pick specific KB documents to ground against, or leave it on "All knowledge bases" to auto-source every READY KB in the project.

The old Review area is now Annotations, and it's grown into a real collaborative workspace for grading calls with your team. Reviewers work from the same queue, label moments straight off the transcript, and disagreements are surfaced instead of buried.

What's new in Annotations:

• Label a moment straight from the transcript. Highlight a turn, tag it against a metric, add a quote, and it lands in the annotation with a jump link back to the exact segment.
• Per-metric rubrics for review sessions. Each metric brings its own scoring guide into the session, so reviewers see the same instructions inline instead of a shared doc nobody reads.
• Inter-annotator agreement across reviewers on the same call, so you can see where your team disagrees before you argue about model accuracy.
• Reviews hub on Library with live auto-score job progress, jumpable review queues, and a lede telling you what needs attention next.
• Alignment detail on follow-ups so a second reviewer coming into a call sees the previous verdicts, the quotes behind them, and where they agreed or split.

Labels aren't just for reporting anymore. When you correct a metric in Studio, that correction becomes ground truth, and your owned LLM metrics fine-tune themselves against it so the next call scores the way you would.

How it works:

• Correct a metric inline. On the call details page and in Studio's EVALUATE view, click a metric result to change the value or verdict. That correction is saved as ground truth against that call.
• Ground-truth panel in Studio shows every example you've labeled for a metric in one place, so you can see the shape of what you've taught it.
• Auto-tune once you have five examples. Owned LLM metrics with five or more ground-truth labels re-score using your examples as few-shots. If agreement with your labels improves, the tuned version sticks. If it regresses, we revert and tell you why.
• Discovery, labeling, and tuning on one surface. The Label calls action opens the picker on the metric card itself, so picking calls, scoring them, and watching alignment move all happen without leaving the metric you're working on.
• Owned LLM metrics only. System metrics and non-LLM metrics don't offer the tune action, so there's no ambiguity about what's actually adjustable.

We shipped three system metric packages that grade parts of a call you couldn't easily grade before, plus a 1-5 Transcription Score that rolls the accuracy family up into a single verdict.

What each package does:

• Property verification: compares the call properties you send us (account IDs, names, dates, CRM context) against what was actually said on the call, then flags mismatches anchored to the turn where they diverged. Nicknames, partial references, and phonetic near-misses are normalized before comparison, so it doesn't fire on "Robert vs Bob".
• Call screening: detects when a receptionist or automated screen picks up an outbound call instead of the person being called, then scores the agent's handling of that interrogation on a 1-5 scale. Ships with an optional brief for what the agent should say when asked who's calling and why.
• Transcription accuracy: grades your customer-supplied realtime transcript against ours, adjudicates every disagreement with an LLM (so we don't flag a case where our post-call transcript was the wrong one), and reports the specific words the transcriber got wrong, split into critical (numbers, names, negations, dates) and non-critical errors.
• Transcription Score: a 1-5 rollup over the accuracy metrics with critical errors as a ceiling, not a weighted term. A call with 0.98 accuracy that got a callback number wrong still can't read as Excellent.

Voicemail simulations no longer live as nodes inside a scripted flow. They're a first-class run template now: pick the Voicemail testing template, tick which greetings you want to test, run.

Why we moved it:

• Eleven greetings out of the box covering the major US carriers and generic voicemail patterns. Multi-select as many as you want per run.
• The old flow-node path silently ran the default carrier greeting regardless of what you configured, on every voicemail step. That's fixed under the new template.
• Existing flows keep working if you have voicemail nodes in production, but new voicemail runs should come through the template.
• Variable interpolation in the expected-message script so a single template can grade dozens of variants.

👉 Read the guide

Two things customers asked for by name: configurable retention on your call data, and a real delete for agents.

What's new:

• Retention windows per project (60 days, 3 months, 6 months, 1 year, or indefinite) in a new Data controls settings tab. Default is indefinite, so nothing changes on your projects until you opt in.
• Soft or hard delete mode. Soft hides the calls, metrics, and traces from the dashboard and is recoverable. Hard permanently purges the underlying records after a 7-day grace period.
• Delete an agent from the platform UI or via DELETE /v1/agent/:agentId. Soft-deleted agents stop accepting new calls immediately across LiveKit and Pipecat, so a deleted agent can't quietly resurrect on the next inbound.

You can now score how human vs robotic your agent actually sounds. Voice Human-Likeness is a new 1-5 metric that listens to the agent's own audio (not the transcript) and judges acoustic delivery against a fixed rubric.

What it gives you:

• Per-agent scores sampled evenly across the call, so scripted monologues never dominate the read.
• Catches smooth robots: TTS voices that pass classic MOS checks but still sound robotic to a real listener.
• Pairs with Voice Naturalness, also new this week. Naturalness (UTMOS-based) grades audio signal quality; Human-Likeness grades whether the delivery reads as human. Together they separate "sounds bad" from "sounds fake."

Frustration and user-effort scores are now folded into a single Customer Reception number. Higher is better, banded 1-5, and it lands on every quality-analysis run without adding another LLM call. It shows up in your metric library automatically, no configuration needed.

Studio's metric page now has a Thresholds section, so you can list, add, rename, archive, and unarchive pass/fail thresholds in one place.

Also new:

• Widget deep-links. "Metric details" on a dashboard threshold widget now jumps straight to the source metric and highlights the specific threshold you clicked from.
• Safe renames. Editing a threshold's name or Pass/Fail labels is never retroactive. Past conversations keep the labels they were graded against.

Studio Evaluate used to expand every picked metric's threshold editor inline, which made a suite of ten metrics feel like fifty. It now renders as a dense checklist where each metric shows a single chip: either the attached threshold or "Set threshold" to add one.

Two side-effects worth calling out:

• Threshold columns appear in the results matrix immediately, no more reload to see them.
• Progress reflects what actually ran, so the bar no longer skews when thresholds are attached.

Environment and Persona background-noise settings now play on calls placed over LiveKit, Telnyx, Daily, and Small WebRTC. If you've configured ambient office chatter or a call-center hum for a persona, it now lands on every transport we support. Loudness is also normalized to a fixed reference level, so a volume: 0.1 setting gives a predictable perceptual result regardless of the source asset.

The call audio player now surfaces where the interesting things happened, not just the transcript. A sentiment ribbon runs under the diarization tracks, and a moments lane above it marks interruptions, agent cutoffs, and any boolean or offset metric that fired during the call.

How it works:

• Hover a marker to see the metric name, value, reasoning, timestamp, and participant.
• Click a marker to seek the audio to that instant and highlight the matching transcript line.
• Toggle legend chips to filter the marker categories you care about.
• Any new segment or turn metric shows up automatically based on its output type. Sentiment paints the ribbon, booleans and offsets drop markers, whole-call scalars stay in the metrics tab.

Every issue in Roark now carries a remediation channel so you don't waste an LLM call generating prompt suggestions for a latency regression. A latency issue points at INFRA or LLM_MODEL, interruption issues point at VOICE_CONFIG, mishearing points at STT, and so on. The suggested-fix panel only auto-runs prompt analysis when PROMPT is actually in the mix; otherwise you get diagnosis-flavored guidance and a link to the right place to fix it.

Opened notifications also went source-agnostic. Trend alerts used to be the only source that pushed a webhook when an issue opened. Now AGENT, MANUAL, API, TRACE_FAILURE, and SINGLE_CALL_ALERT issues all fire the same issue.opened webhook. Existing subscriptions pick this up with no changes on your side.

Also new:

• Sidebar badge with a live count of open and acked issues, hidden at zero.
• Dashboard header chip showing the newest open issue title, or a quiet all-clear.
• "Fix via" chip set on every issue row so you can triage at a glance.

A new 1 to 5 LLM-judged metric in the Conversation Quality package that grades how naturally a call flowed: turn-taking, pacing, responsiveness, and the absence of awkward silences, robotic repetition, or talking over one another. Bands run from Very Choppy to Seamless, with a built-in threshold that fails the two Choppy bands so you can plug it into monitors immediately.

It is transcript-based (it measures conversational naturalness, not audio fidelity) and opt-in per policy, dashboard, or run plan. Add it to a call and you get a holistic "did this flow well?" score in one number, on top of the raw response-time and interruption signals you already have.

A few upgrades that make simulation runs faster to iterate on and more faithful to how your production flows are actually wired.

What's new:

• Preceded-by chained flows. An improv flow can now run a linked scripted flow first at runtime, and the create-run dialog prompts for the preceding flow's variables so the whole chain resolves without hardcoded values.
• Sub-hourly schedules. Run a plan every 15, 30, or 45 minutes (or any custom minute value down to 15). The old floor was hourly.

Roark now answers the simplest question about your voice agent: is it answering right now? A new Health checks page in the Monitor section runs a fixed liveness probe on whatever cadence you pick and renders every monitor as a status-page card.

What you get:

• Status at a glance: Operational / Down / Paused per monitor, a 24-cell probe tick bar, and 30-day uptime pulled from a dedicated ClickHouse fact table (a probe that never connects registers as an outage instead of quietly disappearing).
• Overview widget on the page: 30-day hero plus 7-day and today, with a daily status strip that draws from the same projection the cards do. They can't disagree.
• Uptime on dashboards: a new UPTIME report source auto-provisions an uptime widget on your default dashboard the first time you create a health check.
• Sharper liveness check: the probe's headline is now agent_spoke (did the agent produce any speech at all?) instead of the old who-spoke-last heuristic that false-alarmed on healthy calls where the caller hung up last.
• Cost transparency: the create dialog spells out probe volume for the cadence you pick, so nothing surprising lands on your bill.

Recurring behavioral failures across your production calls now get grouped into Sentry-style issues automatically, no alert configuration required.

How it works:

• Always-on clustering: a scheduled sweep reads the last 24 hours of failing checks and numeric system signals (frustration, sentiment, comprehension failures, conversation loops), asks a model to group them into failure modes, and opens or updates an issue per mode with stable fingerprints and per-call evidence.
• Recent view by default: the issues page now leads with active plus recently-resolved so the trend issues that auto-resolve in an hour aren't hidden the moment you check.
• Issues banner on call detail: when a call is cited as evidence inside an active issue, you see it right there with one click into the full picture.
• Five new default watchers seed on every project (agent going silent, high frustration, negative sentiment trend, comprehension failures, loops), with a one-time backfill for existing projects that never got seeded.

After a simulation run settles, Roark reads the failing checks, the worst calls, and the current agent prompt, then proposes targeted edits with anchors and evidence instead of hand-waving a rewrite. Fix it, re-run the suite, get proof it moved.

What landed:

• Evidence-grounded findings: every suggestion cites the checks and conversations it came from. Anything the model invents is dropped before it reaches your screen.
• Verify loop: focus the analysis on a single failing check, re-run the same suite, and see the movement quantified check by check in the comparison view.
• Prompt versioning: every prompt now carries immutable version history, each conversation records exactly which version ran it, and the optimizer notices when your prompt has drifted since the last analysis.
• Suggestions on issues too: expanding an AI-detected issue runs the same analysis against the calls in evidence and shows the recommended prompt fix inline, so you can go from "here's the pattern" to "here's the edit" without leaving the page.

Ask the in-app assistant to "track how often callers ask about pricing" and it does the whole chain: finds or creates the metric, starts a collector so it runs on every new call, and drops a chart on your dashboard.

How it behaves:

• One confirmation, whole pipeline: metric → collector → report → optional backfill of history, with Roarky narrating what it picked.
• Sensible defaults: auto-places new charts on your default dashboard when you don't name one.
• Honest about empty state: a freshly-charted metric reads empty until data collects. Roarky says so and offers to backfill the history.

Scripted flows now have a branching mode toggle in the editor header, so one flow can either fan out into a simulation per path or collapse into a single call the agent adapts across.

What's new:

• DETERMINISTIC runs one simulation per unique path (the existing behavior).
• ADAPTIVE collapses a flow's own paths into a single call, and links to sub-flows follow whichever path the agent-under-test takes.
• Persona coverage stays orthogonal: five paths across three personas produces three adaptive calls, not one, so you still get real persona variation.
• Simulation tab on call detail: every simulated call now surfaces the flow it ran (adaptive brief or scripted steps), the persona, and the agent endpoint, with deep links out to each.

We rebuilt the entire platform from the ground up around how you actually use it. Metrics are the center of gravity now, the AI agent is one keystroke away from any page, and the navigation, layout, and defaults all reflect a year of customer feedback.

What's different:

• Simulations, rebuilt: a ground-up rebuild of how you test agents, now driven by Customer Flows with improv or scripted callers and ready-made templates (full details below)
• Metric-first: every list, view, and detail page leads with the metrics that matter to you, not raw data you have to interpret yourself
• Agent woven in: ⌘K opens an assistant that knows your project, your data, and your APIs
• Saved views travel with you: column layouts, filters, and sorts persist per user, with inline Save and Discard when you make edits
• Calmer, faster surfaces: switching projects no longer reloads the world, and the chrome got out of the way

This is the foundation we're building on for the rest of the year. Everything below shipped on top of it.

We rebuilt Simulations from the ground up for the new platform. Testing your agents against realistic callers is faster to set up, clearer to reason about, and now built around a new idea: Customer Flows.

What's new:

• Customer Flows replace Scenarios: every simulated caller now runs on a Flow, the direct successor to Scenarios from the old app, with a cleaner model for how a call should unfold
• Improv or Scripted: run an Improv flow where the simulated customer reacts naturally to whatever your agent says, or a Scripted flow where the call follows the exact turns you lay out. Choose per flow depending on whether you're stress-testing open-ended behavior or checking a known path
• Templates to start fast: kick off from a library of ready-made flow templates instead of a blank page, so a useful simulation is minutes away

If you ran Scenarios in the old app, Customer Flows are where they live now, with the improv and scripted split layered on top.

You can now ask questions directly inside the platform and get answers pulled from your own data, without leaving the page you're on.

What you can do:

• Ask in plain English: "show me yesterday's failed calls" or "list my datasets" works out of the box, with the assistant calling the right API behind the scenes
• Pinned to the right rail: hit ⌘K from anywhere, drag the edge to resize, the page stays visible and clickable underneath
• Conversation history: every chat is saved, click History to jump back into a past thread
• Streaming responses: tool calls and answers stream live so you see progress in real time

Read-only for now (lists, lookups, filters). Write actions land next.

You can now upload your own reference material into Roark and attach it to the metrics that need it. Give a judge metric the actual rubric, give a compliance metric the actual policy doc, and let the metric score against your source of truth instead of a prompt approximation of it.

Three ways to bring content in:

• File: upload a PDF, .txt, or .md, PDFs get text-extracted automatically
• Text: paste markdown or free-form text directly
• JSON: paste structured data (taxonomies, decision trees, labeled examples)

Find Knowledge Bases under Configure, then attach one or many to a custom metric from the new Reference Material section in Studio. At eval time we transparently prepend the attached documents to the LLM as ground truth, then your prompt runs against them, no prompt rewriting required.

You can now choose exactly which PII categories to redact from your calls, applied to both the audio and the transcript. Pick what's sensitive in your domain, leave everything else untouched.

What you can redact:

• Financial: credit card numbers, bank accounts, routing numbers
• Government IDs: social security numbers, driver's license, passport
• Contact: phone numbers, email addresses, physical addresses
• Identity: names, dates of birth

Audio gets bleeped in place of the redacted span, transcripts get a [REDACTED:CATEGORY] marker, and the original never leaves the redaction step once it's enabled on a project.

A new system metric that flags when a caller is in a vehicle, both at the segment level and rolled up to the call. Useful for quality scoring, routing decisions, or just understanding the audio conditions your agents are working through.

What you get:

• Boolean + confidence per segment: see exactly when in the call the in-car signal appeared
• Call-level rollup: answers "did this happen at any point in the call", not diluted by quiet stretches
• Powered by a zero-shot audio classifier running on the call recording, no agent-side instrumentation required
• Lives in the new Call Environment package so future ambient-audio signals (hold music, background noise, road noise) slot in next to it

Off by default, enable it via a metric policy on the projects where it matters.

You can now create Formula metrics that combine your existing metrics into composite scores and rules — no code required.

What you can do:

• Weighted scores — (Empathy * 0.4) + (Clarity * 0.3) + (Resolution * 0.3)
• Pass/fail gates — Compliance AND Greeting
• Custom benchmarks — (CSAT + NPS) / 2
• Comparisons — Sentiment == "Positive" AND Empathy > 3

How it works:

1. Create a new metric in your Metric Library and select the Formula calc type
2. Build your formula using the inline builder — start typing to search and insert metrics
3. Formulas are evaluated automatically during call analysis

Under the hood:

• Dependency-aware evaluation — Source metrics are always computed before formulas that reference them
• Deletion protection — Metrics used in formulas cannot be deleted until the formula is updated
• Cycle detection — Circular dependencies are caught at creation time
• Type safety — Math operators only accept numeric metrics; logical operators only accept boolean and classification metrics

A new analysis package that identifies English accents per participant across every segment of a call using ML-based classification.

Metric	Type	What it measures
Accent	Classification	Detected accent per segment and dominant accent at call level, with full probability distribution
Accent Stability	Numeric (0–1)	How consistent the detected accent is across segments

Highlights:

• Per-segment probability distributions — See the full accent breakdown per segment, not just the top-1 prediction
• Stacked probability chart — Visualize accent probabilities over time in the segment view
• 16 English accent variants — American, British, Australian, Canadian, Indian, Irish, Scottish, Welsh, and more
• Threshold support — Set a threshold on Accent Stability to flag calls where the agent's TTS accent drifted

👉 Recipe: Accent Detection & TTS Drift Monitoring

A new analysis package that evaluates whether your AI agents comply with regulatory requirements, safety boundaries, and organizational policies — across healthcare, finance, and legal verticals.

9 compliance metrics out of the box:

Metric	Type	What it measures
Regulatory Adherence	Scale (1–5)	Compliance with industry-specific regulations (HIPAA, PCI-DSS, GDPR, etc.)
Consent & Disclosure	Boolean	Whether the agent obtained required consent and provided necessary disclosures
Prompt Injection Resistance	Boolean	Whether the agent resisted manipulation attempts to override its instructions
Identity Consistency	Boolean	Whether the agent maintained its assigned identity throughout the call
Hallucination Boundary	Scale (1–5)	Whether the agent avoided fabricating information and deferred when unsure
Unauthorized Commitment	Boolean	Whether the agent made promises or commitments outside its authority
Sensitive Data Handling	Scale (1–5)	Whether the agent properly handled PII, PHI, and financial data
Escalation Protocol	Boolean	Whether the agent correctly escalated when required by policy
Scope Adherence	Scale (1–5)	Whether the agent stayed within its defined role and topic boundaries

Key features:

• Segment-level findings — For 5 metrics (prompt injection, identity, unauthorized commitment, escalation, consent), results include the specific agent statements where issues were detected
• Customizable prompts — Every metric accepts optional additional evaluation criteria so you can tailor compliance checks to your organization's specific policies
• Works with policies — Add compliance metrics to metric policies to automatically evaluate every production call

Also in this update:

• Multi-select metric picker — The metric selector now stays open for multi-select with checkboxes, and supports "Select all" at the package level
• View-only metric settings — System metric output configuration (boolean labels, scale ranges) is now visible in the metric library in a read-only mode
• Optional/Required prompt labels — Metric settings now clearly indicate whether the LLM prompt is optional or required

You can now send OpenTelemetry traces to Roark and see exactly what happens inside every turn of your voice AI agent — every STT transcription, every LLM generation, every TTS synthesis, every tool call — with full timing, hierarchy, and context.

Zero-config for Vapi. One function call for LiveKit. Works with anything.

• Vapi — If you have a Vapi integration, traces are collected automatically. No code changes, no exporters to configure. Just make sure Public Logs are enabled in your Vapi dashboard and traces will appear alongside your calls.
• LiveKit — Add a single configure_roark_tracing() call before your agent starts and every span — STT, LLM, TTS, tool calls — flows into Roark automatically.
• Custom / Any platform — Point any OpenTelemetry OTLP HTTP exporter at https://api.roark.ai/v1/otel/v1/traces with your API key. We support TypeScript, Python, Go, and any language with an OTel SDK.

What you get:

• Full turn-by-turn visibility — See exactly how STT, LLM, and TTS are used in each agent turn with span timings and hierarchy
• Latency debugging — Instantly spot slow LLM responses, TTS bottlenecks, or tool call delays
• Tool call inspection — See which tools were invoked, what arguments were passed, and how long they took
• Correlated with your calls — Traces appear on the Tracing tab of every call detail page, right next to transcripts and metrics
• Project-level trace explorer — Browse and search all traces from Observability → Traces

Roark acts as a full OTEL Collector — just send your traces and we handle ingestion, storage, and visualization.

👉 Learn more