Real-time Context Standard

The open standard for context passed between real-time models.

The Real-time Context Standard (RCS) defines how the models in a voice agent pass context to each other beyond plain text. A voice agent is a chain of models: speech-to-text transcribes the user, an LLM writes a reply, and text-to-speech speaks it. Between them they pass text, so what isn’t words is lost along the way, such as how the user sounded, who said what, whether they meant to take a turn or were only saying “mm-hmm”, and how the reply should be said. RCS carries that context in the same text stream, in formats any model in the chain can produce and read.

RCS 0.1 carries how the user sounded and how the agent should speak. Who is speaking and when someone means to speak are on the roadmap.

This page documents RCS 0.1, an alpha. Until 1.0, any version may change its formats, vocabularies and context types in ways that aren’t backward compatible. See Versions.

​

Design Principles

RCS context is written by an LLM as it streams a reply, and read by models that act on it a sentence at a time. Every token of markup is a token the LLM generates before the user hears it, so the formats trade a little expressiveness for as few tokens as they can:

• Few tokens. A tag is a short name in brackets, such as <<warm>> or [[laugh]]. It costs a few tokens where SSML or other XML markup costs several times as many.
• No closing tags across sentences. A delivery tag lasts until the next one, and a span closes within its sentence. Each sentence can be spoken as soon as it’s written, without waiting for the markup to close.
• Safe to cut off. A reply can stop anywhere, at its end or when the user interrupts. What’s left of an unfinished tag is removed, so nothing malformed is spoken.
• Readable inline. Context travels in the text the models already pass each other. The markup stays legible to an LLM that reads it back in later turns.

​

Conformance

MUST, SHOULD and MAY are used as defined in RFC 2119.

A model, with any processor beside it, conforms to RCS as a producer or a consumer of a context type when it meets that type’s requirements. It doesn’t need to support every context type, or both roles.

The grammars are ABNF, with prose in < >.

​

Terms

• Context model: a model in a voice agent that produces or consumes context, such as a speech-to-text (STT) model, an LLM, a text-to-speech (TTS) model, or a classifier.
• Context type: a kind of context RCS carries. Each context type defines its direction, its format, its vocabulary, and how a model declares support for it.
• Producer: the model, or a processor beside it, that writes context of a type.
• Consumer: the model, or a processor before it, that reads context of a type, or acts on it.
• Capabilities: what a producer reports or a consumer performs, declared before the conversation starts, so the models upstream only write what will be used.

​

How Context Flows

A context type travels inline, with the words it describes.

Capabilities are declared out of band. RCS defines the fields of each declaration, listed under each context type; how a declaration reaches the models that read it, and in what encoding, is up to the implementation.

​

Perception

How the user said what they said: their tone, how fast they spoke, and other signals read from the audio or the words.

• Direction: from an STT model, or a classifier model after it, to the LLM.
• Producers: any model that reads how a user turn was spoken.
• Consumers: the LLM, which reads it at the head of the user’s message.

​

Format

The reading comes first in the user’s message, followed by a space and the words:

([tone frustrated=0.86, neutral=0.10],[speech_rate value=4.1]) Can you tell me where my order is?

reading     = "(" signal *( "," signal ) ")"
signal      = "[" name SP ( labels / value ) "]"
labels      = label "=" confidence *( ", " label "=" confidence )
value       = "value=" number
name        = 1*name-char
label       = 1*name-char
name-char   = <any character except whitespace, "=", ",", "[", "]", "(" and ")">
confidence  = DIGIT "." 2DIGIT                      ; 0.00 to 1.00
number      = [ "-" ] 1*DIGIT [ "." 1*DIGIT ] [ "e" ( "-" / "+" ) 1*DIGIT ]

• A signal read as labels gives its likeliest labels, likeliest first, each with a confidence from 0 to 1. A confidence below 0.5 is a guess.
• A signal read as a value gives a measure, such as a rate or a level.
• value, and the e of an exponent, are lowercase. value is reserved: it isn’t a label.

​

Producer Requirements

• A producer MUST write a reading only at the head of the user’s message, followed by one space.
• A producer MUST write only the final reading of a turn. A turn with no reading MUST have no prefix.
• A producer MUST list a signal’s labels likeliest first.
• A producer MUST declare each signal it reports before the conversation starts, and MUST NOT report a signal it didn’t declare.

​

Consumer Requirements

• A consumer MUST read the reading as context about the user’s words, not as words the user said.
• A consumer MUST NOT write a reading in its own output.
• A consumer SHOULD NOT act on a label with a confidence below 0.5.
• A consumer MAY ignore a signal it doesn’t recognize.

​

Declaring Perception

A producer declares each signal it reports before the conversation starts, so the LLM can be told how to read them:

Field	Meaning
name	The signal’s name, as written in a reading, such as tone
description	A short line on how to read it, such as its unit or range, or empty
labels	The labels it can report, for a signal read as labels; empty for values

RCS 0.1 has no shared vocabulary of signals: a producer names its own.

​

Expression

How the agent should say its reply: the delivery of its words, and the sounds and pauses between them.

• Direction: from the LLM, or a classifier model after it, to the TTS model.
• Producers: an LLM taught the tags, or a classifier model that adds tags to an LLM’s plain reply.
• Consumers: the TTS model, and anything that follows the agent’s delivery, such as a client application or a real-time avatar.

​

Tags

Tag	Example	Description
<<delivery>>	<<warm>>	Sets the delivery of the words after it, until the next delivery tag or the end of the response. A delivery can be several words, such as an accent
<<neutral>>	<<neutral>>	Returns to the voice’s baseline, until the next delivery tag or the end of the response
<<delivery: words>>	<<slow: 4 7 2 9>>	A span: speaks just those words in that delivery, added to the delivery around them. <<neutral: words>> speaks them in the baseline. Spans don’t nest
^^words^^	^^did^^ ship	Stresses the words
[[sound]]	[[laugh]]	Makes a sound at that point
[[pause N]]	[[pause 600]]	Pauses for N milliseconds

<<sympathetic>> I'm so sorry about the wait. <<warm>> Good news, though: it ^^did^^ ship yesterday. [[breath]] Your tracking number is <<slow: 4 7 2 9>>.

​

Syntax

delivery-tag = "<<" name ">>"
span         = "<<" span-name ":" *WSP span-words ">>"
emphasis     = "^^" stressed "^^"
sound-tag    = "[[" name "]]"
pause-tag    = "[[pause" SP 1*DIGIT "]]"
name         = *( span-char / ":" )
span-name    = 1*span-char
span-char    = letter / DIGIT / "_" / " " / "'" / "-" / "/"
letter       = <any Unicode letter>
span-words   = 1*<any character except an angle bracket or a line break>
stressed     = 1*<any character except a line break>

• A tag doesn’t cross a line break.
• Where text matches more than one rule, a span takes precedence over a delivery tag and a pause over a sound tag. Stressed words end at the first ^^ after them.
• Names are case-sensitive, and universal names and pause are lowercase. Spaces at either end of a name are ignored.
• An empty <<>> returns to the baseline, as <<neutral>> does. An empty [[]] is left out.
• A span can hold stressed words, but not another span or tag. Stressed words can’t hold a tag.
• RCS 0.1 has no escape. Text that matches a tag is read as one. An unpaired ^^, or a << or [[ left open at the end of a response, is removed. Any other << or [[ that doesn’t start a tag is text.

​

Producer Requirements

• A producer SHOULD write only the deliveries and sounds the consumer declared, by their universal names where one exists.
• A producer SHOULD write a pause only within the consumer’s declared range, and stress only when the consumer declared emphasis.
• A producer MUST NOT write a tag inside a span or inside stressed words.
• A producer MUST write stress as ^^words^^, not as a delivery.

​

Consumer Requirements

• A consumer MUST NOT speak a tag. It MUST leave a tag it can’t perform out of the speech.
• A consumer MUST read a tag’s kind from its brackets, except that a universal name, or a name the consumer declared, keeps its kind in the other brackets: [[warm]] still sets a delivery, and <<laugh>> is still a sound.
• A consumer MUST remove an unfinished tag at the end of a response, or one cut off by an interruption, without speaking it.
• A consumer MUST perform each delivery and sound it declared when it’s written by its universal name, as well as by the consumer’s own name.

The agent SHOULD keep a reply the LLM wrote with tags in the conversation’s context as written, so later turns see how it was said.

​

Universal Names

Deliveries and sounds are named by their universal names, the same whatever the TTS model. A consumer translates each into its own name for it, such as a TTS model that performs warm as warmly.

• One spelling per word: an adjective for a delivery, such as nervous, and a base word for a sound, such as laugh.
• Different words stay distinct, so content and happy are both names.
• Only sounds a voice makes have names. Sound effects, such as applause, have none.
• Any other name passes through as written, for a consumer that takes free text, such as an accent.

Two names are reserved: neutral, the voice’s baseline, and emphasis, the delivery reported for stressed words.

Deliveries (118)

affectionate, agitated, alarmed, amazed, amused, angry, annoyed, anticipating, anxious, apologetic, appalled, ashamed, bored, calm, cautious, compassionate, confident, confused, contemplative, contemptuous, content, crying, curious, cute, deadpan, dejected, delighted, depressed, desperate, determined, disappointed, disdainful, disgusted, dismissive, distant, doubtful, dramatic, elated, embarrassed, empathetic, encouraging, enthusiastic, envious, euphoric, excited, fast, fearful, flirtatious, frustrated, grateful, guilty, happy, hesitant, hopeful, hurt, hysterical, impressed, indifferent, insecure, ironic, jealous, lonely, loud, mad, melancholic, mischievous, mocking, moved, muttering, mysterious, nervous, nostalgic, optimistic, outraged, panicked, peaceful, pessimistic, playful, professional, proud, questioning, reassuring, regretful, rejected, relaxed, relieved, resigned, sad, sarcastic, satisfied, scared, screaming, serene, serious, sheepish, shouting, sincere, singing, skeptical, slow, soft, somber, stern, surprised, sympathetic, thoughtful, threatened, tired, triumphant, trusting, uncertain, unhappy, upset, urgent, warm, whispering, wistful, worried

Sounds (46)

breath, burp, cackle, chuckle, clear throat, cough, cry, deep breath, exhale, gasp, giggle, groan, growl, grunt, gulp, hiccup, hiss, huff, hum, inhale, laugh, laugh harder, light breath, lip smack, moan, pant, scoff, sharp exhale, shriek, shush, sigh, sneeze, sniff, sniffle, snore, snort, sob, start laughing, swallow, tongue click, um, wheeze, whimper, whistle, woo, yawn

​

Reporting Delivery

A consumer that reports what it performed, such as to a client app, reports the delivery of each word as a list of universal names: the delivery in effect, a span’s delivery added to it, and emphasis for a stressed word. <<warm>> it ^^did^^ ship reports did as ["warm", "emphasis"]. An empty list is the voice’s baseline. Each sound and pause is reported on its own, where it is performed.

​

Declaring Expression

A TTS model declares what it performs before the conversation starts, so the producer is offered only those tags:

Field	Meaning
Deliveries	The deliveries it performs, by its own names
Sounds	The sounds it performs, by its own names
Free text	Whether it also takes deliveries, or sounds, described in a few words, declared for each kind on its own
Universal names	Its own name for each universal name, where they differ
Pauses	The shortest and longest pause it makes, or none
Emphasis	Whether it stresses words
Where delivery changes	Anywhere in a response, or only between sentences. A model that changes it only between sentences speaks a span’s whole sentence in the span’s delivery

​

Versions

Version	Changes
0.1	First version: perception and expression.

In 0.1, declarations don’t carry a version.

​

Implementations

• Pipecat 1.15.0 and later implement RCS 0.1 in expressive conversation.
• Pipecat clients receive reported delivery, sounds, pauses and perception readings over RTVI 2.3.0 and later. RCS doesn’t define how context reaches client apps.

​

Roadmap

Context types under consideration for later versions:

• Diarization: who is speaking, for conversations with more than one person.
• Intent: when someone means to speak, and whether what they said needs a response.

​

Contributing

RCS is an open standard, for any framework or model to produce and consume. To propose a change or a new context type, open an issue on GitHub.