> ## Documentation Index > Fetch the complete documentation index at: https://docs.senseaudio.cn/llms.txt > Use this file to discover all available pages before exploring further. # 端到端实时语音模型 > 基于 WebSocket 的实时语音对话协议参考 ## 说明 基于 WebSocket 的实时语音对话接口，支持客户端持续发送语音输入，服务端实时返回用户语音转写、模型文本回复与模型语音回复。文本帧均为 JSON，二进制帧用于传输 PCM 音频数据。 * **接入地址**：`wss://preview.api.senseaudio.cn/ws/v1/realtime/voice-dialog` * **鉴权方式**：Bearer Token，格式 `Authorization: Bearer SENSEAUDIO_API_KEY` * **客户端文本帧**：JSON 文本帧，必须包含 `type` 字段 * **客户端音频帧**：`pcm_s16le`、采样率 `16000Hz`、单声道；建议每 `40ms` 发送一帧（`1280` 字节） * **服务端音频帧**：`pcm_s16le`、采样率 `24000Hz`、单声道 * **计费单位**：按二进制帧传输的 PCM 音频总长度计费，不足 1 秒按 1 秒计 - `start` 必须作为连接后的第一个客户端消息发送，且同一连接内不能重复发送。 - 客户端必须等待服务端返回 `ready` 后，才能开始发送二进制音频帧。 - `update` 不能修改 `model` 和 `audio_setting`。 - `greeting` 非空时会产生独立的开场白轮次及其 `turn.done`。客户端不能把该事件误认为用户语音轮次结束。 ## 请求头 (Request Headers) | 参数名 | 必填 | 说明 | 示例 | | :---------------- | :- | :-------------------------------------- | :------------------ | | **Authorization** | 是 | 鉴权 Token。格式：Bearer SENSEAUDIO\_API\_KEY | Bearer sk-123456... | 本文完整示例面向 Node.js 和 Python 服务端程序。浏览器原生 `WebSocket` API 不能设置自定义 `Authorization` 请求头，也不应把长期 API Key 暴露在浏览器代码中；浏览器场景应通过受控后端代理或专用的短期鉴权机制接入。 ## 接入流程 ``` 1. 客户端建立 WebSocket 连接 ↓ 2. 客户端发送 start 事件，初始化模型与对话配置 ↓ 3. 服务端返回 ready 事件，表示模型连接建立完成 ↓ 4. greeting 非空时，服务端先返回开场白音频及对应的 turn.done ↓ 5. 客户端按实时节奏发送 PCM 二进制音频帧 ↓ 6. 客户端发送 commit 手动提交本轮音频，或由服务端自动判断语音结束 ↓ 7. 服务端返回 user.transcript.done、模型文本回复与模型语音 ↓ 8. 服务端返回 turn.done；模型回复期间客户端可发送 cancel 打断回复 ↓ 9. 客户端发送 end 结束会话，服务端以关闭码 1000 正常关闭连接 ``` ```mermaid theme={null} sequenceDiagram participant Client as 客户端 participant Server as 服务端 Client->>Server: WebSocket Connect (带 Auth Header) Client->>Server: JSON: {"type": "start", "model": "senseaudio-realtime-1.0", "greeting": ""} Server-->>Client: JSON: {"type": "ready", "session_id": "session_50173901836217346"} loop 连接期间每 40ms 一帧 Client->>Server: Binary PCM audio frame (1280 bytes) end Server-->>Client: JSON: {"type": "speech.started"} Server-->>Client: JSON: {"type": "user.transcript.delta", "text": "今天天气怎么样？"} Note over Client,Server: 服务端 VAD 自动判断本轮结束；麦克风帧持续上行 Server-->>Client: JSON: {"type": "user.transcript.done", "text": "今天天气怎么样？"} Server-->>Client: JSON: {"type": "assistant.text.delta", "text": "今天"} Server-->>Client: JSON: {"type": "assistant.audio.start", "response_id": "tts_3", "sample_rate": 24000, "format": "pcm_s16le"} Server-->>Client: Binary PCM audio frames Server-->>Client: JSON: {"type": "assistant.text.done", "text": "今天的天气……"} Server-->>Client: JSON: {"type": "assistant.audio.done", "response_id": "tts_3"} Server-->>Client: JSON: {"type": "turn.done"} Client->>Server: JSON: {"type": "end"} Server-->>Client: Close 1000 ``` ## 客户端事件 ### 1. start - 开始语音对话 `start` 用于初始化实时语音对话，必须作为第一个消息发送。同一连接内重复发送 `start` 会触发 `bad_request`，服务端随后以 `1008` 断开连接。 **请求参数** | 参数名 | 类型 | 必填 | 说明 | 示例 | | :-------------------------- | :------ | :- | :-------------------------------------------- | :------------------------ | | `type` | string | 是 | 固定值：`start` | `start` | | `model` | string | 是 | 模型 ID，当前仅支持 `senseaudio-realtime-1.0` | `senseaudio-realtime-1.0` | | `voice` | string | 否 | 对话音色 ID；不传时由服务端选择默认音色，字面值 `default` 不是有效音色 ID | `f_y_0035_c` | | `instructions` | string | 否 | 系统提示词 | `你是SenseAudio助手` | | `greeting` | string | 否 | 开场白，可以为空；非空时会产生独立的开场白轮次 | `你好，我是SenseAudio语音助手` | | `audio_setting` | object | 否 | 输入音频设置，不传时使用默认配置 | - | | `audio_setting.sample_rate` | integer | 否 | 采样率，当前仅支持 `16000` | `16000` | | `audio_setting.format` | string | 否 | 音频格式，当前仅支持 `pcm_s16le` | `pcm_s16le` | | `audio_setting.channel` | integer | 否 | 声道数，当前仅支持 `1` | `1` | **请求示例** ```json theme={null} { "type": "start", "model": "senseaudio-realtime-1.0", "voice": "f_y_0035_c", "instructions": "你是SenseAudio助手", "greeting": "", "audio_setting": { "sample_rate": 16000, "format": "pcm_s16le", "channel": 1 } } ``` `greeting` 非空时，服务端会在 `ready` 后返回开场白音频，并以一个独立的 `turn.done` 结束该轮。若客户端需要等开场白播放完再上传用户语音，应先等待这个 `turn.done`；完整代码示例将 `greeting` 设为空，以便只演示用户语音轮次。 ### 2. update - 更新对话配置 `update` 用于在已经开始的对话中修改配置。除 `type` 外，至少需要包含一个可更新字段；未包含的字段保持不变。如果配置有误，服务端返回 `error`，不会应用本次更新，连接不会断开。 `update` 不支持修改 `model` 和 `audio_setting`。 **请求参数** | 参数名 | 类型 | 必填 | 说明 | 示例 | | :------------- | :----- | :- | :--------------------------------------- | :--------------- | | `type` | string | 是 | 固定值：`update` | `update` | | `voice` | string | 否 | 更新对话音色 ID，必须使用“音色列表”中的值；字面值 `default` 无效 | `f_y_0035_c` | | `instructions` | string | 否 | 更新系统提示词 | `你是SenseAudio助手` | **请求示例** ```json theme={null} { "type": "update", "voice": "f_y_0035_c", "instructions": "你是SenseAudio助手" } ``` ### 3. 音频流传输（Binary Message） 客户端必须在收到服务端 `ready` 后发送二进制音频帧。 * **格式**：`pcm_s16le` * **采样率**：`16000Hz` * **声道数**：`1` * **推荐帧长**：`40ms`，即 `640` 个采样点、`1280` 字节 * **发送节奏**：按音频实际时长发送，不要将整个文件作为一个大二进制消息突发上传 * **计费规则**：按二进制帧传输的 PCM 音频总长度计费，不足 1 秒按 1 秒计 ### 4. commit - 手动提交本轮音频 `commit` 用于手动提交当前已缓冲的音频，表示此轮用户音频输入结束。后续二进制帧不属于已提交的这一轮，而会进入新的输入缓冲。 ```json theme={null} { "type": "commit" } ``` `commit` 不会关闭 WebSocket 或持续麦克风流。实时客户端通常会继续发送麦克风帧，以支持下一轮输入和打断。 在当前 preview 环境的兼容性测试中，`commit` 后完全停止二进制帧可能长时间收不到完成事件，并最终出现 `fatal / stream idle timeout`。本节实时麦克风示例不主动发送 `commit`，而是持续上传麦克风帧并依赖服务端 VAD 自动分轮。所有上传的二进制帧都可能参与计费；客户端在结束、关闭或出错时必须立即停止采集和发送。 ### 5. cancel - 打断模型回复 `cancel` 用于打断并取消当前模型回复。 ```json theme={null} { "type": "cancel" } ``` ### 6. end - 结束对话 `end` 用于结束当前对话。服务端收到后会以 `1000` 关闭 WebSocket 连接。 ```json theme={null} { "type": "end" } ``` ## 服务端事件 ### ready - 连接就绪 表示服务端到模型的连接已经建立，可以开始发送音频数据。若 `start.greeting` 非空，`ready` 后还会收到一轮开场白事件；该轮也会以 `turn.done` 结束。 ```json theme={null} { "type": "ready", "session_id": "session_50173901836217346" } ``` ### speech.started - 检测到用户开口 表示服务端检测到用户已经开始说话。 ```json theme={null} { "type": "speech.started" } ``` ### user.transcript.delta - 用户转写增量 用户语音转写的增量结果。该结果会随用户语音输入逐渐修正，前端渲染时建议使用最新值替换展示。 ```json theme={null} { "type": "user.transcript.delta", "text": "今天天气怎么样？" } ``` ### user.transcript.done - 用户转写最终稿 用户语音结束后返回的最终转写结果。语音结束包括服务端自动判断结束和客户端手动 `commit` 提交。 ```json theme={null} { "type": "user.transcript.done", "text": "今天天气怎么样？" } ``` ### assistant.text.delta - 模型回复文本增量 模型回复文本的增量片段。客户端需要将本字段追加到之前收到的文本后，得到当前模型回复内容。 ```json theme={null} { "type": "assistant.text.delta", "text": "今天" } ``` ### assistant.text.done - 模型回复文本全文 本轮模型回复文本的完整内容。 ```json theme={null} { "type": "assistant.text.done", "text": "今天的天气情况可以打开天气应用查看。如果你告诉我所在城市，我可以帮你整理出行建议。" } ``` ### assistant.audio.start - 模型回复音频开始 表示模型回复音频开始。之后收到的二进制帧均为本轮模型回复音频。当前服务端输出为单声道 PCM。 ```json theme={null} { "type": "assistant.audio.start", "response_id": "tts_3", "sample_rate": 24000, "format": "pcm_s16le" } ``` ### assistant.audio.done - 模型回复音频结束 表示本轮模型回复音频结束，`response_id` 与 `assistant.audio.start` 中的 `response_id` 一致。 ```json theme={null} { "type": "assistant.audio.done", "response_id": "tts_3" } ``` ### turn.done - 本轮结束 表示一个对话轮次结束。非空 `greeting` 的开场白轮次和每个用户语音轮次都会分别返回 `turn.done`；客户端应维护会话状态，不能无条件在第一个 `turn.done` 后结束连接。 ```json theme={null} { "type": "turn.done" } ``` ### error - 错误通知 发生错误时，服务端返回 `error`。错误分为致命错误和非致命错误；致命错误会在服务端发送该消息后断开 WebSocket，非致命错误不会断开连接。 ```json theme={null} { "type": "error", "code": "bad_request", "message": "客户端发送的 start 消息配置无效" } ``` ## 使用示例 以下示例直接采集系统麦克风，并把模型返回的音频实时播放到扬声器。建议使用耳机，避免扬声器声音被麦克风重新采集后触发误识别或错误打断。 运行前先设置 API Key： ```bash theme={null} export SENSEAUDIO_API_KEY='YOUR_API_KEY' ``` 示例默认连接生产地址。验证 preview 环境时，可额外设置： ```bash theme={null} export SENSEAUDIO_WS_URL='wss://preview.api.senseaudio.cn/ws/v1/realtime/voice-dialog' ``` 示例在收到 `ready` 后持续上传 `16000Hz / 单声道 / pcm_s16le` 麦克风帧，由服务端 VAD 自动判断每轮语音结束，因此不会主动发送 `commit`。模型音频按服务端声明的 `24000Hz / 单声道 / pcm_s16le` 实时播放；普通 `turn.done` 只表示一轮完成，连接会继续用于下一轮。正常使用时按 `Ctrl+C` 发送 `end`。当服务端在模型播报期间返回 `speech.started` 时，示例会发送一次 `cancel` 并清除尚未播放的旧回复。若 `greeting` 非空，示例会先播完开场白再开启麦克风，避免开场白回采形成错误的用户轮次。 依赖：**Node.js 18+；`npm install ws@8 decibri@5`** `decibri@5` 提供预编译音频后端，当前支持 Windows x64、macOS arm64，以及 Linux x64/arm64。Linux 需要可用的 ALSA 环境；macOS 和 Windows 需要允许终端访问麦克风。 ```javascript theme={null} const WebSocket = require('ws') const { Microphone, Speaker } = require('decibri') function parseDevice(value) { if (!value) return undefined return /^\d+$/.test(value) ? Number(value) : value } if (process.argv.includes('--list-devices')) { console.log('麦克风设备：') console.table(Microphone.devices()) console.log('扬声器设备：') console.table(Speaker.devices()) process.exit(0) } const API_KEY = process.env.SENSEAUDIO_API_KEY const WS_URL = process.env.SENSEAUDIO_WS_URL || 'wss://preview.api.senseaudio.cn/ws/v1/realtime/voice-dialog' const GREETING = process.env.SENSEAUDIO_GREETING || '' const VOICE = process.env.SENSEAUDIO_VOICE || '' const INPUT_DEVICE = parseDevice(process.env.SENSEAUDIO_INPUT_DEVICE) const OUTPUT_DEVICE = parseDevice(process.env.SENSEAUDIO_OUTPUT_DEVICE) const INPUT_SAMPLE_RATE = 16000 const INPUT_FRAME_SAMPLES = 640 // 40ms const OUTPUT_SAMPLE_RATE = 24000 const OUTPUT_BLOCK_BYTES = 480 * 2 // 20ms，降低打断后的播放尾音 const MAX_PLAYBACK_BLOCKS = 500 // 最多约 10 秒尚未交给音频后端的数据 const MAX_WS_BUFFERED_BYTES = 64 * 1024 const INPUT_FRAME_MS = INPUT_FRAME_SAMPLES / INPUT_SAMPLE_RATE * 1000 const MAX_CAPTURE_BURST_FRAMES = 5 if (!API_KEY) { throw new Error('请先设置 SENSEAUDIO_API_KEY 环境变量') } class PcmPlayer { constructor(device, onError) { this.options = { sampleRate: OUTPUT_SAMPLE_RATE, channels: 1, dtype: 'int16' } if (device !== undefined) this.options.device = device this.onError = onError this.speaker = null this.ready = null this.tail = Promise.resolve() this.epoch = 0 this.pending = new Map() this.accepting = false this.closed = false } open() { if (!this.ready) this.ready = this.openFor(this.epoch) return this.ready } async openFor(epoch) { const speaker = await Speaker.open(this.options) speaker.on('error', (error) => { if (!this.closed && epoch === this.epoch) this.onError(error) }) if (this.closed || epoch !== this.epoch) { speaker.stop() return null } this.speaker = speaker return speaker } begin() { this.accepting = true } push(data) { if (this.closed || !this.accepting) return const epoch = this.epoch const pending = this.pending.get(epoch) || 0 if (pending >= MAX_PLAYBACK_BLOCKS) { this.accepting = false this.onError(new Error('扬声器播放队列已满，请检查输出设备')) return } const chunk = Buffer.from(data) const ready = this.ready || (this.ready = this.openFor(epoch)) this.pending.set(epoch, pending + 1) const task = this.tail.catch(() => {}).then(async () => { const speaker = await ready if (!speaker || this.closed || epoch !== this.epoch) return await speaker.writeAsync(chunk) }) this.tail = task task.then( () => this.finishTask(epoch), (error) => { this.finishTask(epoch) if (!this.closed && epoch === this.epoch) this.onError(error) } ) } finishTask(epoch) { const count = (this.pending.get(epoch) || 1) - 1 if (count > 0) this.pending.set(epoch, count) else this.pending.delete(epoch) } finish() { this.accepting = false } hasAudio() { return this.accepting || (this.pending.get(this.epoch) || 0) > 0 || Boolean(this.speaker?.isPlaying) } async drain() { const epoch = this.epoch const tail = this.tail await tail.catch(() => {}) if (this.closed || epoch !== this.epoch) return const speaker = await this.ready if (speaker && epoch === this.epoch) await speaker.drainAsync() } async interrupt() { if (this.closed) return this.accepting = false const previousReady = this.ready const oldSpeaker = this.speaker const epoch = ++this.epoch this.speaker = null this.tail = Promise.resolve() if (oldSpeaker) oldSpeaker.stop() // 立即丢弃 native 播放缓冲 this.ready = (async () => { if (previousReady) { try { const previousSpeaker = await previousReady if (previousSpeaker && previousSpeaker !== oldSpeaker) { previousSpeaker.stop() } } catch {} } if (this.closed || epoch !== this.epoch) return null return this.openFor(epoch) })() await this.ready } stop() { if (this.closed) return this.closed = true this.accepting = false this.epoch += 1 if (this.speaker) this.speaker.stop() this.speaker = null } } let mic = null let micOpening = null let startSent = false let ending = false let waitingGreeting = Boolean(GREETING) let responseActive = false let cancelSent = false let suppressAssistant = false let assistantTextStarted = false let downlinkBuffer = Buffer.alloc(0) let forceCloseTimer = null let lastDropWarningAt = 0 let captureTokens = MAX_CAPTURE_BURST_FRAMES let lastCaptureTokenAt = Date.now() const player = new PcmPlayer(OUTPUT_DEVICE, (error) => { fail(`播放模型音频失败：${error.message}`) }) const ws = new WebSocket(WS_URL, { headers: { Authorization: `Bearer ${API_KEY}` }, perMessageDeflate: false }) function sendJson(payload) { if (ws.readyState !== WebSocket.OPEN) return false ws.send(JSON.stringify(payload)) return true } function stopMicrophone() { if (!mic) return mic.removeAllListeners('data') mic.stop() mic = null } function cleanup() { clearTimeout(forceCloseTimer) forceCloseTimer = null stopMicrophone() player.stop() } function fail(reason) { console.error(`\n${reason}`) if (startSent) { endSession('本地运行错误') return } if (ending) return ending = true cleanup() if (ws.readyState === WebSocket.OPEN) ws.close() else if (ws.readyState === WebSocket.CONNECTING) ws.terminate() } function endSession(reason) { if (ending) return ending = true console.log(`\n结束会话：${reason}`) stopMicrophone() player.stop() if (startSent && ws.readyState === WebSocket.OPEN) { ws.send(JSON.stringify({ type: 'end' })) forceCloseTimer = setTimeout(() => { if (ws.readyState !== WebSocket.CLOSED) ws.terminate() }, 2000) } else if (ws.readyState === WebSocket.OPEN) { ws.close() } else if (ws.readyState === WebSocket.CONNECTING) { ws.terminate() } } function onMicrophoneData(chunk) { if (ending || ws.readyState !== WebSocket.OPEN) return if (chunk.length % 2 !== 0) { fail('麦克风返回了长度不是 16-bit 对齐的 PCM 数据') return } // 真实声卡会按时钟产帧；这里额外限制异常虚拟设备的突发速度。 const now = Date.now() captureTokens = Math.min( MAX_CAPTURE_BURST_FRAMES, captureTokens + (now - lastCaptureTokenAt) / INPUT_FRAME_MS ) lastCaptureTokenAt = now const captureTooFast = captureTokens < 1 if (!captureTooFast) captureTokens -= 1 if (captureTooFast || ws.bufferedAmount > MAX_WS_BUFFERED_BYTES) { if (now - lastDropWarningAt > 1000) { console.warn('\n采集或网络速度异常，已丢弃过期麦克风帧') lastDropWarningAt = now } return } ws.send(chunk, { binary: true, compress: false }, (error) => { if (error && !ending) fail(`发送麦克风音频失败：${error.message}`) }) } async function startMicrophone() { if (mic || micOpening || ending) return const options = { sampleRate: INPUT_SAMPLE_RATE, channels: 1, dtype: 'int16', framesPerBuffer: INPUT_FRAME_SAMPLES, highWaterMark: INPUT_FRAME_SAMPLES * 2 * 10 } if (INPUT_DEVICE !== undefined) options.device = INPUT_DEVICE const opening = Microphone.open(options) micOpening = opening let opened try { opened = await opening } finally { if (micOpening === opening) micOpening = null } if (ending) { opened.stop() return } mic = opened mic.on('error', (error) => fail(`麦克风错误：${error.message}`)) mic.on('backpressure', () => { console.warn('\n麦克风采集发生背压，可能已经丢帧') }) mic.on('data', onMicrophoneData) console.log('可以开始说话；按 Ctrl+C 结束会话') } function enqueueAssistantAudio(data) { if (!player.accepting) return downlinkBuffer = downlinkBuffer.length ? Buffer.concat([downlinkBuffer, data]) : Buffer.from(data) let offset = 0 while (downlinkBuffer.length - offset >= OUTPUT_BLOCK_BYTES) { player.push(downlinkBuffer.subarray(offset, offset + OUTPUT_BLOCK_BYTES)) offset += OUTPUT_BLOCK_BYTES } downlinkBuffer = Buffer.from(downlinkBuffer.subarray(offset)) } function finishAssistantAudio() { if (player.accepting && downlinkBuffer.length > 0) { if (downlinkBuffer.length % 2 !== 0) { fail('服务端返回了长度不是 16-bit 对齐的 PCM 音频') return } player.push(downlinkBuffer) } downlinkBuffer = Buffer.alloc(0) player.finish() } function interruptAssistant() { const shouldCancel = responseActive && !cancelSent if (!shouldCancel && !player.hasAudio()) return downlinkBuffer = Buffer.alloc(0) responseActive = false if (shouldCancel) { cancelSent = true suppressAssistant = true sendJson({ type: 'cancel' }) console.log('\n检测到用户打断，已取消当前模型回复') } void player.interrupt().catch((error) => { fail(`重置扬声器失败：${error.message}`) }) } ws.on('open', () => { void (async () => { try { await player.open() if (ending) return const start = { type: 'start', model: 'senseaudio-realtime-1.0', instructions: '你是SenseAudio助手', greeting: GREETING, audio_setting: { sample_rate: INPUT_SAMPLE_RATE, format: 'pcm_s16le', channel: 1 } } if (VOICE) start.voice = VOICE startSent = sendJson(start) } catch (error) { fail(`无法打开音频设备：${error.message}`) } })() }) ws.on('message', (data, isBinary) => { if (isBinary) { if (!suppressAssistant) enqueueAssistantAudio(data) return } let event try { event = JSON.parse(data.toString()) } catch (error) { fail(`服务端返回了无效 JSON：${error.message}`) return } switch (event.type) { case 'ready': if (waitingGreeting) { console.log('连接已就绪，正在播放开场白') } else { void startMicrophone().catch((error) => { fail(`无法打开麦克风：${error.message}`) }) } break case 'speech.started': interruptAssistant() break case 'user.transcript.done': // 新用户轮次开始生成回复，解除上一轮 cancel 的丢弃状态。 suppressAssistant = false cancelSent = false responseActive = true assistantTextStarted = false console.log(`\n你：${event.text || ''}`) break case 'assistant.text.delta': if (suppressAssistant) break responseActive = true if (!assistantTextStarted) { process.stdout.write('\n助手：') assistantTextStarted = true } process.stdout.write(event.text || '') break case 'assistant.text.done': if (suppressAssistant) break responseActive = true if (assistantTextStarted) process.stdout.write('\n') else if (event.text) console.log(`\n助手：${event.text}`) assistantTextStarted = false break case 'assistant.audio.start': if (event.sample_rate !== OUTPUT_SAMPLE_RATE || event.format !== 'pcm_s16le') { fail('服务端输出音频不是预期的 pcm_s16le / 24000Hz') break } downlinkBuffer = Buffer.alloc(0) if (!suppressAssistant) { responseActive = true player.begin() } break case 'assistant.audio.done': finishAssistantAudio() break case 'turn.done': { // 正常情况下 audio.done 已先到达；这里也做一次幂等收尾。 finishAssistantAudio() const greetingDone = waitingGreeting if (assistantTextStarted) process.stdout.write('\n') waitingGreeting = false responseActive = false cancelSent = false suppressAssistant = false assistantTextStarted = false if (greetingDone) { void (async () => { try { await player.drain() await startMicrophone() } catch (error) { fail(`无法开始麦克风会话：${error.message}`) } })() } else { console.log('本轮结束，继续监听麦克风') } break } case 'error': console.error(`\n服务端错误：${event.code || 'unknown'} - ${event.message || ''}`) if (event.code === 'no_active_response' && cancelSent) { cancelSent = false suppressAssistant = false responseActive = false assistantTextStarted = false } break default: console.log('收到未知服务端事件：', event) } }) ws.on('close', (code, reason) => { ending = true cleanup() console.log(`\n连接关闭：${code} ${reason.toString()}`) }) ws.on('error', (error) => { console.error(`\n连接错误：${error.message}`) }) process.once('SIGINT', () => endSession('用户按下 Ctrl+C')) process.once('SIGTERM', () => endSession('收到 SIGTERM')) ``` 将上方代码保存为 `realtime_voice.js`，然后查看设备或直接运行： ```bash theme={null} node realtime_voice.js --list-devices export SENSEAUDIO_INPUT_DEVICE='输入设备名称或编号' export SENSEAUDIO_OUTPUT_DEVICE='输出设备名称或编号' node realtime_voice.js ``` 依赖：**Python 3.9+；`python3 -m pip install websocket-client sounddevice`** `sounddevice` 基于 PortAudio。Windows 和 macOS 的 pip 包通常包含 PortAudio；Linux 如果安装或运行时报 PortAudio 错误，需要先通过系统包管理器安装运行库，例如 Ubuntu / Debian 可执行 `sudo apt-get install libportaudio2`。 ```python theme={null} import json import os import queue import sys import threading import time import sounddevice as sd import websocket def parse_device(name): value = os.environ.get(name) if not value: return None try: return int(value) except ValueError: return value if "--list-devices" in sys.argv: print(sd.query_devices()) raise SystemExit(0) API_KEY = os.environ.get("SENSEAUDIO_API_KEY") WS_URL = os.environ.get( "SENSEAUDIO_WS_URL", "wss://preview.api.senseaudio.cn/ws/v1/realtime/voice-dialog", ) GREETING = os.environ.get("SENSEAUDIO_GREETING", "") VOICE = os.environ.get("SENSEAUDIO_VOICE", "") INPUT_DEVICE = parse_device("SENSEAUDIO_INPUT_DEVICE") OUTPUT_DEVICE = parse_device("SENSEAUDIO_OUTPUT_DEVICE") INPUT_SAMPLE_RATE = 16000 INPUT_BLOCK_FRAMES = 640 # 40ms OUTPUT_SAMPLE_RATE = 24000 OUTPUT_BLOCK_FRAMES = 480 # 20ms，降低打断后的播放尾音 OUTPUT_BLOCK_BYTES = OUTPUT_BLOCK_FRAMES * 2 MIC_QUEUE_BLOCKS = 25 # 最多缓存约 1 秒上行音频 PLAY_QUEUE_BLOCKS = 500 # 最多缓存约 10 秒下行音频 if not API_KEY: raise RuntimeError("请先设置 SENSEAUDIO_API_KEY 环境变量") class RealtimeVoiceClient: def __init__(self): self.stop_event = threading.Event() self.closed_event = threading.Event() self.mic_enabled = threading.Event() self.start_sent = threading.Event() self.fatal_event = threading.Event() self.mic_dropped = threading.Event() self.input_warning = threading.Event() self.output_warning = threading.Event() self.output_busy = threading.Event() self.mic_queue = queue.Queue(maxsize=MIC_QUEUE_BLOCKS) self.play_queue = queue.Queue(maxsize=PLAY_QUEUE_BLOCKS) self.state_lock = threading.Lock() self.send_lock = threading.Lock() self.output_stream_lock = threading.Lock() self.waiting_greeting = bool(GREETING) self.response_active = False self.downlink_audio_active = False self.cancel_sent = False self.drop_current_response = False self.assistant_text_started = False self.playback_epoch = 0 self.downlink_buffer = bytearray() self.ending = False self.fatal_reason = None self.input_stream = None self.output_stream = None self.uplink_thread = None self.output_thread = None self.websocket_thread = None self.ws = websocket.WebSocketApp( WS_URL, header=[f"Authorization: Bearer {API_KEY}"], on_open=self.on_open, on_message=self.on_message, on_close=self.on_close, on_error=self.on_error, ) def set_fatal(self, reason): with self.state_lock: if self.fatal_reason is None: self.fatal_reason = reason self.fatal_event.set() def send_json(self, payload): message = json.dumps(payload, ensure_ascii=False) with self.send_lock: self.ws.send(message) def send_binary(self, payload): with self.send_lock: if self.stop_event.is_set(): return False self.ws.send(payload, opcode=websocket.ABNF.OPCODE_BINARY) return True # PortAudio 实时回调中不能执行网络请求、打印或其他阻塞操作。 def input_callback(self, indata, frames, time_info, status): if status: self.input_warning.set() if self.stop_event.is_set() or not self.mic_enabled.is_set(): return # indata 的内存在回调返回后会被复用，必须复制。 payload = bytes(indata) try: self.mic_queue.put_nowait(payload) except queue.Full: # 实时对话优先保留最新音频，避免积压带来数秒延迟。 self.mic_dropped.set() try: self.mic_queue.get_nowait() self.mic_queue.task_done() except queue.Empty: pass try: self.mic_queue.put_nowait(payload) except queue.Full: pass def uplink_worker(self): try: next_send_at = time.monotonic() while not self.stop_event.is_set(): try: payload = self.mic_queue.get(timeout=0.1) except queue.Empty: continue try: if self.mic_enabled.is_set(): # 即使虚拟设备突发产帧，也按 PCM 实际时长发送。 now = time.monotonic() if next_send_at < now - 1: next_send_at = now if self.stop_event.wait(max(0, next_send_at - now)): return if not self.send_binary(payload): return next_send_at = max(next_send_at, now) + ( len(payload) / (INPUT_SAMPLE_RATE * 2) ) finally: self.mic_queue.task_done() except websocket.WebSocketConnectionClosedException: pass except Exception as error: self.set_fatal(f"发送麦克风音频失败：{error}") def output_worker(self): try: while not self.stop_event.is_set(): try: epoch, payload = self.play_queue.get(timeout=0.1) except queue.Empty: continue try: with self.output_stream_lock: with self.state_lock: valid = ( epoch == self.playback_epoch and not self.drop_current_response ) if valid and not self.stop_event.is_set(): self.output_busy.set() try: if self.output_stream.write(payload): self.output_warning.set() finally: self.output_busy.clear() finally: self.play_queue.task_done() except Exception as error: if not self.stop_event.is_set(): self.set_fatal(f"播放模型音频失败：{error}") def clear_playback_queue(self): while True: try: self.play_queue.get_nowait() except queue.Empty: return else: self.play_queue.task_done() def enqueue_playback(self, epoch, payload): try: self.play_queue.put_nowait((epoch, payload)) except queue.Full: self.set_fatal("扬声器播放队列已满，请检查输出设备") def handle_binary_audio(self, message): payload = bytes(message) with self.state_lock: if ( not self.downlink_audio_active or self.drop_current_response or self.ending ): return epoch = self.playback_epoch self.downlink_buffer.extend(payload) while len(self.downlink_buffer) >= OUTPUT_BLOCK_BYTES: block = bytes(self.downlink_buffer[:OUTPUT_BLOCK_BYTES]) del self.downlink_buffer[:OUTPUT_BLOCK_BYTES] self.enqueue_playback(epoch, block) def finish_downlink_audio(self): with self.state_lock: valid = ( self.downlink_audio_active and not self.drop_current_response and not self.ending ) epoch = self.playback_epoch self.downlink_audio_active = False if valid and self.downlink_buffer: if len(self.downlink_buffer) % 2: self.set_fatal("服务端返回了长度不是 16-bit 对齐的 PCM 音频") else: self.enqueue_playback(epoch, bytes(self.downlink_buffer)) self.downlink_buffer.clear() def interrupt_assistant(self): with self.state_lock: had_audio = ( self.downlink_audio_active or not self.play_queue.empty() or self.output_busy.is_set() ) self.playback_epoch += 1 should_cancel = ( self.response_active and not self.cancel_sent and not self.ending ) self.response_active = False self.downlink_audio_active = False if should_cancel: self.cancel_sent = True self.drop_current_response = True self.downlink_buffer.clear() self.clear_playback_queue() if had_audio and self.output_stream is not None: try: with self.output_stream_lock: self.output_stream.abort(ignore_errors=False) if not self.stop_event.is_set(): self.output_stream.start() except Exception as error: self.set_fatal(f"重置扬声器失败：{error}") if should_cancel: print("\n检测到用户打断，已取消当前模型回复") try: self.send_json({"type": "cancel"}) except websocket.WebSocketConnectionClosedException: pass def enable_mic_after_greeting(self): # turn.done 到达时开场白二进制帧已经全部入队，排空后再开麦避免回声。 self.play_queue.join() if self.stop_event.is_set(): return try: with self.output_stream_lock: # stop() 会等待 PortAudio 和硬件中的待播缓冲全部完成。 self.output_stream.stop(ignore_errors=False) if self.stop_event.is_set(): return self.output_stream.start() except Exception as error: self.set_fatal(f"等待开场白播放完成失败：{error}") return self.mic_enabled.set() print("开场白播放完成，可以开始说话；按 Ctrl+C 结束会话") def request_end(self, reason): with self.state_lock: if self.ending: return self.ending = True self.playback_epoch += 1 print(f"\n结束会话：{reason}") self.mic_enabled.clear() self.stop_event.set() self.downlink_buffer.clear() self.clear_playback_queue() if self.start_sent.is_set(): threading.Thread( target=self.send_end, name="senseaudio-end", daemon=True, ).start() else: self.abort_websocket() def send_end(self): try: self.send_json({"type": "end"}) except Exception: self.abort_websocket() def abort_websocket(self): # 低层 abort 可唤醒卡在 send()/recv() 中的线程。 sock = self.ws.sock if sock is not None: try: sock.abort() except Exception: pass def on_open(self, ws): if self.stop_event.is_set(): self.abort_websocket() return start = { "type": "start", "model": "senseaudio-realtime-1.0", "instructions": "你是SenseAudio助手", "greeting": GREETING, "audio_setting": { "sample_rate": INPUT_SAMPLE_RATE, "format": "pcm_s16le", "channel": 1, }, } if VOICE: start["voice"] = VOICE try: self.send_json(start) self.start_sent.set() except Exception as error: self.set_fatal(f"发送 start 失败：{error}") def on_message(self, ws, message): if self.stop_event.is_set(): return if isinstance(message, (bytes, bytearray, memoryview)): self.handle_binary_audio(message) return try: event = json.loads(message) except json.JSONDecodeError as error: self.set_fatal(f"服务端返回了无效 JSON：{error}") return event_type = event.get("type") if event_type == "ready": if self.waiting_greeting: print("连接已就绪，正在播放开场白") else: self.mic_enabled.set() print("可以开始说话；按 Ctrl+C 结束会话") return if event_type == "speech.started": self.interrupt_assistant() return if event_type == "user.transcript.done": with self.state_lock: # 新用户轮次开始生成回复，解除上一轮 cancel 的丢弃状态。 self.cancel_sent = False self.drop_current_response = False self.response_active = True self.assistant_text_started = False print(f"\n你：{event.get('text', '')}") return if event_type == "assistant.text.delta": with self.state_lock: if self.drop_current_response: return self.response_active = True first_delta = not self.assistant_text_started self.assistant_text_started = True if first_delta: print("\n助手：", end="", flush=True) print(event.get("text", ""), end="", flush=True) return if event_type == "assistant.text.done": with self.state_lock: suppressed = self.drop_current_response text_started = self.assistant_text_started if not suppressed: self.response_active = True self.assistant_text_started = False if suppressed: return if text_started: print() elif event.get("text"): print(f"\n助手：{event['text']}") return if event_type == "assistant.audio.start": if ( event.get("sample_rate") != OUTPUT_SAMPLE_RATE or event.get("format") != "pcm_s16le" ): self.set_fatal("服务端输出音频不是预期的 pcm_s16le / 24000Hz") return self.downlink_buffer.clear() with self.state_lock: self.downlink_audio_active = True if not self.drop_current_response: self.response_active = True return if event_type == "assistant.audio.done": self.finish_downlink_audio() return if event_type == "turn.done": # 正常情况下 audio.done 已先到达；这里也做一次幂等收尾。 self.finish_downlink_audio() with self.state_lock: greeting_done = self.waiting_greeting text_incomplete = self.assistant_text_started self.waiting_greeting = False self.response_active = False self.downlink_audio_active = False self.cancel_sent = False self.drop_current_response = False self.assistant_text_started = False if text_incomplete: print() if greeting_done: threading.Thread( target=self.enable_mic_after_greeting, name="senseaudio-greeting-drain", daemon=True, ).start() else: print("本轮结束，继续监听麦克风") return if event_type == "error": print( "\n服务端错误：" f"{event.get('code', 'unknown')} - {event.get('message', '')}" ) if event.get("code") == "no_active_response": with self.state_lock: self.cancel_sent = False self.drop_current_response = False self.response_active = False self.assistant_text_started = False return print("收到未知服务端事件：", event) def on_close(self, ws, code, reason): self.mic_enabled.clear() self.stop_event.set() self.closed_event.set() print(f"\n连接关闭：{code} {reason or ''}") def on_error(self, ws, error): if not self.ending: print(f"\n连接错误：{error}") def open_audio_devices(self): try: sd.check_input_settings( device=INPUT_DEVICE, channels=1, dtype="int16", samplerate=INPUT_SAMPLE_RATE, ) sd.check_output_settings( device=OUTPUT_DEVICE, channels=1, dtype="int16", samplerate=OUTPUT_SAMPLE_RATE, ) self.input_stream = sd.RawInputStream( device=INPUT_DEVICE, samplerate=INPUT_SAMPLE_RATE, channels=1, dtype="int16", blocksize=INPUT_BLOCK_FRAMES, latency="low", callback=self.input_callback, ) self.output_stream = sd.RawOutputStream( device=OUTPUT_DEVICE, samplerate=OUTPUT_SAMPLE_RATE, channels=1, dtype="int16", blocksize=OUTPUT_BLOCK_FRAMES, latency="low", ) except Exception as error: if self.input_stream is not None: self.input_stream.close() self.input_stream = None if self.output_stream is not None: self.output_stream.close() self.output_stream = None raise RuntimeError( "无法打开 16kHz 麦克风或 24kHz 扬声器；" "请运行 python3 -m sounddevice 检查设备" ) from error def run(self): self.open_audio_devices() self.uplink_thread = threading.Thread( target=self.uplink_worker, name="senseaudio-uplink", daemon=True, ) self.output_thread = threading.Thread( target=self.output_worker, name="senseaudio-playback", daemon=True, ) self.websocket_thread = threading.Thread( target=lambda: self.ws.run_forever( ping_interval=20, ping_timeout=10, ), name="senseaudio-websocket", daemon=True, ) try: self.output_stream.start() self.input_stream.start() self.uplink_thread.start() self.output_thread.start() self.websocket_thread.start() while self.websocket_thread.is_alive(): if self.mic_dropped.is_set(): self.mic_dropped.clear() print("\n警告：网络发送较慢，已丢弃过期麦克风帧") if self.input_warning.is_set(): self.input_warning.clear() print("\n警告：麦克风发生丢帧或输入溢出") if self.output_warning.is_set(): self.output_warning.clear() print("\n警告：扬声器发生输出欠载") if self.fatal_event.is_set(): self.request_end(self.fatal_reason or "本地音频错误") break self.websocket_thread.join(0.1) except KeyboardInterrupt: self.request_end("用户按下 Ctrl+C") finally: if self.websocket_thread.is_alive() and not self.closed_event.wait(2): self.abort_websocket() self.ws.close() self.stop_event.set() self.mic_enabled.clear() self.clear_playback_queue() if self.input_stream is not None: self.input_stream.abort() self.input_stream.close() if self.uplink_thread is not None: self.uplink_thread.join(timeout=1) if self.output_thread is not None: self.output_thread.join(timeout=1) if self.output_stream is not None: self.output_stream.abort() self.output_stream.close() if self.websocket_thread is not None: self.websocket_thread.join(timeout=1) if __name__ == "__main__": RealtimeVoiceClient().run() ``` 将上方代码保存为 `realtime_voice.py`，然后查看设备或直接运行： ```bash theme={null} python3 -m sounddevice export SENSEAUDIO_INPUT_DEVICE='输入设备名称或编号' export SENSEAUDIO_OUTPUT_DEVICE='输出设备名称或编号' python3 realtime_voice.py ``` 两个示例都没有实现声学回声消除（AEC），请优先使用耳机。Node.js 的 `decibri` 会把麦克风输入重采样为 `16000Hz`；其他音频路径能否打开目标采样率取决于操作系统和设备。若初始化失败，请选择支持重采样的系统设备或音频后端，不要直接修改协议规定的输入 `16000Hz` 与当前输出 `24000Hz`。 ## 错误码说明 | 错误码 | 类型 | 说明 | 关闭码 | | :------------------- | :---- | :-------------------------------------------------------------------------------------------------------------------------------- | :----- | | `bad_request` | 致命错误 | JSON 格式错误、`type` 无效、`start` 配置错误（包括音色无效）、重复发送 `start`、`update` 修改了 `model` 或 `audio_setting`，或 `commit` / `cancel` / `end` 携带额外字段 | `1008` | | `model_not_found` | 致命错误 | `start` 中传入的模型错误 | `1008` | | `insufficient_funds` | 致命错误 | 账户没有足够配额完成接下来的语音对话 | `1008` | | `internal_error` | 致命错误 | 服务端没有正确连接到模型服务 | `1011` | | `fatal` | 致命错误 | 服务端发生其他错误 | `1011` | | `empty_audio_buffer` | 非致命错误 | 没有上传音频，或上次手动提交后没有新的音频上传时发送了 `commit`，服务端会忽略本次提交 | 不断开 | | `no_active_response` | 非致命错误 | 模型没有回复时发送了 `cancel`，服务端会忽略本次打断 | 不断开 | | `invalid_voice` | 非致命错误 | `update` 修改的音色不存在，服务端忽略本次更新并继续使用之前的音色 | 不断开 | 鉴权发生在 WebSocket Upgrade 阶段。缺少 Token、Token 无效或鉴权格式错误时，服务端返回 HTTP `401`，不会先建立 WebSocket 再发送 JSON `error`。 ## 音色列表 | 音色ID | 音色名 | 情感 | | :----------- | :--- | :--- | | `m_c_0001` | 可爱萌娃 | 开心 | | `f_m_0009_a` | 气质学姐 | 开心 | | `f_y_0035_a` | 清风少女 | 傲娇 | | `f_y_0035_c` | 清风少女 | 平稳 | | `f_y_0039_c` | 知心少女 | 广告中插 | | `f_y_0039_d` | 知心少女 | 轻松铺陈 | | `f_y_0037_f` | 青春女声 | 主题升华 | | `f_y_0041_c` | 亲切女孩 | 致歉安慰 | | `m_y_0022` | 沙哑青年 | 深情 | | `m_y_0021` | 撒娇青年 | 平稳 | | `m_y_0027` | 粘人男友 | 平稳 | | `m_y_0027_a` | 粘人男友 | 深情 | | `m_y_0028` | 温柔霸总 | 平稳 | | `m_y_0034_d` | 可靠青叔 | 细心提问 | | `m_y_0034_e` | 可靠青叔 | 主题升华 | | `m_y_0036_e` | 利落青年 | 细心提问 | ## 注意事项 1. **首条消息要求**：`start` 必须作为第一个消息发送，且同一连接内只能发送一次。 2. **音频发送时机**：必须等待 `ready` 后再发送二进制音频帧。 3. **开场白轮次**：`greeting` 非空时，应区分开场白的 `turn.done` 与用户语音轮次的 `turn.done`。 4. **文本帧格式**：客户端与服务端的文本帧均为 JSON，且必须包含 `type` 字段。 5. **音频帧格式**：客户端上传音频必须为裸 `pcm_s16le / 16000Hz / 单声道`，建议按 `1280` 字节 / `40ms` 分帧发送。 6. **持续上行与超时**：长连接客户端应持续处理麦克风流，并设置应用层连接与设备超时；结束、关闭或出错时必须停止采集和发送。 7. **服务端音频解析**：模型返回的二进制音频帧按 `assistant.audio.start` 中的 `sample_rate` 与 `format` 解析，当前为单声道；实时播放应使用有界队列，避免阻塞 WebSocket 接收回调。 8. **打断处理**：用户在模型播报期间开口时，可在收到 `speech.started` 后发送一次 `cancel`，同时丢弃已取消回复中尚未播放以及仍在途的音频帧。 9. **回声控制**：示例没有实现 AEC，建议使用耳机，避免模型播报被麦克风回采并触发新的用户轮次。 10. **结束连接**：如需主动结束对话，请发送 `end`，服务端会以关闭码 `1000` 正常关闭连接。 ## 相关资源 基于 WebSocket 的实时文本转语音合成协议。 基于 WebSocket 的实时语音识别协议。 ## AsyncAPI ````yaml api-reference/endpoint/tts/end-to-end-wss.asyncapi.json wss id: wss title: 端到端实时语音模型 description: 实时语音对话 WebSocket 通道。 servers: - id: production protocol: wss host: preview.api.senseaudio.cn bindings: [] variables: [] address: /ws/v1/realtime/voice-dialog parameters: [] bindings: - protocol: ws version: 0.1.0 value: method: GET headers: type: object required: - Authorization properties: Authorization: type: string pattern: ^Bearer\s+\S+$ description: Bearer Token，格式：Bearer 。 additionalProperties: true schemaProperties: - name: method type: string description: GET required: false - name: headers type: object required: false properties: - name: Authorization type: string description: Bearer Token，格式：Bearer 。 required: true operations: - &ref_37 id: receiveStart title: start - 开始语音对话 description: start type: receive messages: - &ref_54 id: start contentType: application/json payload: - name: start - 开始语音对话 description: 初始化模型与对话配置。 type: object properties: - name: type type: string description: start examples: &ref_0 - start required: true - name: model type: string description: senseaudio-realtime-1.0 examples: &ref_1 - senseaudio-realtime-1.0 required: true - name: voice type: string description: 对话音色 ID。省略 voice 时由服务端选择默认音色；字面值 default 不是有效音色 ID。 enumValues: - m_c_0001 - f_m_0009_a - f_y_0035_a - f_y_0035_c - f_y_0039_c - f_y_0039_d - f_y_0037_f - f_y_0041_c - m_y_0022 - m_y_0021 - m_y_0027 - m_y_0027_a - m_y_0028 - m_y_0034_d - m_y_0034_e - m_y_0036_e examples: &ref_2 - f_y_0035_c required: false - name: instructions type: string examples: &ref_3 - 你是SenseAudio助手 required: false - name: greeting type: string description: 开场白，可以为空。非空时会产生独立的开场白轮次及其 turn.done。 examples: &ref_4 - '' required: false - name: audio_setting type: object required: false properties: - name: sample_rate type: integer description: 16000 examples: &ref_5 - 16000 required: false - name: format type: string description: pcm_s16le examples: &ref_6 - pcm_s16le required: false - name: channel type: integer description: 1 examples: &ref_7 - 1 required: false headers: [] jsonPayloadSchema: type: object additionalProperties: false required: - type - model properties: type: type: string const: start examples: *ref_0 x-parser-schema-id: model: type: string const: senseaudio-realtime-1.0 examples: *ref_1 x-parser-schema-id: voice: &ref_9 type: string description: 对话音色 ID。省略 voice 时由服务端选择默认音色；字面值 default 不是有效音色 ID。 enum: - m_c_0001 - f_m_0009_a - f_y_0035_a - f_y_0035_c - f_y_0039_c - f_y_0039_d - f_y_0037_f - f_y_0041_c - m_y_0022 - m_y_0021 - m_y_0027 - m_y_0027_a - m_y_0028 - m_y_0034_d - m_y_0034_e - m_y_0036_e examples: *ref_2 x-parser-schema-id: VoiceId instructions: type: string examples: *ref_3 x-parser-schema-id: greeting: type: string description: 开场白，可以为空。非空时会产生独立的开场白轮次及其 turn.done。 examples: *ref_4 x-parser-schema-id: audio_setting: type: object additionalProperties: false properties: sample_rate: type: integer const: 16000 examples: *ref_5 x-parser-schema-id: format: type: string const: pcm_s16le examples: *ref_6 x-parser-schema-id: channel: type: integer const: 1 examples: *ref_7 x-parser-schema-id: x-parser-schema-id: AudioSetting x-parser-schema-id: StartPayload title: start - 开始语音对话 description: 初始化模型与对话配置。 example: |- { "type": "start", "model": "senseaudio-realtime-1.0", "voice": "f_y_0035_c", "instructions": "你是SenseAudio助手", "greeting": "", "audio_setting": { "sample_rate": 16000, "format": "pcm_s16le", "channel": 1 } } bindings: [] extensions: - id: x-parser-unique-object-id value: start bindings: [] extensions: &ref_11 - id: x-parser-unique-object-id value: wss - &ref_38 id: receiveUpdate title: update - 更新对话配置 description: update type: receive messages: - &ref_55 id: update contentType: application/json payload: - name: update - 更新对话配置 description: 更新后续轮次使用的音色或系统提示词。 type: object properties: - name: type type: string description: update examples: &ref_8 - update required: true - name: voice type: string description: 对话音色 ID。省略 voice 时由服务端选择默认音色；字面值 default 不是有效音色 ID。 enumValues: - m_c_0001 - f_m_0009_a - f_y_0035_a - f_y_0035_c - f_y_0039_c - f_y_0039_d - f_y_0037_f - f_y_0041_c - m_y_0022 - m_y_0021 - m_y_0027 - m_y_0027_a - m_y_0028 - m_y_0034_d - m_y_0034_e - m_y_0036_e examples: *ref_2 required: false - name: instructions type: string examples: &ref_10 - 你是SenseAudio助手 required: false headers: [] jsonPayloadSchema: type: object additionalProperties: false required: - type anyOf: - required: - voice x-parser-schema-id: - required: - instructions x-parser-schema-id: properties: type: type: string const: update examples: *ref_8 x-parser-schema-id: voice: *ref_9 instructions: type: string examples: *ref_10 x-parser-schema-id: x-parser-schema-id: UpdatePayload title: update - 更新对话配置 description: 更新后续轮次使用的音色或系统提示词。 example: |- { "type": "update", "voice": "f_y_0035_c", "instructions": "你是SenseAudio助手" } bindings: [] extensions: - id: x-parser-unique-object-id value: update bindings: [] extensions: *ref_11 - &ref_39 id: receiveCommit title: commit - 提交本轮音频 description: commit type: receive messages: - &ref_56 id: commit contentType: application/json payload: - name: commit - 提交本轮音频 description: 提交当前用户音频缓冲。 type: object properties: - name: type type: string description: commit examples: &ref_12 - commit required: true headers: [] jsonPayloadSchema: type: object additionalProperties: false required: - type properties: type: type: string const: commit examples: *ref_12 x-parser-schema-id: x-parser-schema-id: CommitPayload title: commit - 提交本轮音频 description: 提交当前用户音频缓冲。 example: |- { "type": "commit" } bindings: [] extensions: - id: x-parser-unique-object-id value: commit bindings: [] extensions: *ref_11 - &ref_40 id: receiveCancel title: cancel - 打断模型回复 description: cancel type: receive messages: - &ref_57 id: cancel contentType: application/json payload: - name: cancel - 打断模型回复 description: 打断并取消当前模型回复。 type: object properties: - name: type type: string description: cancel examples: &ref_13 - cancel required: true headers: [] jsonPayloadSchema: type: object additionalProperties: false required: - type properties: type: type: string const: cancel examples: *ref_13 x-parser-schema-id: x-parser-schema-id: CancelPayload title: cancel - 打断模型回复 description: 打断并取消当前模型回复。 example: |- { "type": "cancel" } bindings: [] extensions: - id: x-parser-unique-object-id value: cancel bindings: [] extensions: *ref_11 - &ref_41 id: receiveEnd title: end - 结束会话 description: end type: receive messages: - &ref_58 id: end contentType: application/json payload: - name: end - 结束会话 description: 结束对话并请求正常关闭 WebSocket。 type: object properties: - name: type type: string description: end examples: &ref_14 - end required: true headers: [] jsonPayloadSchema: type: object additionalProperties: false required: - type properties: type: type: string const: end examples: *ref_14 x-parser-schema-id: x-parser-schema-id: EndPayload title: end - 结束会话 description: 结束对话并请求正常关闭 WebSocket。 example: |- { "type": "end" } bindings: [] extensions: - id: x-parser-unique-object-id value: end bindings: [] extensions: *ref_11 - &ref_43 id: sendReady title: ready - 连接就绪 description: ready type: send messages: - &ref_60 id: ready contentType: application/json payload: - name: ready - 连接就绪 description: 模型连接已建立；非空 greeting 会先产生独立开场白轮次。 type: object properties: - name: type type: string description: ready examples: &ref_15 - ready required: true - name: session_id type: string examples: &ref_16 - session_50173901836217346 required: true headers: [] jsonPayloadSchema: type: object required: - type - session_id properties: type: type: string const: ready examples: *ref_15 x-parser-schema-id: session_id: type: string examples: *ref_16 x-parser-schema-id: x-parser-schema-id: ReadyPayload title: ready - 连接就绪 description: 模型连接已建立；非空 greeting 会先产生独立开场白轮次。 example: |- { "type": "ready", "session_id": "session_50173901836217346" } bindings: [] extensions: - id: x-parser-unique-object-id value: ready bindings: [] extensions: *ref_11 - &ref_44 id: sendSpeechStarted title: speech.started description: speech.started type: send messages: - &ref_61 id: speechStarted contentType: application/json payload: - name: speech.started - 检测到用户开口 description: 服务端检测到用户开始说话。 type: object properties: - name: type type: string description: speech.started examples: &ref_17 - speech.started required: true headers: [] jsonPayloadSchema: type: object required: - type properties: type: type: string const: speech.started examples: *ref_17 x-parser-schema-id: x-parser-schema-id: SpeechStartedPayload title: speech.started - 检测到用户开口 description: 服务端检测到用户开始说话。 example: |- { "type": "speech.started" } bindings: [] extensions: - id: x-parser-unique-object-id value: speechStarted bindings: [] extensions: *ref_11 - &ref_45 id: sendUserTranscriptDelta title: user.transcript.delta description: user.transcript.delta type: send messages: - &ref_62 id: userTranscriptDelta contentType: application/json payload: - name: user.transcript.delta - 用户转写增量 description: 返回当前最新的用户语音转写。 type: object properties: - name: type type: string description: user.transcript.delta examples: &ref_18 - user.transcript.delta required: true - name: text type: string examples: &ref_19 - 今天天气怎么样？ required: true headers: [] jsonPayloadSchema: type: object required: - type - text properties: type: type: string const: user.transcript.delta examples: *ref_18 x-parser-schema-id: text: type: string examples: *ref_19 x-parser-schema-id: x-parser-schema-id: UserTranscriptDeltaPayload title: user.transcript.delta - 用户转写增量 description: 返回当前最新的用户语音转写。 example: |- { "type": "user.transcript.delta", "text": "今天天气怎么样？" } bindings: [] extensions: - id: x-parser-unique-object-id value: userTranscriptDelta bindings: [] extensions: *ref_11 - &ref_46 id: sendUserTranscriptDone title: user.transcript.done description: user.transcript.done type: send messages: - &ref_63 id: userTranscriptDone contentType: application/json payload: - name: user.transcript.done - 用户转写最终稿 description: 返回本轮用户语音的最终转写。 type: object properties: - name: type type: string description: user.transcript.done examples: &ref_20 - user.transcript.done required: true - name: text type: string examples: &ref_21 - 今天天气怎么样？ required: true headers: [] jsonPayloadSchema: type: object required: - type - text properties: type: type: string const: user.transcript.done examples: *ref_20 x-parser-schema-id: text: type: string examples: *ref_21 x-parser-schema-id: x-parser-schema-id: UserTranscriptDonePayload title: user.transcript.done - 用户转写最终稿 description: 返回本轮用户语音的最终转写。 example: |- { "type": "user.transcript.done", "text": "今天天气怎么样？" } bindings: [] extensions: - id: x-parser-unique-object-id value: userTranscriptDone bindings: [] extensions: *ref_11 - &ref_47 id: sendAssistantTextDelta title: assistant.text.delta description: assistant.text.delta type: send messages: - &ref_64 id: assistantTextDelta contentType: application/json payload: - name: assistant.text.delta - 模型文本增量 description: 返回模型回复文本的增量片段。 type: object properties: - name: type type: string description: assistant.text.delta examples: &ref_22 - assistant.text.delta required: true - name: text type: string examples: &ref_23 - 今天 required: true headers: [] jsonPayloadSchema: type: object required: - type - text properties: type: type: string const: assistant.text.delta examples: *ref_22 x-parser-schema-id: text: type: string examples: *ref_23 x-parser-schema-id: x-parser-schema-id: AssistantTextDeltaPayload title: assistant.text.delta - 模型文本增量 description: 返回模型回复文本的增量片段。 example: |- { "type": "assistant.text.delta", "text": "今天" } bindings: [] extensions: - id: x-parser-unique-object-id value: assistantTextDelta bindings: [] extensions: *ref_11 - &ref_48 id: sendAssistantTextDone title: assistant.text.done description: assistant.text.done type: send messages: - &ref_65 id: assistantTextDone contentType: application/json payload: - name: assistant.text.done - 模型文本全文 description: 返回本轮模型回复文本全文。 type: object properties: - name: type type: string description: assistant.text.done examples: &ref_24 - assistant.text.done required: true - name: text type: string examples: &ref_25 - 今天的天气情况可以打开天气应用查看。如果你告诉我所在城市，我可以帮你整理出行建议。 required: true headers: [] jsonPayloadSchema: type: object required: - type - text properties: type: type: string const: assistant.text.done examples: *ref_24 x-parser-schema-id: text: type: string examples: *ref_25 x-parser-schema-id: x-parser-schema-id: AssistantTextDonePayload title: assistant.text.done - 模型文本全文 description: 返回本轮模型回复文本全文。 example: |- { "type": "assistant.text.done", "text": "今天的天气情况可以打开天气应用查看。" } bindings: [] extensions: - id: x-parser-unique-object-id value: assistantTextDone bindings: [] extensions: *ref_11 - &ref_49 id: sendAssistantAudioStart title: assistant.audio.start description: assistant.audio.start type: send messages: - &ref_66 id: assistantAudioStart contentType: application/json payload: - name: assistant.audio.start - 模型音频开始 description: 后续二进制帧为本轮模型回复音频。 type: object properties: - name: type type: string description: assistant.audio.start examples: &ref_26 - assistant.audio.start required: true - name: response_id type: string examples: &ref_27 - tts_3 required: true - name: sample_rate type: integer description: 24000 examples: &ref_28 - 24000 required: true - name: format type: string description: pcm_s16le examples: &ref_29 - pcm_s16le required: true headers: [] jsonPayloadSchema: type: object required: - type - response_id - sample_rate - format properties: type: type: string const: assistant.audio.start examples: *ref_26 x-parser-schema-id: response_id: type: string examples: *ref_27 x-parser-schema-id: sample_rate: type: integer const: 24000 examples: *ref_28 x-parser-schema-id: format: type: string const: pcm_s16le examples: *ref_29 x-parser-schema-id: x-parser-schema-id: AssistantAudioStartPayload title: assistant.audio.start - 模型音频开始 description: 后续二进制帧为本轮模型回复音频。 example: |- { "type": "assistant.audio.start", "response_id": "tts_3", "sample_rate": 24000, "format": "pcm_s16le" } bindings: [] extensions: - id: x-parser-unique-object-id value: assistantAudioStart bindings: [] extensions: *ref_11 - &ref_50 id: sendAssistantAudioDone title: assistant.audio.done description: assistant.audio.done type: send messages: - &ref_67 id: assistantAudioDone contentType: application/json payload: - name: assistant.audio.done - 模型音频结束 description: 本轮模型回复音频发送完成。 type: object properties: - name: type type: string description: assistant.audio.done examples: &ref_30 - assistant.audio.done required: true - name: response_id type: string examples: &ref_31 - tts_3 required: true headers: [] jsonPayloadSchema: type: object required: - type - response_id properties: type: type: string const: assistant.audio.done examples: *ref_30 x-parser-schema-id: response_id: type: string examples: *ref_31 x-parser-schema-id: x-parser-schema-id: AssistantAudioDonePayload title: assistant.audio.done - 模型音频结束 description: 本轮模型回复音频发送完成。 example: |- { "type": "assistant.audio.done", "response_id": "tts_3" } bindings: [] extensions: - id: x-parser-unique-object-id value: assistantAudioDone bindings: [] extensions: *ref_11 - &ref_51 id: sendTurnDone title: turn.done description: turn.done type: send messages: - &ref_68 id: turnDone contentType: application/json payload: - name: turn.done - 轮次结束 description: 一个开场白或用户语音轮次结束。 type: object properties: - name: type type: string description: turn.done examples: &ref_32 - turn.done required: true headers: [] jsonPayloadSchema: type: object required: - type properties: type: type: string const: turn.done examples: *ref_32 x-parser-schema-id: x-parser-schema-id: TurnDonePayload title: turn.done - 轮次结束 description: 一个开场白或用户语音轮次结束。 example: |- { "type": "turn.done" } bindings: [] extensions: - id: x-parser-unique-object-id value: turnDone bindings: [] extensions: *ref_11 - &ref_52 id: sendError title: error - 错误通知 description: error type: send messages: - &ref_69 id: error contentType: application/json payload: - name: error - 错误通知 description: 返回致命或非致命错误。 type: object properties: - name: type type: string description: error examples: &ref_33 - error required: true - name: code type: string enumValues: - bad_request - model_not_found - insufficient_funds - internal_error - fatal - empty_audio_buffer - no_active_response - invalid_voice examples: &ref_34 - bad_request required: true - name: message type: string examples: &ref_35 - 客户端发送的 start 消息配置无效 required: true headers: [] jsonPayloadSchema: type: object required: - type - code - message properties: type: type: string const: error examples: *ref_33 x-parser-schema-id: code: type: string enum: - bad_request - model_not_found - insufficient_funds - internal_error - fatal - empty_audio_buffer - no_active_response - invalid_voice examples: *ref_34 x-parser-schema-id: message: type: string examples: *ref_35 x-parser-schema-id: x-parser-schema-id: ErrorPayload title: error - 错误通知 description: 返回致命或非致命错误。 example: |- { "type": "error", "code": "bad_request", "message": "客户端发送的 start 消息配置无效" } bindings: [] extensions: - id: x-parser-unique-object-id value: error bindings: [] extensions: *ref_11 - &ref_42 id: receiveClientAudioFrame title: 客户端 PCM 音频帧 description: 客户端 PCM 音频帧 type: receive messages: - &ref_59 id: pcmAudioFrame contentType: application/octet-stream payload: - type: string format: binary description: 原始 WebSocket Binary Message，不是 JSON 或 Base64 文本。 x-parser-schema-id: name: PCM 音频二进制帧 headers: [] jsonPayloadSchema: &ref_36 type: string format: binary description: 线路上传输原始二进制 PCM；下行帧按 assistant.audio.start 的参数解析。 x-parser-schema-id: title: PCM 音频二进制帧 description: 原始 WebSocket Binary Message，不是 JSON 或 Base64 文本。 example: '{}' bindings: [] extensions: - id: x-parser-unique-object-id value: pcmAudioFrame bindings: [] extensions: *ref_11 - &ref_53 id: sendAssistantAudioFrame title: 模型回复 PCM 音频帧 description: 模型回复 PCM 音频帧 type: send messages: - &ref_70 id: pcmAudioFrame contentType: application/octet-stream payload: - type: string format: binary description: 原始 WebSocket Binary Message，不是 JSON 或 Base64 文本。 x-parser-schema-id: name: PCM 音频二进制帧 headers: [] jsonPayloadSchema: *ref_36 title: PCM 音频二进制帧 description: 原始 WebSocket Binary Message，不是 JSON 或 Base64 文本。 example: '{}' bindings: [] extensions: - id: x-parser-unique-object-id value: pcmAudioFrame bindings: [] extensions: *ref_11 sendOperations: - *ref_37 - *ref_38 - *ref_39 - *ref_40 - *ref_41 - *ref_42 receiveOperations: - *ref_43 - *ref_44 - *ref_45 - *ref_46 - *ref_47 - *ref_48 - *ref_49 - *ref_50 - *ref_51 - *ref_52 - *ref_53 sendMessages: - *ref_54 - *ref_55 - *ref_56 - *ref_57 - *ref_58 - *ref_59 receiveMessages: - *ref_60 - *ref_61 - *ref_62 - *ref_63 - *ref_64 - *ref_65 - *ref_66 - *ref_67 - *ref_68 - *ref_69 - *ref_70 extensions: - id: x-parser-unique-object-id value: wss securitySchemes: - id: bearerAuth name: bearerAuth type: http description: 'Bearer Token 鉴权，格式 `Authorization: Bearer `' scheme: bearer extensions: [] ````