跳转到内容
本站为社区维护的非官方中文翻译,与 Cloudflare, Inc. 无关联。本站不提供账户登录,请勿在任何非官方站点输入您的 Cloudflare 账户信息。官方文档:developers.cloudflare.com

Voice Agent 示例

最后更新 查看 MarkdownAgent 设置

构建 voice agent:聆听用户、用 LLM 思考并实时通过 WebSocket 语音回复。 Beta

本指南结束时你将拥有:

  • 带 speech-to-text 与 text-to-speech 的服务端 voice agent
  • 流式响应的 LLM 驱动 onTurn handler
  • 对话期间 agent 可调用的 tool
  • 带 push-to-talk 风格 UI 的 React client

前置条件

  • 具备 Workers AI 访问权限的 Cloudflare 账户
  • Node.js 18+

1. 创建项目

使用 Vite 和 React 为 Workers 项目搭建脚手架,然后将 agents 更新到包含 agents/voice 的版本:

npm create cloudflare@latest voice-agent -- --template cloudflare/agents-starter
cd voice-agent
npm install agents@latest

Starter 提供可用的 Vite + React + Cloudflare Workers 设置。后续步骤将替换 server 与 client 代码。

2. 配置 wrangler

更新 wrangler.jsonc,包含 Workers AI binding 与 voice agent 的 Durable Object:

{
    "name": "voice-agent",
    // Set this to today's date
    "compatibility_date": "2026-11-20",
    "compatibility_flags": ["nodejs_compat"],
    "main": "src/server.ts",
    "upload_source_maps": true,
    "ai": {
        "binding": "AI"
    },
    "durable_objects": {
        "bindings": [
            {
                "name": "MyVoiceAgent",
                "class_name": "MyVoiceAgent"
            }
        ]
    },
    "migrations": [
        {
            "tag": "v1",
            "new_sqlite_classes": ["MyVoiceAgent"]
        }
    ]
}
name = "voice-agent"
# Set this to today's date
compatibility_date = "2026-11-20"
compatibility_flags = [ "nodejs_compat" ]
main = "src/server.ts"
upload_source_maps = true

[ai]
binding = "AI"

[[durable_objects.bindings]]
name = "MyVoiceAgent"
class_name = "MyVoiceAgent"

[[migrations]]
tag = "v1"
new_sqlite_classes = [ "MyVoiceAgent" ]

3. 构建 server

将 src/server.ts 替换为以下内容。withVoice mixin 为 standard Agent 类添加完整 voice pipeline — STT、句子分块、TTS 与对话持久化。

import { Agent, routeAgentRequest } from "agents";
import { withVoice, WorkersAIFluxSTT, WorkersAITTS } from "agents/voice";
import { streamText, tool, stepCountIs } from "ai";
import { createAI } from "agents/models/ai-sdk";
import { z } from "zod";

const VoiceAgent = withVoice(Agent);

export class MyVoiceAgent extends VoiceAgent {
	transcriber = new WorkersAIFluxSTT(this.env.AI);
	tts = new WorkersAITTS(this.env.AI);

	async onTurn(transcript, context) {
		const ai = createAI({ binding: this.env.AI });

		const result = streamText({
			model: ai("@cf/moonshotai/kimi-k2.6"),
			system:
				"You are a helpful voice assistant. Keep responses concise — you are being spoken aloud.",
			messages: [
				...context.messages.map((m) => ({
					role: m.role,
					content: m.content,
				})),
				{ role: "user", content: transcript },
			],
			tools: {
				get_current_time: tool({
					description: "Get the current date and time.",
					inputSchema: z.object({}),
					execute: async () => ({
						time: new Date().toLocaleTimeString("en-US", {
							hour: "2-digit",
							minute: "2-digit",
						}),
					}),
				}),
			},
			stopWhen: stepCountIs(3),
			abortSignal: context.signal,
		});

		return result.textStream;
	}

	async onCallStart(connection) {
		await this.speak(connection, "Hi there! How can I help you today?");
	}
}

export default {
	async fetch(request, env) {
		return (
			(await routeAgentRequest(request, env)) ??
			new Response("Not found", { status: 404 })
		);
	},
};
import { Agent, routeAgentRequest, type Connection } from "agents";
import {
    withVoice,
    WorkersAIFluxSTT,
    WorkersAITTS,
    type VoiceTurnContext,
} from "agents/voice";
import { streamText, tool, stepCountIs } from "ai";
import { createAI } from "agents/models/ai-sdk";
import { z } from "zod";

const VoiceAgent = withVoice(Agent);

export class MyVoiceAgent extends VoiceAgent<Env> {
    transcriber = new WorkersAIFluxSTT(this.env.AI);
    tts = new WorkersAITTS(this.env.AI);

    async onTurn(transcript: string, context: VoiceTurnContext) {
        const ai = createAI({ binding: this.env.AI });

        const result = streamText({
            model: ai("@cf/moonshotai/kimi-k2.6"),
            system:
                "You are a helpful voice assistant. Keep responses concise — you are being spoken aloud.",
            messages: [
                ...context.messages.map((m) => ({
                    role: m.role as "user" | "assistant",
                    content: m.content,
                })),
                { role: "user" as const, content: transcript },
            ],
            tools: {
                get_current_time: tool({
                    description: "Get the current date and time.",
                    inputSchema: z.object({}),
                    execute: async () => ({
                        time: new Date().toLocaleTimeString("en-US", {
                            hour: "2-digit",
                            minute: "2-digit",
                        }),
                    }),
                }),
            },
            stopWhen: stepCountIs(3),
            abortSignal: context.signal,
        });

        return result.textStream;
    }

    async onCallStart(connection: Connection) {
        await this.speak(connection, "Hi there! How can I help you today?");
    }
}

export default {
    async fetch(request: Request, env: Env) {
        return (
            (await routeAgentRequest(request, env)) ??
            new Response("Not found", { status: 404 })
        );
    },
} satisfies ExportedHandler<Env>;

要点:

  • WorkersAIFluxSTT 处理 continuous speech-to-text — model 检测用户何时说完。
  • WorkersAITTS 将 LLM 响应逐句转为 audio。
  • onTurn 接收 transcript 并返回 stream。Mixin 将 stream 分句并合成每一句。
  • onCallStart 在用户连接时发送问候。
  • context.messages 包含当前转录内容之前已完成的对话历史。
  • 用户 interrupt 或 disconnect 时 context.signal 会被 abort。

4. 构建 client

将 src/client.tsx 替换为使用 useVoiceAgent hook 的 React 组件。Hook 管理 WebSocket 连接、mic 采集、audio 播放与 interrupt 检测。

import { useVoiceAgent } from "agents/voice/react";

function App() {
    const {
        status,
        transcript,
        interimTranscript,
        metrics,
        audioLevel,
        isMuted,
        startCall,
        endCall,
        toggleMute,
    } = useVoiceAgent({ agent: "MyVoiceAgent" });

    return (
        <div>
            <h1>Voice Agent</h1>
            <p>Status: {status}</p>

            <div>
                <button onClick={status === "idle" ? startCall : endCall}>
                    {status === "idle" ? "Start Call" : "End Call"}
                </button>
                {status !== "idle" && (
                    <button onClick={toggleMute}>{isMuted ? "Unmute" : "Mute"}</button>
                )}
            </div>

            {interimTranscript && (
                <p>
                    <em>{interimTranscript}</em>
                </p>
            )}

            {transcript.map((msg, i) => (
                <p key={i}>
                    <strong>{msg.role}:</strong> {msg.text}
                </p>
            ))}

            {metrics && (
                <p>
                    LLM: {metrics.llm_ms}ms | TTS: {metrics.tts_ms}ms | First audio:{" "}
                    {metrics.first_audio_ms}ms
                </p>
            )}
        </div>
    );
}

status 字段循环 "idle" → "listening" → "thinking" → "speaking" → "listening",足以构建响应式 UI。

5. 运行

npm run dev

在浏览器打开应用,选择 Start Call(开始通话) 并说话。你将实时看到 transcript,agent 回复会通过扬声器播放。

添加 pipeline hook

可在 pipeline 各阶段 intercept 并 transform 数据。例如过滤短 transcript(噪声)并在 TTS 前调整发音:

export class MyVoiceAgent extends VoiceAgent {
	transcriber = new WorkersAIFluxSTT(this.env.AI);
	tts = new WorkersAITTS(this.env.AI);

	afterTranscribe(transcript, connection) {
		if (transcript.length < 3) return null;
		return transcript;
	}

	beforeSynthesize(text, connection) {
		return text.replace(/\bAI\b/g, "A.I.");
	}

	async onTurn(transcript, context) {
		return "You said: " + transcript;
	}
}
export class MyVoiceAgent extends VoiceAgent<Env> {
    transcriber = new WorkersAIFluxSTT(this.env.AI);
    tts = new WorkersAITTS(this.env.AI);

    afterTranscribe(transcript: string, connection: Connection) {
        if (transcript.length < 3) return null;
        return transcript;
    }

    beforeSynthesize(text: string, connection: Connection) {
        return text.replace(/\bAI\b/g, "A.I.");
    }

    async onTurn(transcript: string, context: VoiceTurnContext) {
        return "You said: " + transcript;
    }
}

从 afterTranscribe 返回 null 会完全丢弃该 utterance — 适用于过滤噪声或过短 transcript。

使用第三方 provider

更换第三方 STT 或 TTS provider 而无需改动 agent 逻辑:

import { ElevenLabsTTS } from "@cloudflare/voice-elevenlabs";
import { DeepgramSTT } from "@cloudflare/voice-deepgram";

export class MyVoiceAgent extends VoiceAgent {
	transcriber = new DeepgramSTT({
		apiKey: this.env.DEEPGRAM_API_KEY,
	});

	tts = new ElevenLabsTTS({
		apiKey: this.env.ELEVENLABS_API_KEY,
		voiceId: "21m00Tcm4TlvDq8ikWAM",
	});

	async onTurn(transcript, context) {
		return "You said: " + transcript;
	}
}
import { ElevenLabsTTS } from "@cloudflare/voice-elevenlabs";
import { DeepgramSTT } from "@cloudflare/voice-deepgram";

export class MyVoiceAgent extends VoiceAgent<Env> {
    transcriber = new DeepgramSTT({
        apiKey: this.env.DEEPGRAM_API_KEY,
    });

    tts = new ElevenLabsTTS({
        apiKey: this.env.ELEVENLABS_API_KEY,
        voiceId: "21m00Tcm4TlvDq8ikWAM",
    });

    async onTurn(transcript: string, context: VoiceTurnContext) {
        return "You said: " + transcript;
    }
}

后续步骤

Voice Agent API 参考

withVoice、withVoiceInput、React hook、VoiceClient 与所有 provider 的完整参考。

聊天 Agent

用 AIChatAgent 与 useAgentChat 构建文本 AI 聊天。

模型

在 agent 中使用 Workers AI、OpenAI、Anthropic、Gemini 或任意 provider。

这篇文档对您有帮助吗?