语音转文本 API¶

转录 API¶

我们的转录 API 与 OpenAI 的转录 API 兼容；您可以使用 OpenAI 官方 Python 客户端与其进行交互。

注意

要使用转录 API，请使用 pip install vllm[audio] 安装额外的音频依赖项。

代码示例： examples/speech_to_text/openai/openai_transcription_client.py

注意：转录端点目前支持编码器-解码器多模态模型（例如 whisper）的束搜索（beam search），但由于处理编码器/解码器缓存的工作正在积极进行中，因此效率非常低。这是持续优化的重点，将在不久的将来得到妥善处理。

API 强制限制¶

通过 VLLM_MAX_AUDIO_CLIP_FILESIZE_MB 环境变量设置 vLLM 将接受的最大音频文件大小（以 MB 为单位）。默认值为 25 MB。

上传音频文件¶

转录 API 支持上传各种格式的音频文件，包括 FLAC, MP3, MP4, MPEG, MPGA, M4A, OGG, WAV, 和 WEBM。

使用 OpenAI Python 客户端

代码

from openai import OpenAI

client = OpenAI(
    base_url="https://:8000/v1",
    api_key="token-abc123",
)

# Upload audio file from disk
with open("audio.mp3", "rb") as audio_file:
    transcription = client.audio.transcriptions.create(
        model="openai/whisper-large-v3-turbo",
        file=audio_file,
        language="en",
        response_format="verbose_json",
    )

print(transcription.text)

使用带有 multipart/form-data 的 curl

代码

curl -X POST "https://:8000/v1/audio/transcriptions" \
  -H "Authorization: Bearer token-abc123" \
  -F "[email protected]" \
  -F "model=openai/whisper-large-v3-turbo" \
  -F "language=en" \
  -F "response_format=verbose_json"

支持的参数

file：要转录的音频文件（必填）
model：用于转录的模型（必填）
language：语言代码（例如 "en", "zh"）（可选）
prompt：可选的文本，用于引导转录风格（可选）
response_format：响应的格式（"json", "text"）（可选）
temperature：采样温度，介于 0 和 1 之间（可选）

有关支持参数的完整列表（包括采样参数和 vLLM 扩展），请参阅协议定义。

响应格式

针对 verbose_json 响应格式

代码

{
  "text": "Hello, this is a transcription of the audio file.",
  "language": "en",
  "duration": 5.42,
  "segments": [
    {
      "id": 0,
      "seek": 0,
      "start": 0.0,
      "end": 2.5,
      "text": "Hello, this is a transcription",
      "tokens": [50364, 938, 428, 307, 275, 28347],
      "temperature": 0.0,
      "avg_logprob": -0.245,
      "compression_ratio": 1.235,
      "no_speech_prob": 0.012
    }
  ]
}

目前 “verbose_json” 响应格式不支持 no_speech_prob。

额外参数¶

支持以下采样参数。

代码

    use_beam_search: bool = False
    """Whether or not beam search should be used."""

    n: int = 1
    """The number of beams to be used in beam search."""

    length_penalty: float = 1.0
    """Length penalty to be used for beam search."""

    include_stop_str_in_output: bool = False
    """Whether to include the stop strings in output text."""

    temperature: float = Field(default=0.0)
    """The sampling temperature, between 0 and 1.

    Higher values like 0.8 will make the output more random, while lower values
    like 0.2 will make it more focused / deterministic. If set to 0, the model
    will use [log probability](https://en.wikipedia.org/wiki/Log_probability)
    to automatically increase the temperature until certain thresholds are hit.
    """

    top_p: float | None = None
    """Enables nucleus (top-p) sampling, where tokens are selected from the
    smallest possible set whose cumulative probability exceeds `p`.
    """

    top_k: int | None = None
    """Limits sampling to the `k` most probable tokens at each step."""

    min_p: float | None = None
    """Filters out tokens with a probability lower than `min_p`, ensuring a
    minimum likelihood threshold during sampling.
    """

    seed: int | None = Field(None, ge=_LONG_INFO.min, le=_LONG_INFO.max)
    """The seed to use for sampling."""

    frequency_penalty: float | None = 0.0
    """The frequency penalty to use for sampling."""

    repetition_penalty: float | None = None
    """The repetition penalty to use for sampling."""

    presence_penalty: float | None = 0.0
    """The presence penalty to use for sampling."""

    max_completion_tokens: int | None = None
    """The maximum number of tokens to generate."""

支持以下额外参数

代码

    # Flattened stream option to simplify form data.
    stream_include_usage: bool | None = False
    stream_continuous_usage_stats: bool | None = False

    vllm_xargs: dict[str, str | int | float | bool] | None = Field(
        default=None,
        description=(
            "Additional request parameters with string or "
            "numeric values, used by custom extensions."
        ),
    )

翻译 API¶

我们的翻译 API 与 OpenAI 的翻译 API 兼容；您可以使用官方 OpenAI Python 客户端与其交互。Whisper 模型可以将 55 种支持的非英语语言之一的音频翻译成英语。请注意，流行的 openai/whisper-large-v3-turbo 模型不支持翻译。

注意

要使用翻译 API，请使用 pip install vllm[audio] 安装额外的音频依赖项。

代码示例： examples/speech_to_text/openai/openai_translation_client.py

额外参数¶

支持以下采样参数。

    use_beam_search: bool = False
    """Whether or not beam search should be used."""

    n: int = 1
    """The number of beams to be used in beam search."""

    length_penalty: float = 1.0
    """Length penalty to be used for beam search."""

    include_stop_str_in_output: bool = False
    """Whether to include the stop strings in output text."""

    seed: int | None = Field(None, ge=_LONG_INFO.min, le=_LONG_INFO.max)
    """The seed to use for sampling."""

    temperature: float = Field(default=0.0)
    """The sampling temperature, between 0 and 1.

    Higher values like 0.8 will make the output more random, while lower values
    like 0.2 will make it more focused / deterministic. If set to 0, the model
    will use [log probability](https://en.wikipedia.org/wiki/Log_probability)
    to automatically increase the temperature until certain thresholds are hit.
    """

支持以下额外参数

    language: str | None = None
    """The language of the input audio we translate from.

    Supplying the input language in
    [ISO-639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) format
    will improve accuracy.
    """

    hotwords: str | None = None
    """
    hotwords refers to a list of important words or phrases that the model
    should pay extra attention to during transcription.
    """

    to_language: str | None = None
    """The language of the input audio we translate to.

    Please note that this is not supported by all models, refer to the specific
    model documentation for more details.
    For instance, Whisper only supports `to_language=en`.
    """

    stream: bool | None = False
    """Custom field not present in the original OpenAI definition. When set,
    it will enable output to be streamed in a similar fashion as the Chat
    Completion endpoint.
    """
    # Flattened stream option to simplify form data.
    stream_include_usage: bool | None = False
    stream_continuous_usage_stats: bool | None = False

    max_completion_tokens: int | None = None
    """The maximum number of tokens to generate."""

实时 API¶

实时 API 提供基于 WebSocket 的流式音频转录，允许在音频录制时进行实时语音转文本。

注意

要使用实时 API，请使用 uv pip install vllm[audio] 安装额外的音频依赖项。

音频格式¶

音频必须以 base64 编码的 PCM16 音频形式发送，采样率为 16kHz，单声道。

协议概览¶

客户端连接到 ws://host/v1/realtime
服务器发送 session.created 事件
客户端（可选）发送带有模型/参数的 session.update
准备就绪后，客户端发送 input_audio_buffer.commit
客户端发送带有 base64 PCM16 分块的 input_audio_buffer.append 事件
服务器发送带有增量文本的 transcription.delta 事件
服务器发送带有最终文本 + 使用情况的 transcription.done
从第 5 步开始重复下一次话语
（可选）客户端发送带有 final=True 的 input_audio_buffer.commit，以发出音频输入完成的信号。在流式传输音频文件时非常有用。

客户端 → 服务器事件¶

事件	描述
`input_audio_buffer.append`	发送 base64 编码的音频分块：`{"type": "input_audio_buffer.append", "audio": "<base64>"}`
`input_audio_buffer.commit`	触发转录处理或结束：`{"type": "input_audio_buffer.commit", "final": bool}`
`session.update`	配置会话：`{"type": "session.update", "model": "model-name"}`

服务器 → 客户端事件¶

事件	描述
`session.created`	已建立连接，包含会话 ID 和时间戳
`transcription.delta`	增量转录文本：`{"type": "transcription.delta", "delta": "text"}`
`transcription.done`	带有使用统计信息的最终转录
`error`	带有消息和可选代码的错误通知

示例客户端¶

openai_realtime_client.py - 上传并转录音频文件
openai_realtime_microphone_client.py - 实时麦克风转录的 Gradio 演示