OpenAI 兼容服务器¶
vLLM 提供了一个 HTTP 服务器，实现了 OpenAI 的 Completions API、Chat API 等更多功能！此功能允许您使用 HTTP 客户端来提供模型服务并与之交互。
支持的 API¶
我们目前支持以下 OpenAI API
- Completions API (
/v1/completions)- 仅适用于 文本生成模型。
- 注意：
suffix参数不支持。
- Chat Completions API (
/v1/chat/completions) - Chat Completions batch API (
/v1/chat/completions/batch) - Responses API (
/v1/responses,/v1/responses/{response_id},/v1/responses/{response_id}/cancel)- 仅适用于 文本生成模型。
- Embeddings API (
/v1/embeddings)- 仅适用于 嵌入模型。
- Transcriptions API (
/v1/audio/transcriptions)- 仅适用于 自动语音识别 (ASR) 模型。
- Translation API (
/v1/audio/translations)- 仅适用于 自动语音识别 (ASR) 模型。
Completions API¶
在您的终端中，您可以 安装 vLLM，然后使用 vllm serve 命令启动服务器。（您也可以使用我们的 Docker 镜像。）
要调用服务器，请在您喜欢的文本编辑器中创建一个使用 HTTP 客户端的脚本。包含您想要发送给模型的任何消息。然后运行该脚本。下面是使用 官方 OpenAI Python 客户端 的示例脚本。
代码
提示
vLLM 支持一些 OpenAI 不支持的参数，例如 top_k。您可以通过 OpenAI 客户端在请求的 extra_body 参数中将这些参数传递给 vLLM，例如，对于 top_k，设置为 extra_body={"top_k": 50}。
重要
默认情况下，如果 Hugging Face 模型仓库中存在 generation_config.json，服务器会应用它。这意味着某些采样参数的默认值可以被模型创建者推荐的值覆盖。
要禁用此行为，请在启动服务器时传入 --generation-config vllm。
额外参数¶
vLLM 支持一组不属于 OpenAI API 的参数。为了使用它们，您可以在 OpenAI 客户端中将它们作为额外参数传递。或者，如果您直接使用 HTTP 调用，则可以将其直接合并到 JSON 有效负载中。
completion = client.chat.completions.create(
model="NousResearch/Meta-Llama-3-8B-Instruct",
messages=[
{"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"},
],
extra_body={
"structured_outputs": {"choice": ["positive", "negative"]},
},
)
额外 HTTP 头¶
目前只支持 X-Request-Id HTTP 请求头。可以通过 --enable-request-id-headers 启用它。
代码
completion = client.chat.completions.create(
model="NousResearch/Meta-Llama-3-8B-Instruct",
messages=[
{"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"},
],
extra_headers={
"x-request-id": "sentiment-classification-00001",
},
)
print(completion._request_id)
completion = client.completions.create(
model="NousResearch/Meta-Llama-3-8B-Instruct",
prompt="A robot may not injure a human being",
extra_headers={
"x-request-id": "completion-test",
},
)
print(completion._request_id)
API 参考¶
Completions API¶
我们的 Completions API 与 OpenAI 的 Completions API 兼容；您可以使用 官方 OpenAI Python 客户端 与之交互。
代码示例： examples/basic/online_serving/openai_completion_client.py
额外参数¶
支持以下 采样参数。
代码
use_beam_search: bool = False
top_k: int | None = None
min_p: float | None = None
repetition_penalty: float | None = None
length_penalty: float = 1.0
stop_token_ids: list[int] | None = []
include_stop_str_in_output: bool = False
ignore_eos: bool = False
min_tokens: int = 0
skip_special_tokens: bool = True
spaces_between_special_tokens: bool = True
truncate_prompt_tokens: Annotated[int, Field(ge=-1, le=_INT64_MAX)] | None = None
truncation_side: Literal["left", "right"] | None = Field(
default=None,
description=(
"Which side to truncate from when truncate_prompt_tokens is active. "
"'right' keeps the first N tokens. "
"'left' keeps the last N tokens."
),
)
allowed_token_ids: list[int] | None = None
prompt_logprobs: int | None = None
logprob_token_ids: list[int] | None = Field(
default=None,
description=(
"Specific vocab token IDs to return logprobs for at each generated "
"position, in addition to the sampled token. More efficient than "
"requesting the full vocab when only a small fixed label set is "
"needed (e.g. multilabel "
"scoring where each label corresponds to a known vocab id). When "
"set, this explicit token selection takes precedence over the "
"natural top-k selected by `logprobs`. Requires `logprobs` to be "
"set."
),
)
bad_words: list[str] = Field(default_factory=list)
支持以下额外参数
代码
prompt_embeds: bytes | list[bytes] | None = None
add_special_tokens: bool = Field(
default=True,
description=(
"If true (the default), special tokens (e.g. BOS) will be added to "
"the prompt."
),
)
response_format: AnyResponseFormat | None = Field(
default=None,
description=(
"Similar to chat completion, this parameter specifies the format "
"of output. Only {'type': 'json_object'}, {'type': 'json_schema'}"
", {'type': 'structural_tag'}, or {'type': 'text' } is supported."
),
)
structured_outputs: StructuredOutputsParams | None = Field(
default=None,
description="Additional kwargs for structured outputs",
)
priority: int = Field(
default=0,
ge=_INT64_MIN,
le=_INT64_MAX,
description=(
"The priority of the request (lower means earlier handling; "
"default: 0). Any priority other than 0 will raise an error "
"if the served model does not use priority scheduling."
),
)
request_id: str = Field(
default_factory=random_uuid,
description=(
"The request_id related to this request. If the caller does "
"not set it, a random_uuid will be generated. This id is used "
"through out the inference process and return in response."
),
)
return_tokens_as_token_ids: bool | None = Field(
default=None,
description=(
"If specified with 'logprobs', tokens are represented "
" as strings of the form 'token_id:{token_id}' so that tokens "
"that are not JSON-encodable can be identified."
),
)
return_token_ids: bool | None = Field(
default=None,
description=(
"If specified, the result will include token IDs alongside the "
"generated text. In streaming mode, prompt_token_ids is included "
"only in the first chunk, and token_ids contains the delta tokens "
"for each chunk. This is useful for debugging or when you "
"need to map generated text back to input tokens."
),
)
return_token_offsets: bool | None = Field(
default=False,
description=(
"If true, return char-level (start, end) offsets for each "
"token relative to the tokenized source string in the "
"`token_offsets` field of the rendered response. Only "
"supported on the `/v1/completions/render` and "
"`/v1/chat/completions/render` endpoints; ignored on regular "
"generation endpoints. Honored only for Fast (Rust-backed) "
"tokenizers; otherwise `token_offsets` is null. For chat "
"requests, offsets are relative to the templated prompt "
"string (after applying the chat template). Multimodal "
"inputs and pre-tokenized inputs always yield null."
),
)
cache_salt: str | None = Field(
default=None,
description=(
"If specified, the prefix cache will be salted with the provided "
"string to prevent an attacker to guess prompts in multi-user "
"environments. The salt should be random, protected from "
"access by 3rd parties, and long enough to be "
"unpredictable (e.g., 43 characters base64-encoded, corresponding "
"to 256 bit)."
),
)
kv_transfer_params: dict[str, Any] | None = Field(
default=None,
description="KVTransfer parameters used for disaggregated serving.",
)
ec_transfer_params: dict[str, Any] | None = Field(
default=None,
description=(
"ECTransfer parameters used for encoder-cache disaggregated serving."
),
)
vllm_xargs: dict[str, str | int | float] | None = Field(
default=None,
description=(
"Additional request parameters with string or "
"numeric values, used by custom extensions."
),
)
repetition_detection: RepetitionDetectionParams | None = Field(
default=None,
description="Parameters for detecting repetitive N-gram patterns "
"in output tokens. If such repetition is detected, generation will "
"be ended early. LLMs can sometimes generate repetitive, unhelpful "
"token patterns, stopping only when they hit the maximum output length "
"(e.g. 'abcdabcdabcd...' or '\\emoji \\emoji \\emoji ...'). This feature "
"can detect such behavior and terminate early, saving time and tokens.",
)
thinking_token_budget: ThinkingTokenBudget = Field(
default=None,
description=(
"Maximum number of tokens allowed for thinking operations "
"(reasoning models). Non-negative integer sets the limit; "
"-1 means unlimited (treated as unset)."
),
)
聊天 API¶
我们的聊天 API 与 OpenAI 的聊天 Completions API 兼容；您可以使用 官方 OpenAI Python 客户端 与之交互。
我们同时支持 视觉 和 音频 相关的参数；请参阅我们的 多模态输入 指南以获取更多信息。
- 注意：
image_url.detail参数不支持。
代码示例： examples/basic/online_serving/openai_chat_completion_client.py
额外参数¶
支持以下 采样参数。
代码
use_beam_search: bool = False
top_k: int | None = None
min_p: float | None = None
repetition_penalty: float | None = None
length_penalty: float = 1.0
stop_token_ids: list[int] | None = []
include_stop_str_in_output: bool = False
ignore_eos: bool = False
min_tokens: int = 0
skip_special_tokens: bool = True
spaces_between_special_tokens: bool = True
truncate_prompt_tokens: Annotated[int, Field(ge=-1, le=_INT64_MAX)] | None = None
truncation_side: Literal["left", "right"] | None = Field(
default=None,
description=(
"Which side to truncate from when truncate_prompt_tokens is active. "
"'right' keeps the first N tokens. "
"'left' keeps the last N tokens."
),
)
prompt_logprobs: int | None = None
logprob_token_ids: list[int] | None = Field(
default=None,
description=(
"Specific vocab token IDs to return logprobs for at each generated "
"position, in addition to the sampled token. More efficient than "
"`top_logprobs=-1` when only a small fixed label set is needed "
"(e.g. multilabel scoring "
"where each label corresponds to a known vocab id). When set, "
"this explicit token selection takes precedence over the natural "
"top-k selected by `top_logprobs`. Requires `logprobs=True`."
),
)
allowed_token_ids: list[int] | None = None
bad_words: list[str] = Field(default_factory=list)
支持以下额外参数
代码
echo: bool = Field(
default=False,
description=(
"If true, the new message will be prepended with the last message "
"if they belong to the same role."
),
)
add_generation_prompt: bool = Field(
default=True,
description=(
"If true, the generation prompt will be added to the chat template. "
"This is a parameter used by chat template in tokenizer config of the "
"model."
),
)
continue_final_message: bool = Field(
default=False,
description=(
"If this is set, the chat will be formatted so that the final "
"message in the chat is open-ended, without any EOS tokens. The "
"model will continue this message rather than starting a new one. "
'This allows you to "prefill" part of the model\'s response for it. '
"Cannot be used at the same time as `add_generation_prompt`."
),
)
add_special_tokens: bool = Field(
default=False,
description=(
"If true, special tokens (e.g. BOS) will be added to the prompt "
"on top of what is added by the chat template. "
"For most models, the chat template takes care of adding the "
"special tokens so this should be set to false (as is the "
"default)."
),
)
documents: list[dict[str, str]] | None = Field(
default=None,
description=(
"A list of dicts representing documents that will be accessible to "
"the model if it is performing RAG (retrieval-augmented generation)."
" If the template does not support RAG, this argument will have no "
"effect. We recommend that each document should be a dict containing "
'"title" and "text" keys.'
),
)
chat_template: str | None = Field(
default=None,
description=(
"A Jinja template to use for this conversion. "
"As of transformers v4.44, default chat template is no longer "
"allowed, so you must provide a chat template if the tokenizer "
"does not define one."
),
)
chat_template_kwargs: dict[str, Any] | None = Field(
default=None,
description=(
"Additional keyword args to pass to the template renderer. "
"Will be accessible by the chat template."
),
)
media_io_kwargs: dict[str, dict[str, Any]] | None = Field(
default=None,
description=(
"Additional kwargs to pass to the media IO connectors, "
"keyed by modality. Merged with engine-level media_io_kwargs."
),
)
mm_processor_kwargs: dict[str, Any] | None = Field(
default=None,
description=("Additional kwargs to pass to the HF processor."),
)
structured_outputs: StructuredOutputsParams | None = Field(
default=None,
description="Additional kwargs for structured outputs",
)
priority: int = Field(
default=0,
ge=_INT64_MIN,
le=_INT64_MAX,
description=(
"The priority of the request (lower means earlier handling; "
"default: 0). Any priority other than 0 will raise an error "
"if the served model does not use priority scheduling."
),
)
request_id: str = Field(
default_factory=random_uuid,
description=(
"The request_id related to this request. If the caller does "
"not set it, a random_uuid will be generated. This id is used "
"through out the inference process and return in response."
),
)
return_tokens_as_token_ids: bool | None = Field(
default=None,
description=(
"If specified with 'logprobs', tokens are represented "
" as strings of the form 'token_id:{token_id}' so that tokens "
"that are not JSON-encodable can be identified."
),
)
return_token_ids: bool | None = Field(
default=None,
description=(
"If specified, the result will include token IDs alongside the "
"generated text. In streaming mode, prompt_token_ids is included "
"only in the first chunk, and token_ids contains the delta tokens "
"for each chunk. This is useful for debugging or when you "
"need to map generated text back to input tokens."
),
)
return_token_offsets: bool | None = Field(
default=False,
description=(
"If true, return char-level (start, end) offsets for each "
"token relative to the tokenized source string in the "
"`token_offsets` field of the rendered response. Only "
"supported on the `/v1/completions/render` and "
"`/v1/chat/completions/render` endpoints; ignored on regular "
"generation endpoints. Honored only for Fast (Rust-backed) "
"tokenizers; otherwise `token_offsets` is null. For chat "
"requests, offsets are relative to the templated prompt "
"string (after applying the chat template). Multimodal "
"inputs and pre-tokenized inputs always yield null."
),
)
return_prompt_text: bool | None = Field(
default=None,
description=(
"If true, the response will include ``prompt_text`` containing the "
"prompt string produced by chat templating. In streaming mode it "
"is sent only on the first chunk. This is useful for inspecting "
"exactly what was fed into the model."
),
)
return_assistant_tokens_mask: bool = Field(
default=False,
description=(
"If true, the /render response will include an "
"``assistant_tokens_mask`` field — a per-token list of 0/1 "
"values indicating which tokens were assistant-generated. "
"Requires the chat template to use ``{% generation %}`` "
"tags. When the template does not support it, "
"``assistant_tokens_mask`` will be ``null``."
),
)
cache_salt: str | None = Field(
default=None,
description=(
"If specified, the prefix cache will be salted with the provided "
"string to prevent an attacker to guess prompts in multi-user "
"environments. The salt should be random, protected from "
"access by 3rd parties, and long enough to be "
"unpredictable (e.g., 43 characters base64-encoded, corresponding "
"to 256 bit)."
),
)
kv_transfer_params: dict[str, Any] | None = Field(
default=None,
description="KVTransfer parameters used for disaggregated serving.",
)
ec_transfer_params: dict[str, Any] | None = Field(
default=None,
description=(
"ECTransfer parameters used for encoder-cache disaggregated serving."
),
)
vllm_xargs: dict[str, str | int | float | list[str | int | float]] | None = Field(
default=None,
description=(
"Additional request parameters with (list of) string or "
"numeric values, used by custom extensions."
),
)
repetition_detection: RepetitionDetectionParams | None = Field(
default=None,
description="Parameters for detecting repetitive N-gram patterns "
"in output tokens. If such repetition is detected, generation will "
"be ended early. LLMs can sometimes generate repetitive, unhelpful "
"token patterns, stopping only when they hit the maximum output length "
"(e.g. 'abcdabcdabcd...' or '\\emoji \\emoji \\emoji ...'). This feature "
"can detect such behavior and terminate early, saving time and tokens.",
)
响应 API¶
我们的响应 API 与 OpenAI 的响应 API 兼容；您可以使用 官方 OpenAI Python 客户端 与之交互。
代码示例： examples/tool_calling/openai_responses_client_with_tools.py
额外参数¶
请求对象中支持以下额外参数
代码
request_id: str = Field(
default_factory=lambda: f"resp_{random_uuid()}",
description=(
"The request_id related to this request. If the caller does "
"not set it, a random_uuid will be generated. This id is used "
"through out the inference process and return in response."
),
)
media_io_kwargs: dict[str, dict[str, Any]] | None = Field(
default=None,
description=(
"Additional kwargs to pass to the media IO connectors, "
"keyed by modality. Merged with engine-level media_io_kwargs."
),
)
mm_processor_kwargs: dict[str, Any] | None = Field(
default=None,
description=("Additional kwargs to pass to the HF processor."),
)
priority: int = Field(
default=0,
ge=_INT64_MIN,
le=_INT64_MAX,
description=(
"The priority of the request (lower means earlier handling; "
"default: 0). Any priority other than 0 will raise an error "
"if the served model does not use priority scheduling."
),
)
cache_salt: str | None = Field(
default=None,
description=(
"If specified, the prefix cache will be salted with the provided "
"string to prevent an attacker to guess prompts in multi-user "
"environments. The salt should be random, protected from "
"access by 3rd parties, and long enough to be "
"unpredictable (e.g., 43 characters base64-encoded, corresponding "
"to 256 bit)."
),
)
enable_response_messages: bool = Field(
default=False,
description=(
"Dictates whether or not to return messages as part of the "
"response object. Currently only supported for non-background."
),
)
# similar to input_messages / output_messages in ResponsesResponse
# we take in previous_input_messages (ie in harmony format)
# this cannot be used in conjunction with previous_response_id
# TODO: consider supporting non harmony messages as well
previous_input_messages: list[OpenAIHarmonyMessage | dict] | None = None
structured_outputs: StructuredOutputsParams | None = Field(
default=None,
description="Additional kwargs for structured outputs",
)
repetition_penalty: float | None = None
seed: int | None = Field(None, ge=_INT64_MIN, le=_INT64_MAX)
stop: str | list[str] | None = []
ignore_eos: bool = False
vllm_xargs: dict[str, str | int | float | list[str | int | float]] | None = Field(
default=None,
description=(
"Additional request parameters with (list of) string or "
"numeric values, used by custom extensions."
),
)
kv_transfer_params: dict[str, Any] | None = Field(
default=None,
description="KVTransfer parameters used for disaggregated serving.",
)
ec_transfer_params: dict[str, Any] | None = Field(
default=None,
description=(
"ECTransfer parameters used for encoder-cache disaggregated serving."
),
)
chat_template_kwargs: dict[str, Any] | None = Field(
default=None,
description=(
"Additional keyword args to pass to the chat template renderer. "
"Will be accessible by the template."
),
)
响应对象中支持以下额外参数
代码
# These are populated when enable_response_messages is set to True
# NOTE: custom serialization is needed
# see serialize_input_messages and serialize_output_messages
input_messages: ResponseInputOutputMessage | None = Field(
default=None,
description=(
"If enable_response_messages, we can show raw token input to model."
),
)
output_messages: ResponseInputOutputMessage | None = Field(
default=None,
description=(
"If enable_response_messages, we can show raw token output of model."
),
)