结构化输出¶
vLLM 支持使用 xgrammar 或 guidance 作为后端生成结构化输出。本文档将为您展示可用于生成结构化输出的不同选项的一些示例。
在线服务 (OpenAI API)¶
您可以使用 OpenAI 的 Completions 和 Chat API 来生成结构化输出。
支持以下参数,它们必须作为额外参数添加:
guided_choice
: 输出将精确地是所选选项之一。guided_regex
: 输出将遵循正则表达式模式。guided_json
: 输出将遵循 JSON 模式。guided_grammar
: 输出将遵循上下文无关语法。structural_tag
: 在生成的文本中,遵循指定标签集内的 JSON 模式。
您可以在 OpenAI 兼容服务器 页面上查看支持的完整参数列表。
OpenAI 兼容服务器默认支持结构化输出。您可以通过将 --guided-decoding-backend
标志设置为 vllm serve
来指定要使用的后端。默认后端是 auto
,它将尝试根据请求的详细信息选择适当的后端。您也可以选择特定的后端,并附加一些选项。完整的选项集可在 vllm serve --help
文本中找到。
现在,我们来看每个示例,从最简单的 guided_choice
开始。
代码
from openai import OpenAI
client = OpenAI(
base_url="https://:8000/v1",
api_key="-",
)
model = client.models.list().data[0].id
completion = client.chat.completions.create(
model=model,
messages=[
{"role": "user", "content": "Classify this sentiment: vLLM is wonderful!"}
],
extra_body={"guided_choice": ["positive", "negative"]},
)
print(completion.choices[0].message.content)
下一个示例展示了如何使用 guided_regex
。其思想是根据简单的正则表达式模板生成电子邮件地址。
代码
completion = client.chat.completions.create(
model=model,
messages=[
{
"role": "user",
"content": "Generate an example email address for Alan Turing, who works in Enigma. End in .com and new line. Example result: [email protected]\n",
}
],
extra_body={"guided_regex": r"\w+@\w+\.com\n", "stop": ["\n"]},
)
print(completion.choices[0].message.content)
结构化文本生成中最相关的特性之一是能够生成具有预定义字段和格式的有效 JSON。为此,我们可以通过两种不同的方式使用 guided_json
参数:
- 直接使用 JSON Schema
- 定义一个 Pydantic 模型,然后从中提取 JSON Schema(这通常是一个更简单的选项)。
下一个示例展示了如何将 guided_json
参数与 Pydantic 模型一起使用。
代码
from pydantic import BaseModel
from enum import Enum
class CarType(str, Enum):
sedan = "sedan"
suv = "SUV"
truck = "Truck"
coupe = "Coupe"
class CarDescription(BaseModel):
brand: str
model: str
car_type: CarType
json_schema = CarDescription.model_json_schema()
completion = client.chat.completions.create(
model=model,
messages=[
{
"role": "user",
"content": "Generate a JSON with the brand, model and car_type of the most iconic car from the 90's",
}
],
"response_format": {
"type": "json_schema",
"json_schema": {
"name": "car-description",
"schema": CarDescription.model_json_schema()
},
},
)
print(completion.choices[0].message.content)
提示
虽然并非绝对必要,但通常最好在提示中指明 JSON 模式以及字段应如何填充。这在大多数情况下可以显著改善结果。
最后,我们有 guided_grammar
选项,这可能是最难使用的,但它确实非常强大。它允许我们定义完整的语言,如 SQL 查询。它通过使用上下文无关的 EBNF 语法工作。例如,我们可以使用它来定义简化 SQL 查询的特定格式。
代码
simplified_sql_grammar = """
root ::= select_statement
select_statement ::= "SELECT " column " from " table " where " condition
column ::= "col_1 " | "col_2 "
table ::= "table_1 " | "table_2 "
condition ::= column "= " number
number ::= "1 " | "2 "
"""
completion = client.chat.completions.create(
model=model,
messages=[
{
"role": "user",
"content": "Generate an SQL query to show the 'username' and 'email' from the 'users' table.",
}
],
extra_body={"guided_grammar": simplified_sql_grammar},
)
print(completion.choices[0].message.content)
另请参阅:完整示例
推理输出¶
您还可以将结构化输出与
请注意,您可以将推理与任何提供的结构化输出功能结合使用。以下示例使用带有 JSON 模式的功能。
代码
from pydantic import BaseModel
class People(BaseModel):
name: str
age: int
completion = client.chat.completions.create(
model=model,
messages=[
{
"role": "user",
"content": "Generate a JSON with the name and age of one random person.",
}
],
response_format={
"type": "json_schema",
"json_schema": {
"name": "people",
"schema": People.model_json_schema()
}
},
)
print("reasoning_content: ", completion.choices[0].message.reasoning_content)
print("content: ", completion.choices[0].message.content)
另请参阅:完整示例
实验性自动解析 (OpenAI API)¶
本节介绍了 OpenAI Beta 版对 client.chat.completions.create()
方法的包装,该包装提供了与 Python 特定类型更丰富的集成。
截至撰写本文时 (openai==1.54.4
),这是 OpenAI 客户端库中的一项“Beta”功能。代码参考可在此处找到。
对于以下示例,vLLM 使用 vllm serve meta-llama/Llama-3.1-8B-Instruct
进行设置。
这是一个使用 Pydantic 模型获取结构化输出的简单示例。
代码
from pydantic import BaseModel
from openai import OpenAI
class Info(BaseModel):
name: str
age: int
client = OpenAI(base_url="http://0.0.0.0:8000/v1", api_key="dummy")
model = client.models.list().data[0].id
completion = client.beta.chat.completions.parse(
model=model,
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "My name is Cameron, I'm 28. What's my name and age?"},
],
response_format=Info,
)
message = completion.choices[0].message
print(message)
assert message.parsed
print("Name:", message.parsed.name)
print("Age:", message.parsed.age)
ParsedChatCompletionMessage[Testing](content='{"name": "Cameron", "age": 28}', refusal=None, role='assistant', audio=None, function_call=None, tool_calls=[], parsed=Testing(name='Cameron', age=28))
Name: Cameron
Age: 28
这是一个使用嵌套 Pydantic 模型处理逐步数学解法的更复杂示例。
代码
from typing import List
from pydantic import BaseModel
from openai import OpenAI
class Step(BaseModel):
explanation: str
output: str
class MathResponse(BaseModel):
steps: list[Step]
final_answer: str
completion = client.beta.chat.completions.parse(
model=model,
messages=[
{"role": "system", "content": "You are a helpful expert math tutor."},
{"role": "user", "content": "Solve 8x + 31 = 2."},
],
response_format=MathResponse,
)
message = completion.choices[0].message
print(message)
assert message.parsed
for i, step in enumerate(message.parsed.steps):
print(f"Step #{i}:", step)
print("Answer:", message.parsed.final_answer)
输出
ParsedChatCompletionMessage[MathResponse](content='{ "steps": [{ "explanation": "First, let\'s isolate the term with the variable \'x\'. To do this, we\'ll subtract 31 from both sides of the equation.", "output": "8x + 31 - 31 = 2 - 31"}, { "explanation": "By subtracting 31 from both sides, we simplify the equation to 8x = -29.", "output": "8x = -29"}, { "explanation": "Next, let\'s isolate \'x\' by dividing both sides of the equation by 8.", "output": "8x / 8 = -29 / 8"}], "final_answer": "x = -29/8" }', refusal=None, role='assistant', audio=None, function_call=None, tool_calls=[], parsed=MathResponse(steps=[Step(explanation="First, let's isolate the term with the variable 'x'. To do this, we'll subtract 31 from both sides of the equation.", output='8x + 31 - 31 = 2 - 31'), Step(explanation='By subtracting 31 from both sides, we simplify the equation to 8x = -29.', output='8x = -29'), Step(explanation="Next, let's isolate 'x' by dividing both sides of the equation by 8.", output='8x / 8 = -29 / 8')], final_answer='x = -29/8'))
Step #0: explanation="First, let's isolate the term with the variable 'x'. To do this, we'll subtract 31 from both sides of the equation." output='8x + 31 - 31 = 2 - 31'
Step #1: explanation='By subtracting 31 from both sides, we simplify the equation to 8x = -29.' output='8x = -29'
Step #2: explanation="Next, let's isolate 'x' by dividing both sides of the equation by 8." output='8x / 8 = -29 / 8'
Answer: x = -29/8
使用 structural_tag
的示例可在以下链接找到: examples/online_serving/structured_outputs
离线推理¶
离线推理也支持相同类型的结构化输出。要使用它,我们需要在 SamplingParams
类中配置引导式解码,使用 GuidedDecodingParams
。GuidedDecodingParams
中的主要可用选项有:
json
regex
choice
grammar
structural_tag
这些参数可以与上面在线服务示例中的参数以相同的方式使用。下面展示了 choice
参数用法的一个示例:
代码
from vllm import LLM, SamplingParams
from vllm.sampling_params import GuidedDecodingParams
llm = LLM(model="HuggingFaceTB/SmolLM2-1.7B-Instruct")
guided_decoding_params = GuidedDecodingParams(choice=["Positive", "Negative"])
sampling_params = SamplingParams(guided_decoding=guided_decoding_params)
outputs = llm.generate(
prompts="Classify this sentiment: vLLM is wonderful!",
sampling_params=sampling_params,
)
print(outputs[0].outputs[0].text)
另请参阅:完整示例