多模态输入¶
本页面介绍了如何在 vLLM 中向 多模态模型 传递多模态输入。
注意
我们正在积极迭代多模态支持。请参阅 此 RFC 获取即将到来的变更，如果您有任何反馈或功能需求，请在 GitHub 上提交 Issue。
提示
在使用多模态模型进行服务时，请考虑设置 --allowed-media-domains 以限制 vLLM 可以访问的域名，防止其访问可能易受服务器端请求伪造 (SSRF) 攻击的任意端点。您可以为此参数提供域名列表。例如：--allowed-media-domains upload.wikimedia.org github.com www.bogotobogo.com
此外，请考虑设置 VLLM_MEDIA_URL_ALLOW_REDIRECTS=0，以防止 HTTP 重定向被追踪，从而绕过域名限制。
如果您在容器化环境中运行 vLLM，且 vLLM pod 可能对内部网络拥有不受限制的访问权限，那么此限制尤为重要。
离线推理¶
要输入多模态数据，请遵循 vllm.inputs.PromptType 中的架构：
prompt：提示词应遵循 HuggingFace 上记录的格式。multi_modal_data：这是一个字典，遵循 vllm.inputs.MultiModalDataDict 中定义的架构。
图像输入¶
您可以将单张图像传递给多模态字典的 'image' 字段，如下例所示：
代码
from vllm import LLM
llm = LLM(model="llava-hf/llava-1.5-7b-hf")
# Refer to the HuggingFace repo for the correct format to use
prompt = "USER: <image>\nWhat is the content of this image?\nASSISTANT:"
# Load the image using PIL.Image
image = PIL.Image.open(...)
# Single prompt inference
outputs = llm.generate({
"prompt": prompt,
"multi_modal_data": {"image": image},
})
for o in outputs:
generated_text = o.outputs[0].text
print(generated_text)
# Batch inference
image_1 = PIL.Image.open(...)
image_2 = PIL.Image.open(...)
outputs = llm.generate(
[
{
"prompt": "USER: <image>\nWhat is the content of this image?\nASSISTANT:",
"multi_modal_data": {"image": image_1},
},
{
"prompt": "USER: <image>\nWhat's the color of this image?\nASSISTANT:",
"multi_modal_data": {"image": image_2},
}
]
)
for o in outputs:
generated_text = o.outputs[0].text
print(generated_text)
完整示例： examples/offline_inference/vision_language.py
若要替换同一文本提示中的多张图像，您可以改用图像列表进行传递。
代码
from vllm import LLM
llm = LLM(
model="microsoft/Phi-3.5-vision-instruct",
trust_remote_code=True, # Required to load Phi-3.5-vision
max_model_len=4096, # Otherwise, it may not fit in smaller GPUs
limit_mm_per_prompt={"image": 2}, # The maximum number to accept
)
# Refer to the HuggingFace repo for the correct format to use
prompt = "<|user|>\n<|image_1|>\n<|image_2|>\nWhat is the content of each image?<|end|>\n<|assistant|>\n"
# Load the images using PIL.Image
image1 = PIL.Image.open(...)
image2 = PIL.Image.open(...)
outputs = llm.generate({
"prompt": prompt,
"multi_modal_data": {"image": [image1, image2]},
})
for o in outputs:
generated_text = o.outputs[0].text
print(generated_text)
完整示例： examples/offline_inference/vision_language_multi_image.py
如果使用 LLM.chat 方法，您可以直接在消息内容中使用多种格式传递图像：图像 URL、PIL 图像对象或预计算的嵌入。
代码
from vllm import LLM
from vllm.assets.image import ImageAsset
llm = LLM(model="llava-hf/llava-1.5-7b-hf")
image_url = "https://picsum.photos/id/32/512/512"
image_pil = ImageAsset('cherry_blossom').pil_image
image_embeds = torch.load(...)
conversation = [
{"role": "system", "content": "You are a helpful assistant"},
{"role": "user", "content": "Hello"},
{"role": "assistant", "content": "Hello! How can I assist you today?"},
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {"url": image_url},
},
{
"type": "image_pil",
"image_pil": image_pil,
},
{
"type": "image_embeds",
"image_embeds": image_embeds,
},
{
"type": "text",
"text": "What's in these images?",
},
],
},
]
# Perform inference and log output.
outputs = llm.chat(conversation)
for o in outputs:
generated_text = o.outputs[0].text
print(generated_text)
多图像输入可以扩展以执行视频字幕生成。我们以 Qwen2-VL 为例进行展示，因为它支持视频。
代码
from vllm import LLM
# Specify the maximum number of frames per video to be 4. This can be changed.
llm = LLM("Qwen/Qwen2-VL-2B-Instruct", limit_mm_per_prompt={"image": 4})
# Create the request payload.
video_frames = ... # load your video making sure it only has the number of frames specified earlier.
message = {
"role": "user",
"content": [
{
"type": "text",
"text": "Describe this set of frames. Consider the frames to be a part of the same video.",
},
],
}
for i in range(len(video_frames)):
base64_image = encode_image(video_frames[i]) # base64 encoding.
new_image = {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}}
message["content"].append(new_image)
# Perform inference and log output.
outputs = llm.chat([message])
for o in outputs:
generated_text = o.outputs[0].text
print(generated_text)
自定义 RGBA 背景颜色¶
当加载 RGBA 图像（带有透明度的图像）时，vLLM 会将其转换为 RGB 格式。默认情况下，透明像素会被替换为白色背景。您可以使用 media_io_kwargs 中的 rgba_background_color 参数自定义此背景颜色。
代码
from vllm import LLM
# Default white background (no configuration needed)
llm = LLM(model="llava-hf/llava-1.5-7b-hf")
# Custom black background for dark theme
llm = LLM(
model="llava-hf/llava-1.5-7b-hf",
media_io_kwargs={"image": {"rgba_background_color": [0, 0, 0]}},
)
# Custom brand color background (e.g., blue)
llm = LLM(
model="llava-hf/llava-1.5-7b-hf",
media_io_kwargs={"image": {"rgba_background_color": [0, 0, 255]}},
)
注意
rgba_background_color接受 RGB 值列表[R, G, B]或元组(R, G, B)，其中每个值的范围为 0-255。- 此设置仅影响带有透明度的 RGBA 图像；RGB 图像保持不变。
- 如果未指定，出于向后兼容性考虑，将使用默认的白色背景
(255, 255, 255)。
视频输入¶
您可以直接将 NumPy 数组列表传递给多模态字典的 'video' 字段，而不是使用多图像输入。
除了 NumPy 数组，您还可以传递 'torch.Tensor' 实例，如下面的 Qwen2.5-VL 示例所示。
代码
from transformers import AutoProcessor
from vllm import LLM, SamplingParams
from qwen_vl_utils import process_vision_info
model_path = "Qwen/Qwen2.5-VL-3B-Instruct"
video_path = "https://content.pexels.com/videos/free-videos.mp4"
llm = LLM(
model=model_path,
gpu_memory_utilization=0.8,
enforce_eager=True,
limit_mm_per_prompt={"video": 1},
)
sampling_params = SamplingParams(max_tokens=1024)
video_messages = [
{
"role": "system",
"content": "You are a helpful assistant.",
},
{
"role": "user",
"content": [
{"type": "text", "text": "describe this video."},
{
"type": "video",
"video": video_path,
"total_pixels": 20480 * 28 * 28,
"min_pixels": 16 * 28 * 28,
},
]
},
]
messages = video_messages
processor = AutoProcessor.from_pretrained(model_path)
prompt = processor.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True,
)
image_inputs, video_inputs = process_vision_info(messages)
mm_data = {}
if video_inputs is not None:
mm_data["video"] = video_inputs
llm_inputs = {
"prompt": prompt,
"multi_modal_data": mm_data,
}
outputs = llm.generate([llm_inputs], sampling_params=sampling_params)
for o in outputs:
generated_text = o.outputs[0].text
print(generated_text)
注意
'process_vision_info' 仅适用于 Qwen2.5-VL 及类似模型。
完整示例： examples/offline_inference/vision_language.py
音频输入¶
您可以将元组 (array, sampling_rate) 传递给多模态字典的 'audio' 字段。
完整示例： examples/offline_inference/audio_language.py
长音频转录的分块处理¶
诸如 Whisper 之类的语音转文字模型具有处理的最大音频长度（通常为 30 秒）。对于更长的音频文件，vLLM 提供了一个实用程序，可以在静音点处智能地将音频拆分为若干块，以最大限度地减少对语音的切断。
import librosa
from vllm import LLM, SamplingParams
from vllm.multimodal.audio import split_audio
# Load long audio file
audio, sr = librosa.load("long_audio.wav", sr=16000)
# Split into chunks at low-energy (quiet) regions
chunks = split_audio(
audio_data=audio,
sample_rate=sr,
max_clip_duration_s=30.0, # Maximum chunk length in seconds
overlap_duration_s=1.0, # Search window for finding quiet split points
min_energy_window_size=1600, # Window size for energy calculation (~100ms at 16kHz)
)
# Initialize Whisper model
llm = LLM(model="openai/whisper-large-v3-turbo")
sampling_params = SamplingParams(temperature=0, max_tokens=256)
# Transcribe each chunk
transcriptions = []
for chunk in chunks:
outputs = llm.generate({
"prompt": "<|startoftranscript|><|en|><|transcribe|><|notimestamps|>",
"multi_modal_data": {"audio": (chunk, sr)},
}, sampling_params)
transcriptions.append(outputs[0].outputs[0].text)
# Combine results
full_transcription = " ".join(transcriptions)
split_audio 函数：
- 在静音点处拆分音频，以避免切断语音。
- 使用 RMS 能量在重叠窗口内寻找低振幅区域。
- 保留所有音频采样（无数据丢失）。
- 支持任意采样率。
音频通道自动归一化¶
vLLM 会自动为需要特定音频格式的模型归一化音频通道。当使用 torchaudio 等库加载音频时，立体声文件返回形状 [channels, time]，但许多音频模型（特别是基于 Whisper 的模型）期望形状为 [time] 的单声道音频。
支持自动单声道转换的模型：
- Whisper 及所有基于 Whisper 的模型
- Qwen2-Audio
- Qwen2.5-Omni / Qwen3-Omni (继承自 Qwen2.5-Omni)
- Ultravox
对于这些模型，vLLM 会自动：
- 通过特征提取器检测模型是否需要单声道音频。
- 使用通道平均法将多通道音频转换为单声道。
- 处理
(channels, time)格式 (torchaudio) 和(time, channels)格式 (soundfile)。
立体声音频示例：
import torchaudio
from vllm import LLM
# Load stereo audio file - returns (channels, time) shape
audio, sr = torchaudio.load("stereo_audio.wav")
print(f"Original shape: {audio.shape}") # e.g., torch.Size([2, 16000])
# vLLM automatically converts to mono for Whisper-based models
llm = LLM(model="openai/whisper-large-v3")
outputs = llm.generate({
"prompt": "",
"multi_modal_data": {"audio": (audio.numpy(), sr)},
})
无需手动转换 - vLLM 会根据模型的需求自动处理通道归一化。
嵌入 (Embedding) 输入¶
要将预计算的属于某种数据类型（即图像、视频或音频）的嵌入直接输入到语言模型中，请将形状为 (..., 语言模型的隐藏层大小) 的张量传递给多模态字典的相应字段。确切的形状取决于所使用的模型。
您必须通过 enable_mm_embeds=True 启用此功能。
警告
如果传递了不正确的嵌入形状，vLLM 引擎可能会崩溃。仅对可信用户启用此标志！
图像嵌入¶
代码
from vllm import LLM
# Inference with image embeddings as input
llm = LLM(model="llava-hf/llava-1.5-7b-hf", enable_mm_embeds=True)
# Refer to the HuggingFace repo for the correct format to use
prompt = "USER: <image>\nWhat is the content of this image?\nASSISTANT:"
# For most models, `image_embeds` has shape: (num_images, image_feature_size, hidden_size)
image_embeds = torch.load(...)
outputs = llm.generate({
"prompt": prompt,
"multi_modal_data": {"image": image_embeds},
})
for o in outputs:
generated_text = o.outputs[0].text
print(generated_text)
# Additional examples for models that require extra fields
llm = LLM(
"Qwen/Qwen2-VL-2B-Instruct",
limit_mm_per_prompt={"image": 4},
enable_mm_embeds=True,
)
mm_data = {
"image": {
# Shape: (total_feature_size, hidden_size)
# total_feature_size = sum(image_feature_size for image in images)
"image_embeds": torch.load(...),
# Shape: (num_images, 3)
# image_grid_thw is needed to calculate positional encoding.
"image_grid_thw": torch.load(...),
}
}
llm = LLM(
"openbmb/MiniCPM-V-2_6",
trust_remote_code=True,
limit_mm_per_prompt={"image": 4},
enable_mm_embeds=True,
)
mm_data = {
"image": {
# Shape: (num_images, num_slices, hidden_size)
# num_slices can differ for each image
"image_embeds": [torch.load(...) for image in images],
# Shape: (num_images, 2)
# image_sizes is needed to calculate details of the sliced image.
"image_sizes": [image.size for image in images],
}
}
对于 Qwen3-VL，image_embeds 应同时包含基础图像嵌入和 deepstack 特征。
音频嵌入输入¶
您可以像传递图像嵌入一样传递预计算的音频嵌入。
代码
from vllm import LLM
import torch
# Enable audio embeddings support
llm = LLM(model="fixie-ai/ultravox-v0_5-llama-3_2-1b", enable_mm_embeds=True)
# Refer to the HuggingFace repo for the correct format to use
prompt = "USER: <audio>\nWhat is in this audio?\nASSISTANT:"
# Load pre-computed audio embeddings, usually with shape:
# (num_audios, audio_feature_size, hidden_size of LM)
audio_embeds = torch.load(...)
outputs = llm.generate({
"prompt": prompt,
"multi_modal_data": {"audio": audio_embeds},
})
for o in outputs:
generated_text = o.outputs[0].text
print(generated_text)
缓存输入¶
使用多模态输入时，vLLM 通常会根据内容对每个媒体项进行哈希处理，以实现跨请求缓存。您可以选择传递 multi_modal_uuids 来为每个项目提供您自己的稳定 ID，这样缓存就可以在跨请求时重用工作，而无需重新哈希原始内容。
代码
from vllm import LLM
from PIL import Image
# Qwen2.5-VL example with two images
llm = LLM(model="Qwen/Qwen2.5-VL-3B-Instruct")
prompt = "USER: <image><image>\nDescribe the differences.\nASSISTANT:"
img_a = Image.open("/path/to/a.jpg")
img_b = Image.open("/path/to/b.jpg")
outputs = llm.generate({
"prompt": prompt,
"multi_modal_data": {"image": [img_a, img_b]},
# Provide stable IDs for caching.
# Requirements (matched by this example):
# - Include every modality present in multi_modal_data.
# - For lists, provide the same number of entries.
# - Use None to fall back to content hashing for that item.
"multi_modal_uuids": {"image": ["sku-1234-a", None]},
})
for o in outputs:
print(o.outputs[0].text)
使用 UUID，如果您预计对应项会命中缓存，甚至可以完全跳过发送媒体数据。请注意，如果跳过的媒体没有对应的 UUID，或者 UUID 未能命中缓存，请求将会失败。
代码
from vllm import LLM
from PIL import Image
# Qwen2.5-VL example with two images
llm = LLM(model="Qwen/Qwen2.5-VL-3B-Instruct")
prompt = "USER: <image><image>\nDescribe the differences.\nASSISTANT:"
img_b = Image.open("/path/to/b.jpg")
outputs = llm.generate({
"prompt": prompt,
"multi_modal_data": {"image": [None, img_b]},
# Since img_a is expected to be cached, we can skip sending the actual
# image entirely.
"multi_modal_uuids": {"image": ["sku-1234-a", None]},
})
for o in outputs:
print(o.outputs[0].text)
警告
如果多模态处理器缓存和前缀缓存均已禁用，用户提供的 multi_modal_uuids 将被忽略。
在线服务¶
我们兼容 OpenAI 的服务器通过 Chat Completions API 接受多模态数据。媒体输入还支持用户提供可选的 UUID 来唯一标识每个媒体，该 UUID 用于跨请求缓存媒体结果。
重要
使用 Chat Completions API 必须使用聊天模板。对于 HF 格式的模型，默认聊天模板定义在 chat_template.json 或 tokenizer_config.json 中。
如果没有默认聊天模板可用，我们将首先在 vllm/transformers_utils/chat_templates/registry.py 中查找内置的后备方案。如果没有任何后备方案，则会引发错误，您必须通过 --chat-template 参数手动提供聊天模板。
对于某些模型，我们在 examples 中提供了替代的聊天模板。例如，VLM2Vec 使用 examples/pooling/embed/template/vlm2vec_phi3v.jinja，这与 Phi-3-Vision 的默认模板不同。
图像输入¶
图像输入支持遵循 OpenAI Vision API。以下是一个使用 Phi-3.5-Vision 的简单示例。
首先，启动兼容 OpenAI 的服务器：
vllm serve microsoft/Phi-3.5-vision-instruct --runner generate \
--trust-remote-code --max-model-len 4096 --limit-mm-per-prompt.image 2
然后，您可以如下使用 OpenAI 客户端：
代码
import os
from openai import OpenAI
openai_api_key = "EMPTY"
openai_api_base = "https://:8000/v1"
client = OpenAI(
api_key=openai_api_key,
base_url=openai_api_base,
)
# Single-image input inference
# Public image URL for testing remote image processing
image_url = "https://vllm-public-assets.s3.us-west-2.amazonaws.com/vision_model_images/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
# Create chat completion with remote image
chat_response = client.chat.completions.create(
model="microsoft/Phi-3.5-vision-instruct",
messages=[
{
"role": "user",
"content": [
# NOTE: The prompt formatting with the image token `<image>` is not needed
# since the prompt will be processed automatically by the API server.
{
"type": "text",
"text": "What’s in this image?",
},
{
"type": "image_url",
"image_url": {"url": image_url},
"uuid": image_url, # Optional
},
],
}
],
)
print("Chat completion output:", chat_response.choices[0].message.content)
# Local image file path (update this to point to your actual image file)
image_file = "/path/to/image.jpg"
# Create chat completion with local image file
# Launch the API server/engine with the --allowed-local-media-path argument.
if os.path.exists(image_file):
chat_completion_from_local_image_url = client.chat.completions.create(
model="microsoft/Phi-3.5-vision-instruct",
messages=[
{
"role": "user",
"content": [
{
"type": "text",
"text": "What’s in this image?",
},
{
"type": "image_url",
"image_url": {"url": f"file://{image_file}"},
},
],
}
],
)
result = chat_completion_from_local_image_url.choices[0].message.content
print("Chat completion output from local image file:\n", result)
else:
print(f"Local image file not found at {image_file}, skipping local file test.")
# Multi-image input inference
image_url_duck = "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/duck.jpg"
image_url_lion = "https://vllm-public-assets.s3.us-west-2.amazonaws.com/multimodal_asset/lion.jpg"
chat_response = client.chat.completions.create(
model="microsoft/Phi-3.5-vision-instruct",
messages=[
{
"role": "user",
"content": [
{
"type": "text",
"text": "What are the animals in these images?",
},
{
"type": "image_url",
"image_url": {"url": image_url_duck},
"uuid": image_url_duck, # Optional
},
{
"type": "image_url",
"image_url": {"url": image_url_lion},
"uuid": image_url_lion, # Optional
},
],
}
],
)
print("Chat completion output:", chat_response.choices[0].message.content)
完整示例： examples/online_serving/openai_chat_completion_client_for_multimodal.py
提示
vLLM 也支持从本地文件路径加载：您可以在启动 API 服务器/引擎时通过 --allowed-local-media-path 指定允许的本地媒体路径，并在 API 请求中将文件路径作为 url 传递。
提示
无需在 API 请求的文本内容中放置图像占位符 - 它们已由图像内容表示。实际上，您可以通过交替插入文本和图像内容，将图像占位符放置在文本中间。
视频输入¶
除了 image_url，您还可以通过 video_url 传递视频文件。以下是一个使用 LLaVA-OneVision 的简单示例。
首先，启动兼容 OpenAI 的服务器：
然后，您可以如下使用 OpenAI 客户端：
代码
from openai import OpenAI
openai_api_key = "EMPTY"
openai_api_base = "https://:8000/v1"
client = OpenAI(
api_key=openai_api_key,
base_url=openai_api_base,
)
video_url = "http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerFun.mp4"
## Use video url in the payload
chat_completion_from_url = client.chat.completions.create(
messages=[
{
"role": "user",
"content": [
{
"type": "text",
"text": "What's in this video?",
},
{
"type": "video_url",
"video_url": {"url": video_url},
"uuid": video_url, # Optional
},
],
}
],
model=model,
max_completion_tokens=64,
)
result = chat_completion_from_url.choices[0].message.content
print("Chat completion output from image url:", result)
完整示例： examples/online_serving/openai_chat_completion_client_for_multimodal.py
视频帧恢复¶
为了在处理可能损坏或截断的视频文件时提高稳健性，vLLM 支持使用动态窗口前向扫描方法进行可选的帧恢复。启用后，如果目标帧在顺序读取期间加载失败，则将使用成功获取的下一帧（在下一个目标帧之前）来代替它。
要启用视频帧恢复，请通过 --media-io-kwargs 传递 frame_recovery 参数：
# Example: Enable frame recovery
vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \
--media-io-kwargs '{"video": {"frame_recovery": true}}'
参数
frame_recovery：用于启用前向扫描恢复的布尔标志。当为true时，失败的帧将使用动态窗口内（直至下一个目标帧）的下一可用帧进行恢复。默认值为false。
工作原理
- 系统顺序读取帧。
- 如果目标帧获取失败，它将被标记为“失败”。
- 成功获取的下一帧（在到达下一个目标之前）被用于恢复失败的帧。
- 此方法可处理视频中途损坏和视频末尾截断的情况。
在使用 OpenCV 后端时，适用于常见的视频格式，如 MP4。
自定义 RGBA 背景颜色¶
要为 RGBA 图像使用自定义背景颜色，请通过 --media-io-kwargs 传递 rgba_background_color 参数。
# Example: Black background for dark theme
vllm serve llava-hf/llava-1.5-7b-hf \
--media-io-kwargs '{"image": {"rgba_background_color": [0, 0, 0]}}'
# Example: Custom gray background
vllm serve llava-hf/llava-1.5-7b-hf \
--media-io-kwargs '{"image": {"rgba_background_color": [128, 128, 128]}}'
音频输入¶
音频输入支持遵循 OpenAI Audio API。以下是一个使用 Ultravox-v0.5-1B 的简单示例。
首先，启动兼容 OpenAI 的服务器：
然后，您可以如下使用 OpenAI 客户端：
代码
import base64
import requests
from openai import OpenAI
from vllm.assets.audio import AudioAsset
def encode_base64_content_from_url(content_url: str) -> str:
"""Encode a content retrieved from a remote url to base64 format."""
with requests.get(content_url) as response:
response.raise_for_status()
result = base64.b64encode(response.content).decode('utf-8')
return result
openai_api_key = "EMPTY"
openai_api_base = "https://:8000/v1"
client = OpenAI(
api_key=openai_api_key,
base_url=openai_api_base,
)
# Any format supported by librosa is supported
audio_url = AudioAsset("winning_call").url
audio_base64 = encode_base64_content_from_url(audio_url)
chat_completion_from_base64 = client.chat.completions.create(
messages=[
{
"role": "user",
"content": [
{
"type": "text",
"text": "What's in this audio?",
},
{
"type": "input_audio",
"input_audio": {
"data": audio_base64,
"format": "wav",
},
"uuid": audio_url, # Optional
},
],
},
],
model=model,
max_completion_tokens=64,
)
result = chat_completion_from_base64.choices[0].message.content
print("Chat completion output from input audio:", result)
或者，您可以传递 audio_url，它是图像输入中 image_url 的音频对应项。 代码
chat_completion_from_url = client.chat.completions.create(
messages=[
{
"role": "user",
"content": [
{
"type": "text",
"text": "What's in this audio?",
},
{
"type": "audio_url",
"audio_url": {"url": audio_url},
"uuid": audio_url, # Optional
},
],
}
],
model=model,
max_completion_tokens=64,
)
result = chat_completion_from_url.choices[0].message.content
print("Chat completion output from audio url:", result)
完整示例： examples/online_serving/openai_chat_completion_client_for_multimodal.py
嵌入 (Embedding) 输入¶
要将预计算的属于某种数据类型（即图像、视频或音频）的嵌入直接输入到语言模型中，请为每个项目将形状为 (..., 语言模型的隐藏层大小) 的张量传递给多模态字典的相应字段。
重要
与离线推理不同，每个项目的嵌入必须单独传递，以便聊天模板正确应用占位符标记。
您必须通过 vllm serve 中的 --enable-mm-embeds 标志启用此功能。
警告
如果传递了不正确的嵌入形状，vLLM 引擎可能会崩溃。仅对可信用户启用此标志！
图像嵌入输入¶
对于图像嵌入，您可以将 base64 编码的张量传递给 image_embeds 字段。以下示例演示了如何向 OpenAI 服务器传递图像嵌入。 代码
from vllm.utils.serial_utils import tensor2base64
client = OpenAI(
# defaults to os.environ.get("OPENAI_API_KEY")
api_key=openai_api_key,
base_url=openai_api_base,
)
# Basic usage - this is equivalent to the LLaVA example for offline inference
model = "llava-hf/llava-1.5-7b-hf"
embeds = {
"type": "image_embeds",
"image_embeds": tensor2base64(torch.load(...)), # Shape: (image_feature_size, hidden_size)
"uuid": image_url, # Optional
}
# Additional examples for models that require extra fields
model = "Qwen/Qwen2-VL-2B-Instruct"
embeds = {
"type": "image_embeds",
"image_embeds": {
"image_embeds": tensor2base64(torch.load(...)), # Shape: (image_feature_size, hidden_size)
"image_grid_thw": tensor2base64(torch.load(...)), # Shape: (3,)
},
"uuid": image_url, # Optional
}
model = "openbmb/MiniCPM-V-2_6"
embeds = {
"type": "image_embeds",
"image_embeds": {
"image_embeds": tensor2base64(torch.load(...)), # Shape: (num_slices, hidden_size)
"image_sizes": tensor2base64(torch.load(...)), # Shape: (2,)
},
"uuid": image_url, # Optional
}
# Single image input
chat_completion = client.chat.completions.create(
messages=[
{
"role": "system",
"content": "You are a helpful assistant.",
},
{
"role": "user",
"content": [
{
"type": "text",
"text": "What's in this image?",
},
embeds,
],
},
],
model=model,
)
# Multi image input
chat_completion = client.chat.completions.create(
messages=[
{
"role": "system",
"content": "You are a helpful assistant.",
},
{
"role": "user",
"content": [
{
"type": "text",
"text": "What's in this image?",
},
embeds,
embeds,
],
},
],
model=model,
)
# Multi image input (interleaved)
chat_completion = client.chat.completions.create(
messages=[
{
"role": "system",
"content": "You are a helpful assistant.",
},
{
"role": "user",
"content": [
embeds,
{
"type": "text",
"text": "What's in this image?",
},
embeds,
],
},
],
model=model,
)
缓存输入¶
正如离线推理一样，如果您预计所提供的 UUID 会命中缓存，则可以跳过发送媒体。您可以通过如下方式发送媒体：
代码
# Image/video/audio URL:
{
"type": "image_url",
"image_url": None,
"uuid": image_uuid,
},
# image_embeds
{
"type": "image_embeds",
"image_embeds": None,
"uuid": image_uuid,
},
# input_audio:
{
"type": "input_audio",
"input_audio": None,
"uuid": audio_uuid,
},
# PIL Image:
{
"type": "image_pil",
"image_pil": None,
"uuid": image_uuid,
},