跳到内容

llmcompressor.transformers.data.ultrachat_200k

类

UltraChatDataset

UltraChatDataset(
    dataset_args: DatasetArguments,
    split: str,
    processor: Processor,
)

基类：TextGenerationDataset

Ultra Chat 200k 数据集的子文本生成类

参数

  • dataset_args

    (DatasetArguments) –

    数据集加载的配置设置

  • split

    (str) –

    从数据集中加载的拆分，例如 test 或 train[:5%]

  • processor

    (Processor) –

    要在数据集上使用的处理器或分词器

源代码位于 llmcompressor/transformers/data/ultrachat_200k.py 
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
def __init__(
    self, dataset_args: "DatasetArguments", split: str, processor: Processor
):
    dataset_args = deepcopy(dataset_args)
    dataset_args.dataset = "HuggingFaceH4/ultrachat_200k"
    dataset_args.text_column = "messages"

    if split in ["train", "test"]:
        split += "_sft"

    super().__init__(dataset_args=dataset_args, split=split, processor=processor)

    if (
        self.tokenizer is not None
        and getattr(self.tokenizer, "chat_template", None) is None
    ):
        # note that since tokenizer is a member of processor,
        # this change affects processor.apply_chat_template
        self.tokenizer.chat_template = self.DEFAULT_CHAT_TEMPLATE
        logger.warning(
            "tokenizer.chat_template is not set, using default chat template for "
            f"{self.__class__.__name__}"
        )