ggml-org/llama.cpp b11318

on GitHub

vocab : honor BOS/EOS settings for PLaMo-2 and PLaMo-3 (#29734)

• vocab : honor BOS/EOS settings for PLaMo-2 and PLaMo-3

The original tokenizer configs for PLaMo-2 and PLaMo-3 have
add_bos_token: true and add_eos_token: false , but
_set_vocab_plamo() did not write the BOS/EOS metadata. The
PLAMO2 tokenizer path also ignored add_bos/add_eos during
tokenization.

Write the settings from tokenizer_config.json and honor them in
the PLAMO2 tokenization path. GGUFs without these keys keep the
previous behavior.

• Update conversion/base.py

Co-authored-by: Sigbjørn Skjæret sigbjorn.skjaeret@huggingface.co

Co-authored-by: Sigbjørn Skjæret sigbjorn.skjaeret@huggingface.co