ggml-org/llama.cpp b11707

on GitHub

model : Granite switch decision 256 options (#30693)

• feat(convert): Attempt to come up with 256 single-token, semantic option markers

Each switch model needs to identify a collection of single-token option
markers that can be mapped to their respective logits for the output
probability distribution. The simple upper/lower/numbers set only yields 66
unique markers, but the model can support up to 255. The challenge is that
which combinations of characters result in single-token markers is model-
specific, so we do a discovery exercise at conversion time by looking
through the specific model's vocab for single-token sequences of 1+ plain
letter strings (eg A, AA, BB, etc).

Branch: GraniteSwitchDecisionFullOptions
AI-usage: none
Signed-off-by: Gabe Goodhart ghart@us.ibm.com

• docs: Note granite-switch with 256 option markers

Branch: GraniteSwitchDecisionFullOptions
AI-usage: none
Signed-off-by: Gabe Goodhart ghart@us.ibm.com

• style: Remove extra newline

Branch: GraniteSwitchDecisionFullOptions
AI-usage: none
Signed-off-by: Gabe Goodhart ghart@us.ibm.com

• feat(convert): Use the computed labels in the template so c++ side is source of truth

Branch: GraniteSwitchDecisionFullOptions
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart ghart@us.ibm.com

• feat: Use LEV case for GRANITE_SWITCH w/ computed labels

Branch: GraniteSwitchDecisionFullOptions
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart ghart@us.ibm.com

• docs: Consolidate g-s into 255 bucket

Branch: GraniteSwitchDecisionFullOptions
AI-usage: full (Claude + Sonnet 5.5)
Signed-off-by: Gabe Goodhart ghart@us.ibm.com