ggml-org/llama.cpp b10242

on GitHub

CUDA: Add backend sampler for penalties sampler (#25262)

• sampling: enhance penalty handling in common_sampler_init

• Set default value for penalty_last_n based on model context if not specified.
• Ensure penalty_last_n and n_prev are non-negative.
• Update llama_sampler_penalties structure to inherit from llama_sampler_backend and add backend input handling for penalties.
• Implement backend initialization and application logic for penalties, including frequency and presence adjustments.

• tests: add backend penalties sampling tests and utility functions

• Introduced accept_prompt and unique_prompt_tokens functions to handle prompt acceptance and token uniqueness.
• Implemented compare_penalties_logits to compare logits from backend and CPU samplers with penalties.
• Added test_backend_penalties_sampling to validate backend penalties with various configurations.
• Enhanced the test suite for better coverage of penalty handling in sampling.

• sampling: add support for top-k penalties in backend sampling

• sampling: add fix to ensure stable numerical results. Preserve masked logits as -Inf and no longer generate NaN.

• sampling: enhance penalty comparison tests with masking penalties logic

• add comments on padding

• sampling: add comments on modifications

• add the unit test to cover masked-out token as -INF

• validate repeat penalty to ensure it is finite and greater than 0; add tests for invalid values

• refactor: test functions to share logic and be less verbose

• add test to cover case where previously penalized token is not part of candidates

• remove comments

• remove redundant penalty_last_n initialization and validation in common_sampler_init

• add support for penalties in sampler chain with configurable positions

• add validation for penalty parameters and enhance tests for non-finite values

• add context parameter to common_sampler_init and set default for penalty_last_n

• add llama_n_ctx parameter to common_sampler_init for improved sampler initialization

• replace penalty_last_n x n_candidates comparison matrix with a vocabulary-sized count tensor

• add tests for backend penalties sampling without filler entries , token_count.size() == n_active == n_max == 64

• add test for backend penalties sampling after top-p with large history window

• remove as unused

• add is_disabled method, tensor logits reshape, add rest review suggestions

• clarify comment