ggml-org/llama.cpp b12170

on GitHub

moe-cache : LRFU/MQ-like hybrid cache, ClassLRU (#31233)

• moe-cache : LRFU/MQ-inspired hybrid cache

• make the clock advance by N tokens: use the real token count (ne[1] is always 1, ids have n_expert_used entries per token)

• sel is a RESHAPE node, the orig input is in src[0], so sel->src[0]->ne[1] is the count

• c for the class index, last_use for last use time

• explain the roles of these

• compute the key once

• use the key directly instead of re-reading

• show how the score is computed from the age

• remove the branch and clarify exp2 and log2 relationship

• reword the header comment and elaborate on unused slots and sorting

• not a copy, not just skip

• std::array maybe

• Revert "std::array maybe"

This reverts commit 7649b1f.

• half life it is

• add key and iteration helpers

• extract decayed_crf

• bump classes to 8, document the formula, display the cache size in token worth count

• assert the computed number of classes matches the half life