ggml-org/llama.cpp b7756

on GitHub

common : implement new jinja template engine (#18462)

• jinja vm

• lexer

• add vm types

• demo

• clean up

• parser ok

• binary_expression::execute

• shadow naming

• bin ops works!

• fix map object

• add string builtins

• add more builtins

• wip

• use mk_val

• eval with is_user_input

• render gemma tmpl ok

• track input string even after transformations

• support binded functions

• keyword arguments and slicing array

• use shared_ptr for values

• add mk_stmt

• allow print source on exception

• fix negate test

• testing more templates

• mostly works

• add filter_statement

• allow func to access ctx

• add jinja-value.cpp

• impl global_from_json

• a lot of fixes

• more tests

• more fix, more tests

• more fixes

• rm workarounds

• demo: type inferrence

• add placeholder for tojson

• improve function args handling

• rm type inference

• no more std::regex

• trailing spaces

• make testing more flexible

• make output a bit cleaner

• (wip) redirect minja calls

• test: add --output

• fix crash on macro kwargs

• add minimal caps system

• add some workarounds

• rm caps_apply_workarounds

• get rid of preprocessing

• more fixes

• fix test-chat-template

• move test-chat-jinja into test-chat-template

• rm test-chat-jinja from cmake

• test-chat-template: use common

• fix build

• fix build (2)

• rename vm --> interpreter

• improve error reporting

• correct lstrip behavior

• add tojson

• more fixes

• disable tests for COMMON_CHAT_FORMAT_GENERIC

• make sure tojson output correct order

• add object.length

• fully functional selectattr / rejectattr

• improve error reporting

• more builtins added, more fixes

• create jinja rendering tests

• fix testing.h path

• adjust whitespace rules

• more fixes

• temporary disable test for ibm-granite

• r/lstrip behavior matched with hf.js

• minimax, glm4.5 ok

• add append and pop

• kimi-k2 ok

• test-chat passed

• fix lstrip_block

• add more jinja tests

• cast to unsigned char

• allow dict key to be numeric

• nemotron: rm windows newline

• tests ok

• fix test

• rename interpreter --> runtime

• fix build

• add more checks

• bring back generic format support

• fix Apertus

• [json.exception.out_of_range.403] key 'content' not found

• rm generic test

• refactor input marking

• add docs

• fix windows build

• clarify error message

• improved tests

• split/rsplit with maxsplit

• non-inverse maxsplit

forgot to change after simplifying

• implement separators for tojson and fix indent

• i like to move it move it

• rename null -- > none

• token::eof

• some nits + comments

• add exception classes for lexer and parser

• null -> none

• rename global -> env

• rm minja

• update docs

• docs: add input marking caveats

• imlement missing jinja-tests functions

• oops

• support trim filter with args, remove bogus to_json reference

• numerous argument fixes

• updated tests

• implement optional strip chars parameter

• use new chars parameter

• float filter also has default

• always leave at least one decimal in float string

• jinja : static analysis + header cleanup + minor fixes

• add fuzz test

• add string.cpp

• fix chat_template_kwargs

• nits

• fix build

• revert

• unrevert

sorry :)

• add fuzz func_args, refactor to be safer

• fix array.map()

• loosen ensure_vals max count condition, add not impl for map(int)

• hopefully fix windows

• check if empty first

• normalize newlines

Co-authored-by: Alde Rojas hello@alde.dev
Co-authored-by: Sigbjørn Skjæret sigbjorn.skjaeret@scala.com
Co-authored-by: Georgi Gerganov ggerganov@gmail.com