Fangyuan025/Chaty v2.1.2

on GitHub

The model stops re-reading what it just did

Between turns, code mode threw away everything the agent had
actually done. The follow-up arrived with the tool results gone and
"(tools run: read_file, bash)" in their place, so the model no
longer knew what any of those calls had returned and re-read files
it had just read. The exchange now carries over intact — and that is
cheaper than discarding it was: 2035 of 2058 prompt tokens reused,
75ms, against a cold 208ms for the summary it replaces.

A follow-up question no longer re-reads the conversation on MLX
either. Chaty lays a conversation out one message at a time now,
from a layout learned off the model's own template and used only
when it reproduces that template's token ids exactly. Qwen3.5 2B and
Gemma-4 26B both go from 0% to 99% reuse on the second turn's first
step — 1587ms to 95ms, 4292ms to 216ms.

Thinking off no longer costs a full prefill every turn. Chaty
prefills an empty reasoning block after the assistant header, but a
stored turn was rendered without it, so round two diverged right
there — 0% reuse on Qwen3.5 and LFM2, and thinking off is code
mode's default. Now 100%, on both engines.

LFM2's tool calls run. The engine rendered control tokens as
nothing, so <|tool_call_start|>[read_file(path='x')]<|tool_call_end|>
reached Chaty as ordinary prose and the call never fired.

A screenshot the model has already seen is no longer re-encoded.
Three consecutive tool-result rounds on Qwen3.5 4B cost 2826ms,
5596ms and 5741ms before; they now cost about 150ms each. Dropping
stale screenshots turned out to make llama.cpp slower, not faster,
so it keeps them.

Context compaction condenses the work with a model-written summary
instead of a 60-character index per turn, and leaves room to breathe
rather than skimming the limit: the same task on an 8k window went
from 63 rounds and 633 seconds to 18 rounds and 132 seconds. Chat
and code share one token budget now, calibrated against what the
engine actually charges, and attached pictures count toward it.

A follow-up question no longer re-reads the conversation on MLX
either: Qwen3.5 2B and Gemma-4 26B both go from 0% to 99% cache
reuse on the second turn's first step (1587ms to 95ms, 4284ms to
217ms).

A screenshot the model has already seen is no longer re-encoded.
Three consecutive tool-result rounds on Qwen3.5 4B cost 2826ms,
5596ms and 5741ms before; they now cost about 150ms each, and a
fresh screenshot encodes only itself. Dropping stale screenshots
turned out to make llama.cpp slower, not faster, so it keeps them.

A chat reply now travels the way the model wrote it — stripping the
reasoning out of stored turns meant every reply re-read the whole
conversation. Attached pictures count toward the context budget.

Context compaction now condenses the work with a model-written
summary instead of a 60-character index per turn, and leaves room to
breathe rather than skimming the limit: the same task on an 8k
window went from 63 rounds and 633 seconds to 18 rounds and 132
seconds. Chat and code also share one token budget now, calibrated
against what the engine actually charges.

Install

⚠️ macOS first launch — "Apple could not verify…" / "damaged"

Chaty is ad-hoc signed but not notarized (no paid Apple Developer
account), so Gatekeeper flags it on first launch. The app is safe —
everything runs locally. Clear the download quarantine once, in Terminal:

xattr -dr com.apple.quarantine /Applications/Chaty.app

Then open Chaty normally. (Alternatively: try to open it once, then go to
System Settings → Privacy & Security → Open Anyway.)

Full Changelog: v2.1.1...v2.1.2