KTransformers v0.6.4: High-Performance End-to-End Full-Parameter and LoRA Fine-Tuning

KTransformers v0.6.4 is a major upgrade to the KT fine-tuning path for large sparse MoE models. This release extends KT SFT from end-to-end Full-Parameter Fine-Tuning to LoRA, while improving BF16 CPU-GPU heterogeneous training throughput and memory efficiency across host RAM and GPU VRAM.

🚀 Core Highlights

• End-to-end Full-Parameter and LoRA SFT: train large sparse MoE models with Full Fine-Tuning, LoRA, or Hybrid configurations through the familiar LLaMA-Factory workflow.
• High training throughput: in real end-to-end training runs—not isolated kernel benchmarks—KTransformers reached approximately 400 tokens/s with Full-FT and 600 tokens/s with LoRA on Qwen3-30B-A3B using 2× AMD EPYC 9355 CPUs and 2× RTX 5090 GPUs. A separate LoRA setup using one RTX 4090 and an AMX-enabled CPU sustained more than 700 tokens/s. The resulting checkpoints also showed clear gains in end-to-end fine-tuning validation.
• DeepSeek V4 on Ampere GPUs: BF16 attention fallbacks and an FP8 MoE Marlin path expand DeepSeek V4 Flash support to GPUs without native FP8 Tensor Cores.
• RAWINT4 compatibility and acceleration: broader packed-weight compatibility, AVX-VNNI-256 expert loading, and a faster Kimi K2 prefill path.
• Experimental Intel iGPU backend: new SYCL-based GPTQ INT4 MoE inference support for Intel integrated GPUs.

📌 Full-Parameter, LoRA, and Hybrid SFT

This release completes the KT CPU-GPU heterogeneous training workflow for large sparse MoE models:

• Full Fine-Tuning updates KT-managed CPU expert weights together with the model's regular trainable parameters.
• LoRA covers regular linear layers, routers and gates, and fused MoE experts.
• Hybrid training allows Full-FT and LoRA parameters to be trained together.
• Reliable multi-step training supports gradient accumulation, activation checkpointing, distributed execution, optimizer updates, and profiling.

The new training path primarily targets BF16 KT CPU expert training.

👉 Main implementation: ktransformers PR #2094

📌 DeepSeek V4 and Ampere GPUs

The bundled sglang-kt integration adds broader DeepSeek V4 Flash and FP8 MoE support on Ampere GPUs:

• BF16 KV-cache and attention fallbacks for GPUs without native FP8 Tensor Cores.
• FP8 MoE Marlin support for GPU Experts, GPU Prefill, and dynamic expert updates.
• Stability fixes for long-input scheduling, AMXINT4/8 startup, and FP16 FE8M0 scale dequantization.

Related changes:

• DeepSeek V4 Flash inference on Ampere GPUs
• Fix V4 SWA token-pool sizing
• Fix AMXINT4/8 SwiGLU parameter handling
• FP8 MoE Marlin for GPU Expert and GPU Prefill
• FP16 FE8M0 Marlin scale dequantization

📌 RAWINT4 Compatibility and Prefill Acceleration

KTransformers v0.6.4 improves the RAWINT4 CPU Expert path with:

• Support for compressed-tensors checkpoints that store packed INT4 weights in int32.
• Per-expert RAWINT4 loading on AVX-VNNI-256 systems.
• A blocked matrix-multiplication path for faster Kimi K2 RAWINT4 prefill, while keeping the existing decode path for short sequences.

Related changes:

• Normalize compressed RAWINT4 weights
• AVX-VNNI-256 per-expert RAWINT4 loading
• K2 RAWINT4 prefill mat-mat dispatch

📌 Experimental Intel iGPU SYCL Backend

This release introduces an experimental SYCL backend for GPTQ INT4 MoE inference on Intel integrated GPUs. Compatibility and performance depend on the Intel GPU, driver, SYCL toolchain, and model configuration.

👉 Intel iGPU SYCL GPTQ INT4 MoE backend

🔐 Security

The balance_serve scheduler now binds its ZMQ ROUTER endpoint to 127.0.0.1 instead of all network interfaces, reducing exposure to unauthenticated remote pickle deserialization.

• Security fix PR #2091
• GHSA-83vp-v6wg-x93x

📦 Installation

KT SFT

pip install "ktransformers[sft]"

KT SFT continues to use the LLaMA-Factory training entry and YAML workflow.

KT Inference

pip install kt-kernel sglang-kt

Install from Source

git clone https://github.com/kvcache-ai/ktransformers.git
cd ktransformers
git submodule update --init --recursive
cd kt-kernel && ./install.sh
cd .. && ./install.sh

CUDA 12.0 or newer is required. Individual backends may have additional CPU ISA, GPU architecture, driver, or compiler requirements.

📝 Changelog

ktransformers

• feat: end-to-end Full-Parameter, LoRA, and Hybrid SFT (#2094)
• feat: K2 RAWINT4 prefill mat-mat dispatch (#2080)
• feat: Intel iGPU GPTQ INT4 MoE backend (#2089)
• fix: normalize compressed RAWINT4 packed weights (#2075)
• fix: AVX-VNNI-256 per-expert loading (#2092)
• [security]: bind the balance_serve scheduler ZMQ endpoint to loopback (#2091)
• [build]: enable -mavx512vl for AVX512 multi-variant builds (#2021)
• fix: detect ports that are already bound before launch (#2071)
• [build]: improve PyPI release workflow consistency (#2062)
• [docs]: update Qwen3.5 KT LoRA, FAQ, and DeepSeek V4 documentation (#2057, #2029, #2088)

sglang-kt

• Enable DeepSeek V4 Flash inference on Ampere GPUs (#58)
• Fix kt_ep_wrapper import failure (#59)
• Fix V4 SWA token-pool sizing for chunked prefill (#60)
• Fix AMXINT4/8 startup with non-MXFP SwiGLU parameters (#61)
• Add FP8 MoE Marlin for Ampere GPU Experts and GPU Prefill (#62)
• Add FP16 FE8M0 scale dequantization for Marlin (#63)

🌟 Contributors

Thanks to everyone who contributed code, testing, reviews, documentation, and end-to-end validation for this release.

Full Changelog: v0.6.3...v0.6.4

CC: @hermannklie @Oxygen56 @VectorPeak @lutianshu824 @RockmSockmJesus @callmegaga @Anai-Guo @blazingphoenix7 @harrychk @usrlocalben @jdai0 @Illumination111 @yyj6666667