lightvector/KataGo v1.18.0

New backends, major CUDA optimizations, rules fix support

on GitHub

Download the latest neural nets to use with this engine release at https://katagotraining.org/.
Also, for 9x9 boards or for boards larger than 19x19, see https://katagotraining.org/extra_networks/ for networks specially trained for those sizes!

KataGo is continuing to improve at https://katagotraining.org/ and if you'd like to donate your spare GPU cycles and support it, it could use your help there!

This release adds major optimizations to the CUDA backend, and two new backends - ROCm for AMD GPUs, and ONNX for various other GPUs/accelerators. See below!

Getting Started / Choosing a Backend

If you're a new user, this section has tips for getting started and basic usage! For choosing a backend:

NVIDIA GPU

Use CUDA+CUDNN or TensorRT. Both are decent, either may be better performance.

• You'll also have to install CUDA and one of CUDNN or TensorRT from nvidia depending on your choice.
• TensorRT
  • For transformers (which will soon be the best models), CUDA 13 + TensorRT 10.16 is good but older TensorRT versions will often be outperformed by CUDA+CUDNN.
  • TensorRT versions older than 10 are not supported.
• CUDA+CUDNN
  • Faster startup times than TensorRT, still pretty good performance, and this release makes it much faster for transformer models on many GPUs (see below).
  • Use CUDNN >= 9.8.0. The CUDNN 8.9.7 builds will be a LOT slower when running transformer models - consider upgrading if you're still on old CUDNN. We're offering CUDNN 8.9.7 builds only for continuity with prior releases.

AMD GPU

New in this release - try the new ROCm backend, which should be much faster than OpenCL.

• The Linux build requires installing ROCm on your own, builds are offered for ROCm 7.2.4 and ROCm 7.14.0.
• The Windows build does NOT require installing ROCm (you need a reasonably recent AMD (Adrenalin) driver installed), and bundles everything it needs from ROCm 7.13. Download the package for your GPU family:
  • gfx103X - Radeon RX 6000 series (RDNA2)
  • gfx110X - Radeon RX 7000 series (RDNA3), including RDNA3 APUs such as the Radeon 780M
  • gfx1151 - Ryzen AI Max ("Strix Halo") APUs
  • gfx120X - Radeon RX 9000 series (RDNA4)
• The windows builds for ROCm are a bit experimental and were cross-compiled from a machine that doesn't actually have an AMD GPU for testing it - please me know if there are any issues with them!
• The DLLs provided and the rocblas and hipblaslt folders must stay next to katago.exe.
  • The exe is actually the same across these packages and is compiled for more GPUs than these, only the bundled AMD libraries differ. If your GPU is not in the list, for example an RX 5000 series (RDNA1) GPU or a Ryzen AI 300 series APU (gfx1150/gfx1152), it may still work: download the ROCm 7.13 Windows tarball for your GPU family from https://repo.amd.com/rocm/tarball-multi-arch/ (a large download), then copy the DLLs from its bin folder over KataGo's and replace KataGo's rocblas and hipblaslt folders with the ones from there. Copy them into KataGo's own directory rather than relying on PATH, since AMD driver installation can itself leave conflicting copies of some of these DLLs on the system.
  • The first run with a given neural net and board size can take 45 seconds or more before anything seems to happen, while MIOpen tunes its convolution kernels for your GPU - KataGo is not hung. The results are cached under %USERPROFILE%\.miopen\, so later startups are fast.

Intel GPU / NPU

New in this release - try the new ONNX Runtime backend with the OpenVINO execution provider (the onnx-openvino package).

• You must set onnxProvider = openvino in your config.
• Both the Windows and Linux builds are self-contained (ONNX Runtime and the OpenVINO runtime are bundled, no OpenVINO install needed), so beyond downloading them you only need the Intel drivers for your hardware:
  • Intel GPU: on Windows, the normal Intel graphics driver. On Linux, the GPU compute driver (e.g. the intel-opencl-icd package on Ubuntu).
  • Intel NPU: on Windows, the Intel NPU driver, which also comes via Windows Update (Windows 11 only). On Linux, the Intel NPU driver. Also set onnxOpenVINODeviceType = NPU in your config, since the default device is GPU.
• OpenVINO recompiles the model at every startup. To cache the compiled model between runs, set onnxOpenVINOCacheDir to a directory in your config.
• Windows also has a separate onnx-directml package, which runs on any DirectX 12 GPU (Intel, AMD, or NVIDIA) if you set onnxProvider = directml. It is self-contained too, and mostly worth trying if the native backends and OpenCL do not work for you.
• See Compiling.md for the status of the other execution providers it supports.
• ONNX is not currently enabled for contribute on this release, due to some concern about the large surface area of possibly flaky execution providers and newness of backend, but it should work fine for all other KataGo usage.

Other / Old GPUs

If you have some other GPU, or older GPUs on which the above doesn't work, or don't want to install extra stuff, try OpenCL. OpenCL will often work in cases others don't support, but be much slower, particularly for transformer models.

MacOS

For MacOS, use the Metal backend, which you can generally get by installing KataGo from homebrew, which usually updates not too long after KataGo's own release.

CPU

If you need a pure-CPU version of KataGo, use Eigen AVX2. It will be quite slow compared to GPU. If somehow you're on an ancient CPU as well and Eigen AVX2 doesn't work, you can try Eigen, which will be even slower.

Other notes

• +bs50 - these are just for fun, and don't support distributed training but DO support board sizes up to 50x50. They may also be slightly slower and will use much more memory, even when only playing on 19x19, so use them only when you really want to try large boards.

• Linux executables were compiled on a 22.04 Ubuntu machine using AppImage. You will still need to install e.g. correct versions of Cuda/TensorRT or have drivers for OpenCL, etc. on your own. Compiling from source is also not so hard on Linux, see the "TLDR" instructions for Linux here.

Major Changes this Release

New backends

(thanks to @Looong01 @seniorfish)

This release adds a ROCm backend for AMD GPUs using HIP and MIOpen, which should be a much faster option than OpenCL for modern AMD GPUs, especially for transformers, and a new ONNX runtime backend (ONNX Runtime) which supports many kinds of hardware, useful for hardware with no other KataGo backend - in particular Intel GPUs and NPUs via the OpenVINO provider, and DirectML on Windows.

Major CUDA and ROCm performance optimizations

(thanks to wyz @doomoooo) and various others for helping explore the major optimizations possible!)

The CUDA backend received a large round of optimization work (batched/fused matrix multiplies, custom flash attention kernel, fused feed-forward layers, reduced CPU-GPU synchronization, and more), with most of the optimizations shared with the new ROCm backend as well:

• Transformer models can be anywhere from 1.2x to 2x faster in overall search speed depending on the GPU, with the largest gains on the newest GPUs (e.g. NVIDIA Blackwell / RTX 50xx).
• Convolutional models are slightly faster, with the biggest gains in configs using multiple neural net server threads.
• GPU memory usage for transformer models on CUDA is greatly reduced.
• Setting numNNServerThreadsPerModel = 2 (or two threads per-GPU, if multiple GPUs) is now often the best-performing config on both CUDA and ROCm - you can try this if you're optimizing for performance.

Support for upcoming rules fix

Added support for fixing an issue with territory scoring (Japanese-like rules). In some thousand-year-ko situations, the cleanup phase could contain pass fights over the ko mouths in seki, making the score and winrate evaluation wrong. KataGo's formal rules are now updated to version 3, where empty points adjacent to a region in atari no longer count as territory - see the updated rules documentation.

The fix will not immediately take effect, but once contributors to https://katagotraining.org/ upgrade and neural nets are trained sufficiently on the new version, then the issue should be fixed. Older nets will still evaluate with the old behavior.

Other user-facing feature additions/changes:

• Added new command katago benchmarknn that benchmarks raw neural net evaluation throughput without any search, useful for comparing backends and GPU settings in isolation.
• Added better GPU error checks and training data quality safeguards to contribute command.
• Removed the cudaDisableWarmup config option, warmup now always runs.

User-facing bugfixes:

• Fixed a crash in the evalsgf command when printing the lead estimate together with the search graph, and fixed tree-averaged ownership missing from its JSON output.

Dev-facing details and changes:

• Added new command katago dumponnx (TensorRT and ONNX backends) that writes out the ONNX graph KataGo builds for a model, and those two backends can also directly load a .onnx file in place of a .bin.gz model, including ONNX files produced by other tooling. See ONNX_Model_Files.md for the format.
• The CUDA and ROCm backends gained several config knobs that disable individual new optimization paths for debugging (e.g. cudaUseMmaAttention, cudaUseFusedFFN). See the source code for the relevant debugging knobs and flags.

Python scripts changes:

• The Python board implementation is updated to match the territory scoring rules fix, and model configs and export support recording whether a net was trained with the new rule.
• Added options to training scripts to penalize attention logits in transformer models from blowing up, and to check for this at export time for models.