CRITICAL ATTENTION KERNEL BUG FIXES OVER 0.14b

• flyc gfx950 kernel baked scaling factors to in-register fp16/bf16 Q tensor,
  losing precision in proportion to sm_scale * sqrt(head_dim).
  • This is a known problem to Triton kernel, which was fixed in AOTriton 0.7.1b.
• Triton backward kernels mishandle scaling factor, which would produce NaN for
  non-positive scaling factors

The following sections are the original 0.14b release notes, with updated performance numbers.

Overview

AOTriton 0.14 is a feature release in the maintenance-feature cadence
introduced in 0.13: it synchronizes Triton with upstream, adds kernel features,
and updates the tuning database.

Headline changes:

FlyDSL SDPA Kernel Ship as the flyc backend

• Available for gfx950/gfx1201
• Feature Parity with Triton kernel

Gfx950 Performance Boost from 0.13b

d_head=512 is measured with sq/sk=8192.

Geomean over all 100 points: forward 1.41x, backward 1.46x, overall 1.44x.

Gfx1201 Performance Boost from 0.13b

d_head=384/512 is measured with sq/sk=8192 due to VRAM limits.

Geomean over all 98 measured points: forward 1.47x, backward 1.45x, overall
1.46x.

VarlenBits supersedes VarlenType

VarlenBits massively expands the expressibility of specifying varlen inputs:
users can now configure sequence accessing independently for Q and K, along
three choice axes:

• Input tensor shape is BHSD or 1THD
  • Certain varlen inputs are not stacked (T) but padded to S uniformly
• Sequence lengths are uniform, given as cumulative sums, or specified
  individually by an array
• Sequence offsets are uniform, re-used from the cumulative sums above, or
  specified by a second array

Concretely, VarlenBits is defined as (abridged from
include/aotriton/flash.h):

struct VarlenMode {       // one side's three axes, one byte
  uint8_t stacked : 1, length : 2, position : 2, reserved : 3;
};

struct alignas(uint32_t) VarlenBits {
  VarlenMode qmode;         // bits  7:0  -- Q and K decode independently,
  VarlenMode kmode;         // bits 15:8     so mixed addressing is expressible
  uint16_t lse_layout : 2;  // bits 17:16
  uint16_t reserved   : 14; // bits 31:18
};

The extra lse_layout enables both HT and TH memory layouts for the LSE
tensor.

API BREAKING, BUT ABI STABLE. AOTriton has no plan to promise
source-level compatibility, due to the maintenance cost. However, AOTriton
begins delivering ABI compatibility with 0.14b. Applications targeting 0.13b
can upgrade to 0.14b as a drop-in replacement (note that aotriton.images
needs to be replaced in this process as well). The library automatically
translates passed-in 0.13b params structs to 0.14b.

Full Modularization

The source tree is fully modularized: everything flash-specific now lives
under modules/flash/.

pip install . at the root of the source tree installs the Python package
aotriton into the environment, which serves as the family-neutral toolchain
that transforms Triton/FlyDSL kernels into a C++ library with precompiled GPU
kernels.

pytest-gpu-lease: one GPU per pytest-xdist worker

A standalone pytest plugin (#215, #223), shipped in-tree but useful to anyone
running GPU tests under pytest-xdist:

pip install ./python/pytest-gpu-lease

It activates through the pytest11 entry point — no conftest.py wiring.
Each worker leases exactly one GPU through fcntl byte-range locks on a shared
lock file, and tests take it from the gpu_id (int) or gpu_device
('cuda:3') fixture. Set GPU_LEASE_PIN to bypass leasing and pin every
worker to one GPU instead.

The optional watchdog monitors the heartbeat timestamp each pytest-xdist
worker writes into the lock file, and terminates a worker that stops making
progress with a SIGTERM-then-SIGKILL escalation, separated by a grace period.
Monitoring out-of-process sidesteps the limits of in-process stall detection
like pytest-timeout, whose SIGALRM only takes effect at the next bytecode
boundary — which a thread parked in a blocking HIP call never reaches.

export GPU_LEASE_LOCKFILE=/dev/shm/gpu_lease.$$   # both sides must agree
python -m pytest_gpu_lease.watchdog --lockfile "$GPU_LEASE_LOCKFILE" --workers 4 &
pytest -n 4 ...

Major Changes from Release 0.13 Beta

API Changes

• BREAKING VarlenType is replaced by VarlenBits (#222).
  • Spells configurations the enum could not: mixed per-side addressing,
    seqused_k on a packed KV cache, and TransformerEngine's (T, H)
    logsumexp layout (VarlenLseLayout::TH; default stays HT).
  • Covers torch.nn.attention.varlen.
• ABI backward compatibility is now formally supported (#222).
• Backends of the flash module now have names (#230).
  • For use with attn_options.force_backend_index.
  • Declared in the new generated public header <aotriton/flash/backends.h>.
    • OpAttnFwdBackend/OpAttnBwdBackend constants.
  • Also exposed under pyaotriton.v3.flash for the Python testing API.

FlyDSL / flyc Backend (new)

• gfx1201 and gfx950 SDPA kernels developed in FlyDSL are vendored under
  modules/flash/flyc/, cross-compiled ahead of time by
  python/flyc_compile.py, and packed into aotriton.images (#227, #230).
• Operator tuning now probes flyc; the shipped operator database selects it
  for gfx950/gfx1201 training and inferencing on non-fp32 inputs (#239).
• 0.14b must not be built from the released flydsl==0.3.1 wheel — upstream
  LLVM miscompiles register spills and produces wrong kernels rather than
  absent ones. .ci/build_llvm_tarball.sh and .ci/build_flydsl_wheel.sh
  build the pinned LLVM and the compiler wheel from source; the pins live in
  third_party/flydsl-{compiler,kernel,llvm}.txt (#234).

Triton Compiler and Databases

• Triton bumped to ROCm/triton release/internal/3.8.x (3.8.0, aa3cf1a1).
• AITER ASM kernels bumped to v0.1.22 (#238), including the backward odo
  argument ABI change — the vendored dispatchers and the pin move together.
• Kernel tuning database fully re-tuned for gfx950 and gfx1100;
  gfx90a/gfx942/gfx1201 validated against the level-3 UT (#239).
• Operator tuning database updated for gfx950/gfx1201 (#239).
• Partial gfx1150/gfx1151 flash databases for hdim=64, dropout=0.0
  (#200) — attn_fwd averages 61–76% faster than the default configuration.

Kernel Bug Fixes

• attn_fwd: Fix incorrect early exits in persistent dynamic mode (#237,
  #235).
• bwd_kernel_fuse: Fix addressing dO with Out's strides in the dK/dV
  phase (#237, #236).
• attn_fwd: Fix the causal early-exit +inf LSE OOB writes, which corrupted
  the next sequence's LSE under a stacked varlen layout (#222).
• attn_fwd: Fix scaling of bias, which should be cast to fp32 first
  (#228, #232).

Runtime Bug Fixes

• Thread safety: the following functions no longer rely on the caller holding
  the GIL (#224):
  • getGpuFromStream/isArchExperimentallySupported/isArchTechPreview.
  • getMultiProcessorCount.
  • TritonKernel::invoke/TritonKernel::direct_invoke and
    AiterAsmKernel::launch_kernel
    • Now take the device from the stream (hipStreamGetDevice) instead of the
      calling thread's ambient current device, which can differ from the
      stream's.
  • mha_fwd's AiterAsmKernel cache: thread_local dropped, matching
    mha_bwd. It built a separate copy of every kernel for each new thread.
• OnDeviceKernel::get_kernel propagates hipError_t; a missing kernel now
  fails with hipErrorInvalidImage (#238).
  • Otherwise a later call would fail due to a null hipFunction_t.

Build and Infrastructure

• Modularization complete. Flash C++ sources, pybind bindings, tuning
  database, test suite, Triton kernels and tuning logic all live under
  modules/flash/ (#195–#199); v3python/ is deleted and the tuning stack is
  the aotriton.tune package under python/tune/ (#212).
• ROCm installed from pip wheels or a tarball is now supported (available
  since ROCm 7.14): find_package(hip) honours $ENV{ROCM_PATH} instead of
  assuming a system install at /opt/rocm (#207).

Minor Changes from Release 0.13 Beta

• attn_options::KernelSlot gains the FlyDSL kernels (#230). Tuning builds
  only — the enum lives behind AOTRITON_BUILD_FOR_TUNING.
• flyc is registered as op_attn_fwd backend 2 and op_attn_bwd backend 3,
  both metros mixing FlyDSL and Triton steps on one stream (#230).
• FWD_IMPL/BWD_IMPL take a backend name (triton, aiter, flyc,
  triton_split, triton_fuse); V3_API, FORCE_FWD_BACKEND and
  FORCE_BWD_BACKEND are removed (#231).
• test_forward.py is deprecated — use SKIP_BWD=1 test_backward.py (#231).
• New level-0 tests: test_prime_hdim and test_memory_layouts (#231),
  test_bottom_right_fully_masked_rows (#237), test_dropout_layout (#226),
  and test_varlen_bits (#222).
• bwd_preprocess_varlen is fused into bwd_preprocess, changing the kernel
  numbering inside attn_options (#222).
• fast_philox emitted [r0, r2, r1, r3] because tl.join appends a minor
  axis; the four randoms of one offset now land in generation order (#226).
  The API returns the same mask either way — the win is fewer registers and a
  word order other Philox implementations can match.
• "Plain Object Notation" (PON) formalizes the existing k=v;k=v wire format
  used by exaid and the flyc build targets (#225).
• __signature__ gains FLYC_VERSION (#234).
• Flash kernel tuning configuration generators, left as single-config
  placeholders during the ATI migration, are restored (#216).
• Widened the gfx1100 attn_fwd tuning space for inference functionals, and
  dropped the stale gfx11 seqlen > 2048 tuning gate (#220). Tuning-time
  generation only; release builds come from database rows.
• Operator tuning stores a second-best backend (op$best1st/op$best2nd);
  gfx1200 falls back to op$best2nd when the shared gfx1201 row picks flyc,
  since no gfx1200 flyc kernel is shipped (#239).
• Tuner v3.5: kernel-level and operator-level tuning are unified into one
  module, discriminated by a new tuning_level column; dispatch flash_op is
  now dispatch flash --tuning_mode op (#213). The WebUI follows (#214).
  • CAVEAT: no migration path. Every existing tuning database must be
    re-initialized and re-tuned, and the fleet must be drained and restarted.
    Starting from a fresh work directory is recommended.
• amd-smi: resolve the device by PCI BDF so HIP_VISIBLE_DEVICES is
  honoured, read vram_total as MB, and fall back between junction and edge
  temperature sensors (#217).
• Tests can force a CPU reference per test via USE_ADIFFS_TXT=CPUREF or
  AOTRITON_REF_DEVICE_OPTIONS=cpu, working around ROCm 7.14 math_sdp
  defects (#231, #239).
• AOTRITON_DEBUG_SKIP_TRITON_KERNELS builds only the Triton kernels the flyc
  backend needs (#230).
• CMake: .nsv added to the AKS2 packing DEPENDS, fixing stale packs when
  the hsaco manifest changes (#239).
• CI builds CPython >= 3.13 from source in base.Dockerfile; AlmaLinux 8 has
  no such package (#211).

Known Problems

• gfx1150 and gfx1151 are not fully validated against the updated Triton
  compiler, due to hardware availability.
• gfx1250 GPU images are not tested. A tech preview release built with an
  internal compiler is planned for better gfx1250 support.
• The op$best2nd mechanism that protects gfx1200 from gfx1201's flyc
  selection has never been exercised on real gfx1200 hardware.
• gfx950 flyc kernels natively support head dimensions that are multiples of
  32 only.
• flyc hsaco build rules have no proper dependency tracking, the same
  limitation Triton hsaco rules have: neither DSL toolchain ships a -MD
  equivalent.
• On ROCm 7.14.x, Math SDP is unreliable on both gfx950 and gfx1201. It should
  be documented with adiff files, which are not yet written.
• The AITER forward .co packaging globs every *.co, so affine_kernels.zip
  carries uncallable fp8bf16 kernels (+383.9 KiB on gfx950).
• third_party/flydsl-llvm.txt must be emptied once the upstream LLVM fix
  lands, in the same change that bumps flydsl-compiler.txt.

What's Changed (Auto-Generated)

• Modularization Step 1: Relocate flash attention specific C++ code to modules/flash/csrc by @xinyazhang in #195
• Modularization Step 2: Move tuning database to per-family modules/flash/database by @xinyazhang in #196
• Modularization Step 3: Relocate flash attention API bindings to modules/flash/bindings by @xinyazhang in #197
• Modularization Step 4: Make python/test self-contained and migrate tritonsrc dev harness to modules/flash/kernel by @xinyazhang in #198
• Modularization Step 5: Move SDPA test suite to modules/flash/tests by @xinyazhang in #199
• Add gfx1150 and gfx1151 flash tuning databases for hdim=64, dropout=0.0 by @amd-callumm in #200
• Port Fixes found in 0.13.50 Tech Preview to Mainline by @xinyazhang in #207
• ci: build Python >= 3.13 from source in base.Dockerfile by @xinyazhang in #211
• Modularization Step 6: Tuner v3.5 (Phase 1) by @xinyazhang in #212
• Add pytest-gpu-lease package for pytest-xdist GPU assignment by @xinyazhang in #215
• Modularization Step 6: Tuner v3.5 (Phase 2 - Tuning Level Unification) by @xinyazhang in #213
• Modularization Step 6: Tuner v3.5 (Phase 3 - WebUI) by @xinyazhang in #214
• Fix amd-smi usage related issues introduced in Tuner V3.5 by @xinyazhang in #217
• Restore flash kernel tuning config generators in ATI by @nemanjaudovic in #216
• Widen attn_fwd forward tuning space on gfx1100 by @nemanjaudovic in #220
• Replace VarlenType with VarlenBits; add ABI version translation by @xinyazhang in #222
• Fix various race conditions by @xinyazhang in #224
• Add watchdog to pytest-gpu-lease package by @xinyazhang in #223
• FlyDSL Integration PR1: Introduce PON and Deserializable ChoiceView by @xinyazhang in #225
• FlyDSL Integration PR2: Rearrange the Word Order Produced by the Triton fast_philox Kernel by @xinyazhang in #226
• fixed bias and alibi_block f32 asymmtry between the fwd and bwd by @Chi-Chu319 in #228
• FlyDSL Integration PR3: Vendor FlyDSL SDPA Kernels and Add the AOT Compiler Driver by @xinyazhang in #227
• FlyDSL Integration PR4: Prepare ATI Codegen for a Second Language by @xinyazhang in #229
• FlyDSL Integration PR5: Compile, Pack and Dispatch FlyDSL attention kernels as "Flyc" Backend by @xinyazhang in #230
• FlyDSL Integration PR6: Test the flyc Backend, and Fix the gfx950 Slab Bound by @xinyazhang in #231
• FlyDSL Integration PR7: CI Script to Build the FlyDSL/LLVM from Source by @xinyazhang in #234
• unit test added for bias asymmetry by @Chi-Chu319 in #232
• Bump AITER to v0.1.22 by @xinyazhang in #238
• Fix attn_fwd persistent-loop tile skipping and bwd_kernel_fuse dO strides by @xinyazhang in #237
• 0.14 Final Tuning: Triton 3.8 Databases and the Flyc Operator Backend by @xinyazhang in #239
• Tune: remove codegen's dependency on aotriton.tune by @xinyazhang in #243
• Split the Runtime into a ROCm-Independent aotriton_common and a HIP-Dependent Shim by @xinyazhang in #244
• Do not bake scales into the loaded Q tensor in the gfx950 flyc kernel, and fix non-positive scales in the triton backward kernels by @xinyazhang in #247

New Contributors

• @amd-callumm made their first contribution in #200
• @nemanjaudovic made their first contribution in #216
• @Chi-Chu319 made their first contribution in #228

Full Changelog: 0.13b...0.14.2b