github pytorch/FBGEMM v1.9.0
FBGEMM_GPU v1.9.0 Release Notes

latest release: v1.10.0-rc1
40 minutes ago

FBGEMM_GPU v1.9.0 Release Notes

Highlights

TBE GPU Performance

  • Added a small-D CTA kernel specialization for embedding dims ≤128, speeding up common TBE forward/backward shapes (#5711)
  • Shipped an optimized HIP mixed-D warp backward kernel and added mixed-precision support to the HIP warp-per-row backward kernel for faster ROCm training (#5712, #6113)
  • Enabled subwarp-shuffle CTA kernels on ROCm for better warp utilization (#5917)

HBM Streaming & Residency (RES)

  • Introduced a full pipeline for streaming "resident" HBM embedding rows to remote storage — recording touched rows, compacting bitmaps, draining, gathering weights, and shipping to the streamer (#6200, #6221, #6226, #6220, #6227, #6228)
  • Added a res_enabled_tables allowlist and made RES chunk size and consumer/copy-thread counts config-tunable, giving finer control over streaming overhead per table (#6051, #6075, #6217)
  • Added persistent copy/dispatch executors for UVM-hit and HBM-miss lanes to reduce streaming latency (#6079)

Quantization & GenAI

  • Added several new features for HSTU workloads (#4998)
  • Landed Adaptive Regularization support for sparse embeddings, with new frontend dispatch and tests (#5962)
  • Ported u8s8 quantized GEMM to ARM using I8MM and NEON kernels, extending quantized inference to ARM hardware (#6170)
  • Made float16 and bfloat16 distinct types to prevent silent precision bugs (#6049)

Sparse Op Robustness

  • Added a new CUDA kernel for masked_select_jagged_1d (#6190)
  • Closed out int32/HIP grid overflow risks across permute, jagged, and histogram-binning kernels with systematic overflow guards and grid-stride loops (#6081, #6082, #6095)
  • Optimized sparse_permute_2d and reworked group_index_select backward with pre-sorting, caching, and contiguous warp processing for large segments (#5964, #5992)

Platform & Build Health

  • Restored ROCm GPU CI on the MI350 runner and added CUDA 13 family-specific architecture targets, dropping the legacy cu134 build matrix (#6186, #6156, #6139)
  • Added v1.9.0 → torch 2.14 to the compatibility table (#6254)
  • Enabled Pyrefly type checking for deeplearning/fbgemm and expanded compiler warning coverage to catch more issues at build time (#6099, #6144)

Software Requirements

FBGEMM_GPU v1.9.0 has been tested and known to work on the following setups:

  • PyTorch: v2.14
  • CUDA: v12.6, 12.8, 13.0
  • Python: v3.10, 3.11, 3.12, 3.13, 3.14
  • ROCm: v7.0, 7.1

It is recommended to prepare an isolated environment for installing and running FBGEMM_GPU, such as Conda and/or Docker.

Availability

FBGEMM_GPU can be fetched directly from PyPI. The PyPI wheels are built against CUDA 13.0:

# FBGEMM_GPU CUDA variant
pip install fbgemm-gpu==1.9.0

# FBGEMM GenAI variant
pip install fbgemm-gpu-genai==1.9.0

# FBGEMM_GPU CPU variant
pip install fbgemm-gpu-cpu==1.9.0

Alternatively, it can be fetched from PyTorch PIP for a specific CUDA variant:

# FBGEMM_GPU CUDA variant
pip install fbgemm-gpu==1.9.0 --index-url https://download.pytorch.org/whl/cu126/
pip install fbgemm-gpu==1.9.0 --index-url https://download.pytorch.org/whl/cu130/

# FBGEMM GenAI variant
pip install fbgemm-gpu-genai==1.9.0 --index-url https://download.pytorch.org/whl/cu126/
pip install fbgemm-gpu-genai==1.9.0 --index-url https://download.pytorch.org/whl/cu130/

# FBGEMM_GPU CPU variant
pip install fbgemm-gpu-cpu==1.9.0 --index-url https://download.pytorch.org/whl/cpu

Changes

Table Batched Embedding (TBE) Operators

For GPU

  • [Improvement] Refactor CUDA callback functor (#2908)
  • [New] Add a host map option for a UVM tensor alloc (#3073)
  • [New] Add compact_indices op (#3075)
  • [Improvement] Add small-D CTA kernel specialization for max_D <= 128 (#5711)
  • [Improvement] Add optimized TBE backward hip_mixed_d warp kernel (#5712)
  • [Improvement] Enable use_subwarp_shuffle=True for CTA kernel on ROCm (#5917)
  • [New] Adaptive Regularization for sparse embeddings — TBE frontend dispatch + tests (#5962)
  • [Fix] Apply stochastic rounding in the optimized HIP TBE backward kernel (#5972)
  • [Improvement] Add --deterministic flag to TBE benchmark (#5999)
  • [Fix] Add grid-stride + ROCm cap to embedding_inplace_update kernels and pruned_array_lookup_from_row_idx_kernel (#6011)
  • [Fix] Add grid-stride + ROCm cap to tbe_input_combine_with_length_kernel (#6018)
  • [New] Add res_enabled_tables allowlist to scope RES streaming per-table (#6051)
  • [Improvement] Replace TBE type-dispatch macros with C++20 generic-lambda functions (#6052)
  • [Improvement] Drop redundant casts at cap_grid_dim_x in segment_sum/lru_cache/reorder_batched_ad (#6067)
  • [Improvement] Drain the stream queue across N consumer threads (#6069)
  • [Fix] Add grid-stride + ROCm cap to nobag TBE forward (non-index-select) (#6071)
  • [Improvement] Add grid-stride + ROCm cap to grad_indice_weights kernel (#6072)
  • [Improvement] Un-nest tensor_copy_chunk dispatch lambdas (#6074)
  • [Improvement] Make RES chunk size + ship/copy thread counts config-tunable (#6075)
  • [Improvement] Persistent copy/dispatch executors for UVM-hit + HBM-miss lanes (#6079)
  • [Fix] Add ROCm cap to count_unique_indices and find_long_segments launches (#6080)
  • [Fix] Add grid-stride + ROCm cap to nbit inference forward kernel_small_L (#6087)
  • [Fix] Add grid-stride + ROCm cap to nbit pruned hashmap/array lookup kernels (#6092)
  • [Fix] Fix CUDA-13 cudafe++ device-stub const mismatch (backward GWD warp kernel) (#6093)
  • [Improvement] Add grid-stride + ROCm cap to bounds_check_indices_kernel_v1 (#6094)
  • [Fix] Add grid-stride + ROCm cap to lxu_cache_lookup_kernel (#6097)
  • [Fix] Add warp-safe grid-stride + ROCm cap to transpose linearize_index kernels (#6101)
  • [Fix] Create no-FBGEMM_FBCODE stub library to fix no-fbcode streamer test (#6105)
  • [Fix] Add ROCm cap to compact_indices and masked_index (default path) (#6106)
  • [New] Add mixed precision support to the HIP optimized warp per row TBE backward kernel (#6113)
  • [Fix] Move get_maybe_uvm_scalar under FBGEMM_FBCODE for OSS build (#6141)
  • [Fix] Fix TBE writeback dedup crash on CUDA 13 by using 1D torch.unique (#6163)
  • [Improvement] Ship shard RPCs to TrainingPs in parallel (#6168)
  • [New] VLE support on benchmark (#6178)
  • [New] UE support on benchmark (#6179)
  • [Improvement] TBE inference benchmark improvements and scripts (#6185)
  • [Fix] Observe shutdown on teardown instead of running loops to completion (#6189)
  • [Fix] Add an opt-in per-drain sequence number to copy_done_flag (#6196)
  • [Fix] Zero res_count and res_copy_done at construction (#6198)
  • [New] Record every row a lookup touched for HBM streaming (#6200)
  • [Fix] Declare masked_index_select aliasing and register its Meta kernel (#6209)
  • [New] Plumb RES chunk size + consumer/copy-thread config knobs (#6217)
  • [New] Gather the drained rows' weights (#6220)
  • [New] Add sync-free bitmap compaction helper (#6221)
  • [New] Drain the marked rows into row ids (#6226)
  • [New] Ship the drained HBM rows to the streamer (#6227)
  • [Fix] Stream HBM rows from TBEs with no UVM cache (#6228)
  • [Fix] Give each copy_done flag a per-drain token (#6229)
  • [Improvement] Improve error for info_B_num_bits when B overflows (#6236)
  • [New] Add declaration-surface metadata to TBE codegen (#6199) (#6258)

For CPU

  • [Improvement] Add multithreading to table lookup (#5849)
  • [Improvement] Add two-level prefetch into the SVE kernels (#5970)
  • [Improvement] Add two-level prefetch into the NBit autovec kernel (#5971)
  • [Fix] Fix fbgemm::lxu_cache_lookup schema to declare output alias (#6002)
  • [Improvement] Chunk the D2H copy across N copy threads (#6068)
  • [Improvement] Replace raw consumer threads + polled queue with a push executor (#6076)
  • [Improvement] Multi-table CPU pooled coverage for dropped output pre-zero (#6077)
  • [Fix] Skip pruned -1 indices in float autovec/reference SpMDM CPU kernels (#6096)
  • [Improvement] Replace raw dispatcher thread with a named CPUThreadPoolExecutor (#6167)
  • [Fix] Register Meta kernels for get_unique_indices and get_unique_indices_with_inverse (#6208)
  • [Improvement] Skip the unused offsets arange on the CPU nobag TBE fast path (#6213)
  • [Improvement] Add a rolling L1 prefetch to the nbit nobag CPU gather (#6214)
  • [Fix] Fix an off-by-one in the nbit nobag bounds check (#6215)
  • [Improvement] Keep C++ attributes out of generated Torch schemas (#6251)

SSD Table Batched Embedding (TBE) Operators

  • [Fix] Forward-compat sparse_type_to_int fallback for torch.package re-export version blends (#5987)
  • [Fix] derive TBE cache associativity from device warp size (#6037)
  • [Fix] Forward-compat nfp8_dtype fallback for torch.package re-export version blends (#6088)
  • [Fix] Add grid-stride + ROCm cap to ssd_generate/update_row_addrs kernels (#6102)
  • [New] Arch-aware FP8 host-boundary relabel + dispatch guardrails (#6104)
  • [Improvement] Reduce TBE backward arg count — pack weights + aux tensors into TensorLists (#6132)
  • [Improvement] Rename hit-lane copy/dispatch executors + knob to uvm_hit_* for symmetry (#6172)
  • [Improvement] Enable the prefetch pipeline for embedding cache enrichment (#6192)
  • [Fix] Set a 1 minute deadline on test_ssd_cache_flush (#6224)
  • [Fix] Skip RES HBM tests when op absent in OSS wheel (#6233)

GenAI Support and Operators

Triton GEMM Support

  • [Improvement] mx4 quantize: compute per-group scale via exact FP32 bit-construction instead of FP64 exp2 (#6083)
  • [Fix] num_stages=0 -> 1 (#6111)
  • [Improvement] mx4 dequantize: scale via FP32 bit-construction, not FP64 exp2 (#6148)
  • [Fix] Mask the column-scale load in _kernel_scale_fp8_row (#6158)

Quantization Operators

  • [New] Add several features for HSTU. (#4998)
  • [Fix] Add dtype check for rowwise_min_max (#5980)
  • [Improvement] Suppress harness-incompatible opcheck variants (fp8 1d roundtrip, repeat_arange) (#5990)
  • [Fix] Stabilize flaky fused_nbit faketensor opcheck (#6005)
  • [Improvement] Small tweak to FusedNBitRowwiseQuantizedSBHalfToFloatOrHalfNeon (#6021)
  • [Fix] De-flake FastGemvTests (seed + dequantized reference) (#6027)
  • [Improvement] add kWarpSizeHost() for host-side launch configs (#6031)
  • [Fix] Fix HIP grid overflow in _compute_msfp_shared_exponent_cuda_kernel (#6036)
  • [Improvement] Make float16/bfloat16 distinct types (#6049)
  • [Fix] Add grid-stride + ROCm cap to fused_8bit_rowwise quantize kernels (#6109)
  • [Fix] Add grid-stride + ROCm cap to fp8_rowwise quantize kernels (#6110)
  • [Fix] Add grid-stride + ROCm cap to padded_fp8_rowwise quantize kernels (#6112)
  • [Fix] Add ROCm cap to fused_nbit_rowwise dequant launch (#6118)
  • [Improvement] Apply the host warning set to nvcc (#6173)
  • [Fix] Fix differing_executors health check failures in quantize tests (#6205)

Sparse Operators

  • [Improvement] optimize sparse_permute_2d kernel (#5964)
  • [Fix] Exclude warm-ups from group_index_select_2d trace; raise to 50 (#5978)
  • [Fix] Fix HIP grid overflow in batched_dense_vec_jagged_2d_mul_backward (transposed_bmm + outer_prod) (#5984)
  • [Fix] Fix HIP grid overflow in jagged_dense_elementwise_mul_backward (#5985)
  • [Fix] Fix HIP grid overflow in jagged_dense_dense_elementwise_jagged_output_kernel (#5989)
  • [Improvement] Implement pre-sorting, caching and contigous warp processing in group_index_select backward (#5992)
  • [Fix] Add grid-stride + ROCm cap to repeat_arange_kernel (#5995)
  • [Improvement] Un-cascade jagged pt2_compliant_tag tests (#5996)
  • [Improvement] group_index_select: capture onesweep sort + data-driven pattern-group report split (#6000)
  • [Fix] Fix group_index_select bwd sort dispatch: per-group onesweep for large segments (ROCm) (#6001)
  • [Fix] Add grid-stride + ROCm cap to get_source_mask_kernel (#6003)
  • [Fix] Add grid-stride + ROCm cap to jagged_dense_dense_elementwise_jagged_output_opt_search_kernel_ (#6006)
  • [Fix] group_index_select backward correctness fixes (ROCm) (#6008)
  • [Improvement] Simplify redundant grid-cap clamp in embedding_inplace_update (#6017)
  • [Improvement] Add grid-stride loops + ROCm cap to jagged_dense_bmm_kernel (#6019)
  • [Improvement] Add grid-stride loops + ROCm cap to jagged_jagged_bmm_kernel (#6023)
  • [Fix] Add grid-stride + ROCm cap to keyed_jagged_index_select_dim1_kernel and keyed_jagged_index_add_dim1_kernel; fast-fail TORCH_CHECK for index_select_scalar_cumsum_kernel (#6025)
  • [Fix] int64 permute support in permute_1D_sparse_data (#6029)
  • [Fix] Load fbgemm_gpu_embedding_inplace_ops .so in OSS init (#6034)
  • [Fix] Fix HIP grid overflow in jagged_softmax (account for grid.y in cap) (#6038)
  • [Improvement] Add grid-stride + ROCm cap to permute_multi_embs_kernel (#6040)
  • [New] Add grid-stride + ROCm cap to permute_pooled_embs_kernel (#6041)
  • [Fix] Skip >600s-timeout sparse cases (#6047)
  • [Fix] Add ROCm cap to permute_pooled_embs_split host launch (#6050)
  • [Fix] Cap forward grid by T*N for batched_unary_embeddings on ROCm (#6055)
  • [Fix] Add grid-stride + ROCm cap to delinearize_unique_index_kernel (#6058)
  • [Improvement] Drop redundant uint32_t narrowing at cap_grid_dim_x in jagged softmax/batched_dense_vec (#6061)
  • [Improvement] Drop redundant casts at cap_grid_dim_x in jagged elementwise dispatch (#6062)
  • [Fix] Add int32 overflow guards to CUDA sparse permute ops (#6081)
  • [Fix] Fold grid.y/z into permute cap_grid_dim_x to fix HIP 2^32 overflow (#6082)
  • [Improvement] Add permute sparse-data benchmark scripts and --embed-dim option (#6090)
  • [New] Add opt-in host-side validation for CUDA sparse permute ops (#6095)
  • [Improvement] Add grid-stride + ROCm cap to histogram_binning_calibration kernels (#6121)
  • [Fix] Guard permute_2D_sparse_data against undersized permuted_indices (#6122)
  • [Fix] Add grid-stride + ROCm cap to compute_frequency_sequence kernel (#6124)
  • [Fix] Fix over-strict GPU memory gate on opt_search_large_grid test (#6125)
  • [Improvement] Make batched cumsum AOT test traceable (#6126)
  • [Improvement] Move giant Zipf overflow repro out of CI (#6128)
  • [Fix] Gate permute_2D device assert at compile time (fixes 2 GiB relocation overflow) (#6130)
  • [Fix] Add ROCm cap to invert_permute launch (#6134)
  • [Improvement] Bound block-bucketize stress generation (#6135)
  • [Improvement] Gate permute_1D lengths device assert at compile time (#6136)
  • [Improvement] Avoid ATen reduction in masked jagged select (#6140)
  • [Fix] Add ROCm cap to offsets_range launches (#6142)
  • [Fix] Add ROCm cap to pack_segments backward (unpack) launch (#6146)
  • [Fix] Add ROCm cap to block_bucketize + populate_bucketized fallback launches (#6162)
  • [New] Add a CUDA kernel for masked_select_jagged_1d (#6190)
  • [Fix] Relocate permute_multi_embedding index tensors instead of failing (#6243)
  • [Improvement] Mark platform-conditioned GPU parameters unused (#6250)
  • [Improvement] Avoid redundant zero fill in pack_segments (#6257)

Platform Support

CUDA

  • [Improvement] Use CUDA 13 family-specific architecture targets (#6156)
  • [Improvement] Drop CUDA 13.4 (cu134) from all Linux wheel build matrices (#6139)

ARM / AArch64

  • [New] Port u8s8 quantized GEMM to arm with I8MM and NEON compute kernels (#6170)
  • [Fix] Make x86-only conv kernels fail loudly on arm instead of faulting or returning zeros (#6177)
  • [Fix] Skip TransposedRequantizeTest on arm instead of asserting against an empty buffer (#6169)

ROCm / AMD

  • [Fix] [ROCm][CI] Restore ROCm GPU CI: switch to the MI350 runner label (#6186)
  • [Fix] [ROCm] gate warpReduceAllSum DPP path to gfx942/gfx90a/gfx950 (#6014)
  • [Improvement] [ROCm] plumb per-build wave-size flags from CMake into codegen (#6103)
  • [Fix] Fix new rocm version build: use __syncthreads() instead of removed hipcub::CTA_SYNC (#6143)
  • [Fix] Remove unreachable ROCm warp instantiations (#6197)
  • [Improvement] Isolate ROCm index-select boundary coverage (#6138)
  • [Improvement] Update OSS Composable Kernel SHA (#6223)

Build / CI Improvements and Better Engineering

  • [Improvement] Increase Nova timeout to 180 mins (#3102)
  • [Improvement] Shared CPU two-level prefetch helpers + env knobs (#5969)
  • [Improvement] Use Python 3.10+ typing in remaining files (#5975)
  • [Fix] Move x86 CPU test runner to linux.4xlarge.memory to fix py3.12+ OOM (#6010)
  • [Improvement] Migrate contained C10_LIKELY/C10_UNLIKELY to [[likely]]/[[unlikely]] (#6012)
  • [Fix] Clamp cap_grid_dim_x to the 2^31-1 grid-x limit (+ unit test) (#6013)
  • [Fix] Fix MSan uninitialized reads in FBGEMM depthwise convolution and INT64 GEMM packing (#6028) (#6032)
  • [New] Add grid-stride + ROCm cap to recat_copy_async_kernel (#6039)
  • [Fix] Fix build break when using llvm-21 (#6046)
  • [Fix] Add fast-fail TORCH_CHECK for auc_kernel HIP grid overflow (#6054)
  • [Improvement] Install click as part of setup (#6056)
  • [Fix] Skip test_l2_norm_pruning_workflow in OSS CI (#6073)
  • [Improvement] Enable Pyrefly for deeplearning/fbgemm (#6099)
  • [Improvement] Pass third-party include dirs as SYSTEM includes (#6127)
  • [Improvement] Treat asmjit and cpuinfo headers as system headers (#6131)
  • [Improvement] Add fp8 corner case unit test (#6133)
  • [Improvement] Derive nvcc and hipcc warning lists from one source of truth (#6144)
  • [Fix] Downgrade -Werror=maybe-uninitialized on GCC to unblock nightly wheel build (#6147)
  • [Improvement] Add -Wunused-value (#6150)
  • [Improvement] Split four more -Wall-implied warnings across compiler buckets (#6153)
  • [Improvement] Add -Wmissing-braces (#6154)
  • [Improvement] 02c.2 -- HIP and nvcc reconnaissance: warnings on, -Werror off (#6155)
  • [Improvement] Docs: Add fbgemm-ascend to community ports (#6164)
  • [Improvement] Add init and control-flow warnings, split across buckets (#6174)
  • [Improvement] Add three clang-leaning warnings, one of them portable (#6175)
  • [Improvement] Fix B001: Replace bare except with except Exception (#6182)
  • [Improvement] Fix B001: Replace bare except with except Exception (#6183)
  • [Improvement] Add four warnings for deprecated C++ constructs (#6203)
  • [Improvement] Move -Wmismatched-tags to the portable list (#6206)
  • [Improvement] Add four warnings for types and templates (#6207)
  • [Improvement] Add four warnings for conversions and literals (#6210)
  • [Improvement] Add packed-member and object-lifetime warnings (#6211)
  • [Improvement] Add -Wshift-sign-overflow (#6212)
  • [Improvement] Add -Wzero-as-null-pointer-constant, without errors (#6218)
  • [Improvement] Add -Wunused-exception-parameter (#6225)
  • [Improvement] Add -Wshorten-64-to-32, without errors (#6230)
  • [Improvement] Add -Wheader-hygiene, without errors on device code (#6234)
  • [Improvement] Add -Wshadow, without errors (#6239)
  • [Improvement] Add three warnings for unused variables (#6240)
  • [Improvement] Add tensor device relocation helper (#6246)
  • [Improvement] Add four clang-only warnings for Objective-C and C (#6249)
  • [Fix] Guard GPU device queries on CPU lane in verify script (#6252)
  • [Improvement] Add four clang-only warnings that can occur in C++ (#6253)
  • [Improvement] Add v1.9.0 -> torch 2.14 to compat table (#6254)
  • [Improvement] Build FBGEMM release CUDA wheels on linux.12xlarge.memory (#6284)

Tests and Benchmarks

  • [Improvement] Migrate test/tbe/** TBE type imports to fbgemm_gpu.tbe.* leaves (Phase 3.9) (#6066)
  • [New] Add opt-in whole-run multi-stream timing (#5988)
  • [Improvement] Migrate -c fbcode.platform010_cuda_version to the cuda constraint modifier (#6100)
  • [Improvement] Package benchmark utilities in wheels (#6232)

Don't miss a new FBGEMM release

NewReleases is sending notifications on new releases.