FBGEMM_GPU v1.9.0 Release Notes
Highlights
TBE GPU Performance
- Added a small-D CTA kernel specialization for embedding dims ≤128, speeding up common TBE forward/backward shapes (#5711)
- Shipped an optimized HIP mixed-D warp backward kernel and added mixed-precision support to the HIP warp-per-row backward kernel for faster ROCm training (#5712, #6113)
- Enabled subwarp-shuffle CTA kernels on ROCm for better warp utilization (#5917)
HBM Streaming & Residency (RES)
- Introduced a full pipeline for streaming "resident" HBM embedding rows to remote storage — recording touched rows, compacting bitmaps, draining, gathering weights, and shipping to the streamer (#6200, #6221, #6226, #6220, #6227, #6228)
- Added a
res_enabled_tablesallowlist and made RES chunk size and consumer/copy-thread counts config-tunable, giving finer control over streaming overhead per table (#6051, #6075, #6217) - Added persistent copy/dispatch executors for UVM-hit and HBM-miss lanes to reduce streaming latency (#6079)
Quantization & GenAI
- Added several new features for HSTU workloads (#4998)
- Landed Adaptive Regularization support for sparse embeddings, with new frontend dispatch and tests (#5962)
- Ported u8s8 quantized GEMM to ARM using I8MM and NEON kernels, extending quantized inference to ARM hardware (#6170)
- Made float16 and bfloat16 distinct types to prevent silent precision bugs (#6049)
Sparse Op Robustness
- Added a new CUDA kernel for
masked_select_jagged_1d(#6190) - Closed out int32/HIP grid overflow risks across permute, jagged, and histogram-binning kernels with systematic overflow guards and grid-stride loops (#6081, #6082, #6095)
- Optimized
sparse_permute_2dand reworkedgroup_index_selectbackward with pre-sorting, caching, and contiguous warp processing for large segments (#5964, #5992)
Platform & Build Health
- Restored ROCm GPU CI on the MI350 runner and added CUDA 13 family-specific architecture targets, dropping the legacy cu134 build matrix (#6186, #6156, #6139)
- Added
v1.9.0→ torch 2.14 to the compatibility table (#6254) - Enabled Pyrefly type checking for
deeplearning/fbgemmand expanded compiler warning coverage to catch more issues at build time (#6099, #6144)
Software Requirements
FBGEMM_GPU v1.9.0 has been tested and known to work on the following setups:
- PyTorch: v2.14
- CUDA: v12.6, 12.8, 13.0
- Python: v3.10, 3.11, 3.12, 3.13, 3.14
- ROCm: v7.0, 7.1
It is recommended to prepare an isolated environment for installing and running FBGEMM_GPU, such as Conda and/or Docker.
Availability
FBGEMM_GPU can be fetched directly from PyPI. The PyPI wheels are built against CUDA 13.0:
# FBGEMM_GPU CUDA variant
pip install fbgemm-gpu==1.9.0
# FBGEMM GenAI variant
pip install fbgemm-gpu-genai==1.9.0
# FBGEMM_GPU CPU variant
pip install fbgemm-gpu-cpu==1.9.0Alternatively, it can be fetched from PyTorch PIP for a specific CUDA variant:
# FBGEMM_GPU CUDA variant
pip install fbgemm-gpu==1.9.0 --index-url https://download.pytorch.org/whl/cu126/
pip install fbgemm-gpu==1.9.0 --index-url https://download.pytorch.org/whl/cu130/
# FBGEMM GenAI variant
pip install fbgemm-gpu-genai==1.9.0 --index-url https://download.pytorch.org/whl/cu126/
pip install fbgemm-gpu-genai==1.9.0 --index-url https://download.pytorch.org/whl/cu130/
# FBGEMM_GPU CPU variant
pip install fbgemm-gpu-cpu==1.9.0 --index-url https://download.pytorch.org/whl/cpuChanges
Table Batched Embedding (TBE) Operators
For GPU
- [Improvement] Refactor CUDA callback functor (#2908)
- [New] Add a host map option for a UVM tensor alloc (#3073)
- [New] Add compact_indices op (#3075)
- [Improvement] Add small-D CTA kernel specialization for
max_D <= 128(#5711) - [Improvement] Add optimized TBE backward hip_mixed_d warp kernel (#5712)
- [Improvement] Enable
use_subwarp_shuffle=Truefor CTA kernel on ROCm (#5917) - [New] Adaptive Regularization for sparse embeddings — TBE frontend dispatch + tests (#5962)
- [Fix] Apply stochastic rounding in the optimized HIP TBE backward kernel (#5972)
- [Improvement] Add --deterministic flag to TBE benchmark (#5999)
- [Fix] Add grid-stride + ROCm cap to embedding_inplace_update kernels and pruned_array_lookup_from_row_idx_kernel (#6011)
- [Fix] Add grid-stride + ROCm cap to tbe_input_combine_with_length_kernel (#6018)
- [New] Add res_enabled_tables allowlist to scope RES streaming per-table (#6051)
- [Improvement] Replace TBE type-dispatch macros with C++20 generic-lambda functions (#6052)
- [Improvement] Drop redundant casts at cap_grid_dim_x in segment_sum/lru_cache/reorder_batched_ad (#6067)
- [Improvement] Drain the stream queue across N consumer threads (#6069)
- [Fix] Add grid-stride + ROCm cap to nobag TBE forward (non-index-select) (#6071)
- [Improvement] Add grid-stride + ROCm cap to grad_indice_weights kernel (#6072)
- [Improvement] Un-nest tensor_copy_chunk dispatch lambdas (#6074)
- [Improvement] Make RES chunk size + ship/copy thread counts config-tunable (#6075)
- [Improvement] Persistent copy/dispatch executors for UVM-hit + HBM-miss lanes (#6079)
- [Fix] Add ROCm cap to count_unique_indices and find_long_segments launches (#6080)
- [Fix] Add grid-stride + ROCm cap to nbit inference forward kernel_small_L (#6087)
- [Fix] Add grid-stride + ROCm cap to nbit pruned hashmap/array lookup kernels (#6092)
- [Fix] Fix CUDA-13 cudafe++ device-stub const mismatch (backward GWD warp kernel) (#6093)
- [Improvement] Add grid-stride + ROCm cap to bounds_check_indices_kernel_v1 (#6094)
- [Fix] Add grid-stride + ROCm cap to lxu_cache_lookup_kernel (#6097)
- [Fix] Add warp-safe grid-stride + ROCm cap to transpose linearize_index kernels (#6101)
- [Fix] Create no-FBGEMM_FBCODE stub library to fix no-fbcode streamer test (#6105)
- [Fix] Add ROCm cap to compact_indices and masked_index (default path) (#6106)
- [New] Add mixed precision support to the HIP optimized warp per row TBE backward kernel (#6113)
- [Fix] Move get_maybe_uvm_scalar under FBGEMM_FBCODE for OSS build (#6141)
- [Fix] Fix TBE writeback dedup crash on CUDA 13 by using 1D torch.unique (#6163)
- [Improvement] Ship shard RPCs to TrainingPs in parallel (#6168)
- [New] VLE support on benchmark (#6178)
- [New] UE support on benchmark (#6179)
- [Improvement] TBE inference benchmark improvements and scripts (#6185)
- [Fix] Observe shutdown on teardown instead of running loops to completion (#6189)
- [Fix] Add an opt-in per-drain sequence number to copy_done_flag (#6196)
- [Fix] Zero res_count and res_copy_done at construction (#6198)
- [New] Record every row a lookup touched for HBM streaming (#6200)
- [Fix] Declare masked_index_select aliasing and register its Meta kernel (#6209)
- [New] Plumb RES chunk size + consumer/copy-thread config knobs (#6217)
- [New] Gather the drained rows' weights (#6220)
- [New] Add sync-free bitmap compaction helper (#6221)
- [New] Drain the marked rows into row ids (#6226)
- [New] Ship the drained HBM rows to the streamer (#6227)
- [Fix] Stream HBM rows from TBEs with no UVM cache (#6228)
- [Fix] Give each copy_done flag a per-drain token (#6229)
- [Improvement] Improve error for
info_B_num_bitswhen B overflows (#6236) - [New] Add declaration-surface metadata to TBE codegen (#6199) (#6258)
For CPU
- [Improvement] Add multithreading to table lookup (#5849)
- [Improvement] Add two-level prefetch into the SVE kernels (#5970)
- [Improvement] Add two-level prefetch into the NBit autovec kernel (#5971)
- [Fix] Fix
fbgemm::lxu_cache_lookupschema to declare output alias (#6002) - [Improvement] Chunk the D2H copy across N copy threads (#6068)
- [Improvement] Replace raw consumer threads + polled queue with a push executor (#6076)
- [Improvement] Multi-table CPU pooled coverage for dropped output pre-zero (#6077)
- [Fix] Skip pruned -1 indices in float autovec/reference SpMDM CPU kernels (#6096)
- [Improvement] Replace raw dispatcher thread with a named CPUThreadPoolExecutor (#6167)
- [Fix] Register Meta kernels for get_unique_indices and get_unique_indices_with_inverse (#6208)
- [Improvement] Skip the unused offsets arange on the CPU nobag TBE fast path (#6213)
- [Improvement] Add a rolling L1 prefetch to the nbit nobag CPU gather (#6214)
- [Fix] Fix an off-by-one in the nbit nobag bounds check (#6215)
- [Improvement] Keep C++ attributes out of generated Torch schemas (#6251)
SSD Table Batched Embedding (TBE) Operators
- [Fix] Forward-compat sparse_type_to_int fallback for torch.package re-export version blends (#5987)
- [Fix] derive TBE cache associativity from device warp size (#6037)
- [Fix] Forward-compat nfp8_dtype fallback for torch.package re-export version blends (#6088)
- [Fix] Add grid-stride + ROCm cap to ssd_generate/update_row_addrs kernels (#6102)
- [New] Arch-aware FP8 host-boundary relabel + dispatch guardrails (#6104)
- [Improvement] Reduce TBE backward arg count — pack weights + aux tensors into TensorLists (#6132)
- [Improvement] Rename hit-lane copy/dispatch executors + knob to uvm_hit_* for symmetry (#6172)
- [Improvement] Enable the prefetch pipeline for embedding cache enrichment (#6192)
- [Fix] Set a 1 minute deadline on test_ssd_cache_flush (#6224)
- [Fix] Skip RES HBM tests when op absent in OSS wheel (#6233)
GenAI Support and Operators
Triton GEMM Support
- [Improvement] mx4 quantize: compute per-group scale via exact FP32 bit-construction instead of FP64 exp2 (#6083)
- [Fix] num_stages=0 -> 1 (#6111)
- [Improvement] mx4 dequantize: scale via FP32 bit-construction, not FP64 exp2 (#6148)
- [Fix] Mask the column-scale load in _kernel_scale_fp8_row (#6158)
Quantization Operators
- [New] Add several features for HSTU. (#4998)
- [Fix] Add dtype check for rowwise_min_max (#5980)
- [Improvement] Suppress harness-incompatible opcheck variants (fp8 1d roundtrip, repeat_arange) (#5990)
- [Fix] Stabilize flaky fused_nbit faketensor opcheck (#6005)
- [Improvement] Small tweak to FusedNBitRowwiseQuantizedSBHalfToFloatOrHalfNeon (#6021)
- [Fix] De-flake FastGemvTests (seed + dequantized reference) (#6027)
- [Improvement] add kWarpSizeHost() for host-side launch configs (#6031)
- [Fix] Fix HIP grid overflow in _compute_msfp_shared_exponent_cuda_kernel (#6036)
- [Improvement] Make float16/bfloat16 distinct types (#6049)
- [Fix] Add grid-stride + ROCm cap to fused_8bit_rowwise quantize kernels (#6109)
- [Fix] Add grid-stride + ROCm cap to fp8_rowwise quantize kernels (#6110)
- [Fix] Add grid-stride + ROCm cap to padded_fp8_rowwise quantize kernels (#6112)
- [Fix] Add ROCm cap to fused_nbit_rowwise dequant launch (#6118)
- [Improvement] Apply the host warning set to nvcc (#6173)
- [Fix] Fix differing_executors health check failures in quantize tests (#6205)
Sparse Operators
- [Improvement] optimize sparse_permute_2d kernel (#5964)
- [Fix] Exclude warm-ups from group_index_select_2d trace; raise to 50 (#5978)
- [Fix] Fix HIP grid overflow in batched_dense_vec_jagged_2d_mul_backward (transposed_bmm + outer_prod) (#5984)
- [Fix] Fix HIP grid overflow in jagged_dense_elementwise_mul_backward (#5985)
- [Fix] Fix HIP grid overflow in jagged_dense_dense_elementwise_jagged_output_kernel (#5989)
- [Improvement] Implement pre-sorting, caching and contigous warp processing in group_index_select backward (#5992)
- [Fix] Add grid-stride + ROCm cap to repeat_arange_kernel (#5995)
- [Improvement] Un-cascade jagged pt2_compliant_tag tests (#5996)
- [Improvement] group_index_select: capture onesweep sort + data-driven pattern-group report split (#6000)
- [Fix] Fix group_index_select bwd sort dispatch: per-group onesweep for large segments (ROCm) (#6001)
- [Fix] Add grid-stride + ROCm cap to get_source_mask_kernel (#6003)
- [Fix] Add grid-stride + ROCm cap to jagged_dense_dense_elementwise_jagged_output_opt_search_kernel_ (#6006)
- [Fix] group_index_select backward correctness fixes (ROCm) (#6008)
- [Improvement] Simplify redundant grid-cap clamp in embedding_inplace_update (#6017)
- [Improvement] Add grid-stride loops + ROCm cap to jagged_dense_bmm_kernel (#6019)
- [Improvement] Add grid-stride loops + ROCm cap to jagged_jagged_bmm_kernel (#6023)
- [Fix] Add grid-stride + ROCm cap to keyed_jagged_index_select_dim1_kernel and keyed_jagged_index_add_dim1_kernel; fast-fail TORCH_CHECK for index_select_scalar_cumsum_kernel (#6025)
- [Fix] int64 permute support in permute_1D_sparse_data (#6029)
- [Fix] Load fbgemm_gpu_embedding_inplace_ops .so in OSS init (#6034)
- [Fix] Fix HIP grid overflow in jagged_softmax (account for grid.y in cap) (#6038)
- [Improvement] Add grid-stride + ROCm cap to permute_multi_embs_kernel (#6040)
- [New] Add grid-stride + ROCm cap to permute_pooled_embs_kernel (#6041)
- [Fix] Skip >600s-timeout sparse cases (#6047)
- [Fix] Add ROCm cap to permute_pooled_embs_split host launch (#6050)
- [Fix] Cap forward grid by T*N for batched_unary_embeddings on ROCm (#6055)
- [Fix] Add grid-stride + ROCm cap to delinearize_unique_index_kernel (#6058)
- [Improvement] Drop redundant uint32_t narrowing at cap_grid_dim_x in jagged softmax/batched_dense_vec (#6061)
- [Improvement] Drop redundant casts at cap_grid_dim_x in jagged elementwise dispatch (#6062)
- [Fix] Add int32 overflow guards to CUDA sparse permute ops (#6081)
- [Fix] Fold grid.y/z into permute cap_grid_dim_x to fix HIP 2^32 overflow (#6082)
- [Improvement] Add permute sparse-data benchmark scripts and --embed-dim option (#6090)
- [New] Add opt-in host-side validation for CUDA sparse permute ops (#6095)
- [Improvement] Add grid-stride + ROCm cap to histogram_binning_calibration kernels (#6121)
- [Fix] Guard permute_2D_sparse_data against undersized permuted_indices (#6122)
- [Fix] Add grid-stride + ROCm cap to compute_frequency_sequence kernel (#6124)
- [Fix] Fix over-strict GPU memory gate on opt_search_large_grid test (#6125)
- [Improvement] Make batched cumsum AOT test traceable (#6126)
- [Improvement] Move giant Zipf overflow repro out of CI (#6128)
- [Fix] Gate permute_2D device assert at compile time (fixes 2 GiB relocation overflow) (#6130)
- [Fix] Add ROCm cap to invert_permute launch (#6134)
- [Improvement] Bound block-bucketize stress generation (#6135)
- [Improvement] Gate permute_1D lengths device assert at compile time (#6136)
- [Improvement] Avoid ATen reduction in masked jagged select (#6140)
- [Fix] Add ROCm cap to offsets_range launches (#6142)
- [Fix] Add ROCm cap to pack_segments backward (unpack) launch (#6146)
- [Fix] Add ROCm cap to block_bucketize + populate_bucketized fallback launches (#6162)
- [New] Add a CUDA kernel for masked_select_jagged_1d (#6190)
- [Fix] Relocate permute_multi_embedding index tensors instead of failing (#6243)
- [Improvement] Mark platform-conditioned GPU parameters unused (#6250)
- [Improvement] Avoid redundant zero fill in pack_segments (#6257)
Platform Support
CUDA
- [Improvement] Use CUDA 13 family-specific architecture targets (#6156)
- [Improvement] Drop CUDA 13.4 (cu134) from all Linux wheel build matrices (#6139)
ARM / AArch64
- [New] Port u8s8 quantized GEMM to arm with I8MM and NEON compute kernels (#6170)
- [Fix] Make x86-only conv kernels fail loudly on arm instead of faulting or returning zeros (#6177)
- [Fix] Skip TransposedRequantizeTest on arm instead of asserting against an empty buffer (#6169)
ROCm / AMD
- [Fix] [ROCm][CI] Restore ROCm GPU CI: switch to the MI350 runner label (#6186)
- [Fix] [ROCm] gate warpReduceAllSum DPP path to gfx942/gfx90a/gfx950 (#6014)
- [Improvement] [ROCm] plumb per-build wave-size flags from CMake into codegen (#6103)
- [Fix] Fix new rocm version build: use __syncthreads() instead of removed hipcub::CTA_SYNC (#6143)
- [Fix] Remove unreachable ROCm warp instantiations (#6197)
- [Improvement] Isolate ROCm index-select boundary coverage (#6138)
- [Improvement] Update OSS Composable Kernel SHA (#6223)
Build / CI Improvements and Better Engineering
- [Improvement] Increase Nova timeout to 180 mins (#3102)
- [Improvement] Shared CPU two-level prefetch helpers + env knobs (#5969)
- [Improvement] Use Python 3.10+ typing in remaining files (#5975)
- [Fix] Move x86 CPU test runner to linux.4xlarge.memory to fix py3.12+ OOM (#6010)
- [Improvement] Migrate contained C10_LIKELY/C10_UNLIKELY to [[likely]]/[[unlikely]] (#6012)
- [Fix] Clamp cap_grid_dim_x to the 2^31-1 grid-x limit (+ unit test) (#6013)
- [Fix] Fix MSan uninitialized reads in FBGEMM depthwise convolution and INT64 GEMM packing (#6028) (#6032)
- [New] Add grid-stride + ROCm cap to recat_copy_async_kernel (#6039)
- [Fix] Fix build break when using llvm-21 (#6046)
- [Fix] Add fast-fail TORCH_CHECK for auc_kernel HIP grid overflow (#6054)
- [Improvement] Install click as part of setup (#6056)
- [Fix] Skip test_l2_norm_pruning_workflow in OSS CI (#6073)
- [Improvement] Enable Pyrefly for deeplearning/fbgemm (#6099)
- [Improvement] Pass third-party include dirs as SYSTEM includes (#6127)
- [Improvement] Treat asmjit and cpuinfo headers as system headers (#6131)
- [Improvement] Add fp8 corner case unit test (#6133)
- [Improvement] Derive nvcc and hipcc warning lists from one source of truth (#6144)
- [Fix] Downgrade -Werror=maybe-uninitialized on GCC to unblock nightly wheel build (#6147)
- [Improvement] Add -Wunused-value (#6150)
- [Improvement] Split four more -Wall-implied warnings across compiler buckets (#6153)
- [Improvement] Add -Wmissing-braces (#6154)
- [Improvement] 02c.2 -- HIP and nvcc reconnaissance: warnings on, -Werror off (#6155)
- [Improvement] Docs: Add fbgemm-ascend to community ports (#6164)
- [Improvement] Add init and control-flow warnings, split across buckets (#6174)
- [Improvement] Add three clang-leaning warnings, one of them portable (#6175)
- [Improvement] Fix B001: Replace bare except with except Exception (#6182)
- [Improvement] Fix B001: Replace bare except with except Exception (#6183)
- [Improvement] Add four warnings for deprecated C++ constructs (#6203)
- [Improvement] Move -Wmismatched-tags to the portable list (#6206)
- [Improvement] Add four warnings for types and templates (#6207)
- [Improvement] Add four warnings for conversions and literals (#6210)
- [Improvement] Add packed-member and object-lifetime warnings (#6211)
- [Improvement] Add -Wshift-sign-overflow (#6212)
- [Improvement] Add -Wzero-as-null-pointer-constant, without errors (#6218)
- [Improvement] Add -Wunused-exception-parameter (#6225)
- [Improvement] Add -Wshorten-64-to-32, without errors (#6230)
- [Improvement] Add -Wheader-hygiene, without errors on device code (#6234)
- [Improvement] Add -Wshadow, without errors (#6239)
- [Improvement] Add three warnings for unused variables (#6240)
- [Improvement] Add tensor device relocation helper (#6246)
- [Improvement] Add four clang-only warnings for Objective-C and C (#6249)
- [Fix] Guard GPU device queries on CPU lane in verify script (#6252)
- [Improvement] Add four clang-only warnings that can occur in C++ (#6253)
- [Improvement] Add v1.9.0 -> torch 2.14 to compat table (#6254)
- [Improvement] Build FBGEMM release CUDA wheels on linux.12xlarge.memory (#6284)
Tests and Benchmarks
- [Improvement] Migrate test/tbe/** TBE type imports to fbgemm_gpu.tbe.* leaves (Phase 3.9) (#6066)
- [New] Add opt-in whole-run multi-stream timing (#5988)
- [Improvement] Migrate
-c fbcode.platform010_cuda_versionto the cuda constraint modifier (#6100) - [Improvement] Package benchmark utilities in wheels (#6232)