Bug Fixes
- When Tiered Storage is paused and data potentially expires from local storage, there can be gaps between last offset in tiered storage and first offset in local storage. If local storage was truncated in the middle of a segment (i.e. time based retention or via trim-prefix/delete records commands) tiered storage might get stuck with the following exception:
Failed to schedule upload: std::runtime_error (ntp {kafka/foo/0}: log offset N is outside the translation range (starting at M > N)). Fix this by adjusting the upload start offset to the first available and valid offset. Although we might have a bit more data in the segment, other information about that data (i.e. offset translation) is gone with prefix truncation. by @nvartolomei in #26065
Improvements
- #26136 Swap out an internal data structure in the
storagelayer to prevent oversized allocations and crashes when a large number ofsegments are present in apartition. by @WillemKauf in #26137 - PR #26166 [v24.3.x] storage: CORE-10056: Remove contiguous allocations in lock_manager by @wdberkeley
Full Changelog: v24.3.13...v24.3.14