Paper Notes: OSDI’26
Notes on the OSDI’26 papers closest to my own work. I add papers as I read them.
Break on Through to the Other Side: Pooling Memory Elastically with RamRyder
In one sentence.
Memory bandwidth should be allocated as a cloud resource in its own right, just like CPU and memory capacity.
Background.
- Memory allocation in the cloud is not flexible enough: vendors allocate memory capacity in a fixed ratio to vCPUs and offer no way to allocate bandwidth. The authors’ analysis of Alibaba’s cluster traces shows that 90% of servers exhibit average bandwidth utilization below 44.5%.
- Underutilized bandwidth is not necessarily correlated with underutilized capacity, because they exhibit different spatial and temporal patterns.
Approach.
- RamRyder is a software-defined elastic memory system for cloud VMs that adjusts capacity and bandwidth dynamically, by managing memory at channel granularity.
- Components:
- Host user space resource manager: channel allocation
- Hypervisor: virtualize memory channels
- Modified guest kernel: page-to-channel allocation policies
Evaluation.
- Implemented on unmodified commodity servers
- Disable existing DIMM channel interleaving via BIOS/UEFI
- CXL devices reserved as a separate DAX device in user space with CXL configuration tools
- Test setup:
- AMD EPYC Zen 5 with 12 DDR5 channels per socket; each VM’s vCPUs are pinned to their own CCXs, so VMs do not share an L3 cache
- 8 of the 12 channels in one socket are used, each populated with one 32 GB DDR5 DIMM, to match compute-optimized cloud instances
- 4 × 256 GB Samsung CXL 2.0 memory devices
- Debian with Linux 6.15
- RamRyder improves average capacity and bandwidth utilization by 28.6% and 43.2%, respectively, across the cluster while delivering performance comparable to the best case with exclusive access
Food for thought.
- Limitations: the mapping from physical addresses to memory channels is normally hidden from system software, so deployment means turning off channel interleaving in BIOS/UEFI (a boot-time setting), and the guest kernel must be modified
Finding NEMO: Nimble and Expressive Memory Observability
In one sentence.
A small, programmable telemetry engine in the memory controller lets the OS count exactly the memory accesses it cares about, at the granularity it chooses, without spending CPU time.
Background.
- A long-standing need for easy and efficient hardware and software mechanisms that can track memory usage on multi-tier memory systems.
- Existing mechanisms:
- page level access bit tracking, expensive
- hardware cache miss sampling, reduced precision (HeMem)
Approach.
- NEMO, a hardware-software codesigned memory telemetry engine
- On the hardware side, NEMO is a lightweight extension to memory controllers (MCs)
- On the software side, NEMO exposes an OS-facing programming model and Linux driver
- MCs already host QoS and monitoring features; the NEMO pipeline runs alongside, rather than on, the memory datapath.
- To support different granularities of monitoring (cache line to multi-GiB regions), the “match stage” masks and shifts the request’s physical address with OS-set parameters (simple fixed-latency arithmetic, not a hash) and looks the result up in a small translation table to find the counter. For page-hotness tracking, a 2 MiB page granularity is adopted
Evaluation.
- Altera Agilex 7 FPGA I-Series Development Kit with CXL 2.0 Type-3 support in hard IP
- SRAM per NEMO pipeline: ~150 KiB
- Across the full bitstream, NEMO uses 4.4% of the FPGA’s M20K SRAM and 4.8% of its logic
- Speeds up HeMem’s reaction to hot-set changes by 5×, accelerates THP splitting in MEMTIS by 10.4×, and detects noisy neighbors in Linux with 350× lower overhead. These telemetry improvements yield up to 1.7× higher throughput and 23% lower latency across key-value stores and databases
Food for thought.
This paper overlaps a lot with my past work.
- One data point from my own experience: scanning the page table (test-and-clear of the accessed bits) for 1 GB of address space (one VMA of 4 KiB pages, memset first so that every PTE really exists) takes about 1 ms, so 1 TB would take about 1 s, long enough for the accessed bits to go stale. And as the authors point out, one bit per page is too coarse-grained.
- 2 MiB granularity is a common, wise and practical choice for page temperature tracking.
- The memory controller is a compelling place for hardware event monitoring. The prototype already sits in a CXL memory expander’s controller; I would imagine future CXL devices shipping such engines to hint the OS and other system software.
Teaching The Old Dog New Tricks: Building Efficient Data Pipelines for Large-Scale LLM Pre-training (Operational Systems)
In one sentence.
ML workloads usually have fixed ranks and predictable access patterns, which creates room for resource pre-allocation and optimizations.
Background.
- Pre-training data pipeline:
- initialization phase: The training framework issues blocking operations that load full model checkpoints, distributed and replicated across thousands of ranks
- iterative training phase: The dataloader prefetches and pre-processes data for forward passes and backward passes. The training framework periodically persists checkpoints to storage
- companion evaluation phase: Each evaluation cycle loads the latest checkpoint from storage, executes inference and writes back results to storage
- Storage challenges:
- companion evaluation phase is blocked by cross-DC I/O
- initialization phase is bottlenecked by an “I/O storm” where synchronous checkpoint loading triggers massive data contention on a small set of hot files
- data loading shifts from being I/O-bound to compute-bound due to the high cost of transforming video and image data
Approach.
- Predictive checkpoint replication
- Companion evaluations follow a predictable interval, typically every one thousand training steps; instead of issuing thousands of WAN reads for checkpoint shards, they are batched up and cached in the storage tier of the evaluation cluster.
- Allow the storage scheduler to preempt background traffic to make room for high-priority evaluations.
- Proactive Hotspot Prediction
- The training framework explicitly informs the storage system about upcoming access patterns
- The storage tier only increases replication factor for hot metadata and replicated-parameter files
- Storage-Side Transformation Offloading
- Multimodal data loading is compute-bound: the training hosts’ CPUs cannot decode video and images fast enough to keep the GPUs busy
- A deterministic dataloader writes the exact order of upcoming samples into per-step files, so storage nodes know what comes next and use their spare CPUs to decode, crop and resize those samples just in time for the next step
Evaluation.
Reduce wasted GPU hours per evaluation from 16,800 to 4,000, shorten checkpoint loading time at each training start by 40.8%, and reduce training stalls caused by dataloading by 63.2%.
Food for thought.
This work is pretty solid and straightforward. ML workloads tend to be deterministic and that indicates room for prefetching and resource reservation.
Heterogeneous systems + AI storage systems are topics I should dive deeper into.
SBB: Eliminating Centralized Bottlenecks in Userspace Network Runtime
In one sentence.
SBB removes every centralized scheduler from a userspace network runtime: each core receives its own timer and NIC interrupts in user space through Intel’s User Interrupts (UINTR) and balances load itself, so throughput scales with the number of cores.
Background.
- Userspace network runtimes (Shinjuku, Caladan, etc.) must schedule three things, and every prior design centralizes at least one of them, which becomes the bottleneck as worker cores are added:
- Request preemption: a central timer sends preemption signals (IPIs) to workers when a quantum expires
- CPU allocation (lending idle cores to best-effort jobs): a central monitor polls every worker’s NIC queue for backlog, since a polling app that has given up its core cannot notice new packets
- Request load balancing: RSS suffers from skewed flows; a dispatcher thread polling one NIC queue balances well but caps throughput
- Giving these components more cores doesn’t help: the authors measure worse load balancing and no gain elsewhere.
Approach.
- Each worker thread binds to a core with its own TX/RX queues and local timer, and schedules itself:
- Preemption: before each request, the worker arms a one-shot APIC timer that interrupts into user space. With the timer registers mapped into user space, this costs ~50 cycles, versus ~3 µs for
setitimer - CPU allocation: NIC interrupts replace polling. A running latency-critical thread takes them in its user-space handler; otherwise a kernel handler wakes it up, so idle cores can safely run best-effort jobs
- Two-level load balancing: task stealing for temporary imbalance, and flow migration, which reprograms the NIC to move flows off overloaded cores, for persistent imbalance
- Preemption: before each request, the worker arms a one-shot APIC timer that interrupts into user space. With the timer registers mapped into user space, this costs ~50 cycles, versus ~3 µs for
- Both interrupts arrive at the same handler, so SBB enables them in disjoint phases: NIC interrupts only while waiting for packets (which also prevents interrupt storms), the timer only during application-layer work
Evaluation.
- Server: 2 NUMA nodes, each with a 64-core Intel Xeon Platinum 8592 at 1.9 GHz; 100 Gbps Intel E810 NIC
- Built on DPDK, plus a 2,095-line Linux kernel patch
- Baselines: Shinjuku, Concord and TQ for preemption; Skyloft and Caladan for CPU allocation
- SBB achieves 1.7× to 5.2× higher throughput at 48 cores, under the same tail latency target
Food for thought.
- UINTR control plane: the kernel is still involved, registering each core’s notification vector and the user-space handler (SBB also gives the NIC queue and APIC timer interrupts that vector).
- UINTR data path: the CPU first matches an incoming interrupt’s vector against the notification vector; on a match, it skips the IDT and runs the user-space handler directly, without entering ring 0.
Rakaia: Scalable In-Kernel Scheduling for TCP-Based RPCs
In one sentence.
For RPCs, the authors created a set of message semantic APIs based on kernel TCP/IP stack that removes extra I/O threads copying and head-of-line blocking.
Background.
- gRPC big picture:
- NIC RX DMA into the memory, acked in device driver, handled over to TCP/IP stack
- I/O threads were epoll waiting on the socket and wakes up when there is work. I/O threads call read which copies skbs into a userspace buffer, and parses the HTTP frames.
- The complete message is enqueued to a global FIFO and handed to a worker.
- Bottlenecks:
- TCP’s stream-oriented abstraction is a poor fit for message-oriented RPCs: RPCs sharing the same TCP stream are processed sequentially by default, causing HOL blocking
- gRPC must reconstruct message semantics using additional userspace abstractions
Approach.
- Key insight:
- message parsing and scheduling should occur in the same softirq context that processes incoming TCP packets.
- messages must be decoupled from their underlying TCP connections to allow global scheduling
- Each application thread only needs a single Rakaia socket to handle all TCP traffic
- Parsing the message from TCP byte stream in the same softirq context
- Logical centralized queue of messages for scheduling
- Work stealing on empty local queue for load balancing
Evaluation.
- Server: CloudLab xl170, one 10-core Intel Xeon E5-2640v4 at 2.4 GHz (20 hyperthreads); open-loop load over 20 to 5,000 connections, with 20 µs and 100 µs service times
- Achieves up to 5× higher throughputunder-SLO than KCM, Linux’s current in-kernel message API over TCP
- Improves gRPC-Go’s throughput-underSLO by up to 1.56×, and gRPC-C++’s by up to 2.69×
- Improves the throughput-under-SLO for real-world applications including Silo running TPC-C and OpenTelemetry Collector by 1.39× and 1.42×, respectively
Food for thought.
Thinking about Rakaia for UDP-based RPC systems and DPDK