The paper 'ByteExpress: A High-Performance and Traffic-Efficient Inline Transfer of Small Payloads over NVMe', written by Park Jun-hyuk, master's student at the Data-Centric Computing and AI Systems Laboratory (DISCOS) (first author), and Professor Kim Young-jae (corresponding author), has been accepted for publication at The 17th ACM Workshop on Hot Topics in Storage and File Systems (HotStorage) 2025.

The research is set against the background of Computational SSDs (CSD) and Key-Value SSDs (KV-SSD), representative examples of the computational storage concept that has emerged as solid-state drive (SSD) technology advances. These new storage devices aim to reduce data movement and improve performance and efficiency by offloading data processing operations that used to run on the host system — such as data filtering through SQL predicate pushdown, or key-value store PUT/GET operations — into the device itself.

Looking at the SQL predicate pushdown specifications used in existing CSD research, or at the key-value pairs of a typical key-value store workload, the payload moving from host to storage device (the pushdown specification or key-value pair) is only tens to hundreds of bytes in size, and such small data is frequently delivered to the device through kernel-bypass APIs such as NVMe passthrough. Noting this, the research points out that the PCIe-based NVMe protocol — today's mainstream storage device interface — moves data between host and device in 4KB units and is therefore extremely inefficient for these small direct I/Os. The experiments confirm that transferring a 32-byte payload actually causes a 4KB data movement over the PCIe interconnect, producing traffic amplification of more than 130 times, and that transfer performance is likewise bound to that 4KB-unit transfer.

To resolve this problem, Park Jun-hyuk (first author) and Professor Kim Young-jae (corresponding author) propose ByteExpress, a technique that uses the NVMe Submission Queue (SQ) to transfer the payload inline in 64-byte units (the default NVMe SQ entry size); the work was submitted to the HotStorage 2025 workshop and accepted as a regular paper. Rather than transferring payloads in memory-page units as before, ByteExpress places the payload contiguously after the NVMe command in the NVMe SQ in 64-byte units, only as much as needed, achieving both PCIe traffic efficiency and higher performance. Because it uses the NVMe infrastructure largely unmodified, it also has the advantage of maintaining compatibility with existing NVMe drivers and NVMe SSD architectures.

The figure below compares the payload transfer mechanism of standard NVMe (NVMe PRP-based), other solutions proposed for fine-grained I/O (PCIe MMIO-based, NVMe CMD-based) and the new ByteExpress (NVMe SQ-based), showing the superiority of the proposed solution in NVMe compatibility and potential performance gains.

Implementing ByteExpress in the Linux kernel NVMe driver and on the OpenSSD platform achieved, for small payloads of 256 bytes or less, up to 98% less PCIe traffic and more than 40% lower latency than conventional PRP, and up to 39.8% lower latency than existing NVMe-compatible fine-grained data transfer solutions (see the figure below). Applied to real KV-SSD and SQL predicate pushdown workloads, it also delivered higher throughput and consistent performance gains over the conventional approach.