GPU Scheduling and Execution Optimization for LLM Inference

For large-scale LLM inference, we classify the execution characteristics of requests using token-length prediction and continuous batching, and study inference execution and scheduling techniques that maximize GPU utilization and throughput.

Cloud LLM Serving and Infrastructure-Aware KV Cache Management

Considering the hierarchical GPU-CPU-SSD infrastructure of cloud environments, we optimize the placement, migration, and eviction of the KV cache, and study cost-efficient LLM serving architectures that meet SLOs.

VectorDB and Storage Acceleration for RAG

Targeting VectorDB workloads for RAG, we accelerate the storage and retrieval of large-scale embedding data, and study data access and execution scheduling techniques that raise GPU utilization across the retrieval-inference pipeline.

Cross-Layer LLM System Design across Memory, Storage, VectorDB, and GPU

Through cross-layer design that links the LLM runtime, KV cache, VectorDB, file system, storage, and GPU scheduler, we simultaneously improve both the throughput and the tail latency of cloud-based RAG-LLM services.