AI 导读
警报🚨：在 12-16 个并发智能体会话下，启用 DRAM KVCache 卸载优化可使可缓存的 token 数量增加 20-40%🚀。一块 B200 只有 180 GB 的 HBM，扣除权重存储后，每颗芯片仅剩 100 GB HBM 用于 KVCache。 这对长上下文智能体推理负载来说远远不够。因此，借助 SGLang HiCache，你还可以将 KVCache 卸载到 CPU 的 DDR5/LPDDR5X 内存，从而帮助提升性能。(1/2)🧵
正文
ALERT🚨: The number of tokens that can be cached can increase by 20-40%🚀 by enabling DRAM KVCache offloading optimization at 12-16 concurrent agentic sessions. A B200 only has 180 GB of HBM, and after weight storage, only 100 GB of HBM per chip remains for KVCache.
THIS IS NOT ENOUGH FOR LONG-CONTEXT AGENTIC INFERENCE WORKLOADS. Thus, with SGLang HiCache, you can also offload KVCache to the CPU's DDR5/LPDDR5X memory, which helps improve performance. (1/2)🧵
来源：SemiAnalysis · x.com