AWS makes distributed KV caching generally available

On Nov. 18 Amazon made distributed KV caching generally available on ElastiCache for Valkey. AWS said its tests cut self-hosted inference cost per million output tokens by up to 40 percent and time to first token by up to 80 percent, using cache.i8ge.48xlarge nodes, RDMA over Elastic Fabric Adapter, and the LMCache connector for vLLM, SGLang, and TensorRT-LLM. The post states no dollar price. In one shared-document test, AWS said each server handled twice the requests per second and that cost per million output tokens fell 40 percent after paying for the cache node.

/ Sources

/ About this story

Compiled by Venture Atlas from the sources above, using automated AI-assisted research. This is a summary of reporting published elsewhere, not original reporting - follow the source links for the full account. See our editorial standards.

Something wrong here? Email apostructura@gmail.com and we will fix it.

/ Related