On Nov. 18 Amazon made distributed KV caching generally available on ElastiCache for Valkey. AWS said its tests cut self-hosted inference cost per million output tokens by up to 40 percent and time to first token by up to 80 percent, using cache.i8ge.48xlarge nodes, RDMA over Elastic Fabric Adapter, and the LMCache connector for vLLM, SGLang, and TensorRT-LLM. The post states no dollar price. In one shared-document test, AWS said each server handled twice the requests per second and that cost per million output tokens fell 40 percent after paying for the cache node.
/ Sources
/ About this story
Compiled by Venture Atlas from the sources above, using automated AI-assisted research. This is a summary of reporting published elsewhere, not original reporting - follow the source links for the full account. See our editorial standards.
Something wrong here? Email apostructura@gmail.com and we will fix it.
/ Related
- AWS adds managed extensions on HyperPod Slurm clustersSunday, November 22, 2026
- AWS makes Graviton5 T9g instances generally availableSunday, November 22, 2026
- AWS Transform can modernize apps to .NET 11Sunday, November 22, 2026
- CloudWatch Omni can collect Azure logs, metrics, and tracesSunday, November 22, 2026
