SproutStackSproutStack home

KV-Cache

Why streaming the 2nd token is fast.

Teaching this in the lesson? Read the chapter.

Step 1 of 3

1. First token slow

Compute keys/values for whole prompt.

  1. Prefillwhole prompt

Prefer prose? Every animation has a matching chapter, listed above. Diagrams use text labels, so they read aloud with the listen button on the lesson page.