learning-it by example

#17 LLM engineering and evals 5.5 min

Cut LLM Cost and Latency: Caching and Shorter Answers

Same six requests, over two minutes on a laptop. Add a cache and shorter answers, and watch the output tokens and the wait drop, with the real numbers on screen.

1. Why does the video print the number of output tokens for every model call?
2. In the baseline run of 6 requests with a cap of 60 tokens, the model wrote 360 output tokens. What does that tell you?
3. In a Python demo of an Ollama llama3.2 response cache, six requests were three questions asked twice. What did the cache give compared with the baseline?
4. In the Ollama cost demo, why does the shorter answers fix add a sentence asking for a short answer as well as lowering the cap to 40 tokens?
5. In an LLM response cache, what must go into the cache key besides the question?

Practice only: nothing is saved and no score counts towards anything. Each answer comes with the reason.

  1. Show which requests hit the cache

    Copy the code folder first. In cached.py add a print inside the loop that shows each question and whether it came from the cache (used == 0). Predict which of the six lines will say cache before you run python cached.py. This needs Ollama with llama3.2, as in the video.

    A hint

    The first three requests are the three different questions.

  2. Repeat the traffic three times

    In traffic.py change TRAFFIC to QUESTIONS * 3. Predict the number of requests and the number of cache hits that cached.py will print, and the output tokens if every uncached answer again reaches the 60 token cap. Then run it.

    A hint

    Count how many different questions there are.

  3. Cap without asking for brevity

    In concise.py remove the sentence 'Answer in one short sentence.' from the question but keep num_predict at 40, and print each answer. Predict how the answers end, then run it. Compare with what the video says about a cap on its own.

    A hint

    Add print(reply.message.content) inside the loop.

  4. Combine both fixes

    The video did not test the cache and the shorter answers together. Copy cached.py to both.py and make it send the short sentence and use a 40 token cap. Predict the number of cache hits, then run it and compare its tokens with the cached run and the concise run.

    A hint

    Put the same sentence in the cache key by changing the question before it is hashed.

These tasks are for you to try on your own machine. Nothing is checked or saved.

What this does not show

Six made-up requests (three questions, each asked twice) on one small local model, one run on one laptop. Tokens stand in for cost because a local model is free, so no bill was measured. The cache only helps exact repeats, shorter answers can lose detail, and the two fixes were not tested together. The two advice articles are vendor blog posts.

Up next in LLM engineering and evals

More in LLM engineering and evals

All LLM engineering and evals videos

Related videos