The latest update to llama_cpp_canister is making it easier for developers to understand exactly how many AI tokens are used in each on-chain AI request.
Version 0.15.0 introduces detailed token tracking for both run_update and run_query calls. Instead of providing estimates, the system now reports the exact number of prompt tokens used, how many tokens were reused from the cache, how many had to be processed again, and how many new tokens were generated.
The update was created to support ICGPT’s Prompt Cost Lab, a tool that helps developers test AI prompts and compare both their quality and cost. The new token accounting gives developers a more accurate way to measure how expensive each AI request is and where savings can be made.
One of the main benefits is that developers can see whether a prompt is efficiently using cached data or whether it is repeatedly processing the same information. That can help reduce costs for AI applications that run frequently on-chain.
The Prompt Cost Lab also allows users to experiment with different prompt designs, compare results, and evaluate AI responses using automated quality testing methods, including LLM-as-a-Judge evaluations.
The project team says these tools are being developed to help create better and more cost-efficient AI tasks for a planned AI task board that is part of the funnAI roadmap.
The updated llama_cpp_canister has been released as an open-source project on GitHub, where developers can access the latest version and documentation.
Dear Reader,
Ledger Life is an independent platform dedicated to covering the Internet Computer (ICP) ecosystem and beyond. We focus on real stories, builder updates, project launches, and the quiet innovations that often get missed.
We’re not backed by sponsors. We rely on readers like you.
If you find value in what we publish—whether it’s deep dives into dApps, explainers on decentralised tech, or just keeping track of what’s moving in Web3—please consider making a donation. It helps us cover costs, stay consistent, and remain truly independent.
Your support goes a long way.
🧠 ICP Principal: ins6i-d53ug-zxmgh-qvum3-r3pvl-ufcvu-bdyon-ovzdy-d26k3-lgq2v-3qe
🧾 ICP Address: f8deb966878f8b83204b251d5d799e0345ea72b8e62e8cf9da8d8830e1b3b05f
Every contribution helps keep the lights on, the stories flowing, and the crypto clutter out.
Thank you for reading, sharing, and being part of this experiment in decentralised media.
—Team Ledger Life
Community Discussion