Llama_cpp_canister update gives developers a clearer view of AI costs

The latest update to llama_cpp_canister is making it easier for developers to understand exactly how many AI tokens are used in each on-chain AI request.

Version 0.15.0 introduces detailed token tracking for both run_update and run_query calls. Instead of providing estimates, the system now reports the exact number of prompt tokens used, how many tokens were reused from the cache, how many had to be processed again, and how many new tokens were generated.

The update was created to support ICGPT’s Prompt Cost Lab, a tool that helps developers test AI prompts and compare both their quality and cost. The new token accounting gives developers a more accurate way to measure how expensive each AI request is and where savings can be made.

One of the main benefits is that developers can see whether a prompt is efficiently using cached data or whether it is repeatedly processing the same information. That can help reduce costs for AI applications that run frequently on-chain.

The Prompt Cost Lab also allows users to experiment with different prompt designs, compare results, and evaluate AI responses using automated quality testing methods, including LLM-as-a-Judge evaluations.

The project team says these tools are being developed to help create better and more cost-efficient AI tasks for a planned AI task board that is part of the funnAI roadmap.

The updated llama_cpp_canister has been released as an open-source project on GitHub, where developers can access the latest version and documentation.