Contact Us
Contact Us
Cloud Cost Savings

FinOps for AI: How to Track Token, GPU and LLM Costs Before They Blow Your Budget

AI bills are hard to attribute and harder to forecast. How to allocate token, GPU and LLM spend to teams, features and customers before it outruns the budget.

JRC Software Team
7 min read
FinOps for AI: How to Track Token, GPU and LLM Costs Before They Blow Your Budget

The first AI feature went live on a pay-per-token bill small enough that nobody looked at it. A year on there are three models in production, a GPU instance someone started for a fine-tuning experiment in March, a vector database, and a support chatbot customers actually like. The AI line on the invoice has tripled. Finance asks which feature, or which customer, is driving it. Nobody can say.

That's the gap FinOps for AI is meant to close. AI spend doesn't behave like ordinary cloud spend. It's metered in tokens, it moves when someone edits a prompt rather than when infrastructure changes, and part of it often arrives on a separate invoice from a model provider. Without deliberate allocation it all lands in one shared bucket that finance can't forecast and engineering can't optimise.

Below is how we attribute AI costs to teams, features and customers, the handful of metrics worth tracking, and where the savings usually turn up.

AI spend has become the FinOps priority

The FinOps Foundation's State of FinOps 2026 report found that 98% of respondents now manage AI spend, up from 63% in 2025 and 31% in 2024. The same survey names FinOps for AI as the top forward-looking priority, and AI cost management as the number one skill teams need to build.

The reasons are structural. The Foundation's FinOps for AI Overview notes that pricing units like tokens are unfamiliar and change quickly, that GPU capacity and provisioned model throughput can be scarce in some regions, and that AI forecasts are less predictable than cloud forecasts, especially early on. It also points out that applications add system prompts, conversation history and retrieved content to every call, so the tokens you pay for are often far more than what users type.

Put plainly, an AI feature's cost is set as much by engineering choices (prompt length, model choice, caching) as by how many people use it. That makes it a shared problem for product, engineering and finance, not just a line for the cloud team to watch.

Where AI cloud costs actually come from

You can't allocate AI cloud costs until you've found all of them. A pattern we see often is a team watching its model API line closely while bigger costs grow somewhere else.

Cost sourceHow it's billedHow hard to allocate
Hosted model APIs (Amazon Bedrock, Azure OpenAI in Microsoft Foundry)Per input and output token, or per provisioned unit per hourMedium: needs per-application profiles or keys
Direct model provider accountsPer token, on a separate invoiceHigh: often no native tags at all
GPU instances for training, fine-tuning or self-hostingPer instance hourLow if tagged, high if shared
Retrieval stack (vector database, embeddings, search)Storage, compute and per-request, mixedMedium
Logging of prompts and responsesPer GB ingested and retainedOften forgotten entirely
AI add-ons inside SaaS toolsPer seat or usage add-onHigh: sits outside cloud billing

Pricing modes vary a lot too. On Amazon Bedrock, on-demand use is charged per input and output token. Batch inference for selected models is priced 50% below on-demand, the Flex tier is a 50% discount to Standard, and the Priority tier carries a 75% premium. In Microsoft Foundry, provisioned throughput is billed per PTU per hour regardless of tokens consumed, with Azure reservations available on 1-month or 1-year terms.

The same workload can cost very different amounts depending on the mode, so choosing the mode is itself a FinOps decision. Moving an overnight document-summarisation job from on-demand to batch is often a one-line change with a large effect.

FinOps for AI in five allocation steps

The Foundation's advice is to report through showback first, because visibility alone often changes behaviour. This is the order we work in.

  1. Pull every source into one dataset. Cloud billing exports, model provider usage exports and SaaS invoices. The FOCUS specification (version 1.4 at the time of writing) normalises billing data across AI, cloud and SaaS vendors, and AWS and Microsoft both offer exports in FOCUS format, which saves a lot of column-mapping.
  2. Give every application its own meter. On AWS, Bedrock application inference profiles let you attach tags to a profile and track costs for on-demand calls by application. Call the model through the profile's ARN instead of the raw model ID, and remember the tags still have to be activated as cost allocation tags in Billing. On direct provider accounts, issue one API key per application or team and map keys to cost centres.
  3. Tag GPU and retrieval infrastructure. Use the same team, product and environment keys as the rest of the estate. If you haven't standardised tags yet, start with our guide to tagging, showback and accountability.
  4. Attribute to features and customers in your own code. Cloud tags stop at the resource. To know what a feature or a tenant costs, log the model, input and output token counts, and a feature or tenant ID for every call, then join that log to your effective token rate.
  5. Send showback, then decide on chargeback. Each product owner gets a monthly view of their AI spend next to usage and outcomes. Only move to formal chargeback once the numbers are trusted.

Metrics that matter for LLM cost management

Total spend tells you very little. The FinOps for AI Overview sums up its approach as "Measure the use case, not the token." These are what we put on a first dashboard:

  • Cost per successful outcome: total cost of a workflow divided by the runs that met the quality bar. The number a product owner can act on.
  • Cost per inference: total inference cost divided by requests, per feature.
  • Effective cost per million tokens: the blended rate across models, tiers and providers.
  • Cache hit rate: the share of input tokens served from a prompt cache.
  • Model fit: the share of requests sent to a bigger, pricier model than the task needs.

The Foundation is clear that cost targets mean little without the quality they must preserve. Swapping to a cheaper model isn't a saving if resolution rates drop and support tickets climb.

Where the savings usually are

Once costs are visible, most of the levers are engineering ones.

  • Route by difficulty. Send classification and extraction to small models, and keep frontier models for the requests that need them. A simple router with an evaluation set behind it is usually enough to start.
  • Trim the context. System prompts and full conversation history ride along on every call. Summarise older turns and cap the number of retrieved chunks.
  • Cache and batch. Put stable content (system prompt, tool definitions, reference documents) at the start of the prompt so prompt caching can reuse it. Push non-urgent work, such as overnight document processing and evaluation runs, through batch or flexible tiers.
  • Switch off idle GPUs. Notebooks and experiments left running over a long weekend are a classic source of waste. Schedule them, or use managed endpoints that scale to zero.
  • Commit only once usage settles. The Overview advises committing to discounts or reserved capacity once usage is stable. Provisioned throughput bills whether or not you use it, and Microsoft states that reservations don't guarantee capacity. Our Savings Plans vs Reserved Instances guide covers how to size commitments safely.

How JRC Software helps

As part of our cloud cost optimisation service, week one is an inventory. We pull your cloud billing exports, model provider usage reports and SaaS invoices, and find every place AI spend lives, including the ones on a credit card. By the end of it you know your current monthly AI spend by source, and how much of it can be attributed to a team today.

Then we fix the plumbing: inference profiles or per-application keys, consistent tags on GPU and retrieval infrastructure, and request-level logging that ties tokens to features and tenants. Product owners get a monthly unit-cost view and budget alerts.

Finally we work through the optimisation backlog with your engineers (routing, prompt trimming, caching, batch) and test each change against a quality baseline before it ships. Provisioned throughput rarely makes sense in a feature's first few months, before usage settles. Our teams in Melbourne and Sri Lanka keep this affordable for mid-sized businesses.

If your AI bill is growing faster than your ability to explain it, request a cloud and AI cost review and we'll show you where the money goes.

Frequently asked questions

What is FinOps for AI?

FinOps for AI applies the FinOps discipline, shared accountability for technology spend across engineering, finance and the business, to AI workloads. It covers model API tokens, GPU compute, retrieval infrastructure and AI features in SaaS tools. The goal is to allocate those costs to the teams, features and customers that create them, track unit costs such as cost per successful outcome, and optimise without hurting quality.

Why are AI cloud costs harder to forecast than normal cloud costs?

Because AI costs follow token volumes, and token volumes change with prompt design, conversation length and retrieved context, not just user numbers. Model prices and options change often, and a new feature can multiply the calls per user. The FinOps Foundation notes that AI forecasts are less predictable early on and recommends shorter forecast windows with more frequent revisions.

How do I allocate LLM costs to individual customers?

You have to do it in your application, because cloud billing tags stop at the resource. Log the model, input and output token counts, and a customer or tenant ID for every call. Join that log with your effective token rates to get cost per customer. As a bonus, it shows which pricing plans are still profitable once AI features are included.

Should we buy provisioned throughput or reserved capacity for our models?

Only once usage is stable and well understood. Provisioned throughput bills for capacity whether or not you send requests, so it suits predictable, latency-sensitive production traffic. For variable or experimental workloads, pay-per-token or batch modes are usually cheaper. Run on demand first, measure utilisation for a few months, then commit to a baseline you're confident you'll use.

Cloud Cost Savings

Request a cloud and AI cost review

Tell us where you're stuck and we'll come back with practical next steps.

Get in touch

Sources