Opens in a new tab
Talk to Expert

The Real Cost of Running AI: Tokens, GPUs and the Stack

November 4, 2026

A business professional utilising AI to transform his day–to-day operations after understanding the real cost of AI.

AI agent costs come from two meters: tokens at the models layer and GPU-hours at the chips layer. A single query can trigger six model calls behind the scenes, which is why AI bills scale faster than user counts suggest.

Table of contents
Key Takeaways
  • The cost of AI agents is driven by two metres: AI tokens consumed at the models layer, and GPU-hours consumed at the chips layer. Everything else is packaging.
  • The AI stack has five layers: energy, chips, infrastructure, models, and applications. Money flows differently through each one.
  • A single agent conversation often calls the model multiple times behind the scenes. This is why AI bills scale faster than user counts suggest.
  • For Malaysian businesses weighing public APIs against a hosted private model, the maths flips as usage rises. A partner delivering cloud transformation services Malaysia should be able to model both paths against your actual token volume.

A CFO’s first AI bill is usually more interesting than the pilot that produced it. The pilot answered a business question. The bill answered a different one: what did last month of “just testing” cost, and why does next quarter look worse?

Running AI is not one price. It is a stack of five layers, each with its own meter: energy, chips, infrastructure, models, applications. Confusion around AI budgets comes from mixing these up or ignoring the ones vendors do not sell directly to you.

This article walks through what actually drives the cost of AI agents, layer by layer, using verified 2026 pricing for GPUs and per-token API rates. It is written for decision-makers who need to explain the number, not build the model.

Why AI Bills Scale Faster Than People Expect

A user asks an agent one question. Behind the scenes the agent might call the model six or seven times. It rephrases the query. It searches a knowledge base. It plans the steps. It calls a tool. It reads the tool’s output. It drafts a reply. It checks its own draft.

Each of those calls is billed in AI tokens. A token is roughly three-quarters of a word, and both input and output tokens count. Input is what the model reads. Output is what it writes. Output tokens on Claude Opus 4.8 cost $25 per million against $5 per million for input, a five-to-one ratio. Push a lot of context in cheaply and you still pay dearly for the answer that comes back.

The users see one conversation. The bill sees seven. That gap is the single biggest reason AI pilots come in over budget.

Layer 1. Energy: The Binding Constraint

Jensen Huang describes AI as a five-layer cake with energy at the base: intelligence generated in real time requires power generated in real time. Every token a model produces is electrons moving, heat being managed, and energy being converted into computation. There is no abstraction layer beneath it, which makes energy the ceiling on how much intelligence the system can produce.

Layer 2. Chips: Where GPU Costs Show Up

Above energy sit the chips: processors built to turn power into computation at massive parallelism. Enterprise AI in 2026 runs on NVIDIA H100 or newer Blackwell-class GPUs. A single H100 80GB card costs roughly USD 25,000 to 40,000 to buy, and an eight-GPU server exceeds USD 350,000 before power, cooling, and networking.

Most Malaysian businesses do not buy. They rent GPU-hours from a cloud provider. Hourly rates for a single H100 span USD 1.49 to USD 6.98, with hyperscalers at the upper end and specialist neo-clouds at the lower. That is roughly RM 7 to RM 33 per GPU-hour at RM 4.70 to the dollar, and any conversation about a GPU server Malaysia deployment starts here.

Layer 3. Infrastructure: AI Factories, Not Data Centres

Infrastructure is where thousands of GPUs are stitched into one working machine. Land, power delivery, cooling, construction, networking, and the software that orchestrates the whole assembly. Huang calls these systems AI factories. They are not designed to store information. They are designed to manufacture intelligence.

Cost at this layer is where agentic AI actually happens. Orchestration software, GPU drivers, Kubernetes operators, and networking turn a rack of cards into something an application can call. The software itself is cheap. What it produces is more tokens: an agent that searches a knowledge base, summarises the results, then drafts a reply spends three times the tokens of a single-shot chat.

Well-run infrastructure captures discounts the model layer offers but never applies automatically. Anthropic and OpenAI both cut cached input by roughly 90% and batch APIs by 50% off standard rates. Whether a business captures those discounts is decided in the infrastructure layer, not the model.

Layer 4. Models: Where Tokens Become a Bill

The model layer is where the language model itself lives. This is the layer most Malaysian businesses interact with first, usually via an API.

Pricing is per million tokens, and the spread is enormous. Claude Haiku 4.5 sits at USD 1 input and USD 5 output per million tokens. Claude Sonnet 4.6 costs USD 3 and USD 15. Claude Opus 4.8 costs USD 5 and USD 25. On the OpenAI side, GPT-5.4 Nano runs USD 0.20 and USD 1.25, while GPT-5.5 sits at USD 5 input and USD 30 output per million tokens. That is a hundred-fold range across the OpenAI portfolio alone.

The practical implication is that model choice matters more than most teams realise. A cheap model for straightforward summarisation and a premium model for complex reasoning is usually the right split. The other choice is public API versus hosted private model. Public API pricing is low at small volumes. A hosted open model on your own GPUs carries fixed cost but no per-token bill, so at high volumes the maths flips.

Layer 5. Applications: Where Cost Discipline Lives

The top layer is what the user sees: a customer support agent, an internal knowledge assistant, a sales copilot. Huang notes this is where economic value is created, but only if the four layers beneath it have been sized honestly.

Cost discipline at this layer is mostly about prompt design and context management. A prompt that stuffs every possibly relevant document into the model’s context window pays for the full input every call. A prompt that retrieves only the two or three most relevant chunks pays a fraction of that. The same feature can cost ten times more or less depending on how the application is built.

Where a Hosting Partner Changes the Maths

A managed AI infrastructure partner sits across the bottom four layers and handles the parts most businesses do not have the internal capacity for: sizing the GPU footprint, running the infrastructure layer, keeping model versions current, meeting PDPA obligations for data that flows through the stack, and modelling the total cost against actual usage rather than list price.

The commercial question for a Malaysian business is not “public API or private model”. It is at what token volume, and what data-sensitivity level, does hosting become cheaper and safer than paying per token to a provider outside the country. That crossover moves quickly as usage rises.

Every AI pilot reaches a moment where the sandbox bill and the production forecast diverge. Most teams see it in month three, by which point the budget is set on the sandbox number.

Speak to the Net Onboard team if the bill on your AI pilot has started climbing faster than the user count, if leadership is weighing public APIs against hosting a model in Malaysia, or if PDPA and data residency are part of the conversation and no one has a clean answer yet.

Cue Net Onboard, where AmplifyChampion sizes the AI stack against the workload you actually have, models the cost against the volume you actually run, and hosts the layers that need to sit in Malaysia rather than a US region.

Talk to us about cloud transformation services in Malaysia to work through the numbers before the next AI bill sets the wrong expectation.

References:

  1. AI Is a 5-Layer Cake.

    Retrieved 11 August 2026, from https://blogs.nvidia.com/blog/ai-5-layer-cake/

  2. NVIDIA H100 Price Guide 2026: GPU Costs, Cloud Pricing & Buy vs Rent.

    Retrieved 11 August 2026, from https://jarvislabs.ai/blog/h100-price

  3. Anthropic API Pricing in 2026: Complete Guide.

    Retrieved 11 August 2026, from https://www.finout.io/blog/anthropic-api-pricing

  4. OpenAI API Pricing In 2026: Every Model Compared.

    Retrieved 11 August 2026, from https://www.cloudzero.com/blog/openai-pricing/

  5. NVIDIA AI Enterprise Documentation.

    Retrieved 11 August 2026, from https://docs.nvidia.com/ai-enterprise/index.html

Frequently Asked Questions About the Cost of Running AI (FAQs)

  1. What actually drives the cost of running AI agents for a business?

    Two metres run underneath every AI deployment: AI tokens at the models layer and GPU-hours at the chips layer. Everything else is packaging. Tokens are the units the model reads and writes, priced per million, with output tokens costing three to five times more than input. GPU-hours cover the actual silicon running the model. A single user conversation with an AI agent may trigger six or seven model calls behind the scenes, which is why AI bills scale faster than user counts suggest. Orchestration and application design decide whether those calls are efficient or wasteful.

  2. What is an AI token and how does token count affect the bill?

    A token is roughly three-quarters of a word. Both input tokens (what the model reads) and output tokens (what it writes) are billed. Output typically costs three to five times more than input. Claude Opus 4.8 in 2026 charges USD 5 per million input tokens and USD 25 per million output tokens. GPT-5.5 charges USD 5 and USD 30. The practical implication is that verbose prompts and long generated replies compound fast, while prompt caching and batch APIs cut the same work by 50 to 90 percent.

  3. Should a Malaysian business buy GPUs or rent them from a cloud provider?

    In 2026, renting is usually the right answer unless a business runs sustained workloads above roughly 2,750 GPU-hours over the useful life of the card. A single NVIDIA H100 80GB costs USD 25,000 to 40,000 to buy outright and USD 1.49 to USD 6.98 per hour to rent, with specialist providers at the low end and hyperscalers at the high end. Below continuous heavy use, rental wins on cash flow, procurement lead time, and the ability to move up to newer generations without eating depreciation.

  4. Where does most of the money go in the AI stack?

    For businesses using public APIs, the models layer dominates the bill, and the token count decides everything. For businesses hosting their own models, the chips and infrastructure layers carry the largest fixed cost, and utilisation decides the effective rate. In both cases, the infrastructure layer is where cost discipline is enforced, because it decides how many model calls and how many tokens a single user query consumes. Application design is a close second: a prompt that retrieves the right context beats a prompt that stuffs the full context window every time. Energy sits underneath everything and sets a hard ceiling on how much intelligence can be produced at all.

  5. When does a private hosted model become cheaper than paying per token?

    The crossover depends on token volume, model choice, and the utilisation rate of the underlying GPU. As a rough guide, a business processing tens of millions of tokens a day on a mid-tier model usually finds hosted-model economics competitive with public API pricing, especially once PDPA data residency requirements are factored in. Below that volume, public APIs win on flexibility. Above it, the fixed cost of hosting pays back within months rather than years.