From Prompt to Power
A Survey of Models, Chips, and Infrastructure
This course is for anyone with basic training in computer science who wants to understand how modern AI works and what has to happen across its supply chain. No background in machine learning or chip design is required. Concepts are introduced in ordinary language as they appear, so readers without a technical background should still be able to follow most of the course.
The central idea is simple: an AI product is not only a model. Every answer depends on a chain of software, chips, memory, networking, data centers, electricity, manufacturing, and money. A limit or delay anywhere in that chain can affect what the user experiences and what the service costs.
An answer from artificial intelligence, or AI, may feel weightless, but physical machines make it. An AI model is a learned system of calculations that turns an input into a prediction or other output. A large language model is one built to process and generate language at a large scale; it can contain billions of learned numbers. Running it moves data through memory, performs arithmetic, and uses electricity. Those physical limits affect speed and cost.
This course follows that chain from the user outward:
your request
-> a model predicts an answer
-> chips move data and do arithmetic
-> data centers supply power, cooling, and connections
-> someone pays for every second of the process
A chip specification means little until you know the job the chip must do. A price per chip-hour means little until you know how many useful answers that hour produces. Part 0 introduces the units used throughout. The rest of the course follows a compact map:
- Workloads, Parts 1–3: inference, training, and agents.
- The machine, Parts 4–9: why graphics processors work, what is inside them, memory, number formats, manufacturing, and connections.
- Decisions, Parts 10–11: choosing hardware and understanding economics.
AI performance is never one number. A system can have fast arithmetic and slow memory, cheap chips and expensive power, or an impressive maximum speed that useful work never reaches. For repeated work, the most constrained required resource often limits how much the system finishes per second; several stages may still add to the delay of one request.
How facts are handled. Real product, model, and market numbers are sourced, and volatile figures are dated. Examples introduced with “suppose” are illustrative arithmetic, not market quotes. Claims that cannot be supported or usefully qualified are omitted.
Part 0 — A Small Vocabulary for Big Systems
This opening introduces four quantities used throughout the course: arithmetic, stored and moved data, power and energy, and tokens. Keep each amount separate from its rate.
0.1 Work Is Different From Speed
A job has a fixed amount of work, while a machine has a rate at which it can attempt that work. Confusing the two is like confusing miles with miles per hour.
Many AI calculations multiply and add numbers that can represent fractions. Engineers count those individual arithmetic steps as floating-point operations, or FLOPs. Under the convention used in most hardware specifications, one addition or one multiplication is one operation. A combined multiply-and-add is two because it performs both steps. A rate can be written FLOP/s, meaning floating-point operations per second. Because the abbreviations for an amount and a rate are easy to confuse, this course usually writes “operations” and “operations per second” in full.
Suppose a job needs 1,000 operations and a processor actually completes 100 operations each second. The arithmetic alone takes 10 seconds:
time = amount of work ÷ achieved rate
= 1,000 operations ÷ 100 operations per second
= 10 seconds
The word achieved matters. A specification normally gives the highest rate the hardware can reach for a particular kind of number and operation. A real program may spend part of its time waiting for data, another processor, or a result on which its next step depends. A peak operations-per-second figure is therefore a ceiling under stated conditions, not a promise that every program will run at that speed.
0.2 Stored Data Is Different From Moved Data
Memory capacity asks how much data fits. Memory bandwidth asks how much data can pass a named point each second. A warehouse can hold a million boxes and still have one narrow loading door. The first fact describes capacity; the second describes bandwidth.
Suppose a machine must read a 100 GB file from memory and can sustain 50 GB per second. The transfer takes at least two seconds if nothing else uses that path. Doubling the capacity of the memory would let the machine hold more files, but it would not make this transfer faster. Doubling the bandwidth could reduce the transfer time, but it would not create room for a larger file. Later Parts apply this distinction to model weights: the weights must fit before their delivery speed matters.
A bit represents one of two values, conventionally zero or one. Eight bits make one byte. In decimal units, one gigabyte (GB) is exactly one billion bytes; 2³⁰ bytes is one gibibyte (GiB). A quoted bandwidth of 100 GB per second means 100 billion bytes per second, but the label must say whether that rate is a theoretical maximum or a measurement. Network specifications often count bits rather than bytes. Dividing by eight converts the units to bytes; the extra information needed to format and deliver network data can make the useful rate lower. NIST defines decimal and binary data units here.
0.3 Power Is Different From Energy
Electrical power tells you how quickly a device is using energy at a moment in time. It is measured in watts. Electrical energy tells you how much was used over a period of time. Electricity bills normally charge for energy in kilowatt-hours.
Follow one accelerator server through an hour. Suppose it draws one kilowatt while a job runs:
energy = power × time
= 1 kilowatt × 1 hour
= 1 kilowatt-hour
electricity cost = 1 kilowatt-hour × $0.10 per kilowatt-hour
= $0.10
If the same server runs for 10 hours at the same power, it uses 10 kilowatt-hours. Its power did not become “10 kilowatts”; the device drew one kilowatt for longer. Likewise, a device rated at one kilowatt does not necessarily draw that amount at every moment.
The server is not the whole facility. Fans, pumps, power-conversion equipment, networking, and other support systems also use electricity. When someone quotes a power or energy figure, ask three questions: Is it a maximum or a measurement? Over what period? Does it describe one chip, one server, one rack, or the entire data center?
0.4 Models Consume and Produce Tokens
Language models do not receive text as words on a page. A tokenizer first divides the text into pieces called tokens and assigns an integer identifier to each piece. A token might be a whole common word, part of a less common word, punctuation, or even a space joined to nearby text. The exact pieces come from that tokenizer's vocabulary and rules.
For intuition, imagine that a tokenizer divides “unbelievable!” into un, believ, able, and !. Those are illustrative pieces, not a claim about a particular tokenizer. The model receives the corresponding identifiers and later produces identifiers that the service converts back into text. Two models can divide the same sentence differently, so a word count cannot determine an exact token count.
For OpenAI tokenizers, a useful English-only guide is about three-quarters of a word per token. The ratio changes with the language, text, model, and tokenizer. OpenAI explains this estimate and its limits.
A model also contains learned values called parameters. Many parameters are numerical weights that control how strongly one set of numbers affects another; parameters can include other learned adjustments too. A “70-billion-parameter model” contains about 70 billion learned values under its publisher's counting convention. Those values are not 70 billion sentences or database entries. They work together inside the model's calculations.
Tokens describe the input and output length. Parameters describe the size of the model's learned state. A request with more tokens usually requires more work even though the parameter count stays fixed. A model with more parameters usually needs more storage and arithmetic when its design and number format are otherwise comparable. Neither number, by itself, measures answer quality.
Carry two habits forward: attach “per what?” to every rate, and ask whether a figure is a theoretical maximum or a measured result.
Part 1 — Inference: How an Answer Gets Made
Most chat-oriented language models generate text by selecting a next token, adding it to the sequence, and repeating. They do not normally prepare the complete answer in one pass. This process explains streaming text and part of the memory cost of long conversations. Serving requests together can reduce cost, but does not guarantee it. Using a trained model to produce an output is called inference.
1.1 From Pressing Enter to Seeing Text
Pressing Enter does not send words straight into a model and receive a finished paragraph back. A service first prepares the request, then the model predicts one output token at a time. The whole journey has five stages:
- Route the request. A web service may check who is calling, enforce its rules, and send the request to a machine that can access the model.
- Assemble the model input. The service selects instructions, earlier messages, retrieved material, tool results, and the new message. That history is logically part of the current call even when a service caches part of it; the model does not keep human-like memories between independent calls.
- Turn text into numbered pieces. A tokenizer divides text according to its vocabulary and rules, then maps each token to an integer identifier. Vocabularies and splitting rules differ among model families. This conversion is called tokenization.
- Process the input and select the first answer piece. The model produces a numerical score for each token in its vocabulary, and a decoding rule uses those scores to select the next token. It might select “Rise” rather than “Sourdough” in response to “Name my bakery.” Processing the prompt and preparing reusable attention data is called prefill.
- Build the rest one piece at a time. In ordinary autoregressive generation, the selected token becomes part of the sequence and the model runs another step. This answer-building phase is called decode. Generation can stop at an end token, a configured token limit, or another stopping rule. The service then turns token identifiers into readable text.
Input: Name a bakery that sells only sourdough.
Read the full input -> “Rise”
Read the input plus “Rise” -> “and”
Read the input plus “Rise and” -> “Loaf”
Read through “Rise and Loaf” -> ending marker
Output: Rise and Loaf
Because each accepted output token depends on those before it, ordinary answer generation proceeds in order. This creates two phases of serving, explained next.
1.2 The First-Token Pause and the Typing Phase
The pause before an answer and the spacing between later tokens are two different delays. A service can improve one without improving the other, so a single “speed” number can hide what the user will feel.
click Send first token last token
|--------------------------|---|---|---|---|---|--------|
time to first token gaps between output tokens
The first span can include network travel, waiting for a machine, preparing the request, processing the prompt, and selecting the first token. The smaller later spans show the pace of ordinary answer generation.
During prefill, the model processes the prompt tokens together, so much of the work can run in parallel. Longer prompts usually require more work. Time to first token is the elapsed time from submitting a request until receiving the first output token; it can include network travel, waiting, text conversion, request setup, and prefill. NVIDIA defines this measurement boundary in its inference metrics guide.
During ordinary decode, output tokens are committed in order, and each step uses the model's learned parameters again. The elapsed time between output tokens is inter-token latency, which appears as typing speed to the user.
Why can a chip with enormous arithmetic capacity still generate slowly? For one or a few requests, each generation step may have to fetch most of the model's weights from memory and use them only once. Fetching those numbers can take longer than doing the arithmetic. The limit is memory bandwidth: how many bytes memory can supply each second. This pattern is common, not universal.
Consider a deliberately optimistic one-user example. Suppose such a model's weights occupy 70 GB and an accelerator—a processor built to perform AI calculations efficiently—has a peak memory bandwidth of 3,350 GB per second. If each decode step must stream those weights once, weight traffic alone gives an upper bound of 3,350 ÷ 70, or about 48 steps—and therefore about 48 output tokens—per second. Real throughput is lower when the peak bandwidth is not sustained and because cache reads, temporary values, synchronization, and other work also consume time and bandwidth. The bandwidth figure matches NVIDIA's published peak for one H100 server-module configuration, not a measured application result. NVIDIA H100 specifications, accessed July 2026.
The meaning of the estimate matters more than the exact figure. On a specified machine, work limited mainly by arithmetic throughput is compute-bound; work limited mainly by memory traffic is memory-bandwidth-bound. Long or well-batched prefill can be compute-bound, while generating tokens for only a few requests is often memory-bandwidth-bound. Prompt length, model design, number of simultaneous requests, software, and hardware can change the result.
1.3 How a Transformer Uses the KV Cache
The main idea is simple: a transformer predicts one next token, adds the selected token to the sequence, and repeats. To make each repeat less wasteful, serving software keeps some numbers that the model already calculated for earlier tokens. Those saved numbers form the key-value cache, or KV cache.
The cache is not a written summary of the conversation, and it does not make the prediction by itself. The transformer still runs every model layer for each new token. The cache merely prevents those layers from rebuilding the same key and value numbers for all earlier tokens. Each new token must still compare with and read the saved numbers. This is why the cache saves arithmetic while also consuming memory and memory bandwidth. Hugging Face's implementation guide shows the cache being updated and read separately in every attention layer.
First, follow one next-token prediction
Consider this prompt:
The musician tuned the piano before the concert. Then she played the
Suppose the model selects piano as the next token. That outcome is plausible, not guaranteed. To keep the example readable, the words are shown as if each were one token; a real tokenizer may divide the text differently. The model reaches its selection in six broad steps:
- Turn each token into numbers. The token's identifier selects an embedding: a learned list of numbers that gives the model a numerical starting representation for that token. The model also represents where the token occurs in the sequence; designs differ in how they supply that position information to attention.
- Let token positions exchange information. An attention calculation allows each position to combine information from itself and earlier positions. It cannot use later positions that have not been generated. In the example, the representation at the final the can be influenced by earlier positions such as musician, tuned, and piano.
- Refine each position separately. A feed-forward block applies another learned calculation to each token position. Attention moves information among positions; this block transforms the information now held at each position.
- Repeat those two kinds of work. One attention block followed by one feed-forward block makes up the simplified core of a model layer. A transformer passes the representations through many such layers, so later layers work with information already shaped by earlier layers.
- Score every allowed output token. After the final layer, the representation at the last position is mapped to one score for every token in the model's vocabulary. Those scores are converted into probabilities.
- Select and append one token. A decoding rule selects a token, perhaps piano, and appends it to the sequence. The model then runs again to predict what follows piano.
This teaching sequence follows the visual route in 3Blue1Brown's transformer explainer: tokens become numerical representations, attention and feed-forward calculations repeatedly change those representations, the final representation becomes vocabulary scores, and one new token is appended. The original Transformer paper is the primary source for the attention and feed-forward design. The video develops intuition for the transformer; it does not cover KV-cache serving, which is the next step here.
Zoom in: what query, key, and value mean
Inside one attention layer, the model uses learned weights to produce three lists of numbers from a token's current representation: a query, a key, and a value. Together, they form the query-key-value (QKV) part of attention:
- The query represents what the current position is looking for.
- A key represents what a position can be matched on.
- A value carries the information that position can contribute.
These are numerical roles, not labels or sentences written by a person. The model compares a query with the available keys to produce match scores. It turns the scores into weights that add to one, then makes a weighted mixture of the corresponding values. A strong match gives that position's value more influence, but all of these patterns are learned and can differ by layer, attention head, prompt, and model.
An attention head is one set of query-key comparisons and value mixing. A real layer runs several heads in parallel, allowing different learned comparison patterns. The resulting information is combined before the feed-forward block.
The names now explain the cache. A future token creates a new query, compares it with the keys from earlier positions, and mixes the earlier values. It therefore needs the old keys and values again. It does not need the old queries: each old query already did its job when that earlier position was processed. That is why serving software caches K and V, not Q. The Transformer paper formally defines this query-key comparison and weighted use of values.
Prefill builds the first cache
During prefill, the prompt moves through the transformer one layer at a time. Within a layer, hardware can process many prompt positions together while the model's left-to-right rule prevents a position from using later positions. For every prompt token, each attention layer computes a key and a value. Serving software keeps them because a later output token will need them.
The result is not one general-purpose note. It is a separate pair of numerical arrays for every layer:
layer 1 cache: K and V for prompt positions 1 ... N
layer 2 cache: K and V for prompt positions 1 ... N
...
last layer: K and V for prompt positions 1 ... N
The same visible token therefore has different saved numbers at different layers. Those numbers describe how that layer can match and use the position; they are not stored words, model weights, or a plain-language summary.
Decode reads the cache and extends it
After prefill selects the first output token, decode repeats the following process:
- Feed only the newly selected token into the model; the earlier tokens do not need to pass through every layer again.
- In the first layer, calculate a new query, key, and value for that token.
- Compare its new query with the first layer's saved earlier keys and the current token's new key, then use the resulting weights to mix the corresponding earlier and current values.
- Keep the new key and value in the first layer's cache, making that layer's cache one position longer.
- Pass the attention result through the rest of that layer, including its feed-forward block.
- Repeat the same work in every later layer, using that layer's own cache. The current token's representation changes as it moves upward, so each layer calculates its own new query, key, and value.
- Turn the last layer's result into vocabulary probabilities, select one token, and begin the next decode step.
Return to the musician example. If the simplified prompt contains N tokens, prefill creates N key-value positions in every layer and the final prompt position produces the scores from which piano is selected. On the next pass, the model processes the newly selected piano, adds its key and value to every layer's cache, and may select a period. The cache now has N + 1 positions per layer. Processing the period adds one more position and produces another prediction. The sequence therefore grows N → N + 1 → N + 2, while the model avoids rebuilding the first N sets of keys and values.
What the cache changes—and what it does not
| The KV cache does | The KV cache does not |
|---|---|
| keep each layer's earlier keys and values | keep a human-readable transcript or summary |
| avoid recalculating those earlier keys and values | replace the model's learned weights |
| grow as more token positions are retained | make attention over the retained positions free |
| allow identical prompt beginnings to be shared when serving software supports it | automatically remember material omitted from the model's current input |
The model must still run its layers, create the newest query, key, and value, read the retained cache, and select the next token. Cache reuse changes duplicated work; it does not change the transformer's basic next-token task. The cache also belongs to the current sequence. Each active sequence normally needs its own entries, although a serving system may share an identical beginning across requests. Hugging Face documents the per-layer cache and its growth, while the vLLM paper describes one production approach to managing and sharing cache memory.
A concrete memory calculation
Cache size depends on the model's design, retained token count, and bytes used for each saved number. Meta's Llama 3 report lists 80 layers and 8 key/value heads for its 70B model. Its attention calculation splits an 8,192-number token representation across 64 attention heads, giving 8,192 ÷ 64, or 128, numbers per head; each key/value head uses that same width. If each cached number takes two bytes, one retained token requires:
80 layers
× 8 key/value heads
× 128 numbers per head
× 2 arrays (one key and one value)
× 2 bytes per number
= 327,680 bytes per retained token
≈ 0.328 MB per retained token
At 128,000 retained tokens, those saved key and value numbers occupy 41,943,040,000 bytes, or about 41.9 GB in decimal units, for one sequence. For comparison, using the model's rounded 70-billion-parameter label, parameters stored at two bytes each occupy about 140 GB. This comparison does not mean every deployment allocates exactly those amounts: software may reserve cache space in blocks, use a different number format, share an identical prompt beginning, or retain fewer than the model's maximum tokens.
The dimensions come from Table 3 of Meta's Llama 3 report; Meta's Llama 3.1 model card documents the 70B model and its 128K-token context. The durable lesson is the tradeoff: caching avoids repeated calculation, but a long retained sequence occupies substantial memory and still has to be read.
1.4 Serving Many Users Changes the Economics
Serving requests together can make an expensive model cheaper per answer because one weight read can help several requests. The tradeoff is that each request consumes memory and may wait for a group to form.
First consider four users whose conversations are already in the decode phase. Without batching, the accelerator could handle them one after another:
without batching
read a block of weights → produce A's next token
read the same weights → produce B's next token
read the same weights → produce C's next token
read the same weights → produce D's next token
with batching
read a block of weights once
├─ use it for A's next token
├─ use it for B's next token
├─ use it for C's next token
└─ use it for D's next token
Batching means processing several requests in one model operation. The requests do not share answers or conversation histories. They share the opportunity to use the model's fetched weights before those weights are replaced by another block. This reuse performs more useful arithmetic for each byte moved from memory, which can raise throughput: total completed work per unit of time.
A serving system usually cannot wait for four conversations to reach exactly the same point. Requests arrive at different times, prompts have different lengths, and some answers finish early. Modern serving software can therefore remove finished requests and add new ready requests between generation steps. The implementation details vary, but the purpose is the same: keep useful work on the accelerator without making an early request wait too long for a perfectly matched group.
Batching has three limits:
- Waiting: Holding a request while a batch forms can increase its time to first token.
- Memory: Every active sequence needs temporary state, including its own KV-cache entries unless an identical beginning can be shared. More simultaneous requests can exhaust memory before arithmetic is full.
- Uneven work: A long prompt or unfinished answer can outlast shorter requests. Serving software must replace finished work or leave part of the batch idle.
The result is a real tradeoff. A larger batch can lower accelerator cost per token and raise total tokens per second while increasing the delay one user sees. A useful serving report therefore gives both throughput and user-facing latency under a stated request load. The vLLM paper documents one serving system that manages KV-cache memory in blocks and batches changing groups of requests.
1.5 From Chip Rental to Cost per Token
The useful economic question is not “How much does the chip cost per hour?” It is “How many acceptable outputs does that paid hour produce?” A cheap but mostly idle machine can cost more per answer than a more expensive machine kept productively busy.
There are three different prices that people often mix together:
| Price | What it includes |
|---|---|
| Accelerator rental per hour | access to a stated amount of hardware for an hour |
| Provider cost per token | rental or ownership plus the other resources needed to operate the service |
| Customer price per token | the price charged after overhead, service features, and margin |
The following calculation covers only the first row.
The basic conversion is:
raw accelerator-rental cost per million output tokens
= total accelerator price per hour
divided by tokens produced in one hour
multiplied by 1,000,000
Suppose one accelerator rents for $2.20 an hour and the service produces 2,500 output tokens each second across all users it serves at once. It produces 2,500 × 3,600, or 9 million output tokens per hour. Dividing $2.20 by nine gives about $0.24 per million output tokens. The result answers a narrow, hypothetical question: accelerator rental contributes about 24 cents per million output tokens under those assumptions.
It is not the customer price. The provider must also pay for input processing, idle periods, networking, power, staff, failed requests, and profit. A low-delay chat service may use small batches and cost more. An overnight job may use large batches and cost less. Long contexts may run out of memory before arithmetic becomes the limit.
Any serving benchmark—a controlled performance test—should report three results: the delay before the first token, the delay between later tokens, and total tokens per second across all users. It should also name the model, hardware, input and output lengths, and number of simultaneous requests. When only a few requests share a model that uses all its parameter groups, serving often streams much of the weight data for each step. Batching and KV-cache memory then strongly affect how many sequences can share an accelerator.
Part 2 — Training: How a Model Gets Its Numbers
Training changes a model's learned numbers so its predictions improve. It repeats three steps: make a prediction, measure the error, and adjust the learned numbers. Repeating this across enormous amounts of text requires substantial arithmetic and extra memory for the information needed to make each adjustment. The learned state saved at a point in time is called a checkpoint. Initial training commonly starts with randomly chosen values; continued training or fine-tuning starts with values learned earlier.
2.1 A Trained Model Combines a Design With Learned Numbers
Code and configuration describe the model's design. Training learns its parameters: numerical values saved in a checkpoint. Many parameters are entries in matrices called weights; others make different learned adjustments.
One weight participates in a calculation; it should not be read as one stored fact. Consider a made-up dot-product example:
inputs: [2, 5, 1]
weights: [3, -1, 4]
score = (2 × 3) + (5 × -1) + (1 × 4) = 5
The example multiplies matching values and adds the results. Many learned weights are arranged as rectangular arrays called matrices. Multiplying a matrix by a vector or another matrix and summing products is called matrix multiplication.
A neural network applies learned calculations through a sequence of stages called model layers. Most current large language models use a design called a transformer. A transformer repeats blocks that include attention, the mechanism introduced in Part 1 for letting each token use earlier text.
Parameter count is the publisher's count of individual learned numbers. Suppose a 70-billion-parameter model stores every parameter in two bytes. The parameters occupy about 140 GB. For comparison, the cited NVIDIA H100 configuration has 80 GB of memory, so that allocation cannot fit on one H100 without compression, sharding, or moving some data elsewhere. NVIDIA H100 specifications, accessed July 2026.
Inference loads and uses learned parameters. Training or fine-tuning changes them.
2.2 Training Is Repeated Prediction and Correction
A training step does not simply tell the model that an answer was wrong. It must calculate how each learned parameter contributed to the error and then make a small coordinated adjustment. The process is repeated prediction and correction:
For a made-up example, imagine that the training text contains “The cat sat on the mat.” The model sees “The cat sat on the” and assigns probabilities to possible next tokens. If it gives “roof” 40 percent and “mat” 2 percent, it assigns little probability to the observed next token, “mat.” A scoring rule turns that error into a number. For the next-token score used here, lower is better. It is called the training loss.
The computation from input to prediction and loss is the forward pass. Training retains some intermediate values and may later recompute others because the next stage needs them.
The system then works backward through the connected calculations. For each parameter, it estimates whether a small increase or decrease would make the loss better or worse, and by how much. That direction-and-size signal is called a gradient. It is not the update itself. Calculating these signals backward through the model is backpropagation, usually shortened to the backward pass.
Finally, an optimizer uses the gradients and an update rule to change parameters. A widely used optimizer called Adam keeps a moving average of each parameter's gradients and another of its squared gradients, then uses both when calculating the update. The original Adam paper defines these first- and second-moment estimates.
A batch is a group of training examples processed together. A system can combine results from several smaller groups before one optimizer update. The training rule rewards parameters that make the observed answers more likely across examples. This differs from storing sentences as explicit database records, although a model can still memorize and reproduce some training examples.
2.3 One Formula Gives the Rough Arithmetic Bill
For a conventional transformer that uses all its learned parameters for every token, training work grows roughly with both model size and the number of training tokens. Double either one, and the estimated work roughly doubles. A useful estimate is:
training operations ≈ 6 × parameter count × training-token count
The source formula counts the main transformer blocks separately from the learned tables that turn token identifiers into starting representations and final scores. Those tables are usually a small share of a large model's parameters, so using the publisher's total parameter count slightly changes the estimate. The shortcut also omits smaller and design-dependent calculations. Use it only for rough planning. Appendix F of DeepMind's compute-optimal training study gives the full accounting.
Why does the estimate contain the number six? In the simplified accounting, the forward pass uses roughly two operations per parameter for each token: multiply a value by a weight, then add the result to a running total. The backward pass must calculate how the loss changes with both the layer's inputs and its weights, adding roughly twice the forward work. That gives about two operations for the forward pass plus four for the backward pass, or six in total. Real transformers include operations that this shortcut does not count exactly, so six is an estimating rule, not a physical constant.
What does the rule predict for a large run? For an illustrative 70-billion-parameter model trained on 15 trillion tokens:
6 × 70 billion × 15 trillion
= about 6.3 septillion operations (6.3 × 10²⁴)
One septillion is one followed by 24 zeros. To turn that intimidating count into elapsed time, divide the work by the combined achieved rate of the accelerators:
time = total operations
÷ (number of accelerators × peak rate each × achieved share of peak)
Suppose 4,096 accelerators can each perform one quadrillion applicable operations per second using bfloat16, a compact two-byte number format, and the complete run averages 40 percent of that theoretical rate. The division gives about 3.85 million seconds, or 44.5 days. At a hypothetical rental price of $2 per accelerator-hour, accelerator rent alone would be about $8.75 million.
Every figure after “suppose” is an illustrative assumption, not a report of a real training run or market price. The example's purpose is to connect model size and training data to time and money: even thousands of fast chips can remain busy for weeks. Failures, storage, networking, data preparation, and staff add work or cost. The one-quadrillion rate is rounded from NVIDIA's published H100 bfloat16 rate when every value is processed. A larger headline rate in the same table assumes a supported pattern in which the hardware can skip selected zero values. NVIDIA H100 specifications and footnote, accessed July 2026.
2.4 Training Needs Much More Memory Than the Finished Model
The main point comes first: training can need far more memory than serving because it keeps the model plus the information needed to update it. In one documented NVIDIA NeMo recipe, most calculations use bfloat16, shortened to BF16, a two-byte format that retains less detail. Updates keep a more detailed four-byte copy in 32-bit floating point, shortened to FP32. Using more than one format this way is called mixed-precision training. The optimizer in this example, Adam, also keeps two running estimates for every parameter. NVIDIA's NeMo 25.02 guide specifies two-byte parameters and gradients alongside four-byte master parameters and optimizer state, and the Adam paper defines its two per-parameter running estimates.
This illustrative setup keeps five items for every learned parameter. The table explains why each one exists before adding their sizes:
| Item kept for one parameter | Why it exists | Memory in this example |
|---|---|---|
| BF16 working parameter | used for most forward and backward arithmetic | 2 bytes |
| gradient | measures how the loss changes as the weight changes | 2 bytes |
| more precise master weight | receives the optimizer's update | 4 bytes |
| Adam's recent-gradient average | helps smooth the update direction | 4 bytes |
| Adam's recent-squared-gradient average | helps scale the size of the update | 4 bytes |
| Total | persistent state in this recipe | 16 bytes |
The gradient was introduced in Section 2.2, and the last two rows are Adam's running estimates. The important idea is that this recipe uses smaller values for much of the arithmetic while retaining more detail for updates. Together, its five items use 16 bytes per parameter. Seventy billion parameters therefore require 1.12 trillion bytes, or 1.12 TB, before temporary results and software overhead. This is one worked recipe, not a law: other systems use different formats, optimizers, or ways of dividing and relocating data.
An H100 with 80 GB of memory would require at least fourteen-way perfect division just to hold that 1.12 TB. A real run needs more devices because temporary values, communication buffers, uneven splits, and software must also fit. NVIDIA H100 specifications, accessed July 2026.
Splitting large arrays so that each processor stores only a share is called sharding. It saves memory on each processor, but the processors must exchange data when a calculation needs the complete array.
Training also creates intermediate arrays of numbers as the input passes through the network. These values are called activations. The backward pass needs selected activations, or enough information to recompute them. Their memory depends on the model design, number and length of examples, layer width, number storage, how work is divided, and which values are saved.
The system can save a selected subset of intermediate values and recompute omitted forward operations during the backward pass. This activation checkpointing, also called recomputation, trades extra arithmetic for lower activation memory. The cost and saving depend on which operations are repeated; there is no universal percentage.
Inference and training therefore have different memory bills. Inference may keep one compact copy of each weight plus the per-conversation cache from Part 1. The particular 16-byte training setup above keeps eight times as many persistent bytes per weight as two-byte inference weights, plus activations. Other training setups have different ratios. A chip that serves a model well is not automatically the best chip for training it.
2.5 Thousands of Processors Become One Fragile Machine
Adding processors shortens a training run only when the extra parallel work outweighs communication, synchronization, imbalance, data-loading, and failure overhead. At large scale, network and reliability can materially limit completed work.
Start with a four-accelerator example. Each accelerator has an identical copy of the model but reads a different quarter of a training batch. All four perform the forward and backward passes at the same time. They now have four different gradient arrays because they saw different examples. Before any copy updates its weights, the accelerators add or average those arrays and give the combined result back to every participant. Only then can all four apply the same update and remain identical. This arrangement is data parallelism.
The extra processors reduce the calculation each one performs on training examples, but they add a communication step and a wait for the slowest participant. Other ways of dividing the model create other exchanges:
There are several intuitive ways to divide a run:
| How the work is divided | What must be exchanged | Technical name |
|---|---|---|
| processors handle different examples; ordinary data parallelism gives each a model copy | gradients combined before an optimizer update | data parallelism |
| different processors hold different groups of layers | activations move forward; signals showing how the loss changes move backward | pipeline parallelism |
| processors partition operations inside a layer | partial arrays combined through frequent communication | tensor parallelism |
| processors hold different pieces of parameters, gradients, or optimizer state | pieces requested, combined, or moved when a calculation needs them | state sharding |
Large runs combine these approaches. Data-parallel gradient communication commonly occurs once per update or group of accumulated batches, while tensor-parallel communication can occur several times inside every transformer block. The latter is usually more sensitive to link delay, but message size, the ability to calculate while data moves, the connection layout, and the number of processors all matter. Part 9 explains the resulting network hierarchy.
The table is a map, not a vocabulary test. Processors can divide the examples, groups of layers, operations inside a layer, or stored parameters and training state.
Thousands of processors also make individually rare failures routine. In one 54-day Llama 3 training snapshot, Meta recorded 466 interruptions: 47 planned and 419 unexpected. Unexpected interruptions therefore averaged one every 3.1 hours. The run used clusters of up to 16,384 H100 accelerators. Meta's Llama 3 report, Section 3.3.4.
To recover, the system periodically saves enough durable state—such as model, optimizer, schedule, and progress information—to resume from a saved point. This recovery checkpoint differs from activation checkpointing: one supports failure recovery, while the other reduces memory during a calculation. More frequent recovery checkpoints can reduce repeated work after a failure but add storage, input/output, and synchronization overhead. The cited Meta report calls the share of wall-clock time spent on useful training “effective training time” and reports more than 90 percent.
2.6 Training Continues After the Big Text Run
A model trained only to continue broad text has learned a powerful next-token task, but it has not automatically learned the behavior people expect from an assistant. If a prompt contains a question and several possible continuations, the model's basic objective rewards likely continuations, not necessarily the most helpful, honest, or safe response.
Post-training changes that behavior after the large general training run. A simplified pipeline looks like this:
- Show examples of desired behavior. Trainers prepare prompts and suitable responses. Updating the model on those pairs is supervised fine-tuning.
- Compare possible responses. People, automated checks, or another model may rank or score candidate answers. This creates preference or reward information.
- Train toward preferred behavior. A method uses that information to update the model. Reinforcement learning from human feedback and direct preference optimization are two different approaches; a real pipeline may use either, both, or neither.
- Evaluate and repeat. Developers test capabilities and unwanted behavior, revise data or objectives, and produce another checkpoint.
The InstructGPT paper documents one pipeline using supervised fine-tuning followed by reinforcement learning from human feedback, while the direct-preference-optimization paper introduces a different way to learn directly from preferred and rejected responses. These papers are examples, not a universal recipe.
For tasks with checkable outcomes, a system can generate candidate solutions and evaluate them with tests. Reinforcement learning is a broader family of methods that improves behavior to increase an expected reward; not every generate-and-filter process is reinforcement learning. Sampling candidates can make inference a substantial part of later training.
A released checkpoint contains fixed learned state that an inference service loads, often after conversion, sharding, or quantization. Ordinary inference does not update those parameters. A provider may separately retain interactions and later use selected data in another training run, subject to its policy.
The distinction closes the loop:
training = use data and an objective to update learned parameters
inference = use fixed parameters to compute outputs
Training is a concentrated project with a large arithmetic, memory, communication, and reliability bill. Inference is the repeated cost paid whenever the file is used. Together they explain why AI hardware needs fast arithmetic, spacious memory, quick links, dependable operation, and abundant power.
Part 3 — From a Prompt to Finished Work
In this course, an AI agent means software that repeatedly calls a model and can use external tools. The model proposes what to say or do next. A coordinating program, called the agent runtime here, keeps task state, runs approved tools, returns their results to the model, and decides when to stop. The model itself does not click a browser or run a command; it produces text or a structured tool request that ordinary software may execute. A short answer may take one model call, while research or coding can require many calls with searches, file reads, edits, and tests between them. “Agent” has no single universal industry definition; this is the course's working meaning. OpenAI documents the model–tool–result flow here, and Anthropic distinguishes fixed workflows from agents here.
3.1 The Whole Agent in One Loop
The loop has five parts.
- The runtime gathers the material the model needs: the user's request, relevant conversation history, instructions, available tools, and recent tool results. This bundle is the model context.
- The model reads that context and produces either an answer or a proposed tool call. A tool call is a structured request such as “search for this phrase,” “read this file,” or “run these tests.”
- A well-designed runtime checks that the requested tool exists, validates its arguments, and enforces the user's permissions. A consequential action, such as sending an email or spending money, may also require explicit approval.
- If the call is allowed, the tool runs outside the model. The search engine searches, the browser clicks, or the test runner runs. The tool returns an observation: a page, a screenshot, command output, a file, or an error.
- The runtime adds that observation to the task state and calls the model again. The loop ends when the model gives a final answer, the user must make a decision, or a safety or resource limit stops the task.
The loop is easier to see in a research task. Suppose the user asks for a comparison with sources. The first model call may propose two searches. The runtime runs them and returns the results. The next model call may notice that one source is a marketing page and request a more authoritative document. A later call may use a calculator. Only after the evidence is adequate does the model write the answer.
3.2 What Happens During One Model Call
An agent task can contain many model calls, and every call is a fresh inference request. The runtime must assemble the material needed for that call; the model does not silently remember the result of an earlier call that the runtime leaves out.
Follow a simple research task. The user asks, “Compare two accelerators and cite the official specifications.”
- First model call: The runtime supplies the request, instructions, and descriptions of available search tools. During prefill, the model processes those tokens. During decode, it produces a structured request to search for the first specification.
- Tool work: The runtime checks the request and runs the search outside the model. The result might contain several links and snippets.
- Second model call: The runtime sends the original goal plus the useful search result back to the model. This call has its own prefill and decode. The model may request the official page rather than relying on a snippet.
- More tool and model calls: The loop repeats for the second accelerator, calculations, and source checks.
- Final model call: The runtime provides the evidence needed for the answer. The model generates the cited comparison one token at a time.
call 1 context: goal + instructions + tool descriptions
call 2 context: goal + needed history + search result
call 3 context: goal + needed history + official specification
final context: goal + selected evidence + conclusions to check
The later contexts do not have to contain a verbatim copy of everything that happened. The runtime can retain exact evidence, summarize old discussion, reopen a source, or discard an irrelevant result. Each choice has a cost: dropping information can remove a needed fact, while carrying everything makes later prompts longer.
The KV cache from Part 1 helps while a particular sequence is being processed and can sometimes let serving software reuse an identical prompt beginning. It is not durable agent memory. The runtime still has to decide what information belongs in a later model call.
Long agent tasks can therefore cost more in two ways: they make more calls, and later calls may process more input tokens. A rough task bill is the sum of every call's input and output cost plus paid tools and other services. Retries add to the bill even when they produce no useful final result.
End-to-end time also includes work outside the model. Tests, searches, websites, approval waits, and other services may take longer than inference. This is why the meaningful performance measure for an agent is usually the time, cost, and success rate of a completed and checked task—not tokens per second in one model call.
3.3 Four Tasks, One Loop
| Kind of task | Typical shape | What usually slows it down | A useful success measure |
|---|---|---|---|
| Plain chat | One model call, no tools | Reading the prompt and generating the reply | Time to a correct answer |
| Research | Several calls with search and source reading | Finding trustworthy evidence and carrying it into context | Accurate answer with working citations |
| Coding | Repeated search, read, edit, and test cycles | Choosing the right files, test time, and recovery from mistakes | A verified fix, not lines of code produced |
| Browser work | Repeated page-state or screenshot, decision, and action cycles | Page loads, changing interfaces, and permission checks | Correct completion without an unsafe action |
The economic lesson is simple: a low token price does not guarantee a low task price. Suppose one model takes 20 failed steps while another completes the same task in four; the lower token price may still produce the higher total task cost. The numbers are illustrative, not benchmark results. For agent work, track the cost, elapsed time, and success rate of the whole task.
3.4 Further Reading
- OpenAI's function-calling guide shows the same model-call, tool, result, and model-call loop for software developers.
- Anthropic's guide to effective agents explains the difference between fixed workflows and systems in which the model chooses its next action.
Checkpoint. An agent is not a new kind of model. It is a repeated cycle in which the model proposes, ordinary software checks and acts, and the result returns to the model.
Part 4 — Why GPUs Suit Many Neural-Network Calculations
AI models repeat the same calculations across large arrays of numbers. A graphics processing unit, or GPU, is built to do many similar calculations at once; a central processing unit, or CPU, is built to handle a wider variety of tasks and step-by-step decisions quickly. Neither is universally better: AI systems usually use GPUs for parallel arithmetic and CPUs for operating systems, request handling, and serial control.
GPUs became a major AI-compute platform because their hardware and software fit many neural-network calculations. Other purpose-built processors use different designs for related work.
4.1 Two Ways to Be Fast
Imagine a supermarket deciding how to shorten its lines. It could hire one exceptional cashier who handles a difficult customer quickly, or it could open 100 ordinary checkout lanes for 100 customers with similar baskets. The first approach reduces the waiting time for one complicated job. The second increases the number of jobs finished each minute.
Computer designers call those goals latency and throughput. Latency is the elapsed time for a specified job. Throughput is completed work per unit of time. Both require a clear measurement boundary and unit.
High-performance CPU cores devote substantial hardware to following irregular program logic and finding other instructions to run when one instruction must wait. Those features can reduce delay for work with many decisions and dependencies. CPUs also contain several cores and can perform parallel work, so “CPU favors latency” describes a tendency, not a definition.
A GPU organizes many small units into groups that normally receive the same instruction for different data. Control hardware, nearby memory, and specialized matrix units support that work. Performance depends on supplying enough similar work and avoiding too many different decision paths or waits for data.
4.2 Graphics Supplied the First Big Parallel Job
GPUs did not begin as AI processors. They grew around a graphics problem: a screen contains many picture elements, or pixels, and many of those pixels need the same kind of calculation on different data.
Consider drawing a lit red ball. Software first describes the ball's shape, position, and surface. The graphics processor then has to determine which screen locations the ball covers and what color each visible location should be. For thousands or millions of locations, it can apply related calculations to different coordinates, colors, and lighting inputs. Not every graphics step is independent, but the workload supplies far more similar operations than one general-purpose CPU core could efficiently perform in sequence.
The useful progression is:
many screen locations
→ apply related calculations to different data
→ run many small workers in parallel
→ finish the image at an interactive rate
Neural networks presented a different job with a similar hardware opportunity. Instead of calculating colors for many screen locations, they repeatedly multiply and add numbers across large arrays. The data means something different, but the processor can again apply related instructions to many pieces of it at once.
Neural networks turned out to contain many suitable operations. In 2012, an image-recognition network called AlexNet was trained on two GPUs and won a major image-classification contest by a large margin. The original AlexNet paper reports the hardware and results. Image models and language models differ, but both contain large array operations that can exploit parallel hardware.
4.3 Why Models Create So Much Repeated Arithmetic
Part 2 described a model as learned parameters organized into layers. Many of those parameters sit in rectangular tables of numbers called matrices. A matrix multiplication combines rows from one table with columns from another to make a new table.
Start with one row and one column:
row from the input: [2, 5, 1]
column of learned weights: [3, -1, 4]
(3 × 2) + (-1 × 5) + (4 × 1) = 5
That multiply-and-add chain produces one number in the output table. The next input row can be combined with the same weight column to produce another output number. The same input row can also be combined with another weight column. A large layer repeats this pattern across many rows and columns, creating a large supply of related calculations.
weight columns
C1 C2 C3
input row R1 → O11 O12 O13
input row R2 → O21 O22 O23
input row R3 → O31 O32 O33
O23 is built from input row R2 and weight column C3.
Many output positions can be worked on at the same time because they use different row-and-column pairs. If a sum is long, workers can also calculate pieces of that sum and combine their partial results. This is the parallel structure that GPUs exploit.
Efficient software does not fetch every number separately for every output. It loads small blocks of input values and weights into faster on-chip storage, uses each block for several output numbers, then moves to the next block. Part 5 follows one of these blocks, called a tile, through the accelerator.
Attention and feed-forward blocks in a transformer use several matrix multiplications, but a model layer also normalizes values, applies other functions, coordinates attention, and moves data. “Transformers contain much matrix multiplication” is accurate; “a transformer is only matrix multiplication” is not. The original Transformer paper specifies the learned matrix projections used in attention and feed-forward blocks.
4.4 Moving Numbers Often Costs More Than Using Them
Arithmetic is only half the job. Before a calculator can multiply two numbers, the numbers must reach it. Bringing a value from memory can take more time and energy than performing the multiplication that uses it. The exact difference depends on the chip and on how far the value travels, so there is no honest universal ratio. The durable conclusion is simpler: reusing a nearby value is cheaper than fetching it again from farther away.
Think of a kitchen. Chopping an onion is quick if the onion is already on the counter. If the cook must walk to a warehouse for every slice, adding more knives will not make dinner arrive sooner. A GPU has many “knives.” Its central design problem is keeping ingredients close enough to use them.
Matrix multiplication shows how reuse helps. Suppose a small block of input numbers contributes to eight different output blocks. A poor program could fetch that input block from large memory eight times. A better program fetches it once into small nearby memory and reuses it eight times. The arithmetic is unchanged, but seven long trips disappear. This is why data layout and software can change speed even when the chip's peak arithmetic stays the same.
That is why the next parts focus so heavily on memory and packaging. Several kinds of small memory sit on the chip, while high-bandwidth memory beside it supplies much larger capacity. Software can keep temporary values in the smaller memories and reduce trips to the larger one.
Relative to a specified program and machine, work limited by memory traffic is memory-bandwidth-bound; work limited by arithmetic throughput is compute-bound. Large matrix multiplications can be compute-bound, while generating text for only a few requests often becomes memory-bandwidth-bound. These are common cases, not permanent labels for a model.
4.5 Why CPUs Did Not Simply Keep Getting Faster
For decades, making transistors smaller let designers fit more of these microscopic electronic switches on a chip while also raising clock speeds. Voltage could fall along with size, helping keep heat under control.
By about 2005, lowering voltage had become much harder and clock-speed growth slowed because of power and heat limits. IBM's history of Robert Dennard describes this transition. Moore's law is a separate observation about growth in the number of components that could be integrated economically; it is not a physical law. Intel summarizes Moore's observation and its later revision.
Power limits encouraged designers to spend additional transistors on multiple cores and units specialized for particular calculations. GPUs already existed; the power limit accelerated parallel and specialized computing rather than causing GPUs to be invented.
Further reading:
- 3Blue1Brown's neural-network lessons show visually how repeated multiply-and-add operations become a network.
- Mark Horowitz's energy-cost presentation compares the energy used by arithmetic and by data movement in one documented chip technology. Its figures are historical examples, not constants for current hardware.
What to remember: A GPU is not a universally faster CPU. Its throughput-oriented execution and memory system make it effective when a workload exposes enough similar parallel operations. Many neural-network operations do; other portions can remain limited by serial work, memory, or communication. The next challenge is feeding the parallel hardware with data.
Part 5 — What Happens Inside an AI Accelerator
An accelerator is useful only when its arithmetic units receive data and ready work. The practical limit may be arithmetic throughput, memory traffic, or coordination among many pieces of work. This Part explains those three possibilities before introducing any vendor-specific names.
Think of a factory. Arithmetic units are workers. Tiny on-chip memories are workbenches. The accelerator's large device memory is the stockroom. A GPU scheduler is control hardware that chooses which ready group of work runs next. This analogy describes the roles, not a literal map of every accelerator.
5.1 A GPU Repeats the Same Small Factory Many Times
GPU designers repeat processing blocks across the chip's piece of silicon, called a die. Repetition matters because one block can handle one portion of an array while other blocks handle other portions at the same time.
Follow one piece of a matrix multiplication:
- Software divides the large matrices into small blocks of work.
- Control hardware assigns a ready block of work to a processing block on the GPU.
- Workers in that processing block load needed numbers into registers and small shared working memory.
- Arithmetic paths and matrix engines multiply and add those numbers.
- The block writes completed results to the larger device memory or passes them to another calculation.
Each processing block contains arithmetic paths, scheduling hardware, registers for immediate values, and a small amount of on-chip working memory. Different blocks can be at different points in this sequence while sharing a larger memory system.
A cache keeps copies of data from a slower memory. It helps when a later access finds the needed value there, so software often arranges work to reuse nearby data. Exact block and memory names vary by vendor; the stable idea is repeated parallel processing supported by a hierarchy of local storage. NVIDIA's programming guide and AMD's hardware glossary document two concrete designs.
5.2 Many Workers Can Share One Instruction
One hardware path can perform an operation on one set of values. GPUs reduce control overhead by giving a common instruction to a group of small workers, which apply it to different data. Group names and sizes vary by vendor.
Imagine a group of bakers receiving the instruction “add two cups of flour,” each for a different bowl. One instruction coordinates many useful actions. This is efficient when every bowl needs the same step. It becomes inefficient when half the bakers need to whisk while the other half need to wait; the group may have to perform both paths in turn.
The rule underneath matters more than the labels: GPUs are most efficient when workers in a group need the same instructions. If they take different decision paths, the hardware must handle those paths separately while some workers sit idle. Modern NVIDIA hardware does not require every worker to advance in literal “lockstep,” so that familiar shorthand is misleading. NVIDIA GPU programming guide.
5.3 The Chip Switches Jobs While Data Is in Transit
Device memory can take much longer to respond than an arithmetic operation. A GPU cannot remove that delay, so it keeps multiple thread groups ready. When one group waits for data, the scheduler can issue work from another ready group.
Think of a restaurant server with several tables. After placing one table's order with the kitchen, the server does not stand still until the food appears. They take an order from another table, deliver a drink to a third, and return when the first meal is ready. Enough independent tables keep the server productive despite each kitchen delay.
Engineers call this latency hiding: working on something else during a delay. A simplified timeline looks like this:
group A: request data ───────── wait ───────── calculate
group B: calculate → request data ───── wait
group C: calculate → request data
processor: useful A/B/C work fills some of the waiting periods
The memory response did not become faster. The processor merely found independent work that was ready. Latency hiding succeeds only when the program supplies enough such work and each group leaves enough registers and on-chip memory for other groups to remain ready. If one group consumes nearly all nearby storage, too few alternatives may fit. If every group waits on the same missing result, there is nothing to switch to. NVIDIA's GPU programming guide documents hardware scheduling and the resource limits on active thread groups.
5.4 Special Matrix Engines Do the Main AI Calculation
Modern AI accelerators contain specialized circuits for the repeated matrix work from Section 4.3. NVIDIA calls its versions Tensor Cores; other vendors use other names. A tensor core is not a separate processor that runs a whole model. It is an arithmetic unit inside the GPU that performs a supported matrix operation on small blocks of numbers.
The accelerator handles a large matrix in pieces:
- Divide the input and weight matrices into small rectangular blocks called tiles.
- Load matching tiles from large device memory into faster on-chip storage.
- Give those tiles to a matrix engine, which performs many multiply-and-add steps.
- Add the partial result to the output tile. A large output may need contributions from several pairs of input tiles.
- Reuse any tile that contributes to another output, then store the completed output tile.
large input matrix large weight matrix
┌───┬───┬───┐ ┌───┬───┐
│ A │ B │ C │ │ D │ E │
└───┴───┴───┘ └───┴───┘
↓ choose matching tiles
matrix engine performs a small matrix operation
↓ repeat and add partial results
one output tile
The exact tile shapes, accepted number formats, and instructions depend on the product. The stable idea is that dedicated wiring can perform a common block of matrix arithmetic more efficiently than issuing every multiply and addition as an unrelated instruction.
Specification sheets usually report the highest theoretical matrix rate for a stated number format—the rule used to encode numbers in bits—and may assume a supported pattern of zeros. Code must use a supported operation and keep the unit supplied with data to approach that rate. The number therefore does not describe arbitrary code or guarantee application speed. NVIDIA's H100 table illustrates how the stated rate changes with format and sparsity; its programming guide describes tensor-core matrix operations.
5.5 Local Memory Has Several Levels
Within one accelerator, storage levels trade capacity and chip area against access time and bandwidth:
closest to arithmetic
registers immediate values for individual threads
on-chip working memory small reusable tiles for one processing block
caches copies of data from larger memory
device memory model weights and other large arrays
farther from arithmetic
Registers are tiny storage locations used directly by arithmetic instructions. The next levels use a fast on-chip memory technology explained in Part 6. Large device memory uses other technologies introduced there. Exact sizes, names, and access costs depend on the accelerator.
Memory on another accelerator, the main computer's memory, and storage are separate pools reached through additional links. They are not extra rungs with guaranteed increasing capacity, and their order can change by product and by whether the reader compares delay or transfer rate. Fast software loads useful data into nearby storage, reuses it, and avoids unnecessary transfers.
5.6 Every Operation Faces One of Two Ceilings
To predict whether a piece of code will run well, ask two questions:
- How much arithmetic does it perform?
- How much data must it move to perform that arithmetic?
If an operation uses each fetched value many times, the arithmetic units may become the limit. Adding faster memory will not help much because the calculators are already full. The operation is compute-bound.
If an operation does little work with each fetched value, data arrives too slowly to fill the calculators. Adding more arithmetic units will not help much because the existing ones are waiting. The operation is memory-bound.
Engineers measure work per byte transferred across a named memory boundary as arithmetic intensity. The simplified roofline model gives this upper bound:
attainable arithmetic rate is no greater than the smaller of:
1. peak arithmetic throughput
2. memory bandwidth × arithmetic operations per byte
Large matrix operations in training and prompt processing often reuse values enough to become compute-bound. When only a few requests share a model that uses all of its parameter groups, token generation can do relatively little work per byte of weights read and become memory-bandwidth-bound. Model design, context length, number of simultaneous requests, software, and hardware can change either result.
Use a small illustrative machine to see the rule. Suppose it can perform at most 1,000 operations per second and its memory supplies 100 bytes per second.
- An operation that does two arithmetic steps per byte can be supplied with at most
100 × 2 = 200operations per second. Memory sets the lower ceiling, so the operation is memory-bound. - An operation that does 20 arithmetic steps per byte has a memory ceiling of
100 × 20 = 2,000operations per second. The machine can perform only 1,000, so arithmetic sets the lower ceiling and the operation is compute-bound.
The same operation can move from one side to the other when software increases reuse, the number format reduces bytes, or the machine changes. The label belongs to a particular workload and implementation on a particular system.
Real programs can fall below both ceilings because of coordination, scheduling, or other overhead. The roofline tells you which resource sets the best possible limit; it does not promise that software reaches it.
5.7 Better Software Can Remove Entire Memory Trips
A GPU kernel is a small program the GPU runs across many parallel workers. A simple implementation may run one kernel, save an intermediate result to large device memory, then run another kernel that immediately reads it back.
Kernel fusion combines compatible steps so the intermediate result can be eliminated or remain in registers, on-chip working memory, or cache:
separate steps:
device memory → step 1 → device memory → step 2 → device memory
fused steps:
device memory → step 1 + step 2 → device memory
Fusion can reduce memory traffic and the work needed to start separate programs, but using too many on-chip resources can make it slower. FlashAttention is a real example: it reorganizes attention so less data moves between large memory and small on-chip memory while producing effectively the same result. Reordering calculations can still create tiny numerical differences.
This is why software support is part of practical hardware performance. Libraries and compilers choose matrix operations, data layouts, memory reuse, fusion, and scheduling. Peak specifications give a ceiling under stated conditions; an application reaches only what its full implementation sustains.
Further reading:
- The original Roofline report defines the two-ceiling performance model.
- The FlashAttention paper is a primary-source case study in reducing memory traffic.
What to remember: A GPU groups arithmetic into repeated processing blocks, keeps several thread groups ready to cover delays, and uses matrix engines for supported operations. Whether compute, local memory traffic, or coordination is the tightest limit depends on the workload and implementation.
Part 6 — Memory: What Fits and How Fast It Moves
Memory has two separate jobs. It must be large enough to hold the model and its temporary working data, and it must deliver those numbers quickly enough to keep the processor useful. The first property is memory capacity. The second is bandwidth.
Think of a water system. Capacity is the size of the tank; bandwidth is the width of the pipe. A large tank with a narrow pipe stores plenty but delivers slowly. A wide pipe attached to a small tank delivers quickly until the tank runs out. AI systems often need both a large tank and a wide pipe.
6.1 Fast Memory Is Small; Large Memory Is Farther Away
No single kind of memory gives a processor unlimited capacity, immediate access, low power, and low cost. Designers therefore build a hierarchy: a little fast storage close to the arithmetic and much more storage farther away.
Registers, caches, and small working memories on a processor commonly use static random-access memory, or SRAM. An SRAM cell keeps its bit while power remains available without the periodic refresh used by DRAM. That makes it useful for fast on-chip access, but its circuit uses more transistors and chip area for each stored bit. Filling a large accelerator package with enough SRAM for a large model would therefore require far more silicon area and cost than present designs can justify.
Large device memories use dynamic random-access memory, or DRAM. A DRAM cell stores charge that must be sensed and refreshed. Its denser design fits more bits in a given area, but access involves more delay than the small on-chip memories. In high-end accelerators, DRAM is manufactured on separate memory dies and connected to the processor through the package. Samsung's memory overview explains DRAM storage and refresh and compares its density with SRAM.
The useful analogy is a desk and a nearby library. The desk is fast to reach but holds only today's working papers. The library holds far more, but every visit takes time. Good programs bring a useful set of papers to the desk, work with them repeatedly, and return them only when necessary. Sections 5.4 and 5.7 described exactly this strategy with matrix tiles and fused calculations.
High-bandwidth memory, introduced next, is DRAM arranged and connected through an unusually wide interface. It is not a different storage principle.
6.2 High-Bandwidth Memory Uses Many Short Wires in Parallel
High Bandwidth Memory, or HBM, raises transfer rate mainly by moving data over a very wide set of short connections. Instead of asking a comparatively small number of wires to run at ever-higher rates, the package carries many pieces of data in parallel.
One HBM stack is assembled in stages:
- Several DRAM dies store the actual bits.
- The dies are stacked, and vertical electrical paths connect the layers.
- A base or interface layer connects the stack to the package.
- Dense package wiring links the stack to the accelerator's compute die.
- A memory controller on the processor coordinates reads and writes across the wide interface.
The memory is close to the processor in the package, but it is not part of the processor die. Data still has to travel through memory cells, the stack, package wiring, and the processor's memory system before arithmetic units can use it.
Graphics Double Data Rate, or GDDR, memory devices are commonly mounted around a GPU package on a circuit board. GDDR uses narrower connections that transfer data at a higher rate on each wire than HBM. Neither name alone establishes which product is faster, larger, or cheaper; exact capacity and bandwidth depend on the memory generation, number of devices, interface width, and complete system.
That package wiring is not always one full silicon layer. Current designs can use a large silicon wiring layer, fine patterned wiring made from other materials, or small local silicon bridges. HBM capacity therefore depends on memory-die fabrication, stacking, package routing, assembly, and test—not simply attaching a conventional memory module. SK hynix explains the stacked memory and vertical connections, while TSMC documents several package-wiring approaches.
6.3 Capacity Answers “Will the Model Fit?”
Suppose a model contains 70 billion parameters and stores each one in two bytes. The weight payload is 70 billion × 2 = 140 billion bytes, or 140 GB in decimal units. Those weights alone exceed the 80 GB in the cited H100 configuration and fit within the 192 GB in an AMD MI300X. NVIDIA H100 specifications and the AMD MI300X data sheet provide those capacities. Complete inference also needs temporary workspaces and usually a KV cache, so 192 GB does not guarantee that every group of requests and context length fits.
The order of the check matters:
available device memory
− model weights
− KV cache for active sequences
− temporary workspaces and software allocations
= remaining safety margin
If that result is negative, the service cannot simply “run a little slower” on the same allocation. It must change what is stored or where it is stored. Section 1.3 calculated about 41.9 GB of KV cache for one 128,000-token sequence on a specific 70-billion-parameter model. This is why a model whose weights fit can still run out of memory as context length or batch size grows.
If all required data does not fit, a system can split selected state across accelerators, move some state to slower memory, reduce numerical precision, shorten the context, or choose a smaller model. These choices can add communication, delay, software work, or quality loss. Capacity can therefore change the architecture and cost of the system.
Training needs more than weights. It commonly stores gradients, optimizer state such as Adam's running averages, and intermediate arrays called activations. Systems can divide some state among accelerators. Activation checkpointing, also called recomputation, saves only selected intermediate results and recreates others during the backward pass, trading extra calculation for lower memory use. PyTorch documents that trade.
The plain-language rule is: when memory is scarce, engineers either divide the data, store it more compactly, or repeat some calculations instead of storing their results.
6.4 Bandwidth Answers “How Fast Can the Model Be Read?”
When only a few requests share a model that uses most of its parameters for every output token, each generation step may stream most of the weights while doing relatively little work per byte. Memory bandwidth can then be the limit. Serving more requests together can share a weight read, and other model designs can change how much data moves.
If a decode step must read a 140 GB weight file once, an ideal 3,000 GB/s memory interface could supply at most about 21.4 complete weight-file reads per second: 3,000 ÷ 140. This uses decimal units and assumes peak bandwidth, one full read per step, no competing traffic, and no communication or arithmetic overhead. One low-batch stream will normally be slower. Aggregate tokens per second can be higher when a batch shares each read, so this is not a universal token-rate ceiling.
This example gives the bandwidth number a question and a baseline. “Three terabytes per second” sounds enormous in isolation. Compared with a 140-gigabyte model that must be revisited token after token, it becomes a limit the reader can feel.
The widening gap between processor arithmetic and memory-system speed is called the memory wall. Caches and data reuse reduce external-memory traffic, lower precision reduces bytes moved, and wider interfaces such as HBM raise bandwidth. Which response helps depends on the workload. Wulf and McKee introduced the term.
6.5 Memory Supply Can Limit Accelerator Supply
An accelerator vendor cannot ship an HBM product merely because a processor die exists. The required memory must pass through its own chain:
make DRAM dies
→ test them
→ assemble and connect the stack
→ test the stack
→ qualify it for the customer's platform
→ assemble and test the complete accelerator package
An HBM stack contains multiple memory dies, many vertical connections, and an interface layer that joins it to the package. A failure in a die, connection, stack assembly, or later package can reduce the number of usable finished parts. Testing at several stages can reduce the chance of assembling a component already known to have failed, but it cannot remove every integration risk.
HBM production and customer qualification—testing that a component meets a customer's platform requirements—are specialized stages. Too little production, too many rejected stacks, or slow qualification can therefore reduce usable accelerator supply even when processor dies are available. Supply claims should always name the HBM generation and date.
For a reader following the industry, the practical questions are straightforward: How much memory does each accelerator carry? How quickly can it move that memory? How many tested stacks can suppliers produce? Those three questions connect the model, the hardware, and the supply chain without requiring a catalog of HBM version names.
Further reading:
- SK hynix's HBM explanation covers stacked DRAM, vertical connections, and the wide interface.
- The AMD MI300X data sheet is a concrete example of how capacity and bandwidth appear in a product specification.
What to remember: Capacity determines whether weights and working data fit. Bandwidth limits how quickly memory can supply data. HBM raises bandwidth with stacked DRAM and a wide package interface, adding demanding memory, assembly, test, and packaging stages.
Part 7 — Why AI Uses Smaller Numbers
Many training and inference systems store or compute some arrays with fewer bits while retaining more detail where it is needed. Fewer bits can reduce storage and memory traffic and can increase throughput on matching hardware. Smaller formats can erase fine differences between nearby values or fail to hold numbers outside their range. Either problem can change model quality, so the usable format depends on the model, task, and operation.
The practical question is not “What is the smallest format?” It is “Which format meets the measured quality target for this workload?” A performance claim is incomplete without the format, quality result, and measurement conditions.
7.1 A Number Format Divides Bits Between Range and Detail
Computers store numbers as patterns of bits, the zero-or-one values introduced in Part 0. More bits allow more possible patterns. Designers must decide what those patterns mean.
For values that can be extremely large or small, computers commonly use a scheme similar to scientific notation. The decimal expression 3.14 × 10² separates significant digits from scale. A binary floating-point format likewise records a sign, a scale, and meaningful digits.
The two jobs are range and detail. Range determines how large or small a magnitude the format can represent. Detail determines how closely it can distinguish nearby values at a given scale.
Imagine a tiny decimal format that keeps only three significant digits. It can store 3.14, but the value 3.14159 must be rounded. Near 3, it cannot distinguish every possible value between 3.14 and 3.15. A format with more significant digits can preserve more of that difference. Separately, a larger exponent field lets the format move the decimal—or, in a computer, binary—point across a wider range of magnitudes.
Giving more bits to the exponent therefore allows a wider range of sizes. Giving more bits to the significant digits preserves finer differences between nearby values. Reducing the total number of bits makes storage and supported arithmetic cheaper, but it increases rounding and the chance that a value is too large or too small to represent.
The IEEE 754 standard, published by the Institute of Electrical and Electronics Engineers, defines a 32-bit format called binary32. It is commonly called FP32 and uses four bytes. BF16 uses 16 bits, keeps a similarly wide range, and records less fine detail. FP8 names a family of one-byte formats with different balances between range and detail. Four-bit formats use half a byte per value only when packed tightly; scaling information and other metadata add storage. IEEE 754, Google's BF16 guide, and the FP8 formats paper define these distinctions.
7.2 Smaller Numbers Save Space, Movement, and Arithmetic
For 70 billion parameters, the packed weight payload alone requires roughly the following in decimal gigabytes. Metadata, padding, KV cache, workspaces, and other runtime state are excluded:
two bytes per weight: 140 gigabytes
one byte per weight: 70 gigabytes
half a byte per weight: 35 gigabytes
Reducing the payload from two bytes to one changes 140 GB to 70 GB, so the weights alone fit within an 80 GB device. The complete service may not fit once metadata, KV cache, workspaces, and other allocations are included. If weight traffic is the only limit, halving those bytes gives an ideal bound near twice the weight-delivery rate; real speedup may be smaller. Supported low-precision matrix units can also have higher stated throughput.
Quantization maps values from a larger set into a smaller set of representable values. Picture taking many precise temperature readings and recording each one at the nearest whole degree. The record becomes simpler, but several slightly different readings now share one stored value. AI quantization uses more sophisticated mappings and commonly stores scaling information for groups of values, but the central tradeoff is the same: fewer stored choices mean less data and more approximation.
Quantization may apply to weights, temporary values, or the KV cache. It can reduce storage and traffic and accelerate supported calculations, but quality and speed depend on the method, model, task, and hardware. PyTorch's quantization overview explains the mapping and scale used to represent a larger numeric range with fewer bits.
Training repeatedly updates weights and must represent gradients and accumulated optimizer state, so mixed-precision recipes often retain more precision for selected values. Inference often permits aggressive weight-only quantization, but acceptable precision remains model- and task-specific. “Supports 4-bit AI” does not establish that an entire training or inference workload used four bits at acceptable quality.
7.3 How to Read an Accelerator Specification Honestly
Start with the workload, then compare matching measurements. Five questions remove most of the confusion:
- What kind of numbers entered the calculation, and what kind held the running total? Compare rates only when both formats and the operation are the same.
- Does the peak assume that zeros appear in the exact pattern the hardware can skip? This is called structured sparsity. Compare results under the same condition; arbitrary zeros do not automatically earn the advertised speedup.
- Is the total for one processor package, one server, or a whole rack? A larger box naturally produces a larger headline.
- Does the workload need arithmetic, memory capacity, or memory bandwidth most? Peak operations per second cannot answer the other two questions.
- What happened on a real model at an acceptable quality and response time? A benchmark is more informative than a theoretical ceiling when its setup matches the intended use.
Report power or energy beside throughput and latency because a power-constrained facility may care about work per watt or per rack. Report software and benchmark versions because application performance depends on the available kernels, compilers, and libraries.
A useful comparison therefore holds operation, format, sparsity assumption, workload quality, and measurement conditions constant. It reports arithmetic throughput, memory capacity and bandwidth, link bandwidth and latency, power, application throughput, and response latency.
Further reading:
- The Institute of Electrical and Electronics Engineers' floating-point overview is the formal source for conventional formats.
- NVIDIA's low-precision primer explains how multiplication with smaller formats is combined with scaling and a more detailed running total.
What to remember: Fewer bits can reduce packed size and memory traffic and can raise throughput on supported hardware. They can also reduce measured model quality. Compare the same operation, format, sparsity assumption, workload, and quality target—not the largest number on a slide.
Part 8 — How an Accelerator Is Manufactured
This Part focuses on high-end data-center accelerators that use HBM. A die is one piece of processed silicon containing a circuit. A package protects and connects one or more dies to the rest of the system; some accelerator packages also contain nearby memory stacks. Advanced packaging is the separate assembly work that joins these pieces with dense wiring.
Finished supply therefore depends on processor dies, qualified memory, package assembly, and test. A shortage, high rejection rate, or qualification problem at any required stage can limit finished accelerators even when the other parts are available. Intel defines die and package, while TSMC documents current HBM package structures.
8.1 Start With What the Workload Requires
An accelerator package is easier to understand when you start with the problems it must solve. The package is not an arbitrary collection of silicon pieces; each major part answers a workload need.
The compute die contains the matrix engines, general arithmetic paths, control hardware, and on-chip memory introduced in Part 5. HBM stacks supply much larger memory capacity and bandwidth. Dense wiring below and between those components carries many closely spaced signals that would be difficult to route across an ordinary server board. Other circuits bring in power and connect the package to the rest of the server.
Seen from above, the result resembles a small city center. The main processor sits in the middle. Memory stacks sit around it. A dense wiring layer underneath carries traffic between them, and a larger base connects the whole assembly to a server.
This city picture describes one common HBM accelerator arrangement, not every accelerator. A package need not contain memory, and different products place and connect their components differently.
8.2 From a Silicon Wafer to Individual Dies
Chip fabrication turns a blank silicon wafer into many copies of an intricate circuit. It does not carve a complete transistor with one tiny tool. Instead, the factory builds and connects structures layer by layer through a repeated sequence of coating, patterning, adding, and removing material.
A simplified journey is:
- Prepare the wafer. Fabrication starts with a circular wafer of highly purified silicon. For scale, Intel says the modern wafers it uses are 12 inches, or 300 millimeters, across. Intel's manufacturing glossary supplies that example.
- Add a light-sensitive coating. This coating changes where light reaches it.
- Project one circuit pattern. Lithography uses light and a patterned mask to expose selected areas. Developing the coating leaves a temporary pattern on the wafer.
- Change the exposed material. Other equipment can remove material, add a thin material, or change selected electrical properties. The exact operation depends on the layer being built.
- Repeat for many layers. Transistors form first, and later conducting layers connect them into circuits. Each layer must line up with the earlier ones.
- Test and separate the dies. Probe tests identify dies that meet specified electrical limits. The wafer is cut, or diced, into individual dies. Acceptable dies continue to packaging and further tests.
Leading-edge processes use extreme-ultraviolet light for selected critical layers; other layers can use different lithography systems. ASML explains the coat, expose, develop, and repeated layer process, including deposition, etching, and other steps omitted from this introduction.
The steps above omit hundreds of specialized operations and inspections. The mental model is what matters: one wafer carries many repeated circuits, each circuit is built through many aligned layers, and a finished wafer still has to be tested and divided. Some designs can disable a faulty repeated block and still meet a supported lower configuration; dies that meet no supported specification are discarded.
Many accelerator companies design chips but hire a specialist manufacturer, called a foundry, to make them. This is why one foundry's plans can affect several competing chip companies.
8.3 “3 Nanometer” Is a Generation Name, Not a Measurement
Foundries give manufacturing generations names such as 7 nanometer, 5 nanometer, or 3 nanometer. This named generation is a process node. Modern node names do not measure one physical feature, and labels from different foundries cannot be compared directly.
That means a chip sold as “3 nanometer” does not contain one important part that is simply three nanometers wide. Decades ago, node labels tracked particular device dimensions more closely. Modern transistors and their wiring have several relevant dimensions, and manufacturers use node names to distinguish a generation of design and manufacturing technology.
For a specific design, compare how many circuits fit, how fast and energy-efficient it is, what share of dies pass testing, and what it costs. A newer process may improve some combination of those properties, but the result depends on the circuit and design choices; development, masks, wafers, and packaging can also cost more.
When two products use different nodes, replace “which nanometer number is smaller?” with four questions:
- How does the complete chip perform on the same workload?
- How much power does it draw to produce that result?
- How large and costly is the finished die and package?
- How many usable parts can the manufacturer produce?
Smaller labels therefore do not guarantee a cheaper finished accelerator. A designer may use the available transistors for more arithmetic, control, memory, or links, subject to area, power, and cost limits. On-chip memory has also become harder to shrink. Treat the node as one manufacturing input, not a product performance score. Intel explains modern node naming.
8.4 Bigger Dies Are Powerful but Harder to Produce
Two limits make a large die expensive.
First, a lithography scanner can print only a limited rectangle at a time. This reticle limit constrains the size of a conventional die made in one exposure area. ASML system specifications.
Second, microscopic defects occur during manufacturing. Picture cutting brownies from a tray that contains a few burnt spots. If the brownies are small, each burnt spot ruins one small piece and many good pieces remain. If the brownies are huge, the same spot can ruin a much more valuable piece. Large dies have the same problem.
The circle creates a separate loss. Rectangular dies near a wafer's edge do not fit completely, so a larger rectangle also tends to produce fewer complete candidates from one wafer. Defects then reduce that candidate count further. The exact result depends on die dimensions, wafer layout, defect distribution, and which faults the design can tolerate; the analogy explains the direction, not a yield formula.
Die yield is the share of manufactured dies that meet required tests. Designers may include redundant or disable-able blocks so a die with a localized defect can still meet a supported configuration. Manufacturers then test and sort acceptable parts into product or performance grades; that sorting is binning. A lower bin may be fully functional but unable to meet a higher speed or power grade.
These effects are why a wafer count does not translate directly into the same number of premium parts and why related products may expose different numbers of working units. Yield, defect tolerance, and binning are related but not interchangeable ideas. Intel's manufacturing glossary.
8.5 Chiplets Trade One Large Problem for Several Smaller Ones
When one large die becomes too hard to print or too wasteful, designers can split it into several smaller dies and connect them inside one package. These pieces are called chiplets.
The brownie analogy shows one potential benefit: a localized defect affects a smaller piece. Dies can be tested before assembly to reduce the chance of packaging a bad part. A package can also combine dies made with different processes—for example, newer arithmetic dies with mature circuits that move data into and out of the package. Total economics still depend on how many package assemblies pass testing.
Imagine a design that needs four repeated arithmetic regions. A one-die version places all four on one large rectangle. A chiplet version manufactures four smaller arithmetic dies and joins acceptable ones in a package. A defect that ruins one small die does not automatically discard the other three before assembly, and the factory can test the pieces first. But the finished package now needs working links among all four pieces, enough wiring to carry their data, and a way to deliver power and remove heat across the assembly.
Chiplets add die-to-die communication, package routing, power, thermal, assembly, and test requirements. Crossing a die boundary can cost more delay and energy than communication within one die, although the exact cost depends on the interface and package. Hardware, firmware, compilers, or application software must manage whichever parts of this layout they can see. Chiplets move part of the problem from one large die into integration. UCIe documents one standardized die-to-die interface.
8.6 Advanced Packaging Joins the Processor to Memory
Fabrication produces separate compute and memory dies. Advanced packaging turns those pieces into one usable accelerator by placing, connecting, protecting, powering, and testing them as an assembly.
Follow one read from memory. A memory cell in an HBM die supplies bits through the stack's vertical connections. Signals pass through the stack's interface and across dense package wiring to the compute die's memory controller. The controller delivers the requested data into the processor's memory hierarchy, where arithmetic units can finally use it. Every physical connection on that path must work at the required rate.
An ordinary server board cannot directly route all of the closely spaced signals needed by a wide HBM interface. Vendors therefore use denser package wiring, such as a silicon interposer, fine patterned wiring made from other materials, or small embedded silicon bridges. An interposer is an intermediate routing structure between components; it carries signals but does not perform the model's arithmetic.
The assembly sequence varies by product, but the dependency is universal: good compute dies and good memory stacks are not finished accelerators until the package joins them and passes test. Packaging therefore has its own equipment, capacity, yield, and qualification limits. TSMC documents several current wiring approaches and their processor-to-HBM arrangements.
A finished HBM accelerator therefore depends on processor-die fabrication and test, memory-die fabrication, HBM stacking and test, package assembly and test, and later board or server integration. A shortage, high rejection rate, or qualification delay at any required stage can limit usable systems.
Further reading:
- ASML's lithography overview explains how circuit patterns reach a wafer.
- TSMC's CoWoS overview distinguishes silicon-, redistribution-layer-, and bridge-based package structures.
What to remember: A usable HBM accelerator depends on processor dies, memory, packaging, test, and system integration. Larger dies can reduce die yield. Chiplets can reduce one large-die risk but add integration work. The tightest required production stage can constrain finished supply.
Part 9 — Why Multiple Accelerators Need Fast Connections
Large models and training jobs are often divided across accelerators. Communication then becomes part of the calculation, and a processor can sit idle while it waits for another one.
Designers try to keep frequent, delay-sensitive communication inside a tightly coupled accelerator group and use the broader cluster network across servers or groups. The first arrangement is called scale-up; the second is scale-out. They describe system organization and communication scope, not simply two physical distances.
9.1 Scale-Up Makes Nearby Accelerators Behave Like One Larger Machine
Imagine four people assembling one puzzle. If they sit at the same table, they can pass pieces and point to matches with little delay. If each person sits in a different building, the same coordination becomes much slower even if every person works equally fast.
Scale-up networks connect a tightly coupled accelerator group with links designed for high bandwidth and low delay. NVIDIA calls its version NVLink and uses switch chips to extend it across supported systems. Other vendors use other designs. A scale-up group may be inside one server or extend across a rack; software must still divide and coordinate the work.
Scale-out networks connect servers or scale-up groups through switches. Ethernet is a standardized family of data-center network technologies. InfiniBand is a switched network design built for high data-transfer rates and short delays. A scale-out path often provides less bandwidth to each accelerator and more delay than the local scale-up network, but exact rates, sharing, and connection layout depend on the deployed system. NVIDIA documents NVLink, and the InfiniBand Trade Association documents InfiniBand.
scale-up: a tightly coupled accelerator group
[GPU] ===== [GPU]
|| ||
[GPU] ===== [GPU]
scale-out: many servers and racks, network switches between them
[server] --- [switch] --- [switch] --- [server]
Neither layer replaces the other. A large cluster—a group of connected computers managed for shared work—can use tightly coupled accelerator groups and then connect those groups over a broader network.
9.2 Compare Each Communication Path Directly
For a specified system, compare local HBM, the scale-up fabric, and the scale-out network separately. Local HBM commonly supplies the most bandwidth per accelerator, but there is no universal ratio or ordering for every metric. Product, direction, topology, link count, and whether a figure is per port or an aggregate all matter.
The safe lesson is conditional: when the same value is already in local memory, using it usually avoids a link transfer. When data is remote, ask how many bytes cross which link, how often, and whether the quoted rate is theoretical or measured.
That is why “the model fits across eight GPUs” is not a performance result. If the devices exchange large arrays after nearly every layer, communication can dominate running time. If they work independently for longer periods and exchange small summaries, a slower path may be adequate.
Consider a simplified model layer divided across two accelerators. Accelerator A holds the left half of a weight matrix and accelerator B holds the right half. Both receive the layer's input and calculate different parts of the output at the same time. If the next calculation needs the complete combined result, the two devices must exchange or combine their partial arrays before either can continue.
same layer input
├─→ accelerator A uses weight shard A ─→ partial result A ─┐
└─→ accelerator B uses weight shard B ─→ partial result B ─┤
↓
exchange/combine
↓
next calculation
The arithmetic may take less time because two processors share it, yet the layer now includes a communication step. If that exchange takes longer than the saved calculation time, adding the second accelerator makes the layer slower. Real tensor-parallel layouts split particular matrices and use different collective operations, but the dependency is the same: the next step sometimes cannot begin until remote partial results arrive. The Megatron-LM paper gives a primary-source example of splitting transformer-layer matrix operations across GPUs.
Placement should follow measured communication: keep high-volume, delay-sensitive exchanges on paths that meet their needs, and spread more independent work across the broader network. The connection layout is part of the workload design, not a decorative server specification.
9.3 Group Communication Has Repeated Patterns
In data-parallel training, each worker processes a different group of examples and computes gradients: arrays describing how the loss changes with the model parameters. Before the next update, the workers combine those arrays so their model copies remain in agreement.
An all-reduce combines every participant's array with a chosen operation and returns the combined array to every participant. Suppose four workers calculate one illustrative gradient value:
worker A: 2 worker B: 4 worker C: 6 worker D: 8
all-reduce with addition
↓
every worker receives: 2 + 4 + 6 + 8 = 20
framework forms the average if needed: 20 ÷ 4 = 5
Real gradients are large arrays, not one number. A communication library divides those arrays into blocks, moves blocks along available paths, and combines them. The operation is collective because every worker participates and every worker needs the result. Addition is common for gradients, but the standard supports other reduction operations. The message-passing standard defines all-reduce.
An all-to-all moves different data rather than one combined result. Each participant divides its input into one block for every destination, sends those distinct blocks, and receives one from every other participant. Afterward, workers do not all hold the same array. This pattern creates widespread traffic. If a model routes many more tokens to one specialized section than another, one destination may receive more data and work even though the communication pattern permits every pair to exchange.
before: A holds [to A, to B] B holds [to A, to B]
all-to-all
after: A holds [A→A, B→A] B holds [A→B, B→B]
Communication libraries improve how messages are split, sent, and overlapped with calculation, but the hardware and connection layout still set limits. Results also depend on how much each worker calculates, how much data it sends, how long it waits for others, how evenly work is divided, and how much software overhead remains. Adding accelerators therefore need not produce proportional speedup. NVIDIA's collective-operations guide.
9.4 Topology Is the Map of Who Connects to Whom
Topology is the connection map: which processors and switches are joined by which links. Small local systems can provide direct or switched paths among every accelerator. Large clusters use layers of switches rather than a separate physical cable for every pair.
The beginner does not need to memorize tree layouts or switch names. The useful questions are: Can the accelerators that exchange the most data reach one another without a crowded path? What happens when several groups communicate at once? Can the system keep working when a link or switch fails?
For buyers, the connection map changes the product. One accelerator rented alone, eight with fast local links, and many servers with a well-built cluster network may use the same chip but support different workloads. A price comparison that omits the links is incomplete.
Further reading:
- NVIDIA's NVLink overview shows one commercial scale-up design.
- The NVIDIA collective-operations guide illustrates common group exchanges after the plain-language introduction here.
What to remember: Multiple accelerators must communicate. Scale-up fabrics connect a tightly coupled group; scale-out networks connect servers or groups. Keep high-volume, delay-sensitive communication on paths with the required bandwidth and latency, and verify the topology and application result rather than assuming accelerator count predicts speed.
Part 10 — Choosing Hardware by Workload
There is no universally best AI chip. The best choice is the system that completes a particular job at the required speed, reliability, and cost.
This is why a list of product names ages badly and teaches little. A chip that is excellent for training a large model may be wasteful for serving a small one. A design that wins a benchmark may be unusable if the model does not fit in memory or the software team cannot run its code. Start with the work, not the logo.
10.1 Ask What Job the Chip Must Do
Hardware should be chosen for a measured job, not for “AI” in general. Training, interactive chat, and high-volume offline generation can reward different system qualities.
Training creates or updates a model by applying matrix operations and adjusting learned parameters. Large runs often divide that work across many chips and exchange data among them, so they depend on arithmetic performance, chip-to-chip communication, enough memory for training state, and software that remains reliable over a long run. A small training job may use only one chip.
Inference uses a finished model to answer requests. The first phase reads the prompt; the second produces the answer one token at a time. Prompt processing can make heavy use of the arithmetic units. Token generation often spends more time moving the model's learned numbers from memory than doing arithmetic. The best inference system depends on the mix of short and long prompts, the number of users served together, and the response speed promised to each user.
Three example buyers can therefore reach different answers:
| Buyer | Job that matters | First questions to ask |
|---|---|---|
| Research lab | finish one large training run reliably | Do training state and activations fit across the system? Can the chips exchange gradients and partial results quickly? |
| Chat service | answer unpredictable requests without long pauses | What are time to first token and inter-token latency at the expected request load? How many KV caches fit? |
| Overnight document processor | finish a known batch by morning | What total throughput and cost does the system achieve when latency can be relaxed? |
A benchmark for one row cannot automatically answer the others. “Fastest” is incomplete until the job, scale, and response-time target are named.
10.2 Ask What Must Fit in Memory
Accelerators need nearby memory for model parameters and temporary working data. High-end training and large-model inference accelerators commonly use high-bandwidth memory, or HBM; other designs use graphics memory or share system memory.
Capacity answers a yes-or-no question: do the model and its working data fit at the same time? Count the weights, KV cache or training state, temporary workspaces, and a safety margin. If the total exceeds available memory, an operator can split work across devices, move some data to slower memory, use fewer bits, compress the conversation cache, shorten the retained context, or choose a smaller model. Depending on the remedy, the cost may appear as more communication, delay, software work, or reduced model quality.
Bandwidth answers a speed question: how quickly can memory supply data? When a model uses most of its parameters to generate one token at a time, it commonly reads much of that data on each step. More bandwidth can then improve response speed even when peak arithmetic does not change. Batching, compression, caching, and models that activate only selected parameters change the balance.
10.3 Ask How Far Data Must Travel
One chip rarely tells the whole story. Large models and large training runs use groups of chips, so intermediate results must move between them. The connection among nearby chips is the accelerator interconnect. If it cannot keep up, expensive processors wait for data instead of working.
Use three distances as a checklist:
- Already on the chip: Registers, on-chip memory, caches, and local device memory are the closest sources. Software should reuse values here when possible.
- On another nearby accelerator: A scale-up link can carry partial layer results, shared state, or other frequent exchanges inside a tightly coupled group.
- Across servers or racks: A scale-out network connects a larger cluster. It is necessary for large jobs but introduces switches, shared paths, and more opportunities for delay or congestion.
Suppose a model fits only after its weights are split across two accelerators. Capacity has been solved, but every cross-device dependency now becomes part of response time. A slightly slower chip with a stronger relevant path can beat a faster chip that spends more time waiting. The correct comparison measures the complete model on the complete system.
The relevant question is not simply “How fast is the network?” First ask whether important communication stays inside one tightly connected server or rack. Then ask what bandwidth, delay, and competition for shared links apply when traffic crosses the broader network between racks. A system that keeps frequent communication on the stronger path can outperform one with a higher arithmetic rating but weaker connections.
10.4 Ask Whether the Software Is Ready
A processor is useful only when supporting software can turn a model into efficient chip instructions. That support includes translators that convert programs for the chip, tested implementations of common calculations, tools for dividing work across chips, and systems for serving requests.
General-purpose graphics processing units, or GPUs, are supported by major machine-learning frameworks, but “supported” does not guarantee equal speed or reliability. The practical question is whether the exact model and its distributed features run well on the proposed hardware today, and how much software work is required.
Google's Tensor Processing Units and Amazon's Trainium chips are examples of processors designed specifically for machine learning and integrated with their owners' cloud software. Whether they are a good fit depends on measured results for the buyer's model and on the cost of adapting software to that environment.
These are rules of thumb, not product guarantees:
| Approach | Possible fit | Potential advantage | What must be tested |
|---|---|---|---|
| GPU platform | Models or tools that already support that vendor's software | Familiar frameworks and a broad product range | Does the exact workload meet speed, memory, reliability, and cost targets? |
| Cloud-provider accelerator | A supported workload already committed to that cloud | Hardware, compiler, and service are operated together | What code changes, service limits, and platform dependence are required? |
| Workload-specialized system | A stable workload with a measured bottleneck | May optimize that bottleneck | Which models, operations, and scales fall outside its strengths? |
| Older or smaller accelerator | A model that fits and has a loose performance target | May cost less to rent or buy | Does measured end-to-end performance still meet the requirement? |
10.5 Compare Systems Under the Same Conditions
Headline arithmetic is not a buying decision. Two performance numbers are comparable only when they use the same model, number format, input and output lengths, number of requests served together, response-time target, software and optimization level, power mode, and amount of hardware.
Write those conditions into a one-page test before looking at vendor results. For interactive inference, an illustrative requirement might use 1,000-token prompts and 200-token answers. It might require at least 95 out of every 100 requests to receive a first token within two seconds, then ask for successful requests and total cost per hour. The point is to define success in the buyer's unit before selecting a benchmark.
Power and availability belong in the comparison too. Suppose one system is 10 percent faster but cannot be delivered for six months: it cannot finish today's job. The numbers are illustrative. A system that requires a rack your data center cannot cool is not a deployable option. Rental price, engineering time, and failure rate all belong beside speed.
Start with the workload
|
+-- Do the model and working data fit?
|
+-- Can the chips exchange data fast enough?
|
+-- Does the software run the real model well?
|
+-- Can the system meet the speed, power, and price target?
10.6 Make the Decision in This Order
A disciplined purchase or rental decision follows the same order as this Part:
- Write down the real workload and success target.
- Calculate what must fit in memory, including working data and a safety margin.
- Identify which data must cross chip, server, or rack boundaries.
- Run the exact model with the intended software, not only a small operation that favors the hardware.
- Measure user-facing latency, total throughput, quality, reliability, power, and complete cost under the same conditions.
Reject any option that fails a required condition before ranking the survivors. A system that does not fit the model, meet the response-time promise, or run the software reliably is not made viable by a low price or a high peak rate.
Further reading:
- Google's Tensor Processing Unit architecture guide explains why a processor specialized for matrix operations differs from a general processor.
- Amazon Web Services' Trainium documentation shows how compute, memory, connections, and programming support appear in one design.
- MLPerf Inference provides controlled benchmark results. The conditions and system description matter as much as the winning number.
Checkpoint. Choose hardware by asking what work it must finish, what must fit in memory, how far data must travel, whether the software is ready, and what the complete system costs.
Part 11 — The Economics of Compute
A hard-to-replace bottleneck can raise the cost of AI compute or slow expansion, but it does not set price by itself. Demand, competition, contracts, financing, operating costs, and bargaining power matter too. The constrained input may be an accelerator, high-bandwidth memory, advanced packaging, networking, or a powered and cooled site, and it can change over time.
Scarcity alone is not enough. An input sets the pace only when substitutes are poor. A shortage of one cable matters little if another cable works. A shortage of memory that the chip requires is different: without it, the chip cannot ship. The same logic applies to an available GPU with no powered building in which to run it.
11.1 From Factory to Useful Output
AI compute passes through several functions. A chip factory makes the processor. Memory makers produce nearby memory. Advanced packaging joins the required dies and dense wiring. A server maker adds networking, storage, power equipment, and cooling. A data-center operator supplies a building and grid connection. A cloud operator finances and rents equipment. A model provider turns machine time into tokens, answers, or completed tasks. This is a functional map, not a claim that every company or contract follows one linear chain; one firm may own several stages.
One required stage can limit output from the whole chain. For an illustrative example, suppose packaging plants can assemble 10,000 devices while chip factories can make 15,000 suitable processors. No more than about 10,000 finished devices emerge unless production changes or a substitute appears. Paying more for extra processors does not solve that packaging limit. Scarce capacity may command a premium, although contracts and competition determine where any extra profit is captured.
The conclusion is practical: when a price rises, look upstream for the input that cannot be added quickly or replaced easily. Do not assume the most famous company or the most expensive component is the current constraint.
11.2 What One GPU-Hour Costs
A GPU-hour means one graphics processor used for one hour. A simplified owner-cost model separates equipment, electricity, operations and maintenance, and financing. Real accounting may also allocate buildings, cooling, networking, taxes, insurance, failures, and idle capacity.
The purchase price is spread across the hours the owner expects to sell. The share of available hours that customers actually pay for therefore matters. This sales rate is different from the fraction of time the chip is busy or the fraction of its peak arithmetic it achieves. An idle GPU still ages and ties up capital.
Consider an illustrative machine with $40,000 of equipment and installation cost assigned to each GPU. If the owner expects four useful years and sells 70 percent of the available hours, the equipment cost is about $1.63 per sold hour:
$40,000 ÷ (4 years × 8,760 hours × 70 percent sold) = $1.63 per sold GPU-hour
Assume that share of the server draws 1.2 kilowatts while a paid job is running, that idle equipment is powered down, and that electricity costs $0.08 per kilowatt-hour. Electricity then adds about $0.10 per sold hour. If maintenance and staff add $0.30 and financing adds $0.45, the illustrative owner cost is about $2.48 per sold hour.
These figures are assumptions, not a market quote. Their purpose is to show the levers. If the share of hours sold falls from 70 percent to 35 percent, the equipment portion doubles from $1.63 to $3.26. Electricity per paid job stays at $0.10 only under the stated assumption that idle equipment is powered down; real idle draw and facility overhead change the result. A provider may therefore accept a lower hourly price from a customer who commits to steady use because the commitment reduces demand risk.
11.3 From GPU-Hours to Token Cost
Customers usually do not want GPU-hours. They want model output. Converting between the two requires knowing how much useful output the system delivers in an hour.
Suppose a serving system costs $3 per GPU-hour and produces 600 output tokens each second across all users being served together. In one hour it produces 2.16 million output tokens. The GPU rental portion is therefore about $1.39 per million output tokens:
600 tokens/second × 3,600 seconds = 2.16 million tokens/hour
$3.00 / 2.16 million = $1.39 per million output tokens
The phrase “across all users” is essential. One user may see 40 tokens per second while the machine serves many users at once. Serving more requests together can lower cost, but only until waiting time becomes unacceptable. The economic problem is to fill the machine without making the product feel slow.
The calculation is incomplete by design. A provider also pays for prompt processing, conversation state, CPUs, networking, failed requests, engineering, support, and profit. Input and output tokens can have different costs because reading a prompt and generating a reply stress the system differently.
Agent tasks push the unit one step further. Their cost includes model calls, paid tool use, retries, and any compute or service capacity that remains reserved while the task waits. A useful comparison is dollars per verified fix, research report, or resolved support case. Tokens remain an input; the finished task is the product.
11.4 Training and Inference Are Different Businesses
Training and inference both use accelerators, but buyers often purchase different outcomes. The table shows common patterns, not strict boundaries; training services can be continuous, and one inference failure can affect many users.
| Training | Inference | |
|---|---|---|
| Goal | Produce a model's learned settings | Use those settings to answer requests |
| Shape of demand | A large campaign with a start and end | Continuous traffic that rises and falls |
| What failure means | A delayed run can waste work across a large cluster | A failed request harms one or a group of users |
| Main system concern | Keep many chips doing useful work together | Meet response-time targets while keeping machines full |
| Best economic unit | Cost and time for a successful training run | Cost per token, request, or completed task |
A simple training estimate is accelerators multiplied by hours and hourly price. For example, 1,000 accelerators used for 30 days at $2.50 per hour cost $1.8 million in accelerator rent. Storage, networking, staff, preparation runs, and failed time come on top.
Inference rewards steady paid demand. Traffic arrives throughout the day, and users expect a quick response. Operators can serve several requests together, reuse repeated prompt prefixes when the model and software support caching, and route requests to suitable hardware. Their challenge is to reduce cost without crossing the response-time or quality line that customers notice.
When inference volume is high, a small unit-cost reduction repeats across many requests. That creates an incentive for some large cloud and model companies to build custom chips for stable workloads. Development cost, software support, and the share of available hours sold determine whether that investment pays off.
11.5 Why Market Prices Move
Market price is not the same as production cost. Cost helps determine what a supplier can sustain; price also reflects how strongly buyers want the product, how many alternatives they have, and how supply is sold.
Imagine that two providers own the same number of identical machines. Provider A sells mostly hour by hour and risks long idle periods. Provider B has a customer committed to steady use for a year. Provider B may offer a lower hourly rate because the contract makes future paid use more predictable. The lower rate does not prove its electricity or hardware is cheaper. The customer is taking some demand and commitment risk in exchange for the discount.
Four forces often contribute to price differences:
- Scarce hardware can command a premium.
- Steady demand lets a provider keep more machines earning revenue.
- Better memory, connections, software, and operation can increase useful output from the same chip.
- Long commitments shift demand risk from the provider to the customer and can help finance equipment.
A fifth issue is what the quote includes. One price may include a tightly connected group of accelerators, local storage, and support; another may describe a bare device with separate charges or weaker links. A low hourly price can therefore be poor value if the system completes less work or excludes resources the buyer needs.
The buyer should compare cost per useful result at the needed service level. The seller must ask whether the price covers equipment, operations, idle time, financing, and risk over the expected life of the system.
11.6 Power Can Be Cheap and Still Be the Bottleneck
In the illustrative calculation above, electricity is a smaller hourly line item than equipment and financing. That relationship changes with hardware, the share of hours sold, power price, cooling, and accounting. Access to electricity can still constrain expansion because a large site needs permits, equipment that changes voltage, a grid connection, backup systems, and cooling.
The International Energy Agency's analysis describes grid connections as a material constraint on new data centers. Forecasts will change, but the mechanism is durable: money can buy hardware faster than a constrained region can build transmission and connect a new site.
This resolves the example's apparent contradiction. Energy for one sold GPU-hour cost about 10 cents under its assumptions, yet dependable power capacity in the required place and time can be scarce. The energy bill and the ability to connect and draw power are different constraints.
Think of water again. The water itself may be inexpensive, but a new factory cannot operate without a pipe large enough to deliver the required flow. Building that pipe can require permits, equipment, construction, and time. A data center likewise needs not only kilowatt-hours over the year but also enough power available at the moment its equipment draws it, plus the electrical and cooling infrastructure that makes the draw safe and dependable.
11.7 Reading the Economics
Start with the result being sold. Is the buyer paying for an hour of access, a million tokens, a completed task, or a training campaign? Each unit places different costs and risks on the seller.
Then find the input that cannot be expanded or substituted on the required schedule. Check whether the quoted price includes networking, storage, support, and a reliable level of service. Finally, divide price by useful output under the buyer's real response-time and quality requirements.
- The International Energy Agency's Energy and AI report gives the physical setting for data-center electricity demand and grid constraints.
- Epoch AI's data-center cost model shows how equipment, construction, and power assumptions can be separated.
- Public cloud and specialist-cloud pricing pages are useful observations, but every figure should be dated and checked for region, commitment length, availability, and bundled resources.
Final checkpoint. A hard-to-replace bottleneck can constrain supply and affect compute prices alongside demand, contracts, competition, and financing. A GPU-hour price becomes meaningful only after it is connected to useful output and the conditions under which that output was measured.
Complete Glossary
Complete for This Course, Not for the Industry
This glossary covers the technical terms the course uses or requires to explain its main ideas. It is intentionally not a catalog of every AI product, company, acronym, or architecture name. The course should still explain each necessary term when it first appears; this section is a compact reference.
Accelerator
A processor designed to perform a particular class of work efficiently. In this course, an AI accelerator may be a GPU or another processor built for neural-network calculations. Source.
Activation
An intermediate array of numbers produced while a neural network processes an input. Training must retain some activations, or recreate them later, to calculate how the model should change. Source.
Activation checkpointing and recomputation
A training method that saves only selected intermediate results and recreates, or recomputes, the others during the backward calculation. It exchanges extra arithmetic for lower memory use; it is different from saving a recovery checkpoint. Source.
Adam
A training update method that keeps running averages of recent gradients and squared gradients for each learned parameter. Those extra arrays are part of the optimizer state and consume memory. Source.
Advanced packaging
Manufacturing work that uses dense wiring to assemble one or more dies into a package. Some advanced packages also include nearby memory. It is separate from fabricating the silicon dies. Source.
Agent and agent runtime
In this course, an agent is software that can call a model repeatedly and let it request external tools. The agent runtime is the ordinary software that keeps state, checks and executes allowed tool requests, returns results, and decides when to stop. Source.
All-reduce
A group communication operation in which every participant contributes an array, the arrays are combined with an operation such as addition, and every participant receives the combined result. Distributed training commonly uses it to combine gradients. Source.
All-to-all
A group communication operation in which every participant sends a different block of data to every participant and receives one block from each. Its cost depends on the amount of data, the connection map, and the implementation. Source.
Arithmetic intensity
The number of arithmetic operations performed for each byte moved across a named memory boundary. A value is high or low only relative to a particular machine and memory level. Source.
Attention
A neural-network calculation that lets a token draw different amounts of information from selected token positions. During ordinary text generation, a token may use itself and earlier tokens but not later ones. Source.
Attention head
One parallel set of learned calculations that creates query, key, and value vectors and performs attention with them. A layer can run several heads so it can learn different comparison patterns, then combine their outputs. Source.
Autoregressive generation
Generating a sequence one accepted item at a time, with each new item depending on the items already accepted. Ordinary language-model decoding generates tokens this way. Source.
Backpropagation and backward pass
Backpropagation applies the chain rule backward through a neural network to calculate how the loss changes with each parameter. The computation that performs this work is commonly called the backward pass; an optimizer later uses the resulting gradients to update the parameters. Source.
Bandwidth and memory bandwidth
Bandwidth is a maximum or measured data-transfer rate over a stated connection. Memory bandwidth is the rate between a processor and a named level of memory; every figure should say whether it is theoretical or measured and which directions or operations it includes. Source.
Batch and batching
A batch is a group of examples or active requests processed together. Batching can reuse data and raise total throughput, but larger batches also use more memory and can make an individual request wait. Source.
Benchmark
A controlled test used to compare performance under stated conditions. A useful benchmark names the workload, hardware, software, numerical format, input and output sizes, number of simultaneous requests, and measurement rules. Source.
Bfloat16 (BF16)
Bfloat16 is a 16-bit floating-point format. It covers roughly the same range of magnitudes as 32-bit floating point but records fewer fine distinctions between nearby values. Source.
Binning
Testing manufactured chips and sorting them into product grades according to the specifications they meet, such as working features, frequency, or power. A lower bin is not necessarily a defective chip. Source.
Bit and byte
A bit represents one of two values, conventionally zero or one. A byte contains eight bits; in decimal units, one gigabyte is one billion bytes. Source.
Cache
Storage that keeps copies of data near a consumer so a later access may avoid a slower source. A cache helps only when it contains the needed data. Source.
Memory capacity
The amount of data a memory system can hold. Memory capacity answers whether the required weights and working data fit; it does not say how quickly they can be read. Source.
Checkpoint
A saved collection of model or training state. It may be a released model or a recovery snapshot used to resume a run; it may also be split across files or machines. This meaning is different from activation checkpointing, which reduces temporary memory during one calculation. Source.
Chiplet
A die designed to be combined with other dies in one package. Chiplets may divide one function or provide different functions and do not have to be physically tiny. Source.
Cluster
A group of connected computers that work together on a shared task or service. Calling a system a cluster does not by itself specify its connection speed, size, or reliability. Source.
Compute-bound
A condition in which available arithmetic throughput is the main performance limit for a specified program and machine. Memory traffic, communication, or dependencies may be separate limits in another implementation. Source.
Model context
The information represented in the input supplied for the current model calculation. A service may build the model context from instructions, messages, retrieved material, and tool results; information outside that supplied input is not automatically available to the model. Source.
CPU
Central processing unit: a general-purpose processor that runs operating systems and a wide variety of programs. CPUs can perform parallel work, but high-performance cores also devote substantial hardware to reducing delay on irregular and decision-heavy code. Source.
Customer qualification
Testing and approval by a customer that a component meets the customer's platform and reliability requirements before broader use. Passing a supplier's internal tests does not automatically complete a customer's qualification process. Source.
Data parallelism
A training method in which workers process different examples and combine their gradients or updates. The model parameters may be copied on each worker or divided by a sharded variant. Source.
Decode
In autoregressive language-model serving, the phase that generates output tokens after the prompt has been processed. Accepted output tokens are produced in sequence, although an implementation may test several candidates at once. Source.
Die
One piece of semiconductor material containing a fabricated circuit. A die is cut from a processed wafer and may later be combined with other dies inside a package. Source.
DRAM
Dynamic random-access memory: memory that stores each bit in a cell whose charge must be refreshed. GDDR and HBM are DRAM families; DRAM is denser but generally slower than the on-chip SRAM used for small memories. Source.
Embedding
A learned vector representation of an item. In this course, a token embedding is the learned list of numbers selected by a token identifier before transformer layers add context to that representation. Source.
Ethernet
A standardized family of wired network technologies used from local networks to data centers. A quoted Ethernet rate is a link rate; packet and protocol overhead can make useful application throughput lower. Source.
Floating-point
A way to encode numbers using a sign, a scale called an exponent, and significant digits. Different floating-point formats trade storage and precision against the range of magnitudes they can represent. Source.
Feed-forward block
The part of a transformer layer that applies the same learned neural-network calculation separately to each token position after attention has exchanged information among positions. Transformer designs vary in the exact calculation they use. Source.
Floating-point operation (FLOP)
A floating-point operation, or FLOP, is one counted arithmetic operation under a stated convention; a fused multiply-and-add is commonly counted as two. FLOP/s is a rate, and it is incomplete unless the number format, operation, and any sparsity assumption are stated. Source.
Forward pass
The computation from a model's input through its prediction and, during training, its loss. Training retains or later recreates selected intermediate results so the backward pass can calculate gradients. Source.
Foundry
A semiconductor manufacturer that fabricates circuits, often from designs supplied by other companies. A pure-play foundry primarily manufactures customer designs rather than selling its own chip designs. Source.
FP8
A family of one-byte floating-point formats that divide their bits differently between numerical range and fine detail. A claim using FP8 must name the exact format, operation, and accumulation method. Source.
32-bit floating point (FP32)
The common name for the standardized 32-bit binary floating-point format. Each value uses four bytes and records more numerical detail than BF16, but more precision does not guarantee a better or faster model result. Source.
GDDR
Graphics Double Data Rate: a family of high-data-rate DRAM used for graphics and some AI systems. Its capacity and bandwidth depend on the generation and system design. Source.
GPU
Graphics processing unit: a processor architecture developed for graphics and organized for high-throughput parallel work. Many neural-network operations suit its matrix and vector hardware, but performance still depends on software and data movement. Source.
Gradient
Numbers showing how small changes to the parameters would change the training loss. An optimizer uses gradients to calculate an update; a gradient is not the update itself. Source.
HBM
High Bandwidth Memory: stacked DRAM with a very wide connection, commonly placed in the same package as a high-end accelerator. HBM raises bandwidth but adds demanding memory-stacking, packaging, and testing work. Source.
Inference
Using a trained model to calculate an output from new input without performing an ordinary training update to its parameters. Output may be text, a prediction, an image, an embedding, or another model result. Source.
InfiniBand
A switched network design built for high data-transfer rates and short delays. It is widely used in high-performance-computing and AI clusters and can move data directly between computers' memory with little work from their CPUs. Source.
Accelerator interconnect
A connection that carries data among nearby accelerators in a tightly coupled system. Its bandwidth, delay, connection map, and software support affect how well work can be split across chips. Source.
Interposer
An intermediate wiring structure that creates dense connections among dies inside a package. Interposers can be built in different ways, and not every HBM package uses the same kind. Source.
Inter-token latency
The elapsed time between two generated output tokens. It differs from time to first token and from an average time-per-token measure calculated over a whole response. Source.
Kernel
A small program that a GPU runs across many parallel workers. The word has other meanings in operating systems, so this course uses it only in the GPU sense. Source.
Kernel fusion
Combining compatible calculations into one GPU kernel so a temporary result can stay on the chip or be eliminated instead of being written to large device memory and read back. Fusion can use more on-chip resources and is not always faster. Source.
Key-value cache (KV cache)
The key and value arrays already computed by each attention layer for earlier tokens. Reusing them during decode avoids recomputing those earlier keys and values, but the cache consumes more memory as the retained sequence grows. Source.
Latency
The elapsed time from the start of an operation or request to a stated completion point. A latency figure is incomplete unless it says exactly what starts and ends the measurement. Source.
Latency hiding
Keeping other work ready so a processor can run it while one group waits for data or an earlier operation. This reduces idle time but does not make the underlying delay disappear. Source.
Model layer
One stage of a neural network that transforms input values into output values using learned parameters and other operations. Modern models usually apply many model layers in sequence. Source.
Large language model
A language model with a very large number of learned parameters. “Large” has no universal numerical threshold. Source.
Lithography
A chip-manufacturing process that uses light and a patterned mask to define features in a light-sensitive coating on a wafer. Deposition, etching, and other steps then build the physical structures. Source.
Training loss
A numerical score for a model's error on an example or batch. Training calculates gradients of the loss and uses them to guide parameter updates. Source.
Matrix and matrix multiplication
A matrix is a rectangular array of numbers. Matrix multiplication combines rows from one matrix with columns from another to produce a new matrix; many neural-network layers spend much of their arithmetic on this operation. Source.
Memory-bound and memory-bandwidth-bound
A condition in which moving data through a named memory connection is the main performance limit for a specified program and machine. Changing the batch, software, or hardware can change whether the same operation is memory-bound. Source.
Memory wall
The growing mismatch between how quickly processors can perform arithmetic and how quickly memory systems can supply data. Caches, data reuse, lower precision, and wider memory connections address parts of this problem. Source.
Mixed-precision training
Training that uses more than one numerical format, choosing each according to the operation and numerical need. A common pattern uses compact values for much of the arithmetic and more precise values for selected updates or accumulated results. Source.
AI model
A learned system of calculations that maps an input to a prediction or other output. Training adjusts an AI model's parameters; inference uses a trained model to produce outputs. Source.
Moore's law
An industry observation that the number of components economically placed on an integrated circuit tended to grow rapidly over time. It is not a physical law and does not promise that every program becomes faster at the same rate. Source.
Neural network
A model built from connected layers whose behavior depends on learned parameters. “Neural” is historical terminology; the model is a mathematical computation, not a biological brain. Source.
Number format
The rule that says how a pattern of bits represents a number. Formats differ in storage size, range, precision, and the operations that hardware supports. Source.
Optimizer
A training algorithm that uses gradients and an update rule to change model parameters. Some optimizers, including Adam, keep extra state from earlier steps. Source.
Package
The structure that protects one or more dies and connects them to the rest of a system. Some accelerator packages also include nearby HBM, but memory is not part of every chip package. Source.
Parameter
A numerical value adjusted by training, commonly a weight or bias. Parameter count describes how many learned values a model has but does not by itself determine quality, file size, or active computation. Source.
Pipeline parallelism
A method that assigns sequential groups of model layers to different devices. Intermediate results move forward through the stages, while training gradients move backward; idle gaps and unequal stage times reduce efficiency. Source.
Electrical power and electrical energy
Electrical power is the rate of energy use and is measured in watts; one watt equals one joule per second. Electrical energy is power accumulated over time, so one kilowatt used for one hour equals one kilowatt-hour. Source.
Precision
How finely a numerical representation distinguishes nearby values. Precision is different from range, which describes the smallest and largest magnitudes the format can represent. Source.
Direct preference optimization
A post-training method that learns from comparisons between preferred and less-preferred responses without first fitting a separate reward model. It is different from reinforcement learning from human feedback. Source.
Prefill
The language-model inference phase that processes the supplied input tokens and prepares the initial KV cache before output-token generation. Time to first token also includes possible network, queue, tokenization, and scheduling delay. Source.
Process node
A foundry's name for a generation of transistor and wiring technology. Modern labels such as “3 nanometer” do not measure one physical feature and cannot be compared directly across foundries. Source.
Prompt
Input supplied to a generative model for one call. A service may combine user content with instructions, earlier messages, retrieved material, and tool results before constructing the model's input. Source.
Quantization
Mapping values from a larger set into a smaller set of representable values, often for model weights or intermediate results. It can reduce storage and memory traffic, but speed and quality effects depend on the method, model, and hardware. Source.
Query-key-value (QKV)
Three numerical roles in attention. A query is compared with keys to obtain match weights; those weights determine how the corresponding values are mixed into the attention output. Learned calculations create Q, K, and V from the current token representations; they are not human-written labels. Source.
Register
A tiny storage location used directly by processor instructions for values needed immediately. Registers are much smaller and closer to arithmetic hardware than device memory. Source.
Reinforcement learning
A family of methods in which an agent learns what actions to take so as to increase expected cumulative reward. Human preferences can provide a reward signal in one application, but they are not part of the general definition. Source.
Reticle limit
Informal shorthand for the maximum rectangular field a lithography scanner exposes at once. It constrains the size of a conventional die made with one exposure field. Source.
Roofline model
A performance bound that compares a machine's peak arithmetic rate with its memory bandwidth multiplied by arithmetic intensity. The lower of those two ceilings limits the attainable arithmetic rate in the simplified model. Source.
Scale-out
Connecting multiple servers or tightly connected accelerator groups through a broader network. Scale-out describes system organization, not a guarantee about distance, speed, or network design. Source.
Scale-up
Connecting processors within a tightly coupled group using links designed for high bandwidth and low latency. The group may be inside a server or extend across a rack; the term describes system organization rather than distance alone. Source.
GPU scheduler
Control hardware that chooses which ready group of GPU work runs next. It can issue another ready group while one group waits for data. Source.
Sharding
Dividing specified model data or training state across workers so each holds only a share. It lowers the memory needed on one worker but creates communication when another worker needs a piece. Source.
SRAM
Static random-access memory: fast memory technology used for small on-chip caches and working memories. It uses more chip area per stored bit than DRAM, so processors include much less of it. Source.
Structured sparsity
A pattern in which zeros appear in the exact arrangement that supported hardware can skip. Arbitrary zeros do not automatically qualify a calculation for an advertised sparse rate. Source.
Supervised fine-tuning
Additional training on examples that pair an input with a desired output. It can teach a pretrained model a task or response style without defining every later post-training method. Source.
Tensor core
NVIDIA's name for specialized GPU units that perform supported matrix multiply-and-accumulate operations. Their advertised rate depends on the numerical format and may also assume a supported sparsity pattern. Source.
Tensor parallelism
A method that splits operations inside a model layer, commonly a matrix multiplication, across devices. The devices must exchange partial results frequently, so connection performance matters. Source.
Tile
A small rectangular block cut from a larger array of numbers. GPU programs move and reuse tiles in nearby memory so many calculations can share the same loaded data. Source.
Throughput
The amount of completed work per unit time, such as requests or output tokens per second. It differs from latency, the time one specified request takes; batching can raise throughput while increasing waiting. Source.
Time to first token
The elapsed time from submitting a request until its first generated output token arrives. Depending on the measurement boundary, it may include network transfer, queueing, tokenization, scheduling, and prompt processing. Source.
Token
An identifier produced by a model's tokenizer, representing a whole word, part of a word, punctuation, or another text fragment. Token-to-word ratios vary with language, text, model, and tokenizer. Source.
Tokenization and tokenizer
A tokenizer applies a fixed vocabulary and splitting rules to map input into token identifiers; tokenization is that conversion process. Different model families can split the same text differently. Source.
Tool call
A structured request from a model asking ordinary software to run a named external function with specified inputs. The surrounding application—not the model—must validate permissions, execute the function, and return its result. Source.
Topology
The pattern of connections among processors, servers, and switches. Together with link rates, routing, and software, topology affects available paths, congestion, delay, and failure behavior. Source.
Training
Adjusting a model's parameters using data so that a stated error or reward measure improves. Neural-network training commonly uses a training loss, backpropagation, and an optimizer; only large runs require many machines. Source.
Transformer
A neural-network design introduced in 2017 that arranges attention and other calculations in repeated blocks. Most current large language models use transformer-based designs, with many variations. Source.
Transistor
A semiconductor device that can switch or control an electrical signal. Modern processors combine many transistors to build arithmetic, memory, and control circuits. Source.
Wafer
A circular slice of semiconductor material on which a factory builds many copies of a circuit. The processed wafer is tested and cut into individual dies. Source.
Weight
A learned parameter that a model multiplies by another value; many weights are entries in matrices. Not every parameter is a weight. Source.
Vector
An ordered one-dimensional list of numbers. A token embedding and each query, key, or value inside one attention head are vectors; the number of entries depends on the model design. Source.
Vocabulary
The set of token pieces and identifiers available to a tokenizer and its model. The tokenizer maps input text to identifiers from this set, and a text-generating model assigns an output score to each allowed vocabulary token. Source.
Die yield
The share of manufactured dies that meet the required tests. Die yield depends on the design, defect rate, process, and test criteria; it is not the same as binning acceptable parts into product grades. Source.