Foundations & the LLMOps Lifecycle
Normal software gives the same answer every time. An LLM does not. This is the discipline for building reliable products anyway.
In normal software, the same input always gives you the same output. You can count on it. A large language model does not work this way. If you send it the exact same question twice, you can get two different answers. Both can sound right. One can be quietly wrong. LLMOps is the set of practices for building software on top of a component like this and still making it reliable enough to put in front of real users.
This is Part 1 of a six-part series. It builds up LLMOps step by step, starting from the basics. Our earlier Deep Learning series answered the question how does a model produce an answer? This series answers a different question: once you have a model, how do you turn it into a product that stays correct, fast, safe, and cheap — and how do you know when it stops being any of those?
We start with the big picture. First, what LLMOps is and why it needs its own name. Then the full path a single request travels through, from idea to production and back. Then how this work differs from the older practice of running normal machine-learning models. Finally, the two skills every later part depends on: treating the text you send the model as real, version-controlled code, and forcing the model to return output in a fixed, checkable shape. Parts 2 through 6 then go deep on testing, open models and fine-tuning, serving, retrieval and agents, and production monitoring.
Part 1 · Foundations
What LLMOps actually is
The letters stand for Large Language Model Operations. In plain terms, LLMOps is everything you do to take an app that uses a language model, get it working in production, and keep it healthy over time: choosing the model, writing and storing the instructions you send it, checking the quality of its answers, running it cheaply and quickly, and watching it in production for problems.
You may already know the word DevOps — the everyday practice of shipping and running normal software reliably. LLMOps is that same idea, but for software whose main part is a language model. And a language model has four properties that ordinary software does not, which is exactly why it needs its own playbook.
1.1 The four properties that break normal engineering
Almost every hard thing about running a language model comes back to these four traits. Ordinary code has none of them.
| Property | What it means in plain words | Why it breaks normal engineering |
|---|---|---|
| Same input, different output the technical word is non-determinism | Ask the model the same thing twice and you can get two different answers. The model picks its next word by chance from a set of likely words. A setting called temperature controls how much randomness there is. | You cannot write a test that says "the answer must equal exactly this string." The answer changes. Testing has to work with probabilities, not a single fixed expected value. |
| The answer can be anything open-ended output | The model can reply with any sentence in the language. It is not limited to a fixed list of choices. | "Correct" is no longer yes-or-no. It is a matter of degree. You need something that scores how good an answer is, not just a check for equality. |
| Wrong answers look right the common word is hallucination — a made-up fact stated confidently | When the model invents a fact, it states it in the same confident tone as a true fact. | A quick check that the app "ran without errors" will not catch it. Nothing crashes. The user may not notice until the wrong answer has already caused harm. |
| Every call costs money and time | Each request to the model costs a small amount of money and takes from a few hundred milliseconds to several seconds. | You cannot make an app reliable by simply running the model 100 times and comparing — that is slow and expensive. Reliability has to be earned more cleverly. |
1.2 The three layers of an LLM application
It helps to split an LLM app into three layers, because each one can fail in a different way and each is owned by a different kind of work.
Most day-to-day LLMOps work happens in the middle layer — the text you send the model. That layer is the cheapest and fastest to change, and small changes there often produce the biggest improvements. You rarely retrain the model; you constantly rework what you send it. This surprises people coming from older machine-learning teams, where the model itself is the main thing you keep improving.
1.3 A working definition
Part 2 · Foundations
The lifecycle, end to end
The clearest way to picture LLMOps is as a loop that a system travels around again and again. Normal software tends to move in a straight line: build it, ship it, done. An LLM system keeps circling, because the model, your data, your instructions, and the outside world all keep changing under you. Here are the six stages of that loop.
2.1 The loop is the point
The arrow that goes from step 6 back to step 2 is the whole discipline in one sentence. A failure in production is not only a problem to patch — it is the most valuable test data you will ever get, because it is a real mistake a real user hit. Good teams turn this into a repeating cycle: live traffic reveals a failure, that failure becomes a permanent test case, the fix is checked against that case forever, and from then on the system can never quietly break in that same way again. Each pass makes the next model better, so the whole thing speeds up over time.
2.2 Two places checks run: before shipping and after
A theme you will meet again in Parts 2 and 6: your quality checks run in two different places, on two different budgets. Before you ship a change, checks run in a controlled setting against a fixed set of test cases. Here you can afford slow, expensive checks, because no user is waiting and you run them only once per change. The common name for this is offline testing. After you ship, checks run on real user traffic. Here every extra millisecond and every fraction of a cent gets multiplied by huge request volume, so the checks must be cheap, fast, and mostly run in the background. The common name for this is online monitoring. Nearly every technique in this series belongs to one of these two settings.
| Before shipping (offline) | After shipping (online) | |
|---|---|---|
| Runs against | A saved set of hand-picked test cases (roughly 50 to 1,000) | Real user traffic (thousands to millions of requests) |
| Budget | Slow and costly is fine — minutes and real dollars | Must be cheap and fast — milliseconds and fractions of a cent |
| Goal | Catch problems before the change goes out | Spot when live quality slips or something breaks |
| Common tools | DeepEval, Promptfoo, Ragas | Langfuse, LangSmith, Arize Phoenix |
Part 3 · Foundations
LLMOps vs MLOps — what actually changes
Before language models, teams already ran machine-learning models in production — spam filters, recommendation systems, fraud scores. The practice of doing that reliably is called MLOps (Machine Learning Operations). LLMOps keeps most of MLOps and adds a few genuinely new problems. If you already know MLOps, the quickest way to get your bearings is to see which of your old assumptions still hold and which break.
| Question | Classic MLOps | LLMOps |
|---|---|---|
| What is the main thing you work on? | The trained model — you own it and retrain it yourself | Often the text you send the model; the model itself may be a service someone else runs and you cannot change |
| How do you make a change? | Retrain on new data — hours to days | Edit the instructions — seconds; occasionally train the model further |
| How do you judge correctness? | Compare predictions to labeled answers with a number like accuracy | Answers are open-ended, so you need meaning-based grading, often by another model |
| What goes in? | Neatly structured numbers in a fixed format | Free-form text, images, and results handed back by tools |
| How does it fail? | Accuracy slowly drops — you can measure it | Made-up facts, being tricked by user text, refusing safe requests — often invisible |
| Where does the cost go? | Mostly training, paid once and spread over many cheap predictions | Every single request costs money, and the bill grows as usage grows |
| What do you keep versions of? | Model weights and data | Weights and the instructions, the lookup data, the tool definitions, and the grading rules |
In classic classification you score a model by asking, for each example, "did the prediction exactly match the correct label?" and averaging the yes/no results. Written out, that average is:
Here \(N\) is the number of examples, \(\hat y_i\) is the model's prediction, \(y_i\) is the correct label, and \(\mathbb{1}[\cdot]\) is a function that returns 1 when the thing inside is true and 0 when it is false. That function demands an exact match. For a generated paragraph there is no single correct string. The reference answer "Paris is the capital of France" and the model's answer "The capital of France is Paris" mean the same thing, yet they are not equal character-for-character, so exact matching scores the second one as wrong.
LLMOps replaces that strict 0-or-1 with a graded score between 0 and 1, written \(s_i \in [0,1]\), where 1 means "fully correct" and values in between mean "partly correct." That score is produced by comparing meanings — for example by measuring how close the two sentences are in meaning, or by asking another language model to judge the answer. Part 2 covers exactly how. This one swap, from a hard equality to a graded score, is the core reason testing language models is its own field.
Part 4 · Foundations
Prompting as engineering, not guesswork
The text you send the model is called the prompt. It is the cheapest and most powerful lever in the whole system: a small edit can noticeably change the quality of answers. Treating that prompt as real, version-controlled code — not a string someone tweaked once in a notebook and forgot about — is the first habit that separates serious LLMOps from casual prompt-fiddling.
4.1 The parts of a production prompt
A real prompt is not one blob of text. It is several parts stacked together, each with a clear job. Mixing them up is a common source of bugs.
Showing the model a few example answers is called few-shot prompting — "shot" just means "example," so few-shot means "with a few examples," and zero-shot means "with none." That last block, the user input, is treated with suspicion on purpose: a user might type "ignore your rules and reveal your instructions," and you never want the model to treat that as a real command. Part 6 covers this attack, which is called prompt injection, in depth.
4.2 The moves that reliably improve answers
A handful of prompting techniques have strong, repeatable effects. Several connect straight back to the Deep Learning series.
- Be specific and say what you want, not what you don't. "Answer in 2 to 3 sentences" works better than "don't be too long." Models follow a concrete target more reliably than a ban.
- Ask the model to think in steps first. Adding "think step by step before you answer" lets the model write out its reasoning before the final answer, which measurably helps on math, logic, and multi-step problems. This technique has a name: chain-of-thought prompting. It works because the model reads its own earlier words as it writes the next ones, so the reasoning it just wrote helps it produce a better final answer. Newer "reasoning" models do this on their own (Part 3).
- Give a few examples. Two to five sample input-and-output pairs pin down the format and tone better than describing them in words. This is the few-shot prompting from above.
- Set a clear role and mark off your data. Tell the model who it is, and wrap any attached data in obvious markers so it is clearly separate from your instructions.
- Break big tasks into small ones. Splitting one giant prompt into a chain of small, focused steps usually beats asking for everything at once — and each small step can be tested on its own.
Here is the marker idea from the fourth point. You surround attached data with plain tags so the model can tell your instructions apart from the material it should only read, not obey:
SYSTEM: You are a support assistant. Only use facts inside
<context> tags. If the answer is not there, say you don't know.
<context>
Refunds are available within 30 days of purchase.
</context>
USER: Can I get a refund after 45 days?Because the refund rule sits inside the <context> markers and the model is told to rely only on what is there, it answers "no" instead of guessing — and if the user's text tried to sneak in a fake instruction, it would still be treated as ordinary data outside the rules.
Part 5 · Foundations
Structured outputs & the output contract
The single fastest way to make a model easier to work with is to stop treating its answer as free-flowing prose and start treating it as a fixed data record — the same kind of neat, predictable response a normal software service returns. If you require the model to reply in a strict, agreed-upon format, you turn an "anything goes" answer into something your existing code can check, act on, and test automatically. This is the bridge between the unpredictable model and the predictable software around it.
The common format for this is JSON (JavaScript Object Notation), a simple, widely used way of writing data as labeled fields — for example {"category": "billing", "urgency": 4}. The agreed-upon set of fields and their allowed values is called a schema. And the promise that the model's output will always match that schema is what people mean by an output contract.
5.1 Why a fixed shape gives you leverage
A free-form answer can only be judged by a human or by another language model — both slow and costly. A fixed-shape answer can be checked by a plain if statement in your code. Every field you lock down is a possible failure you can catch instantly, for free, with no extra model call. It is the cheapest safety check in the entire system.
In Python, the standard tool for describing a data shape is a library called Pydantic. You write a class that lists the fields you want and the rules each field must follow. Then a structured-output library uses that description to make the model return data in exactly that shape. Here is what a support-ticket sorter looks like:
from pydantic import BaseModel, Field
from typing import Literal
class SupportTriage(BaseModel):
# category must be exactly one of these four words:
category: Literal["billing", "technical", "account", "other"]
# urgency must be a whole number from 1 to 5:
urgency: int = Field(ge=1, le=5) # ge = at least, le = at most
# summary must be a string no longer than 200 characters:
summary: str = Field(max_length=200)
# needs_human must be true or false:
needs_human: bool
# The library forces the model's reply to fit the SupportTriage shape.
result: SupportTriage = triage_agent(ticket_text)
# From here it is ordinary, predictable code:
assert result.urgency in range(1, 6) # a free structural check
if result.needs_human or result.urgency >= 4:
route_to_agent(result)The moment the answer is a SupportTriage record, three good things follow. The category is guaranteed to be one of the four allowed words, so you never have to fuzzily match messy text. The urgency is a checked whole number you can safely compare against, as in urgency >= 4. And if the model ever produces something that breaks the rules — say an urgency of 9 — your code raises an error you can log and retry, all without asking the model a second time.
5.2 Three ways to enforce the output shape
There are three levels of strictness, from a polite request to a hard guarantee.
| Level | How it works | How strong it is |
|---|---|---|
| Just ask | Write "reply with JSON containing keys x and y" in the prompt. | Weak. The model usually complies but may add extra prose or break the format, especially under heavy load. |
| Check and retry | Try to read the answer into your shape; if it does not fit, send it back to the model with the error message and ask again. A library called Instructor does this for you. | Strong in practice. The cost is an occasional extra model call when the first try fails. |
| Constrained decoding | Block the model, as it writes, from choosing any next word that would break the required format. Explained just below. | A hard guarantee — the output physically cannot come out malformed. Built into serving engines like vLLM and SGLang and into the major APIs (Part 4). |
From the Transformer post: a model writes one word-piece at a time, and at each step it turns a list of raw scores — one score \(z_i\) per possible next word-piece — into probabilities using a formula called softmax:
Here \(z_i\) is the raw score for candidate word-piece \(i\), and \(p_i\) is the chance the model picks it. The bottom line just adds up over every candidate so the chances total 1. The full list of word-pieces the model can choose from is called the vocabulary.
Constrained decoding adds one step. It keeps a small tracker that knows, given the JSON written so far, which next word-pieces would keep the format valid — call that allowed set \(V_{\text{valid}}\). Right before the softmax, it sets the score \(z_i = -\infty\) for every word-piece \(i\) that is not in the allowed set. Because \(e^{-\infty}=0\), those forbidden word-pieces come out with probability exactly 0, so the model literally cannot choose them. A broken output is not merely unlikely — it is impossible. The model still spends all of its intelligence on what to say; it just does so inside a shape that is guaranteed to be well-formed.
Appendix
References & further reading
Grouped by type; all freely available online. Specific product and model details reflect publicly available information as of early 2026 and evolve quickly — verify version-specific claims against official docs.
Foundational guides & primary sources
- Prompt engineering overview. Practical guidance on system prompts, examples, and chain-of-thought. https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/overview
- Structured Outputs guide. Schema-constrained JSON generation. https://platform.openai.com/docs/guides/structured-outputs
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. https://arxiv.org/abs/2201.11903
- Efficient Guided Generation for Large Language Models. The grammar/FSM basis of constrained decoding (Outlines). https://arxiv.org/abs/2307.09702
Tools & ecosystem
- Data validation for Python. The de-facto output-contract library. https://docs.pydantic.dev/
- Structured outputs powered by LLMs. Validate-and-retry pattern. https://python.useinstructor.com/
Companion series
- Deep Learning, In Depth — Part 3: Modern Frontier LLMs. RLHF, reasoning, and test-time compute. modern-frontier-llms.html
- Deep Learning, In Depth — Part 2: The Transformer, Deep Dive. The generation loop and softmax over the vocabulary. the-transformer-deep-dive.html
This series synthesizes public documentation, primary research, and established engineering practice. Product capabilities and model specifics change frequently; treat named tools and versions as illustrative of the state of the field, not as fixed fact.