The Core Idea: Programs need structure, not prose. Structured outputs make the model answer in a schema you define, and tool calling lets it request actions that your code performs, as long as you validate and limit everything.
- Beginner: Why Free Text Is Not Enough
A chat answer written for a human is easy to read but hard for a program to use. Imagine your app asks a model to classify a review and the model replies Sure! I think this one is positive, probably. Your code cannot reliably pull the label out of that sentence.
Structured outputs solve this by making the model reply in a fixed format, usually JSON that follows a schema you define. Tool calling goes one step further: the model can ask your program to do something, such as look up the weather or query a database.
- Beginner: JSON and Schemas
A schema describes the shape you expect: which fields exist, their types and which are required. For example, a classification result might need a text label, a number confidence and a list of tags.
| Field | Type | Meaning |
|---|---|---|
| label | string | One of positive, neutral, negative |
| confidence | number | From 0 to 1 |
| tags | list of strings | Topics mentioned in the review |
There are three levels of strictness, from weakest to strongest:
| Method | How it works | Reliability |
|---|---|---|
| Ask nicely in the prompt | Reply with JSON only. | Usually works, sometimes fails |
| JSON mode | The provider forces syntactically valid JSON | Valid JSON, but the fields may still be wrong |
| Schema-constrained output | The provider forces output that matches your exact schema | Highest reliability |
- Medium: How Constrained Decoding Works
Schema-constrained output does not rely on the model choosing to behave. At every step the system looks at the format rules and blocks every token that would break them, by setting its probability to zero before sampling. If the schema says the next character must be a quote or a digit, no other token can be chosen.
Important: A valid format does not mean a correct answer. The JSON can be perfectly shaped and still wrong. Always check the content, not only the shape.
- Medium: Tool Calling, the Model Asks and Your Code Acts
In tool calling (also called function calling) you describe a set of tools to the model: a name, what it does, and a schema for its arguments. The model never runs anything itself. It only replies with which tool to call and with what arguments. Your program runs the tool and sends the result back.
Why this is powerful: Tools give a model fresh data, calculators, databases and actions. The model supplies the language skill, your code supplies the facts and actions.
- Advanced: Safety, Validation and Failure Handling
Anything the model returns is untrusted input, just like text typed by a stranger. Treat tool calls with the same care as a public web form:
| Risk | Defense |
|---|---|
| Invalid or missing fields | Validate every response against the schema and retry or repair on failure |
| Wrong or dangerous arguments | Check values (ranges, allowed lists) before running the tool |
| Too much power | Give each tool the minimum permission it needs. Prefer read-only tools |
| Prompt injection through tool results | Web pages or emails can contain hidden instructions. Treat tool output as data, never as commands |
| Irreversible actions | Ask the human to confirm before sending money, deleting data or emailing people |
| Endless loops | Limit the number of tool calls per request |
- Visual Explanation: The Tool Loop and the Three Gates
The figure shows one full tool call in two turns, then the three checks every response should pass before your program trusts it.
Click to see why valid is not the same as correctCompare ▼
Part 1 follows the conversation: the model requests a tool, your code runs it, the model writes the answer. Part 2 lists the gates a response must pass. Open the bar for the difference between valid and correct.
- Python Implementation: A Validator and a Tool-Calling Loop
This script validates three model replies against a small schema, then runs a complete tool-calling loop with a fake model so you can see every step without needing an API key.
import json # 1. A tiny JSON-schema style validator for structured outputsschema = {"label": str, "confidence": float, "tags": list} def validate(text): try: data = json.loads(text) except json.JSONDecodeError as e: return None, "invalid JSON: " + e.msg for key, kind in schema.items(): if key not in data: return None, "missing key: " + key if not isinstance(data[key], kind): return None, "wrong type for " + key return data, "ok" answers = [ '{"label": "positive", "confidence": 0.93, "tags": ["battery", "screen"]}', '{"label": "positive", "confidence": "high", "tags": []}', 'Sure! Here is the JSON: {"label": "positive"}',]for a in answers: data, status = validate(a) print(status.ljust(34), "|", a[:45]) # 2. A tool-calling loop: the model asks for a tool, our code runs it, the result goes backdef get_weather(city): return {"Paris": "18C, cloudy", "Delhi": "33C, sunny"}.get(city, "unknown city") def add(a, b): return a + b TOOLS = {"get_weather": get_weather, "add": add} def fake_model(messages): last = messages[-1] if last["role"] == "user": # first turn: decide to call a tool return {"tool_call": {"name": "get_weather", "arguments": {"city": "Delhi"}}} return {"text": "It is " + last["content"] + " in Delhi right now."} # second turn: answer messages = [{"role": "user", "content": "What is the weather in Delhi?"}]reply = fake_model(messages)while "tool_call" in reply: call = reply["tool_call"] result = TOOLS[call["name"]](**call["arguments"]) # we run the tool, not the model print("Model asked for", call["name"], call["arguments"], "-> result:", result) messages.append({"role": "tool", "content": result}) reply = fake_model(messages)print("Final answer:", reply["text"])Only the first reply passes. The second has the wrong type for confidence, and the third is not valid JSON because of the extra sentence. In the loop, notice that our code, not the model, runs the tool and passes the result back.
Interactive Knowledge Check
Key Points
The Big Picture
Schema + constrained decoding + validation + least-privilege tools = reliable, safe automation
With these two ideas an LLM stops being a chat box and becomes a component that other software can depend on. The same loop is the heart of AI agents.
Up Next: Long documents and long chats raise a new problem: attention gets expensive. In Long Context and Efficient Attention you will see how FlashAttention, sliding windows and other tricks make long inputs practical.