Structured Outputs and Tool Calling

Learn how to get reliable JSON from a model with schemas and constrained decoding, how tool calling lets a model request actions that your code runs, and how to validate and secure everything the model returns.

28 minIntermediateCode Examples

The Core Idea: Programs need structure, not prose. Structured outputs make the model answer in a schema you define, and tool calling lets it request actions that your code performs, as long as you validate and limit everything.

  1. Beginner: Why Free Text Is Not Enough

A chat answer written for a human is easy to read but hard for a program to use. Imagine your app asks a model to classify a review and the model replies Sure! I think this one is positive, probably. Your code cannot reliably pull the label out of that sentence.

Structured outputs solve this by making the model reply in a fixed format, usually JSON that follows a schema you define. Tool calling goes one step further: the model can ask your program to do something, such as look up the weather or query a database.

Free textHard to parse
Reply: Sure! It seems positive.
Problem: Your code must guess where the answer is.
StructuredEasy to parse
Reply: a JSON object with label and confidence
Benefit: Your code reads fields directly.

  1. Beginner: JSON and Schemas

A schema describes the shape you expect: which fields exist, their types and which are required. For example, a classification result might need a text label, a number confidence and a list of tags.

FieldTypeMeaning
labelstringOne of positive, neutral, negative
confidencenumberFrom 0 to 1
tagslist of stringsTopics mentioned in the review

There are three levels of strictness, from weakest to strongest:

MethodHow it worksReliability
Ask nicely in the promptReply with JSON only.Usually works, sometimes fails
JSON modeThe provider forces syntactically valid JSONValid JSON, but the fields may still be wrong
Schema-constrained outputThe provider forces output that matches your exact schemaHighest reliability

  1. Medium: How Constrained Decoding Works

Schema-constrained output does not rely on the model choosing to behave. At every step the system looks at the format rules and blocks every token that would break them, by setting its probability to zero before sampling. If the schema says the next character must be a quote or a digit, no other token can be chosen.

1
Model scores tokens: As usual, it scores every token in the vocabulary.
2
Grammar filter: Tokens that would break the schema are removed.
3
Sample from what is left: Only valid tokens remain, so the result is always well-formed.

Important: A valid format does not mean a correct answer. The JSON can be perfectly shaped and still wrong. Always check the content, not only the shape.

  1. Medium: Tool Calling, the Model Asks and Your Code Acts

In tool calling (also called function calling) you describe a set of tools to the model: a name, what it does, and a schema for its arguments. The model never runs anything itself. It only replies with which tool to call and with what arguments. Your program runs the tool and sends the result back.

1
You describe the tools: For example get_weather(city).
2
User asks: What is the weather in Delhi?
3
Model requests a call: It returns get_weather with city set to Delhi.
4
Your code runs it: You call your real weather function.
5
Result goes back: The model reads the result and writes the final answer.

Why this is powerful: Tools give a model fresh data, calculators, databases and actions. The model supplies the language skill, your code supplies the facts and actions.

  1. Advanced: Safety, Validation and Failure Handling

Anything the model returns is untrusted input, just like text typed by a stranger. Treat tool calls with the same care as a public web form:

RiskDefense
Invalid or missing fieldsValidate every response against the schema and retry or repair on failure
Wrong or dangerous argumentsCheck values (ranges, allowed lists) before running the tool
Too much powerGive each tool the minimum permission it needs. Prefer read-only tools
Prompt injection through tool resultsWeb pages or emails can contain hidden instructions. Treat tool output as data, never as commands
Irreversible actionsAsk the human to confirm before sending money, deleting data or emailing people
Endless loopsLimit the number of tool calls per request
Parallel tool calls
Some models can ask for several independent tools at once, for example the weather in two cities. Your code can run them together and return all the results in one go.

  1. Visual Explanation: The Tool Loop and the Three Gates

The figure shows one full tool call in two turns, then the three checks every response should pass before your program trusts it.

Part 1 · The Tool-Calling LoopThe model asks, your code acts
Turn 1 · DecideModel
User: weather in Delhi?
↓
Model sees tool list: get_weather(city)
↓
Reply: call get_weather with city = Delhi
Turn 2 · AnswerModel again
Your code ran the tool
↓
Result sent back: 33C, sunny
↓
Final text: It is 33C and sunny in Delhi
Part 2 · Three Gates for Safe OutputNever trust, always check
Gate 1 · Parse
Valid JSON?
If not, retry or ask the model to fix it.
Gate 2 · Validate
Matches the schema?
Correct fields, types and allowed values.
Gate 3 · Authorize
Allowed to run?
Check permissions and confirm risky actions.
Click to see why valid is not the same as correctCompare ▼
A schema guarantees shape, such as a number between 0 and 1. It cannot guarantee that the number is the right one. Add checks that use real data, for example confirming that a returned product id exists.

Part 1 follows the conversation: the model requests a tool, your code runs it, the model writes the answer. Part 2 lists the gates a response must pass. Open the bar for the difference between valid and correct.

  1. Python Implementation: A Validator and a Tool-Calling Loop

This script validates three model replies against a small schema, then runs a complete tool-calling loop with a fake model so you can see every step without needing an API key.

structured_output_and_tools.pyPython 3.10+ · standard library only
import json # 1. A tiny JSON-schema style validator for structured outputsschema = {"label": str, "confidence": float, "tags": list} def validate(text):    try:        data = json.loads(text)    except json.JSONDecodeError as e:        return None, "invalid JSON: " + e.msg    for key, kind in schema.items():        if key not in data:            return None, "missing key: " + key        if not isinstance(data[key], kind):            return None, "wrong type for " + key    return data, "ok" answers = [    '{"label": "positive", "confidence": 0.93, "tags": ["battery", "screen"]}',    '{"label": "positive", "confidence": "high", "tags": []}',    'Sure! Here is the JSON: {"label": "positive"}',]for a in answers:    data, status = validate(a)    print(status.ljust(34), "|", a[:45]) # 2. A tool-calling loop: the model asks for a tool, our code runs it, the result goes backdef get_weather(city):    return {"Paris": "18C, cloudy", "Delhi": "33C, sunny"}.get(city, "unknown city") def add(a, b):    return a + b TOOLS = {"get_weather": get_weather, "add": add} def fake_model(messages):    last = messages[-1]    if last["role"] == "user":                           # first turn: decide to call a tool        return {"tool_call": {"name": "get_weather", "arguments": {"city": "Delhi"}}}    return {"text": "It is " + last["content"] + " in Delhi right now."}   # second turn: answer messages = [{"role": "user", "content": "What is the weather in Delhi?"}]reply = fake_model(messages)while "tool_call" in reply:    call = reply["tool_call"]    result = TOOLS[call["name"]](**call["arguments"])    # we run the tool, not the model    print("Model asked for", call["name"], call["arguments"], "-> result:", result)    messages.append({"role": "tool", "content": result})    reply = fake_model(messages)print("Final answer:", reply["text"])
EXPECTED OUTPUT (your numbers may differ slightly)
ok | {"label": "positive", "confidence": 0.93, "ta
wrong type for confidence | {"label": "positive", "confidence": "high", "
invalid JSON: Expecting value | Sure! Here is the JSON: {"label": "positive"}
Model asked for get_weather {'city': 'Delhi'} -> result: 33C, sunny
Final answer: It is 33C, sunny in Delhi right now.

Only the first reply passes. The second has the wrong type for confidence, and the third is not valid JSON because of the extra sentence. In the loop, notice that our code, not the model, runs the tool and passes the result back.

Interactive Knowledge Check

Question 1: In tool calling, who actually runs the tool?
Question 2: A reply is valid JSON that matches your schema. Can you fully trust it?

Key Points

✓Structured outputs make model replies machine-readable, usually as JSON that follows a schema.
✓Strictness levels: prompt only, JSON mode, then schema-constrained decoding (most reliable).
✓Constrained decoding blocks tokens that would break the schema before sampling.
✓In tool calling the model only requests a call. Your code runs the tool and returns the result.
✓Treat model output as untrusted: validate shape and values, limit permissions, confirm risky actions.
✓Tool results can contain injected instructions, so treat them as data and not as commands.

The Big Picture

Structured Output in One Line

Schema + constrained decoding + validation + least-privilege tools = reliable, safe automation

With these two ideas an LLM stops being a chat box and becomes a component that other software can depend on. The same loop is the heart of AI agents.

Up Next: Long documents and long chats raise a new problem: attention gets expensive. In Long Context and Efficient Attention you will see how FlashAttention, sliding windows and other tricks make long inputs practical.