Structured Output vs Free Text: When Should an LLM Return JSON?
In short: Use free text when a person reads the answer. Use a JSON schema (structured output) when your code must parse fields and hand them to a screen, a database, or another API. JSON mode only means the text is valid JSON. It does not mean the fields match what you asked for. A schema fixes the shape. It does not make the values true.
What this means for you
- Chat reply a human reads → free text.
- Your code branches on fields → JSON schema, then validate.
- The model must call your function → tool calling, not a prose answer you regex.
Most broken AI features fail at the boundary between the model and the rest of the product. The sentence looked fine. The parser did not. JSON Schema is the language for describing that boundary: a declarative description of structure and constraints, checked by a validator. Providers now let you send that schema with the request so the model is constrained while it generates, instead of hoping a prompt that says "reply in JSON" is enough.
Three outputs, not two
Free text is the default chat completion. Use it for explanations, drafts, and support answers a person will read. Do not scrape fields out of a paragraph with a regular expression. The wording will change.
JSON mode is the middle step. OpenAI's structured outputs guide says JSON mode ensures valid JSON, except for edge cases you still have to handle, and it does not guarantee a schema. The same guide says to instruct the model to produce JSON, and that the API rejects the request if the context never mentions JSON. If the response hits the token limit, you can get a partial object. Treat JSON mode as "it parses," then validate the fields yourself and retry when they are missing.
Structured outputs add the schema. OpenAI calls this the evolution of JSON mode: both can produce valid JSON, and only structured outputs ensure schema adherence, on models that support them. The guide recommends structured outputs instead of JSON mode when possible. Anthropic's structured outputs describe the same idea as constrained decoding: the response follows a JSON schema you send in output_config.format, so you are not retrying because a required field was missing or a type was wrong. Schemas are a subset of JSON Schema. Check the provider's supported keywords before you paste a schema from another system.
Which shape to ask for
| Your situation | Ask for |
|---|---|
| A person reads the reply in chat | Free text |
| You render named parts in a UI (title, steps, score) | JSON schema |
| You write a row or call another API with those fields | JSON schema, then your own validator |
You only need something JSON.parse accepts | JSON mode, plus validation and a retry |
| The model must query your data or take an action | Tool or function calling with a strict input schema |
| You need both an action and a structured final answer | Strict tools and a response schema together |
OpenAI separates the last two cases. Function calling is for connecting the model to tools in your system. A response schema is for structuring what the model says back, for example so a tutoring UI can place each part of the answer in a different component. Anthropic documents the same split: JSON outputs shape the reply, strict tool use validates tool names and inputs, and you can enable both on one request. Choosing a protocol for how an agent discovers those tools is a different decision, covered in MCP vs custom API integration.
What a schema does not guarantee
OpenAI's announcement is explicit: structured outputs do not prevent every model mistake. Values inside the object can still be wrong, including a bad step in a calculation. If the model refuses an unsafe request, or if generation stops because it hit the token limit, the output may not follow the schema. Handle the refusal field and incomplete responses in code. Do not assume every 200 response is your object.
Anthropic adds a narrower caveat: enum and const strings are not guaranteed to match capitalization. A value can differ only by case, the request still completes, and there is no special stop reason. Compare those fields case-insensitively, and do not define two enum values that differ only by case.
So the schema is a contract for shape. Truth, tone, and policy still belong in evaluation. How to build that check is in LLM evaluation. Keep the schema small enough that a wrong value is obvious: a few required fields beats a nested document the model can fill with plausible junk.
A build order that stays debuggable
- Write down the one object your next system needs. Name, type, and which fields are required. That object is the schema.
- Prefer the provider's schema-constrained mode over "please return JSON" in the prompt. Keep JSON mode only where structured outputs are not available, and validate after.
- Reject or retry on refusal, truncation, and schema errors. Log the raw response when validation fails so you can see the miss.
- Map the object onto your UI or database explicitly. Do not spread model JSON straight into a table because the keys happened to match today.
- Add examples in the instructions when values are wrong even though the shape is right. OpenAI suggests examples or smaller subtasks for that class of mistake.
How long this takes depends on how many shapes you define, whether your existing schema uses keywords the provider rejects, and whether screens and storage already expect those fields. That scope is set during discovery.
Wiring the schema, the validation, and the product path around it is part of AI integration: the model call is the small piece, and the contract with the rest of the system is the work.
Frequently asked questions
What is the difference between structured output and free text?
Free text is prose a person reads. Structured output is model text constrained to a schema, usually JSON, so your code can read named fields without guessing. Use free text for chat answers. Use a schema when the next step is software: a form, a database row, or an API call.
Is JSON mode the same as structured outputs?
No. OpenAI describes JSON mode as a guarantee of valid JSON, not of a particular schema. Structured outputs are the stricter option: the response is valid JSON and it follows the schema you sent, on models that support it. OpenAI recommends structured outputs instead of JSON mode when you can use them. If you only have JSON mode, validate the object and retry when fields are missing.
When should you use tool calling instead of a response schema?
Use tool or function calling when the model must invoke your code, such as a database query or an action in your product. Use a response schema when you want the model's answer to the user, or an extracted record, in a fixed shape. OpenAI splits those two cases. Anthropic documents the same split: JSON outputs shape the reply, and strict tool use validates tool inputs. You can use both in one request.
Does a JSON schema stop the model from being wrong?
No. A schema constrains shape, types, and required fields. OpenAI says structured outputs do not prevent mistakes inside the values, such as a wrong step in a calculation. Anthropic also notes that enum strings may differ from your schema only by capitalization, so compare those values case-insensitively. Test the content with the same evaluation set you use for other model behavior.
How long does it take to add structured outputs?
It depends on how many response shapes you define, whether the schema matches a subset the provider accepts, and whether the UI or database already expects those fields. That scope is set during discovery. One extraction endpoint is a different project from schemas across a multi-step workflow.
Sources
- OpenAI — Structured model outputs — Structured outputs adhere to a JSON schema. JSON mode only guarantees valid JSON. Use function calling to reach tools, and a response schema to shape the answer.
- OpenAI — Introducing Structured Outputs — Schema match does not stop value mistakes. Refusals and hitting the token limit can mean the output does not follow the schema.
- Anthropic — Structured outputs — JSON outputs and strict tool use solve different problems. Constrained decoding targets schema-compliant JSON. Enum values may differ only by capitalization.
- JSON Schema — What is JSON Schema? — JSON Schema is a declarative language for the structure and constraints of JSON data. A validator checks an instance against that schema.
Need help putting this into practice?
Tech Programmer builds and ships this work for startups and enterprises. Tell us what you are trying to do and we will tell you what it takes.
Related reading
- MCP vs Custom API Integration: When Should an AI Agent Use Each?The Model Context Protocol (MCP) is an open standard so AI applications can discover and call tools, data, and prompts through a shared server. A custom API or in-process tool call is code your product owns: smaller tool lists, per-user auth, and tighter control. Prefer custom tools when one product agent is the only caller. Prefer MCP when several clients — Claude, ChatGPT, an IDE, an internal copilot — must reuse the same capabilities. A practical path is to ship the domain API first, then add a thin MCP server once a second real client appears.
- LLM Evaluation: How to Test AI Features Before (and After) They ShipLLM evaluation measures whether your AI application behaves as intended on realistic inputs — correctness, faithfulness, safety, and task success — not whether the text sounds fluent. Start with traces and a small golden set, combine deterministic checks with calibrated LLM-as-a-judge, gate prompt and model changes in CI, and keep production failures feeding the next regression suite.
- AI Agent Security: The Real Risks in 2026, and How to Actually Manage ThemAI agents don't usually fail because someone broke in — they fail because they were talked into doing the wrong thing, or given more access than anyone meant to grant. Here is what the 2026 breach data actually shows, and the controls that catch it before it costs you data or trust.