Introduction
Prompt engineering is often described as an art, but for anyone shipping large language model (LLM) features into production, it behaves a lot more like an engineering discipline: it has failure modes, measurable trade-offs, and techniques that compose. The same model, given the same underlying task, can swing from unusable to production-grade depending entirely on how the input is structured. That gap is not a quirk - it is the direct consequence of how these models are trained, and understanding it is what separates people who "get lucky with prompts" from people who can systematically debug and improve them.
This article walks through the core techniques that make up the modern prompt engineering toolkit: basic prompting (zero-shot and few-shot), instruction prompting, self-consistency sampling, chain-of-thought reasoning (in both its zero-shot and few-shot forms), tree of thoughts, structured output generation, system prompt design, and automatic prompt design. Each section explains the mechanism behind the technique, when to reach for it, and how to implement it with real code. The goal is not to catalog tricks, but to build a mental model of why these techniques work, so you can reason about new problems rather than just copying patterns.
Why Prompting Is a Real Engineering Problem
It helps to start with what a prompt actually does mechanically. An LLM is a next-token predictor trained on a huge distribution of text, then fine-tuned (via supervised fine-tuning and reinforcement learning from human feedback) to follow instructions and behave conversationally. When you write a prompt, you are not "asking a question" in the way you would ask a colleague - you are conditioning a probability distribution over the next tokens. The model has no persistent goal, no internal plan, and no guarantee of consistency across two runs unless you engineer the input and the sampling process to constrain it. Every technique in this article is, at its core, a way of shaping that conditioning so the resulting distribution puts more probability mass on the outputs you actually want.
This matters because naive prompting fails in predictable ways. Ambiguous instructions produce plausible-but-wrong answers because the model fills gaps with the most statistically common completion, not the one that matches your specific intent. Multi-step reasoning tasks fail because the model, absent instruction, tends to jump straight to an answer, and that shortcut path is often lower quality than a reasoning path would be. Format requirements fail because "natural" text generation does not respect strict schemas unless you push it to. Prompt engineering techniques exist because each of these failure modes has a structural fix - not a magic phrase, but a change in how information is presented to the model.
There is also a systems dimension to this. In a production pipeline, a prompt is a piece of infrastructure: it has inputs, outputs, versioning, regression tests, and cost/latency implications. A chain-of-thought prompt that improves accuracy by 15% but triples token usage might be the wrong trade-off for a latency-sensitive endpoint but the right one for a batch analysis job. Framing prompting as an engineering surface - with the same rigor you'd apply to an API contract - is the mindset this article assumes throughout.
Basic Prompting: Zero-Shot and Few-Shot
Zero-shot prompting is the simplest possible interaction: you describe the task in natural language and let the model produce a result without showing it any examples. This works because instruction-tuned models have seen enormous numbers of task descriptions paired with correct completions during training, so a clear task description alone can activate the right "mode" of behavior. Zero-shot is attractive because it is cheap - no extra tokens spent on examples - and it generalizes well to novel task phrasings. Its weakness shows up on tasks with subtle output conventions: formatting quirks, domain-specific terminology, or edge cases that a plain description does not fully disambiguate.
Few-shot prompting addresses that weakness by including a small number of input-output examples directly in the prompt before the actual query, a technique popularized at scale by the GPT-3 paper ("Language Models are Few-Shot Learners," Brown et al., 2020), which showed that large models could perform new tasks from a handful of demonstrations without any weight updates. The examples act as an implicit specification: they show format, tone, level of detail, and edge-case handling that would be tedious or ambiguous to describe in prose. In practice, few-shot prompting is most valuable when the desired output has structural regularities - a specific JSON shape, a particular citation style, a consistent classification taxonomy - that are far easier to demonstrate than to explain.
The trade-off is token cost and a phenomenon worth watching for: example ordering and selection bias. Models can be sensitive to the order in which examples appear and can overfit to superficial patterns in the examples (for instance, always predicting the label that appeared most recently) rather than the underlying task logic. This is why production few-shot systems typically randomize or rotate example sets across calls and validate that accuracy is not an artifact of example placement rather than genuine task understanding.
# Zero-shot vs. few-shot prompting with the Anthropic API
import anthropic
client = anthropic.Anthropic()
def classify_zero_shot(review: str) -> str:
response = client.messages.create(
model="claude-sonnet-4-5",
max_tokens=10,
messages=[{
"role": "user",
"content": f"Classify the sentiment of this review as positive, negative, or neutral. "
f"Respond with a single word.\n\nReview: {review}"
}]
)
return response.content[0].text.strip()
def classify_few_shot(review: str) -> str:
examples = (
"Review: 'This blender broke after two uses.' -> negative\n"
"Review: 'Exactly what I expected, does the job.' -> neutral\n"
"Review: 'Best purchase I've made this year!' -> positive\n"
)
response = client.messages.create(
model="claude-sonnet-4-5",
max_tokens=10,
messages=[{
"role": "user",
"content": f"{examples}\nReview: '{review}' ->"
}]
)
return response.content[0].text.strip()
Instruction Prompting and System Prompt Variants
Instruction prompting is the practice of writing the task specification the way you would write a well-scoped ticket for a competent but literal-minded engineer: explicit about the goal, the constraints, the output format, and the things to avoid. The difference between "summarize this" and "summarize this in three bullet points, each under 20 words, focused on financial impact only" is the difference between a prompt that samples from a huge space of plausible summaries and one that constrains the model to the narrow region you actually need. Well-constructed instructions typically separate four things: the role or persona, the task itself, any constraints or exclusions, and the exact output format. Keeping these as distinct, labeled sections (rather than blending them into a single paragraph) measurably reduces the model's tendency to drop a requirement.
The system prompt is a specific implementation of instruction prompting: a message slot, distinct from the user turn, that sets standing behavior for the entire conversation rather than a single request. Anthropic's and OpenAI's chat APIs both expose a system role for exactly this reason - it lets you separate "who you are and how you behave" from "what the user is asking right now," which keeps per-turn prompts simpler and makes behavior more consistent across many user inputs. System prompt variants generally fall into a few recognizable patterns: persona framing ("You are a senior security auditor reviewing pull requests"), guardrail framing ("Never suggest network calls in code samples; flag them as TODOs instead"), and output-contract framing ("Always respond with valid JSON matching this schema, and nothing else"). Combining these in a single system prompt is common, but each pattern should be reasoned about separately, because they fail differently: a weak persona produces bland output, a weak guardrail produces occasional policy violations, and a weak output contract produces malformed responses that break downstream parsers.
A subtlety worth calling out is that system prompts are not magically more obeyed than user-turn instructions - instruction-following strength depends on the specific model and how it was fine-tuned, and can be probabilistically overridden by sufficiently persistent or adversarial user input. Engineering teams building anything user-facing should treat the system prompt as a strong prior, not an unbreakable contract, and pair it with output-side validation.
Chain-of-Thought Reasoning and Self-Consistency
Chain-of-thought (CoT) prompting, introduced in "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" (Wei et al., 2022), is based on a simple but consequential observation: for tasks that require multi-step reasoning - arithmetic, logic puzzles, multi-hop question answering - models produce substantially better answers when they are prompted to generate intermediate reasoning steps before the final answer, rather than jumping straight to a conclusion. The mechanism is intuitive once you remember that these models generate left to right: if the correct answer depends on information that only becomes explicit partway through a reasoning chain, forcing that chain to be written out gives the model "access" to its own intermediate conclusions as conditioning context for the next token, in a way that a single-shot answer cannot replicate.
Few-shot CoT supplies this behavior via worked examples: each demonstration in the prompt shows not just the input and final answer, but the reasoning steps connecting them, teaching the model the expected reasoning style through demonstration rather than instruction. Zero-shot CoT, described in "Large Language Models are Zero-Shot Reasoners" (Kojima et al., 2022), showed something more surprising: simply appending a phrase like "Let's think step by step" to a zero-shot prompt - with no worked examples at all - substantially improves performance on many reasoning benchmarks. This works because the phrase steers the model into a "reasoning mode" that its instruction-tuning data associates with explicit, sequential problem-solving, without needing to spend tokens on full demonstrations.
Self-consistency sampling, from "Self-Consistency Improves Chain of Thought Reasoning in Language Models" (Wang et al., 2022), builds on CoT by addressing its main weakness: any single reasoning chain can go wrong, and there is no way to know from one sample alone whether it did. The technique samples multiple independent CoT completions for the same question at a non-zero temperature, then takes the majority-vote answer across them, treating the final answers (not the reasoning text itself) as votes. The intuition is that wrong reasoning paths tend to diverge in different, idiosyncratic ways, while correct reasoning paths tend to converge on the same answer even when the intermediate steps are phrased differently - so majority voting filters out noise. This is a genuine accuracy/cost trade-off: sampling five or ten completions instead of one multiplies token cost roughly proportionally, so self-consistency is best reserved for tasks where correctness matters more than latency or spend, such as offline evaluation, high-stakes classification, or math and logic tasks in an agent's planning step.
# Self-consistency sampling: majority vote over independent CoT completions
from collections import Counter
import re
import anthropic
client = anthropic.Anthropic()
def cot_sample(question: str, temperature: float = 0.8) -> str:
response = client.messages.create(
model="claude-sonnet-4-5",
max_tokens=500,
temperature=temperature,
messages=[{
"role": "user",
"content": f"{question}\n\nLet's think step by step, then give the final "
f"numeric answer on its own line prefixed with 'ANSWER:'."
}]
)
return response.content[0].text
def extract_answer(text: str) -> str | None:
match = re.search(r"ANSWER:\s*(.+)", text)
return match.group(1).strip() if match else None
def self_consistency(question: str, n_samples: int = 7) -> str:
answers = []
for _ in range(n_samples):
completion = cot_sample(question)
answer = extract_answer(completion)
if answer:
answers.append(answer)
if not answers:
raise ValueError("No valid answers extracted from any sample.")
return Counter(answers).most_common(1)[0][0]
Tree of Thoughts: Extending Chain-of-Thought
Chain-of-thought assumes reasoning is fundamentally linear: one thought leads to the next, and the model either gets the chain right or wrong. Tree of Thoughts (ToT), introduced in "Tree of Thoughts: Deliberate Problem Solving with Large Language Models" (Yao et al., 2023), generalizes this by treating reasoning as a search problem over a tree of intermediate "thoughts," where each node is a partial solution state. At each step, the model generates several candidate next thoughts (branching), a separate evaluation step scores or compares those candidates, and a search strategy - breadth-first search, depth-first search, or a simple beam search - decides which branches to keep exploring and which to prune. This turns the model from a single-pass generator into the propose-and-evaluate engine at the center of an explicit search algorithm, which is a meaningfully different architecture even though it reuses the same underlying LLM calls as CoT.
The practical payoff shows up on tasks where a single reasoning path is prone to getting locally stuck - puzzles with backtracking requirements, creative writing with multiple viable directions, or planning tasks where an early wrong commitment invalidates everything downstream. The Game of 24 and Mini Crosswords tasks used in the original ToT paper are good illustrations: both have a large branching factor early on, and greedy single-path CoT frequently commits to a promising-looking first move that turns out to be a dead end several steps later, whereas ToT's evaluation-and-pruning step catches that earlier. The cost of this power is substantial: because you are running multiple generate-and-evaluate cycles per problem instead of one, ToT can require an order of magnitude more model calls than a single CoT prompt, so it is best reserved for problems with a real combinatorial structure, not applied as a default reasoning upgrade.
Structured Output Generation
Structured output prompting exists to solve an integration problem: downstream code almost never wants free-form prose, it wants a JSON object, a specific enum value, or a row that fits a database schema, and getting an LLM to reliably emit exactly that shape is a distinct skill from getting it to reason well. The most robust approach layers three things together: an explicit schema description in the prompt (ideally an actual JSON Schema or a Pydantic/Zod model definition, not just a prose description), a strong instruction that the output must contain nothing but the structured payload, and - wherever the API supports it - a model-level constraint such as Anthropic's and OpenAI's native structured-output or tool-use modes, which constrain the sampling process itself rather than relying purely on instruction-following.
Tool use (also called function calling) deserves particular attention here because it is the most reliable structured-output mechanism available on modern LLM APIs: rather than asking the model to produce JSON as free text and hoping it is well-formed, you define a tool with a strict input schema, and the API's decoding process is constrained to only emit arguments matching that schema. This eliminates an entire class of failures - trailing commentary, markdown code fences wrapping the JSON, minor schema violations - that plague pure prompt-based JSON generation. Even when using this mechanism, it remains good practice to validate the returned payload against your schema in code (using something like Pydantic in Python or Zod in TypeScript) rather than trusting it blindly, since edge cases and API version differences can still produce payloads that are syntactically valid JSON but semantically wrong for your business logic.
// Structured output via tool use, validated with Zod on the way out
import Anthropic from "@anthropic-ai/sdk";
import { z } from "zod";
const client = new Anthropic();
const TicketSchema = z.object({
priority: z.enum(["low", "medium", "high", "critical"]),
category: z.string(),
summary: z.string().max(200),
requiresEscalation: z.boolean(),
});
async function triageTicket(ticketText: string) {
const response = await client.messages.create({
model: "claude-sonnet-4-5",
max_tokens: 300,
tools: [{
name: "classify_ticket",
description: "Classify a support ticket into a structured triage record.",
input_schema: {
type: "object",
properties: {
priority: { type: "string", enum: ["low", "medium", "high", "critical"] },
category: { type: "string" },
summary: { type: "string", maxLength: 200 },
requiresEscalation: { type: "boolean" },
},
required: ["priority", "category", "summary", "requiresEscalation"],
},
}],
tool_choice: { type: "tool", name: "classify_ticket" },
messages: [{ role: "user", content: ticketText }],
});
const toolUse = response.content.find((block) => block.type === "tool_use");
if (!toolUse || toolUse.type !== "tool_use") {
throw new Error("Model did not return a tool_use block.");
}
return TicketSchema.parse(toolUse.input);
}
A second, lighter-weight pattern worth knowing is delimiter-based extraction: asking the model to wrap the exact payload you need inside unambiguous markers (like <answer>...</answer>) even within an otherwise free-text response. This is less strict than tool use but is useful when you deliberately want the model to show reasoning and a final structured answer in the same response, and you will parse out just the delimited section downstream.
Automatic Prompt Design
Every technique so far assumes a human is writing and iterating on the prompt. Automatic Prompt Engineer (APE), from "Large Language Models Are Human-Level Prompt Engineers" (Zhou et al., 2022), reframes prompt writing itself as an optimization problem that an LLM can help solve. The core loop has three stages. First, given a small set of example input-output pairs for a task, an LLM is used as an "inference model" to propose a pool of candidate instructions that could plausibly explain the mapping from inputs to outputs - effectively asking the model to reverse-engineer what instruction would have produced these examples. Second, each candidate instruction is scored by using it to prompt a model on a held-out set of the same task's examples and measuring how well the resulting outputs match the expected outputs, using a chosen scoring function such as exact-match accuracy, execution accuracy for code tasks, or a log-probability-based proxy score that avoids needing to generate a full completion for every candidate. Third, the highest-scoring candidates are selected, and optionally refined further through an iterative variant that generates paraphrases or mutations of the best-performing candidates and re-scores them, similar in spirit to a evolutionary search.
The practical value of this approach is that it turns "prompt tuning" from a manual, intuition-driven process into something closer to hyperparameter search, which is valuable when you have a task with enough labeled examples to score candidates meaningfully and enough scale (many downstream calls) to justify the upfront optimization cost. It also surfaces genuinely non-obvious instructions: the original APE paper found automatically discovered prompts that outperformed prompts written by human experts on several benchmark tasks, precisely because the search process isn't biased by the same linguistic habits a human prompt engineer brings to the table. The approach has clear limits, though - it requires a labeled dataset and a well-defined scoring function, which many real-world tasks (open-ended writing, subjective quality judgments, tasks without a clean "correct answer") do not have, and the search itself consumes a nontrivial number of model calls, so it is best treated as an offline optimization step run once per task rather than something done per request.
# Simplified automatic prompt design loop: propose candidates, score, select best
import anthropic
client = anthropic.Anthropic()
def propose_candidates(examples: list[tuple[str, str]], n: int = 8) -> list[str]:
formatted = "\n".join(f"Input: {i}\nOutput: {o}" for i, o in examples)
response = client.messages.create(
model="claude-sonnet-4-5",
max_tokens=600,
messages=[{
"role": "user",
"content": (
f"Here are input-output examples for a task:\n\n{formatted}\n\n"
f"Propose {n} distinct candidate instructions that, if given to a "
f"language model, would plausibly produce this input-to-output mapping. "
f"List them as a numbered list, one per line, no extra commentary."
)
}]
)
lines = response.content[0].text.strip().split("\n")
return [line.split(".", 1)[-1].strip() for line in lines if line.strip()]
def score_candidate(instruction: str, held_out: list[tuple[str, str]]) -> float:
correct = 0
for inp, expected in held_out:
response = client.messages.create(
model="claude-sonnet-4-5",
max_tokens=100,
messages=[{"role": "user", "content": f"{instruction}\n\nInput: {inp}"}]
)
if response.content[0].text.strip().lower() == expected.strip().lower():
correct += 1
return correct / len(held_out)
def automatic_prompt_search(train: list[tuple[str, str]], held_out: list[tuple[str, str]]):
candidates = propose_candidates(train)
scored = [(c, score_candidate(c, held_out)) for c in candidates]
return max(scored, key=lambda pair: pair[1])
Trade-offs and Common Pitfalls
The techniques in this article are not free - every one of them trades some combination of latency, token cost, implementation complexity, and reliability for improved accuracy or control, and picking the wrong point on that trade-off curve is the most common real-world mistake. Chain-of-thought and self-consistency inflate token usage substantially, sometimes by an order of magnitude when you factor in multiple samples; applying them to a simple classification task that a well-written zero-shot or few-shot prompt already handles correctly is pure waste. Tree of Thoughts compounds this further, since it multiplies the number of model calls by the branching factor and search depth, and teams sometimes reach for it because it sounds powerful without first confirming that their task actually has the combinatorial, backtracking-prone structure it is designed for.
A second pitfall is treating prompt engineering as a one-shot activity rather than something requiring regression testing. Prompts are code, and like code, small changes can have non-local effects: rewording an instruction to fix one failure case can silently break a different case that was previously working, especially with few-shot examples where reordering or swapping a single example can shift accuracy on unrelated inputs. Teams that skip building even a small held-out evaluation set for their prompts tend to discover this the hard way, in production, after a "minor tweak."
A third, subtler pitfall is over-trusting structured output guarantees. Even with tool use or JSON mode constraining the model's syntax, nothing constrains the model's semantic correctness - it can return perfectly valid JSON with a hallucinated field value, an internally inconsistent set of fields, or a confident answer to a question it should have flagged as unanswerable. Structured output solves the parsing problem, not the correctness problem, and conflating the two leads to downstream systems that fail silently because "the JSON parsed fine" was mistaken for "the answer was right."
Finally, there's a false economy in over-engineering prompts for edge cases that a better system architecture would solve instead. Elaborate prompt gymnastics to make a model reliably do arithmetic, for instance, will generally lose to simply giving the model a calculator tool and letting it delegate that subtask - prompt engineering is one lever among several (retrieval, tool use, fine-tuning, validation layers) and should be weighed against those alternatives rather than treated as the only available fix.
Best Practices for Applying These Techniques
Start every non-trivial prompting task by writing down, in plain language, exactly what a correct answer looks like and what a subtly wrong answer looks like - this single step surfaces most of the ambiguity that later causes model failures, and it gives you the seed of an evaluation set before you write a single line of prompt. From there, default to the cheapest technique that could plausibly work: a clear zero-shot instruction first, few-shot examples if format or edge-case handling is the bottleneck, and chain-of-thought only once you've confirmed the task genuinely requires multi-step reasoning rather than better instructions. Escalating in this order keeps token costs proportional to actual task difficulty instead of applying maximum machinery everywhere out of habit.
Treat your prompts as versioned artifacts with tests, the same way you'd treat a database migration or an API contract. Keep a small but representative held-out set of inputs with known-good outputs, and re-run it whenever you change a prompt, a model version, or a sampling parameter - this catches the silent regressions described in the pitfalls section before they reach production. For anything with a hard reliability requirement, pair prompt-level techniques with code-level validation: schema validation on structured output, sanity checks on numeric ranges, and explicit fallback behavior (retry with a different prompt, escalate to a human, return a "low confidence" flag) when validation fails, rather than assuming the model got it right because it responded fluently.
Reserve self-consistency and tree-of-thought search for the specific cases they were designed for - tasks with meaningful answer variance across samples, or a genuine branching solution space - and measure the accuracy gain against the multiplied cost before committing to them in a production path rather than an offline evaluation. Finally, if you are optimizing a high-volume, well-labeled task, treat automatic prompt design as a legitimate offline engineering step, not a novelty: running an APE-style search once and locking in the winning instruction can outperform hours of manual prompt tweaking, and it produces an artifact (the scored candidate) that is easier to justify and reproduce than "a prompt someone iterated on until it felt right."
Key Takeaways
- Start with the cheapest technique that could work - zero-shot, then few-shot - and only escalate to chain-of-thought or tree-of-thoughts once you've confirmed the task genuinely needs multi-step reasoning or branching search.
- Use tool use / function-calling schemas for structured output instead of asking for free-text JSON; validate the result in code regardless, since syntactic validity does not imply semantic correctness.
- Reserve self-consistency sampling for tasks where correctness matters more than latency or cost, since it multiplies token usage roughly linearly with the number of samples.
- Version your prompts and maintain a held-out evaluation set - prompts are code, and small wording changes can silently regress cases that previously worked.
- Consider automatic prompt design (APE-style search) for high-volume, well-labeled tasks where the upfront cost of generating and scoring candidates is justified by the number of downstream calls that will reuse the winning prompt.
Conclusion
Prompt engineering, stripped of its mystique, is the discipline of shaping the conditioning input to a next-token predictor so that its output distribution reliably lands where you need it. Zero-shot and few-shot prompting establish the baseline; instruction and system prompts add explicit structure and standing behavior; chain-of-thought and self-consistency address multi-step reasoning and its inherent variance; tree of thoughts extends that into genuine search over a solution space; structured output makes the results usable by the rest of your system; and automatic prompt design turns the entire process into something that can, in the right conditions, be optimized rather than hand-tuned.
None of these techniques is a substitute for good system design. The engineers who get the most value from LLMs treat prompting as one component in a larger architecture - alongside retrieval, tool use, validation, and evaluation - and choose each technique deliberately, based on the actual failure mode they're trying to fix, rather than reaching for the most sophisticated-sounding option by default. That deliberate, trade-off-aware approach is what turns prompt engineering from trial-and-error into an engineering practice you can actually reason about, test, and improve over time.
References
- Brown, T. et al. (2020). Language Models are Few-Shot Learners. arXiv:2005.14165.
- Wei, J. et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv:2201.11903.
- Kojima, T. et al. (2022). Large Language Models are Zero-Shot Reasoners. arXiv:2205.11916.
- Wang, X. et al. (2022). Self-Consistency Improves Chain of Thought Reasoning in Language Models. arXiv:2203.11171.
- Yao, S. et al. (2023). Tree of Thoughts: Deliberate Problem Solving with Large Language Models. arXiv:2305.10601.
- Zhou, Y. et al. (2022). Large Language Models Are Human-Level Prompt Engineers. arXiv:2211.01910.
- Anthropic. Prompt Engineering Overview. docs.claude.com/en/docs/build-with-claude/prompt-engineering/overview.
- Anthropic. Tool Use (Function Calling) Documentation. docs.claude.com.
- Pydantic documentation. docs.pydantic.dev.
- Zod documentation. zod.dev.