In partnership with |  |
|
Reading Time: 4 minutes |
Hey Prompt Lover, |
Seventeen newsletters in. Still March. Still inside The Prompt Report. |
Today we close Module 6 with something most prompting content never mentions. Not because it's obscure. Because most people don't realize it exists as a separate discipline. |
Here's the distinction that changed how I diagnose broken workflows. |
Prompt engineering gets AI to reason correctly toward the right answer. |
Answer engineering gets the right answer out of the output in the format you actually need. |
They fail independently. And fixing an answer engineering problem with prompt engineering changes is why a lot of prompt iteration goes nowhere. |
|
Here's What Happened |
Six months ago I built a content classification system for a client. Four categories. One correct label per piece of content. Simple enough. |
The prompt was working. The reasoning was correct. I could read the outputs and see the AI was landing on the right answer every time. |
But the output format was a mess. |
Sometimes it returned the category name. |
Sometimes the category name plus a confidence score. |
Sometimes a full sentence explaining why. Sometimes all three. Sometimes a clarifying question. |
I spent hours fixing the prompt. Better format instructions. Stricter output requirements. More examples. |
Nothing held consistently. |
Then I read the Answer Engineering section of The Prompt Report and realized I'd been solving the wrong problem entirely. One structural line at the end of the prompt fixed it in ten minutes. |
|
The Real Problem Here |
When output format is inconsistent, most people add another instruction to the prompt. |
"Return only the label." "Do not include explanations." "One word only." |
These help. They don't always hold. |
On complex tasks the model's tendency to explain overrides format instructions because explanation feels more complete than a single word. You're asking the model to suppress a tendency. It does it most of the time. Not all of the time. |
Answer engineering stops asking. It builds the extraction structurally so it works regardless of what the model writes around it. |
|
What You'll Learn In This Newsletter |
By the end of this issue, you'll have: |
• The three components of answer engineering and how each one fails |
• A structural fix that makes output format consistent without relying on instructions |
• A two-minute diagnostic for knowing which layer to fix before you touch anything |
Let's get started. |
Attio is the AI CRM for modern teams. |
|
Connect your email and calendar, and Attio instantly builds your CRM. Every contact, every company, every conversation, all organized in one place. |
Then Ask Attio anything: |
Prep for meetings in seconds with full context from across your business Know what’s happening across your entire pipeline instantly Spot deals going sideways before they do
|
No more digging and no more data entry. Just answers. |
Start your free trial → |
|
The Three Components |
Answer Shape — The form the answer takes. Single token. JSON. Binary. Number. Leave it undefined and format instructions are your only control. Define it structurally and the model has a bounded target. |
Answer Space — The domain of valid answers. Open space means anything goes. Closed space means "choose from these options only." Closed spaces produce more consistent outputs because they narrow the statistical weight toward your valid labels. |
Answer Extractor — The rule that pulls the answer from whatever the model produces. Three types: |
Regex — search for the first or last instance of a valid label in the output. Works when the answer is buried in explanation. |
Verbalizer — map model output to your labels. "+" maps to "Positive." "Great" maps to "High quality." Useful when model language doesn't match your label format. |
Separate LLM — run a second lightweight prompt to extract just the answer from a complex output. Slower. More reliable at scale. |
|
Quick Reality Check |
The research found that adding a verbalizer improved classification performance more than adding examples or improving the prompt on several benchmarks. The model was reasoning correctly the whole time. |
The answer was getting lost in translation between output and label. One mapping layer fixed it. The problem was never the prompt. |
|
The Prompt That Works |
|
▼ COPY THIS TEMPLATE: |
[Your normal prompt here — role, context, examples, reasoning instructions, all of it]
Output constraint: Your response must end with a final answer line in exactly this format:
FINAL ANSWER: [Option A / Option B / Option C / Option D]
Choose only from the options listed. No additions. No modifications. The final answer line must be the last line of your response.
[Extraction logic for automated workflows:]
From the response above, extract only the text after "FINAL ANSWER:" on the last line. Return it exactly as written.
|
|
How To Use This Prompt |
Step 1: Build your normal prompt first. Answer engineering is a layer on top, not a replacement. |
Step 2: Add the output constraint exactly as written at the end. The FINAL ANSWER anchor creates a consistent extraction point regardless of what comes before it. |
Step 3: Define your answer space explicitly. List every valid option. If it's binary, list both. If it's a scale, list all points. |
Step 4: Test ten outputs before committing. Any format failure in ten tells you where to adjust before you run it at scale. |
Here's a fully filled-in example using a content classification task: |
|
▼ EXAMPLE PROMPT — FILLED IN: |
Role: You are a content strategist who specializes in categorizing marketing content for B2B SaaS companies.
Task: Classify the following piece of content into one of four categories based on where it belongs in the buyer journey.
Categories:
Awareness — Content that introduces a problem or concept to someone who doesn't know they have it yet
Consideration — Content that helps someone evaluate solutions to a problem they've already identified
Decision — Content that helps someone choose between specific vendors or products
Retention — Content written for existing customers to help them get more value from a product they already use
Example 1: Input: "5 Signs Your Team Is Losing Hours to Manual Data Entry" Classification: Awareness Reasoning: The reader doesn't know they have a data entry problem yet. This content names and frames it.
Example 2: Input: "How to Evaluate Project Management Tools for Remote Teams" Classification: Consideration Reasoning: The reader already knows they need a solution. This helps them assess their options.
Now classify this: Input: "Why [Company Name] Customers Renew at 94% — And What That Means For Your Team"
Walk through your reasoning before giving your final answer. Identify which category fits and why the others don't.
Output constraint: Your response must end with a final answer line in exactly this format:
FINAL ANSWER: Awareness / Consideration / Decision / Retention
Choose only from the options listed. No additions. No modifications. The final answer line must be the last line of your response.
|
|
The Diagnostic Worth Running First |
Before touching anything, run this check on three outputs from your broken prompt. |
Find the answer in each output. Is the AI reasoning toward the right answer but formatting it wrong? |
Yes → Answer engineering problem. Add the structural anchor. |
No → Prompt engineering problem. Fix the prompt first, then add the anchor. |
Both wrong → Fix the prompt first. A well-engineered extraction layer on top of wrong reasoning produces wrong answers faster and more consistently. |
Two minutes. Saves hours of fixing the wrong layer. |
|
The Bigger Lesson Here |
When output is inconsistent, most people assume prompt problem. Sometimes it is. Sometimes the reasoning is fine and the extraction is broken. |
Check the reasoning before you touch the prompt. If the answer is right and the format is wrong, you don't need a better prompt. You need a structural anchor the extraction logic can find every time. |
Different problem. Different fix. Stop mixing them up. |
|
Try This Right Now |
Find one task where AI output format is inconsistent. Classification, scoring, labeling, anything requiring a specific structured answer. |
Run the diagnostic. Three outputs. Is the reasoning right? |
If yes, add the FINAL ANSWER anchor. Test ten outputs. Watch the consistency change. |
|
What's Coming Next |
Module 6 is done. |
Next we move into Module 7 and start with one of the most counterintuitive findings in the entire paper. |
English prompts consistently outperform prompts written in the task language — even when the output needs to be in a completely different language. Not sometimes. Consistently. Across multiple models. |
If you work with non-English content at all, next newsletter is not one to miss. |
Module 7 starts Monday. |
|
Reply With Your Results |
Run the diagnostic this week and reply with what you found. Prompt problem or extraction problem. And whether the anchor fixed it. |
I read every reply. |
— Prompt Guy |
