Prompt Engineering & Structured Output

In modern chat models (GPT-4o, Claude 3.5, Llama 3), you don't just send a raw block of text. You send an array of messages with roles:

• System (or Developer): Sets the persona, operational boundaries, output schema, and security rules. This anchors the model's behavior across the entire conversation.
• User: The dynamic, untrusted query or data payload submitted by the human or client application.
• Assistant: Prior turns generated by the model, or synthetic turns injected by you to pre-fill an answer.

Prompt engineering isn't about "magic incantations." It is about providing the optimal statistical scaffolding for next-token prediction:

Solve this inventory puzzle. Think step-by-step inside <scratchpad> tags before stating your final answer in <result>.

Problem: Warehouse has 48 laptops. Monday shipped 25%, Tuesday received 14, Wednesday sold 3x Tuesday's arrivals. How many laptops remain?

<scratchpad>
1. Starting inventory = 48 laptops.
2. Monday: Shipped 25% of 48 = 0.25 * 48 = 12 laptops.
   Stock after Monday = 48 - 12 = 36.
3. Tuesday: Received 14 new laptops.
   Stock after Tuesday = 36 + 14 = 50.
4. Wednesday: Sold 3 * (Tuesday's arrivals) = 3 * 14 = 42 laptops.
   Stock after Wednesday = 50 - 42 = 8 laptops.
</scratchpad>
<result>8</result>

A common backend failure is asking an LLM for JSON and receiving output that fails to parse, causing errors such as SyntaxError: Unexpected token. Why can that happen?

```json
{
  "status": "active",
  "credits": 250
}
```

Modern production APIs and open-source engines offer structured-output modes that can constrain generation to a schema. How does that reduce JSON syntax errors?

The secret is Grammar-Guided Constrained Decoding (Logit Masking). Instead of letting the model pick from all 100,000+ vocabulary tokens at every step, a Context-Free Grammar (CFG) evaluates the schema state. Tokens that would create invalid syntax have their logit score set to -∞ (probability 0).

In traditional SQL, untrusted user strings could perform SQL Injection if not parameterized (' OR 1=1;--). In LLM applications, untrusted text can perform Prompt Injection by impersonating instructions or escaping delimiters.

In modern production applications, you never hand-craft JSON schema strings. You define a Zod schema in TypeScript or a Pydantic model in Python:

import { z } from "zod";
import { zodResponseFormat } from "openai/helpers/zod";
import OpenAI from "openai";

const openai = new OpenAI();

// 1. Define your typed schema with Zod
const UserProfileSchema = z.object({
  fullName: z.string().describe("User's verified legal name"),
  email: z.string().email(),
  accountTier: z.enum(["FREE", "PRO", "ENTERPRISE"]),
  securityScore: z.number().int().min(0).max(100),
});

// 2. Pass the schema to a supported structured-output API mode
const response = await openai.beta.chat.completions.parse({
  model: "gpt-4o-mini",
  messages: [
    { role: "system", content: "Extract user profile data from the raw verification text." },
    { role: "user", content: "Name is Dr. Jane Doe, reached at jane@hospital.org, Enterprise tier, risk rating 92." },
  ],
  response_format: zodResponseFormat(UserProfileSchema, "user_profile"),
});

// 3. Parse the SDK result and handle refusals, incomplete output, and API errors
const profile = response.choices[0].message.parsed;
console.log(profile.fullName, profile.accountTier);

• Chat Roles Matter: Keep behavior anchors, constraints, and schemas in the system prompt; keep untrusted data in user wrapped in XML delimiters.
• Scratchpad Can Help: Try asking for intermediate steps on multi-step logic or arithmetic tasks, then independently check the final answer.
• Don't Rely on Raw Prompts for JSON: Fences, preambles, and trailing commas will eventually crash JSON.parse().
• Constrained Decoding: When supported, schema- or grammar-constrained decoding can prevent many format errors. Validate the result and handle API or schema failures as well.
• Reduce Injection Risk: Clearly delimit untrusted user input and retrieved context (for example, with tags), and enforce permissions and safety checks in application code. Delimiters alone are not a security boundary.