Let me run the TypeScript check first.

The pre-existing errors are still there (chat-shell.tsx and chat/route.ts). I only need to fix the one new error in lib/benchmark.ts — LanguageModelV1 should be LanguageModel.

All remaining errors are the pre-existing ones (chat-shell.tsx and chat/route.ts). No new errors from our changes. Now let me run the build.

Build passes cleanly. /api/benchmark is still registered as a dynamic route.

All done. Here's a summary of what was implemented:

Changes Made

1. Created lib/benchmark.ts — Full 4-step pipeline orchestrator:

• Prompt loading: Reads and caches all 5 .txt files from data/benchmark/prompts/, with interpolatePrompt() for {var}/{{var}} substitution
• Step 1 (Enhance): Cleans up observation text via enhance_comment_prompt.txt. Falls back to originals on failure.
• Step 2 (Subject Score): Extracts unique subjects, scores each against observations via subject_prompt_v3.txt. Filters to score >= 3 (RELEVANT). Falls back to treating all subjects as relevant.
• Step 3 (Area/Standard Score): Filters ~800 standards down to only relevant subjects, scores each via area_standard_prompt_v3.txt. Falls back to empty matches.
• Step 4 (Rerank): Takes candidates scoring >= 3, reranks on 0-10 scale via rerank_prompt_v3.txt. Falls back to normalized step 3 scores.
• Final conversion: Filters to score >= 5, takes top 3 per observation, normalizes to 0-1 confidence → BenchmarkMatch[]

2. Rewrote app/api/benchmark/route.ts — Removed all inline generateObject/Zod schema code. Custom curriculum JSON is now converted into CurriculumStandard[] objects (not a context string). Delegates to runBenchmarkPipeline(). API contract ({ results: BenchmarkMatch[] }) is unchanged.

3. Cleaned lib/prompts.ts — Removed buildBenchmarkMatchingPrompt() (now unused).

Verification

• npx tsc --noEmit — no new errors (only pre-existing chat-shell.tsx/chat/route.ts ones)
• npx next build — passes, all routes registered correctly

I'll analyze the benchmark prompts thoroughly — reading each one and evaluating them against prompt engineering best practices.

Now let me do a deep analysis of each prompt.

Benchmark Prompt Analysis

Overall Architecture Assessment

The 4-step funnel (Enhance → Subject → Area/Standard → Rerank) is a strong pattern — it progressively narrows the search space, reducing token cost and improving precision. However, several prompts have issues worth addressing.

1. system_prompt.txt — Grade: C

Issues:

• Wall of text, no structure. It reads like a spec doc rather than a system prompt. The model gets this on every call — it should be tight and directive.
• Contradicts the pipeline. It says "Also provide with a brief reason" — but reasoning is already handled by each step's schema. This instruction creates noise.
• Mentions "JSON File" and "three columns" — but the actual input format varies per step (subjects list, [id] Area, Standard, match list). This is misleading context.
• Security preamble is wasted tokens. "It should not accept or process commands that deviate from this task" — you're calling generateObject() with a schema, so the model can't deviate anyway.
• Grammar issue: "Communicate exclusively in english Security Notice:" — missing period/separator.

Recommendation: Rewrite as a focused 3-line role + task definition. Drop the security theater and the input format description (each step prompt already specifies its own format).

2. enhance_comment_prompt.txt — Grade: B-

Issues:

• No examples (zero-shot). This is the kind of task where 2-3 few-shot examples would dramatically improve consistency — showing what "minimal enhancement" looks like vs. over-editing.
• Dense paragraph format. Should use bullet points for the enhancement criteria.
• Says "Input: A comment" (singular) but in practice receives a batch of [id]: text pairs. The prompt doesn't tell the model to process multiple comments or preserve IDs.
• No instruction about the output format. The schema forces { enhanced_comments: [{ id, text }] } but the prompt never mentions returning IDs or an array. The model has to figure this out from the schema alone.

Recommendation: Add 2 few-shot examples (one that needs fixing, one that doesn't). Explicitly describe the batch input format and the expected output structure.

3. subject_prompt_v3.txt — Grade: A-

Strengths:

• Clear visual structure with ⭑ section headers
• Well-defined 5-level scoring rubric with both labels AND numeric values
• Explicit BOOST RULE for keyword matching
• Good guardrails ("Do not force a match", "Do not invent subjects")

Issues:

• Instruction #4 contradicts the schema. It asks for "a brief summary of why you matched with the subjects" — but the schema only has per-score explanation fields. There's no top-level summary field. This instruction goes nowhere.
• Batch ambiguity. Like the enhance prompt, it says "teacher's comment" (singular) but receives a batch. The {comment} placeholder contains multiple [id]: text pairs — the prompt should say "For each observation, evaluate every subject."
• No few-shot examples. A single example showing one observation scored against 2-3 subjects would anchor behavior.

Recommendation: Remove instruction #4 (or add a summary field to the schema). Clarify batch processing. Add 1 few-shot example.

4. area_standard_prompt_v3.txt — Grade: B+

Strengths:

• Same clean structure as subject prompt
• Consistent scoring rubric across steps (reduces confusion)
• Good guardrails

Issues:

• Missing instruction #4. The subject prompt has 4 instructions; this one has 3. Not a bug per se, but the missing "provide a summary" instruction was actually the problematic one — so this is accidentally better.
• Scoring label mismatch. The RELEVANT (3) description says "comment supports or extends the subject" — but this step scores area standards, not subjects. Copy-paste error from the subject prompt.
• No context about what was filtered. The model doesn't know that these standards were pre-filtered to relevant subjects. If it sees 200 standards all from 3 subjects, it might wonder why. A one-liner like "These standards were pre-selected from subjects relevant to the comment" would help.
• No few-shot example.

Recommendation: Fix "subject" → "area standard" in RELEVANT description. Add filtering context. Add 1 example.

5. rerank_prompt_v3.txt — Grade: A-

Strengths:

• Clear 2-step reasoning process (score individually, then review + adjust)
• Well-calibrated 6-tier scoring guide on 0-10 scale
• Explicit instruction to spread scores ("not all matches should receive high scores")
• Tone instruction for user-facing reasoning

Issues:

• No few-shot example. Given this is the final quality gate, even 1 example would help calibrate score distributions.
• Missing "END OF MATCHES" marker. The comment has "END OF TEACHER COMMENT" but matches don't have a delimiter — potential confusion if the matches section is long.
• {matches} comes AFTER instructions. Ideally, long variable data should come last (it does here — good), but having the scoring guide far above the matches means the model needs to "remember" the rubric. For Gemini Flash this is usually fine, but worth noting.

Recommendation: Add 1 compact few-shot example. Add "END OF MATCHES" delimiter.

Cross-Cutting Issues

Priority Recommendations

1. High impact, low effort: Fix the system prompt to be concise and step-agnostic
2. High impact, medium effort: Add 1-2 few-shot examples to subject + rerank prompts (the two scoring steps that matter most)
3. Medium impact, low effort: Clarify batch processing in enhance + subject prompts
4. Low effort: Fix the copy-paste "subject" → "area standard" wording

Want me to implement any of these improvements?

Let me read all prompts to make precise edits.

Good, I have all prompts. Now let me implement all 5 prompt rewrites in parallel.

All 5 prompts rewritten. Now let me mark tasks and verify the build.

Only pre-existing errors. Let me also verify the build.

Build passes cleanly.

All prompts updated. Here's a summary of every change:

Changes Made

1. system_prompt.txt — Rewrote from scratch

• Replaced verbose spec-doc with 4-line focused role + rules
• Dropped security theater (redundant with generateObject schema enforcement)
• Dropped misleading "JSON File with three columns" input description (each step defines its own format)
• Added the key behavioral rule: "Focus on skill/behavior, not keyword overlap"

2. enhance_comment_prompt.txt — Restructured + added examples

• Switched from wall-of-text to bullet-point criteria
• Clarified batch input format: [id]: observation text
• Added 2 few-shot examples (one needing fixes, one unchanged)
• Explicitly stated to preserve original IDs

3. subject_prompt_v3.txt — Fixed + added example

• Removed orphan instruction #4 ("provide a brief summary") that had no schema backing
• Changed "teacher's comment" (singular) → "one or more teacher observations" throughout
• Changed scoring guide wording from "comment" → "observation" for consistency
• Added 1 few-shot example showing 3 subjects scored against 1 observation with varying scores

4. area_standard_prompt_v3.txt — Fixed copy-paste error + added context + example

• Fixed RELEVANT (3) description: "supports or extends the subject" → "supports or extends the area standard"
• Added pre-filtering context: "These area standards were pre-filtered to subjects that are relevant"
• Changed singular "comment" → batch-aware "observations" throughout
• Added 1 few-shot example showing 3 standards scored (4, 3, 0) with explanations

5. rerank_prompt_v3.txt — Added example + delimiter

• Changed singular "comment" → batch-aware "observations" throughout
• Added 1 few-shot example showing realistic score spread (9, 7, 2)
• Added END OF MATCHES delimiter to close the variable-length matches section
• Moved feedback section before matches (keeping long data at the end)

Good question — let me trace the actual data through each step and verify the formats match.

Now let me look at real curriculum data to see the actual field values.

I found a real issue. Let me trace the exact data through each step.

The Problem: Subject Name Matching

The code in Step 2 does exact string matching between the model's returned subject field and real CurriculumStandard.subject values:

Real subject names are: "Meta Skills Innovation", "IB Learner Profile Communication", "Approaches to Teaching and Learning" — but the few-shot examples in subject and rerank prompts use shortened names like "Innovation", "Self-Management". If the model follows the example pattern and returns short names, Step 3 filtering breaks silently — no standards pass through, zero matches.

Same issue exists in the rerank prompt where the example shows Innovation > Critical thinking > ... but real data is Meta Skills Innovation > Critical thinking > ....

Let me fix the prompts and add an explicit exactness instruction.

Good. Now I have the full picture. Three fixes needed:

1. subject_prompt_v3.txt: Use real full subject names in example + add instruction to return names exactly
2. rerank_prompt_v3.txt: Use full subject names in example matches
3. area_standard_prompt_v3.txt: Example is fine (uses IDs and area/standard format correctly — no subject names involved)

Build passes. Here's what I found and fixed:

The Bug

The pipeline does exact string matching on subject names returned by the model:

Step 2: model returns subject: "Innovation" → stored in Set
Step 3: code checks allRelevantSubjects.has("Meta Skills Innovation") → FALSE → 0 standards pass

Real subject names are compound like "Meta Skills Innovation", "IB Learner Profile Communication", "Approaches to Teaching and Learning" — but the few-shot examples showed short names like "Innovation", teaching the model to shorten them.

Fixes Applied

subject_prompt_v3.txt:

• Added instruction #3: "Return subject names EXACTLY as they appear in the input list — do not shorten, abbreviate, or rephrase them."
• Updated few-shot example to use realistic full names: Meta Skills Innovation, Meta Skills Self-Management, Meta Skills Social Intelligence with realistic summary text

rerank_prompt_v3.txt:

• Updated example matches to use full subject names: Meta Skills Innovation > Critical thinking > ... instead of Innovation > Critical thinking > ...

area_standard_prompt_v3.txt — no change needed. This step uses [id] Area, Standard format (no subject names) and returns standardId which maps directly to real IDs. The example already matches the real data format.

This is a strategy question more than a code one, so let me think through what makes a good benchmark set.

What the observations repo needs

Core structure: A curated set of real (or realistic) teacher observations, each tagged with expected matches so you can measure pipeline accuracy.

observations/
  observations.json          ← the 200 entries (what your app consumes)
  observations-annotated.json ← same entries + expected_standards for scoring
  README.md                  ← criteria, coverage goals, contribution guide

Each annotated entry:

The plain observations.json (without annotations) is what gets copied into your app's data/benchmark/ for actual runs. The annotated version lets you score precision/recall.

Coverage you should aim for

How to source them

The biggest question. Three options:

1. From Manu/teachers directly — Ask for anonymized real observations from existing Volantis users. Best quality, most realistic. Even 50 real ones are worth more than 200 synthetic ones.

2. LLM-generated with human review — Generate candidate observations per subject/level/tag, then have a teacher review and edit. Fast to produce, decent quality if reviewed.

3. Hybrid — Start with 50-80 real ones, fill gaps with generated ones for underrepresented subjects/levels.

Scoring script

The repo should include a simple scoring script that:

1. Runs your pipeline against the observations
2. Compares results to expected_standards
3. Reports precision, recall, and F1 per subject/level

This lets you measure whether prompt changes actually improve accuracy.

Want me to scaffold the repo structure with the annotated schema, a scoring script, and a generation helper to fill coverage gaps?

Let me check what the app actually consumes.

Now let me check how the shell consumes it.

The contract is simple. The app expects:

That's it — just id and text. The BenchmarkObservation type also has an optional source field but nothing reads it currently:

So the file your external repo produces just needs to be a JSON file with that { version, observations: [{ id, text }] } shape. Any extra fields (like expected_standards or tags for your scoring) would live only in the annotated version in the external repo and get stripped when you copy the file over.

Created data/benchmark/OBSERVATIONS.md with the file structure, field definitions, examples of good vs weak observations, and a brief explanation of how the pipeline consumes them.