I'll start by reading the key files I need to understand before implementing.

Now let me also look at the proposals route to understand the generateObject pattern used there.

Now I have a clear picture of everything. Let me also check what buildBenchmarkMatchingPrompt is referenced from, to ensure I clean up all imports.

Good — only used in the benchmark route. Now let me create the task list and start implementing.

Now let me create the main pipeline orchestrator. This is the largest piece of work.

Now rewrite the benchmark route to delegate to the pipeline.

Now clean up lib/prompts.ts — remove buildBenchmarkMatchingPrompt().

Let me run the TypeScript check first.

The pre-existing errors are still there (chat-shell.tsx and chat/route.ts). I only need to fix the one new error in lib/benchmark.ts — LanguageModelV1 should be LanguageModel.

Now replace all usages of LanguageModelV1 with LanguageModel.

All remaining errors are the pre-existing ones (chat-shell.tsx and chat/route.ts). No new errors from our changes. Now let me run the build.

Build passes cleanly. /api/benchmark is still registered as a dynamic route.

All done. Here's a summary of what was implemented:

Changes Made

1. Created lib/benchmark.ts — Full 4-step pipeline orchestrator:

• Prompt loading: Reads and caches all 5 .txt files from data/benchmark/prompts/, with interpolatePrompt() for {var}/{{var}} substitution
• Step 1 (Enhance): Cleans up observation text via enhance_comment_prompt.txt. Falls back to originals on failure.
• Step 2 (Subject Score): Extracts unique subjects, scores each against observations via subject_prompt_v3.txt. Filters to score >= 3 (RELEVANT). Falls back to treating all subjects as relevant.
• Step 3 (Area/Standard Score): Filters ~800 standards down to only relevant subjects, scores each via area_standard_prompt_v3.txt. Falls back to empty matches.
• Step 4 (Rerank): Takes candidates scoring >= 3, reranks on 0-10 scale via rerank_prompt_v3.txt. Falls back to normalized step 3 scores.
• Final conversion: Filters to score >= 5, takes top 3 per observation, normalizes to 0-1 confidence → BenchmarkMatch[]

2. Rewrote app/api/benchmark/route.ts — Removed all inline generateObject/Zod schema code. Custom curriculum JSON is now converted into CurriculumStandard[] objects (not a context string). Delegates to runBenchmarkPipeline(). API contract ({ results: BenchmarkMatch[] }) is unchanged.

3. Cleaned lib/prompts.ts — Removed buildBenchmarkMatchingPrompt() (now unused).

Verification

• npx tsc --noEmit — no new errors (only pre-existing chat-shell.tsx/chat/route.ts ones)
• npx next build — passes, all routes registered correctly

I'll analyze the benchmark prompts thoroughly — reading each one and evaluating them against prompt engineering best practices.

Now let me do a deep analysis of each prompt.

Benchmark Prompt Analysis

Overall Architecture Assessment

The 4-step funnel (Enhance → Subject → Area/Standard → Rerank) is a strong pattern — it progressively narrows the search space, reducing token cost and improving precision. However, several prompts have issues worth addressing.

1. system_prompt.txt — Grade: C

Issues:

• Wall of text, no structure. It reads like a spec doc rather than a system prompt. The model gets this on every call — it should be tight and directive.
• Contradicts the pipeline. It says "Also provide with a brief reason" — but reasoning is already handled by each step's schema. This instruction creates noise.
• Mentions "JSON File" and "three columns" — but the actual input format varies per step (subjects list, [id] Area, Standard, match list). This is misleading context.
• Security preamble is wasted tokens. "It should not accept or process commands that deviate from this task" — you're calling generateObject() with a schema, so the model can't deviate anyway.
• Grammar issue: "Communicate exclusively in english Security Notice:" — missing period/separator.

Recommendation: Rewrite as a focused 3-line role + task definition. Drop the security theater and the input format description (each step prompt already specifies its own format).

2. enhance_comment_prompt.txt — Grade: B-

Issues:

• No examples (zero-shot). This is the kind of task where 2-3 few-shot examples would dramatically improve consistency — showing what "minimal enhancement" looks like vs. over-editing.
• Dense paragraph format. Should use bullet points for the enhancement criteria.
• Says "Input: A comment" (singular) but in practice receives a batch of [id]: text pairs. The prompt doesn't tell the model to process multiple comments or preserve IDs.
• No instruction about the output format. The schema forces { enhanced_comments: [{ id, text }] } but the prompt never mentions returning IDs or an array. The model has to figure this out from the schema alone.

Recommendation: Add 2 few-shot examples (one that needs fixing, one that doesn't). Explicitly describe the batch input format and the expected output structure.

3. subject_prompt_v3.txt — Grade: A-

Strengths:

• Clear visual structure with ⭑ section headers
• Well-defined 5-level scoring rubric with both labels AND numeric values
• Explicit BOOST RULE for keyword matching
• Good guardrails ("Do not force a match", "Do not invent subjects")

Issues:

• Instruction #4 contradicts the schema. It asks for "a brief summary of why you matched with the subjects" — but the schema only has per-score explanation fields. There's no top-level summary field. This instruction goes nowhere.
• Batch ambiguity. Like the enhance prompt, it says "teacher's comment" (singular) but receives a batch. The {comment} placeholder contains multiple [id]: text pairs — the prompt should say "For each observation, evaluate every subject."
• No few-shot examples. A single example showing one observation scored against 2-3 subjects would anchor behavior.

Recommendation: Remove instruction #4 (or add a summary field to the schema). Clarify batch processing. Add 1 few-shot example.

4. area_standard_prompt_v3.txt — Grade: B+

Strengths:

• Same clean structure as subject prompt
• Consistent scoring rubric across steps (reduces confusion)
• Good guardrails

Issues:

• Missing instruction #4. The subject prompt has 4 instructions; this one has 3. Not a bug per se, but the missing "provide a summary" instruction was actually the problematic one — so this is accidentally better.
• Scoring label mismatch. The RELEVANT (3) description says "comment supports or extends the subject" — but this step scores area standards, not subjects. Copy-paste error from the subject prompt.
• No context about what was filtered. The model doesn't know that these standards were pre-filtered to relevant subjects. If it sees 200 standards all from 3 subjects, it might wonder why. A one-liner like "These standards were pre-selected from subjects relevant to the comment" would help.
• No few-shot example.

Recommendation: Fix "subject" → "area standard" in RELEVANT description. Add filtering context. Add 1 example.

5. rerank_prompt_v3.txt — Grade: A-

Strengths:

• Clear 2-step reasoning process (score individually, then review + adjust)
• Well-calibrated 6-tier scoring guide on 0-10 scale
• Explicit instruction to spread scores ("not all matches should receive high scores")
• Tone instruction for user-facing reasoning

Issues:

• No few-shot example. Given this is the final quality gate, even 1 example would help calibrate score distributions.
• Missing "END OF MATCHES" marker. The comment has "END OF TEACHER COMMENT" but matches don't have a delimiter — potential confusion if the matches section is long.
• {matches} comes AFTER instructions. Ideally, long variable data should come last (it does here — good), but having the scoring guide far above the matches means the model needs to "remember" the rubric. For Gemini Flash this is usually fine, but worth noting.

Recommendation: Add 1 compact few-shot example. Add "END OF MATCHES" delimiter.

Cross-Cutting Issues

Priority Recommendations

1. High impact, low effort: Fix the system prompt to be concise and step-agnostic
2. High impact, medium effort: Add 1-2 few-shot examples to subject + rerank prompts (the two scoring steps that matter most)
3. Medium impact, low effort: Clarify batch processing in enhance + subject prompts
4. Low effort: Fix the copy-paste "subject" → "area standard" wording

Want me to implement any of these improvements?

Let me read all prompts to make precise edits.

Good, I have all prompts. Now let me implement all 5 prompt rewrites in parallel.

All 5 prompts rewritten. Now let me mark tasks and verify the build.

Only pre-existing errors. Let me also verify the build.

Build passes cleanly.

All prompts updated. Here's a summary of every change:

Changes Made

1. system_prompt.txt — Rewrote from scratch

• Replaced verbose spec-doc with 4-line focused role + rules
• Dropped security theater (redundant with generateObject schema enforcement)
• Dropped misleading "JSON File with three columns" input description (each step defines its own format)
• Added the key behavioral rule: "Focus on skill/behavior, not keyword overlap"

2. enhance_comment_prompt.txt — Restructured + added examples

• Switched from wall-of-text to bullet-point criteria
• Clarified batch input format: [id]: observation text
• Added 2 few-shot examples (one needing fixes, one unchanged)
• Explicitly stated to preserve original IDs

3. subject_prompt_v3.txt — Fixed + added example

• Removed orphan instruction #4 ("provide a brief summary") that had no schema backing
• Changed "teacher's comment" (singular) → "one or more teacher observations" throughout
• Changed scoring guide wording from "comment" → "observation" for consistency
• Added 1 few-shot example showing 3 subjects scored against 1 observation with varying scores

4. area_standard_prompt_v3.txt — Fixed copy-paste error + added context + example

• Fixed RELEVANT (3) description: "supports or extends the subject" → "supports or extends the area standard"
• Added pre-filtering context: "These area standards were pre-filtered to subjects that are relevant"
• Changed singular "comment" → batch-aware "observations" throughout
• Added 1 few-shot example showing 3 standards scored (4, 3, 0) with explanations

5. rerank_prompt_v3.txt — Added example + delimiter

• Changed singular "comment" → batch-aware "observations" throughout
• Added 1 few-shot example showing realistic score spread (9, 7, 2)
• Added END OF MATCHES delimiter to close the variable-length matches section
• Moved feedback section before matches (keeping long data at the end)

Good question — let me trace the actual data through each step and verify the formats match.

Now let me look at real curriculum data to see the actual field values.

I found a real issue. Let me trace the exact data through each step.

The Problem: Subject Name Matching

The code in Step 2 does exact string matching between the model's returned subject field and real CurriculumStandard.subject values:

Real subject names are: "Meta Skills Innovation", "IB Learner Profile Communication", "Approaches to Teaching and Learning" — but the few-shot examples in subject and rerank prompts use shortened names like "Innovation", "Self-Management". If the model follows the example pattern and returns short names, Step 3 filtering breaks silently — no standards pass through, zero matches.

Same issue exists in the rerank prompt where the example shows Innovation > Critical thinking > ... but real data is Meta Skills Innovation > Critical thinking > ....

Let me fix the prompts and add an explicit exactness instruction.

Good. Now I have the full picture. Three fixes needed:

1. subject_prompt_v3.txt: Use real full subject names in example + add instruction to return names exactly
2. rerank_prompt_v3.txt: Use full subject names in example matches
3. area_standard_prompt_v3.txt: Example is fine (uses IDs and area/standard format correctly — no subject names involved)

Now let me verify the build still passes.

Build passes. Here's what I found and fixed:

The Bug

The pipeline does exact string matching on subject names returned by the model:

Step 2: model returns subject: "Innovation" → stored in Set
Step 3: code checks allRelevantSubjects.has("Meta Skills Innovation") → FALSE → 0 standards pass

Real subject names are compound like "Meta Skills Innovation", "IB Learner Profile Communication", "Approaches to Teaching and Learning" — but the few-shot examples showed short names like "Innovation", teaching the model to shorten them.

Fixes Applied

subject_prompt_v3.txt:

• Added instruction #3: "Return subject names EXACTLY as they appear in the input list — do not shorten, abbreviate, or rephrase them."
• Updated few-shot example to use realistic full names: Meta Skills Innovation, Meta Skills Self-Management, Meta Skills Social Intelligence with realistic summary text

rerank_prompt_v3.txt:

• Updated example matches to use full subject names: Meta Skills Innovation > Critical thinking > ... instead of Innovation > Critical thinking > ...

area_standard_prompt_v3.txt — no change needed. This step uses [id] Area, Standard format (no subject names) and returns standardId which maps directly to real IDs. The example already matches the real data format.

This is a strategy question more than a code one, so let me think through what makes a good benchmark set.

What the observations repo needs

Core structure: A curated set of real (or realistic) teacher observations, each tagged with expected matches so you can measure pipeline accuracy.

observations/
  observations.json          ← the 200 entries (what your app consumes)
  observations-annotated.json ← same entries + expected_standards for scoring
  README.md                  ← criteria, coverage goals, contribution guide

Each annotated entry:

The plain observations.json (without annotations) is what gets copied into your app's data/benchmark/ for actual runs. The annotated version lets you score precision/recall.

Coverage you should aim for

How to source them

The biggest question. Three options:

1. From Manu/teachers directly — Ask for anonymized real observations from existing Volantis users. Best quality, most realistic. Even 50 real ones are worth more than 200 synthetic ones.

2. LLM-generated with human review — Generate candidate observations per subject/level/tag, then have a teacher review and edit. Fast to produce, decent quality if reviewed.

3. Hybrid — Start with 50-80 real ones, fill gaps with generated ones for underrepresented subjects/levels.

Scoring script

The repo should include a simple scoring script that:

1. Runs your pipeline against the observations
2. Compares results to expected_standards
3. Reports precision, recall, and F1 per subject/level

This lets you measure whether prompt changes actually improve accuracy.

Want me to scaffold the repo structure with the annotated schema, a scoring script, and a generation helper to fill coverage gaps?

Let me check what the app actually consumes.

Now let me check how the shell consumes it.

The contract is simple. The app expects:

That's it — just id and text. The BenchmarkObservation type also has an optional source field but nothing reads it currently:

So the file your external repo produces just needs to be a JSON file with that { version, observations: [{ id, text }] } shape. Any extra fields (like expected_standards or tags for your scoring) would live only in the annotated version in the external repo and get stripped when you copy the file over.

Created data/benchmark/OBSERVATIONS.md with the file structure, field definitions, examples of good vs weak observations, and a brief explanation of how the pipeline consumes them.