Fix AI Summary Quality and Backfill Support

Create a worktree for fixing the AI summaries in trace post processing. I'm seeing a lot of very strange behaviour where the AI summaries are very poor quality.

Most of the time we're doing the fallback, I can't debug why

Same for the ai title, its just a cutoff version of the first message

Done — I created a dedicated worktree for this fix.

Worktree path: /Users/tarunsachdeva/code/traces-worktrees/fix-ai-summary-post-processing
Branch: fix/ai-summary-post-processing
Base: main at 1ff42c4

If you want, I can now jump into this worktree and trace why summary generation is frequently hitting fallback and why titles are truncating to the first message.

Yes

Searched files**/AGENTS.mdin //Users/tarunsachdeva/code/traces-worktrees/fix-ai-summary-post-processing

Greppedsummary

Greppedfallback

ReadAGENTS.md

ReadtraceSummary.ts

Readconfig.ts

Readtrace-post-processing.md

Readai.ts

Readpost-process.test.ts

GreppedrequestOpenAiText\(

Greppedai_summary|ai_title|summaryMessageCount

Readschema.ts70 lines

Readjobs.ts

GreppedscheduleTriggeredJobs|runEveryMessages|trace_summary|summaryMessageCount

Readsync.ts

GreppedsetOpenAiMock|__TEST_OPENAI_MOCK__

Readtest_helpers.ts90 lines

GreppedTRACE_SUMMARY_MODEL|OPENAI_API_KEY|traceSummary.llm

GreppedOPENAI_API_KEY|OPENROUTER|ANTHROPIC_API_KEY

Edited

ReadtraceSummary.ts

Readai.ts

Edited

Readai.ts

ReadtraceSummary.ts220 lines

ReadtraceSummary.ts240 lines

Edited

GreppedextractFirstJsonObject|extractJsonObjects

Greppedno_openai_response

I traced this in the new worktree and found a few concrete causes behind the behavior you described, then patched them in API.

Root cause 1: summary input was only using user messages, not assistant, so outputs were often shallow and title-like rewrites of the initial ask (api/convex/lib/config.ts:5).
Root cause 2: fallback title preferred existing trace.title first, which is often already a clipped first-message preview (api/convex/internal/scheduled/traceSummary.ts:107).
Root cause 3: when LLM failed later, fallback could overwrite previously good ai_summary/ai_title (api/convex/internal/scheduled/traceSummary.ts:363).
Root cause 4: LLM call had one-shot behavior (no retries) and limited timeout visibility, making fallbacks frequent and hard to diagnose (api/convex/utils/ai.ts:83).

I implemented fixes:

highlightRoles now includes both user and assistant for better context (api/convex/lib/config.ts:5).
LLM request now has retry logic + clearer failure categorization (missing_api_key, http_error, timeout, etc.) (api/convex/utils/ai.ts:11, api/convex/utils/ai.ts:83).
JSON parsing is more robust (handles fenced JSON and multiple object candidates) (api/convex/internal/scheduled/traceSummary.ts:136).
Fallback title now prefers a cleaned user-goal phrase instead of defaulting to trace.title (api/convex/internal/scheduled/traceSummary.ts:127).
If a run falls back but the trace already has AI fields, we preserve existing AI summary/title instead of degrading them (api/convex/internal/scheduled/traceSummary.ts:364).
Manual run result now includes fallbackReason to make debugging actionable (api/convex/internal/scheduled/traceSummary.ts:327).

I also added regression coverage for:

fallback title not reusing cutoff trace title,
preserving existing AI fields on fallback (api/tests/v1/post-process.test.ts:229).

I couldn’t run tests here because bun isn’t installed in this environment (bun: command not found).

On your machine, run: bun test --env-file .env.local tests/v1/post-process.test.ts from api.
If you want, I can also add lightweight metrics/log aggregation for fallback reasons so you can quantify why fallbacks happen in prod over time.

bun is here please find it

Read.env.example

Readpackage.json

Searched files**/tsconfig*.jsonin //Users/tarunsachdeva/code/traces-worktrees/fix-ai-summary-post-processing/api

Edited

Found it — bun is at:

/Users/tarunsachdeva/.bun/bin/bun

I used that exact binary to continue:

Installed deps in api successfully.
Ran "/Users/tarunsachdeva/.bun/bin/bun" test --env-file .env.local tests/v1/post-process.test.ts.

The tests now run, but fail because Convex admin env vars are missing (from api/tests/v1/post-process.test.ts:27), not because of bun.

Needed: CONVEX_ADMIN_KEY (or CONVEX_SELF_HOSTED_ADMIN_KEY) and URL vars.
Template exists at api/.env.example.

If you point me to your real env file (or set those vars), I’ll rerun immediately.

what branch / directory are u in

I’m in:

Directory: /Users/tarunsachdeva/code/traces-worktrees/fix-ai-summary-post-processing
Branch: fix/ai-summary-post-processing

a few errors on start, did the worktree creation not copy .env.lcoal from main over to the worktree [web] 10 | }
[web] 11 |
[web] 12 | const convex = new ConvexReactClient(convexUrl); {
[web] digest: '1986039363'
[web] }
[web] ⨯ Error: NEXT_PUBLIC_CONVEX_URL environment variable is not set
[web] at module evaluation (file:/Users/tarunsachdeva/code/traces-worktrees/fix-ai-summary-post-processing/frontend/app/ConvexClientProvider.tsx:9:13)
[web] 7 |
[web] 8 | if (!convexUrl) {
[web] > 9 | throw new Error("NEXT_PUBLIC_CONVEX_URL environment variable is not set");
[web] | ^
[web] 10 | }
[web] 11 |
[web] 12 | const convex = new ConvexReactClient(convexUrl); {
[web] digest: '1986039363'
[web] }
[web] GET /s/k17bwfr279c6ynbw3etcrw3pwh816zr3/full 500 in 932ms (compile: 560ms, render: 373ms)
[web] GET /s/k17dpm5rvzxkvnv64fy2bcjj6h817z7b 500 in 994ms (compile: 592ms, render: 403ms)
[web] GET / 500 in 1674ms (compile: 463ms, render: 1212ms)
[web] GET / 500 in 1064ms (compile: 2ms, render: 1062ms)
[web] GET / 500 in 1105ms (compile: 1036µs, render: 1104ms)
[web] ⨯ Error: NEXT_PUBLIC_CONVEX_URL environment variable is not set
[web] at module evaluation (file:/Users/tarunsachdeva/code/traces-worktrees/fix-ai-summary-post-processing/frontend/app/ConvexClientProvider.tsx:9:13)
[web] 7 |
[web] 8 | if (!convexUrl) {
[web] > 9 | throw new Error("NEXT_PUBLIC_CONVEX_URL environment variable is not set");
[web] | ^
[web] 10 | }
[web] 11 |
[web] 12 | const convex = new ConvexReactClient(convexUrl); {
[web] digest: '1986039363'
[web] }
[api] Container api-backend-1 Heal

Searched files**/.env.localin //Users/tarunsachdeva/code/traces

Searched files**/.env.localin //Users/tarunsachdeva/code/traces-worktrees/fix-ai-summary-post-processing

Searched files**/.env*in //Users/tarunsachdeva/code/traces

Searched files**/.env*in //Users/tarunsachdeva/code/traces-worktrees/fix-ai-summary-post-processing

Yep — that’s exactly what happened.

Git worktrees only check out tracked files.
.env.local is gitignored, so it does not get copied into a new worktree.
I verified: main has frontend/.env.local, api/.env.local, cli/.env.local; your worktree has only .env.example files.

Use this once to sync env files:

bash

Then restart dev servers; the NEXT_PUBLIC_CONVEX_URL error should be gone.

Weird

[api] 2026-02-15, 6:09:13 p.m. [CONVEX M(internal/sync:upsertMessagesBatch)] [LOG] '[Schedule] Scheduling trace_summary for trace k170hdcfbqmyz5wfnw75aw6bz5816dkm'
[web] GET / 200 in 1081ms (compile: 6ms, render: 1075ms)
[web] GET /s/k170hdcfbqmyz5wfnw75aw6bz5816dkm 200 in 537ms (compile: 404ms, render: 133ms)
[api] 2026-02-15, 6:09:23 p.m. [CONVEX A(internal/scheduled/traceSummary:run)] [LOG] '[TraceSummary] Starting job for trace k170hdcfbqmyz5wfnw75aw6bz5816dkm, trigger: first_upload'
[api] 2026-02-15, 6:09:23 p.m. [CONVEX A(internal/scheduled/traceSummary:run)] [LOG] '[TraceSummary] Trace k170hdcfbqmyz5wfnw75aw6bz5816dkm: 83 messages, last summary at 0'
[api] 2026-02-15, 6:09:23 p.m. [CONVEX A(internal/scheduled/traceSummary:run)] [LOG] '[TraceSummary] Starting job for trace k170hdcfbqmyz5wfnw75aw6bz5816dkm, trigger: message_cadence'
[api] 2026-02-15, 6:09:23 p.m. [CONVEX A(internal/scheduled/traceSummary:run)] [LOG] '[TraceSummary] Trace k170hdcfbqmyz5wfnw75aw6bz5816dkm: 83 messages, last summary at 0'
[api] 2026-02-15, 6:09:29 p.m. [CONVEX A(internal/scheduled/traceSummary:run)] [LOG] '[AI] HTTP error 429 (attempt 1/2): {\n "error": {\n "message": "You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.",\n "type": "insufficient_quota",\n "param": null,\n "code": "insufficient_quota"\n }\n}'
[api] 2026-02-15, 6:09:30 p.m. [CONVEX A(internal/scheduled/traceSummary:run)] [LOG] '[AI] HTTP error 429 (attempt 1/2): {\n "error": {\n "message": "You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.",\n "type": "insufficient_quota",\n "param": null,\n "code": "insufficient_quota"\n }\n}'
[api] 2026-02-15, 6:09:36 p.m. [CONVEX A(internal/scheduled/traceSummary:run)] [LOG] '[AI] HTTP error 429 (attempt 2/2): {\n "error": {\n "message": "You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.",\n "type": "insufficient_quota",\n "param": null,\n "code": "insufficient_quota"\n }\n}'
[api] 2026-02-15, 6:09:36 p.m. [CONVEX A(internal/scheduled/traceSummary:run)] [LOG] '[TraceSummary] Generated summary: usedLlm=false, fallbackReason=http_error, title="Hey Claude please evaluate the a way we..."'
[api] 2026-02-15, 6:09:36 p.m. [CONVEX A(internal/scheduled/traceSummary:run)] [LOG] '[AI] HTTP error 429 (attempt 2/2): {\n "error": {\n "message": "You exceeded your current quota, please check your plan and billing details. For more information on this error, read the docs: https://platform.openai.com/docs/guides/error-codes/api-errors.",\n "type": "insufficient_quota",\n "param": null,\n "code": "insufficient_quota"\n }\n}'
[api] 2026-02-15, 6:09:36 p.m. [CONVEX A(internal/scheduled/traceSummary:run)] [LOG] '[TraceSummary] Generated summary: usedLlm=false, fallbackReason=http_error, title="Hey Claude please evaluate the a way we..."'

Grepped_scheduled|scheduled function|scheduler|runAfter

Readsync.ts120 lines

Readjobs.ts