Fix AI Summary Quality and Backfill SupportPublic

Done — I created a dedicated worktree for this fix.

• Worktree path: /Users/tarunsachdeva/code/traces-worktrees/fix-ai-summary-post-processing
• Branch: fix/ai-summary-post-processing
• Base: main at 1ff42c4

If you want, I can now jump into this worktree and trace why summary generation is frequently hitting fallback and why titles are truncating to the first message.

I traced this in the new worktree and found a few concrete causes behind the behavior you described, then patched them in API.

• Root cause 1: summary input was only using user messages, not assistant, so outputs were often shallow and title-like rewrites of the initial ask (api/convex/lib/config.ts:5).
• Root cause 2: fallback title preferred existing trace.title first, which is often already a clipped first-message preview (api/convex/internal/scheduled/traceSummary.ts:107).
• Root cause 3: when LLM failed later, fallback could overwrite previously good ai_summary/ai_title (api/convex/internal/scheduled/traceSummary.ts:363).
• Root cause 4: LLM call had one-shot behavior (no retries) and limited timeout visibility, making fallbacks frequent and hard to diagnose (api/convex/utils/ai.ts:83).

I implemented fixes:

• highlightRoles now includes both user and assistant for better context (api/convex/lib/config.ts:5).
• LLM request now has retry logic + clearer failure categorization (missing_api_key, http_error, timeout, etc.) (api/convex/utils/ai.ts:11, api/convex/utils/ai.ts:83).
• JSON parsing is more robust (handles fenced JSON and multiple object candidates) (api/convex/internal/scheduled/traceSummary.ts:136).
• Fallback title now prefers a cleaned user-goal phrase instead of defaulting to trace.title (api/convex/internal/scheduled/traceSummary.ts:127).
• If a run falls back but the trace already has AI fields, we preserve existing AI summary/title instead of degrading them (api/convex/internal/scheduled/traceSummary.ts:364).
• Manual run result now includes fallbackReason to make debugging actionable (api/convex/internal/scheduled/traceSummary.ts:327).

I also added regression coverage for:

• fallback title not reusing cutoff trace title,
• preserving existing AI fields on fallback (api/tests/v1/post-process.test.ts:229).

I couldn’t run tests here because bun isn’t installed in this environment (bun: command not found).

1. On your machine, run: bun test --env-file .env.local tests/v1/post-process.test.ts from api.
2. If you want, I can also add lightweight metrics/log aggregation for fallback reasons so you can quantify why fallbacks happen in prod over time.

Found it — bun is at:

• /Users/tarunsachdeva/.bun/bin/bun

I used that exact binary to continue:

• Installed deps in api successfully.
• Ran "/Users/tarunsachdeva/.bun/bin/bun" test --env-file .env.local tests/v1/post-process.test.ts.

The tests now run, but fail because Convex admin env vars are missing (from api/tests/v1/post-process.test.ts:27), not because of bun.

• Needed: CONVEX_ADMIN_KEY (or CONVEX_SELF_HOSTED_ADMIN_KEY) and URL vars.
• Template exists at api/.env.example.

If you point me to your real env file (or set those vars), I’ll rerun immediately.

I’m in:

• Directory: /Users/tarunsachdeva/code/traces-worktrees/fix-ai-summary-post-processing
• Branch: fix/ai-summary-post-processing

Yep — that’s exactly what happened.

• Git worktrees only check out tracked files.
• .env.local is gitignored, so it does not get copied into a new worktree.
• I verified: main has frontend/.env.local, api/.env.local, cli/.env.local; your worktree has only .env.example files.

Use this once to sync env files:

Then restart dev servers; the NEXT_PUBLIC_CONVEX_URL error should be gone.

Yep, this log is showing two separate issues:

• Duplicate scheduling bug: same trace queued both first_upload and message_cadence before initial summary is written.
• LLM is unavailable: OpenAI returns 429 with code: insufficient_quota, so it always falls back.

I patched both in this branch:

• Prevent duplicate pre-summary runs by only allowing scheduling when count transitions 0 -> >0 if summaryMessageCount is missing (api/convex/internal/sync.ts:167).
• Make HTTP failure reason explicit (insufficient_quota) and avoid retrying that specific case (api/convex/utils/ai.ts:11, api/convex/utils/ai.ts:85, api/convex/utils/ai.ts:150).
• Improved fallback title cleanup so greetings like “Hey Claude …” are stripped first (api/convex/internal/scheduled/traceSummary.ts:84).

What to do now:

1. Restart the API dev process from this worktree so Convex picks up latest code in fix/ai-summary-post-processing.
2. Reproduce one upload and check logs:
   • You should no longer see both first_upload + message_cadence for the same initial burst.
   • If quota is still exhausted, fallback will be expected and logged as fallbackReason=insufficient_quota.