Completed Phases 2 and 3: Connectors, Retrieval

Summary

Closed, Completed

Understand
Assessed current codebase against spec phases 0 and 1.
Understand
Verify
Confirmed Phase 0 and Phase 1 completion with tests and acceptance criteria.
Verify
Plan
Outlined Phase 2 build list including Slack and Notion connectors and webhook handling.
Plan
Build
Implemented Phase 2 connectors, webhook routes, slim ingestion, and source-diversified retrieval logic.
Build
Verify
Live-verified Phase 2 acceptance gates with real Composio data.
Verify

check we are where we are currently standing against the spec.md. I think we are past phase zero and phase one, and we can successfully move on to phase two. Let's cook.

I'll start by reading the spec and assessing where the code actually stands against it.

Let me read the spec and survey the current state of the code.

Now let me survey what's actually built. Let me map the codebase structure against Phase 0/1/2 requirements.

I see a sentry-setup worktree already exists — Phase 2 work may have started. Let me check the git state and verify Phase 1 actually holds (build + tests) before declaring it done.

The sentry-setup worktree is actually feature/observability (Phase 5 observability work, not the Phase 2 Sentry connector — different thing). Let me verify Phase 1 genuinely holds by running typecheck and tests, while a subagent maps feature completeness.

Typecheck is clean. Let me run the test suite and have a subagent do a thorough completeness assessment of Phases 0/1 in parallel.

Typecheck clean, 9/9 tests pass. Now let me get a thorough assessment of Phase 0/1 acceptance criteria against the actual code.

The subagent confirms it thoroughly: Phase 0 and Phase 1 are genuinely DONE — typecheck clean, 9/9 tests pass, all acceptance deliverables present in code, and commit 3dbde66 ("Composio live integration verified against real data") shows the live path was exercised. The only "not done" items are correctly deferred to later phases (rerank→P5, cache→P5, graph→P3, full UI→P6).

Before I start cooking on Phase 2, let me check one thing about branching — there's unmerged Greptile work and a review gate to respect.

Here's where we stand, confirmed:

Phase 0 — DONE. Migrations 0001–0004 (context_item, context_chunk with 8 hash partitions, entity/edge, pg_trgm/pgvector, RLS, hybrid_search, sync_state), typed Supabase clients, server-only service role.

Phase 1 — DONE end to end. Auth (NextAuth Google, uuid-v5 tenant id, middleware), connector contract + Gmail/Calendar/Linear, Trigger.dev ingest job (idempotent, per-item isolation), full retrieval path (plan → hybrid_search → rollup → assemble → grounded cited synthesis with refuse-when-thin), minimal Ask UI. Typecheck clean, 9/9 tests green, and the live Composio path was already verified (commit 3dbde66).

Everything still "missing" is correctly deferred (rerank/cache/breaker → P5, graph → P3, full UI → P6). So yes — clear to move to Phase 2.

Branching note: main has all of Phase 1. The Greptile fix (b89dd7f) and the observability PR #2 are still unmerged (your 5/5 gate). Phase 2 extends the ingestion pipeline, so I'll branch Phase 2 off the current tip (fix/greptile-ingest-pr1) to build on the latest ingestion core rather than stale main — not merging PR1 myself.

One real fork before I cook: Phase 2 mandates Slack + one of Notion/GitHub/Sentry. That second pick changes what I build and which account you'll connect for the demo, so let me get it from you.

Notion it is — good pick, it stresses the enrich/gloss + chunking path and sets up Phase 3 graph.

Phase 2 build list:

lib/connectors/slack.ts (load/poll/slim + handleEvent) and lib/connectors/notion.ts (load/poll/slim)
Register both (registry + composio auth-config map + SourceName type)
app/api/webhooks/[source]/route.ts — HMAC-verified Slack event ingestion
trigger/slim.ts — periodic id-only pass flipping is_deleted
Defensive count assertions in fetch

Let me read the existing connector + ingestion patterns so I match them exactly before writing anything.

SourceName and composio auth-config already include slack/notion — good scaffolding. Let me read the ingestion wiring (trigger jobs, run, normalize, enqueue) and gmail (the query/slim pattern) to match conventions.

Now the slim acceptance requires deleted items to "disappear from answers" — I need to verify the retrieval path actually filters is_deleted. Let me check rollup/search and the connect/oauth wiring.

rollup.ts:34 already filters is_deleted, so the slim acceptance ("disappears from answers") is satisfied the moment slim flips the flag. Good — the retrieval side needs no change. Let me check the enrich gating (so Notion long docs get glossed/chunked correctly), migration 0001 for is_deleted + a slim-friendly index, and .env.example for signing-secret conventions.

I have the full picture. Everything aligns: enrich.ts:12 already treats notion as unstructured (gloss-eligible), chunk.ts handles long docs, rollup.ts:34 filters is_deleted, and .env.example already has WEBHOOK_SECRET_SLACK + Slack/Notion auth-config ids. The retrieval side needs zero changes.

Let me set up the work. Entering an isolated worktree first (required before edits), then I'll build.

The worktree branched off origin/main, which is missing the Greptile ingestion fix (b89dd7f). I want Phase 2 on top of the latest ingestion core, so let me rebase this fresh worktree branch onto fix/greptile-ingest-pr1.

Now at b89dd7f with the Greptile ingestion fixes. Now let me build Phase 2. I'll start with the shared helpers, then the two connectors, then webhooks + slim wiring.

Let me create the count-assertion helper and the Slack HMAC verifier first (independent pieces).

Now the Slack connector — load/poll/slim via channel history, plus handleEvent for webhook event-mode:

Now the Notion connector — search-by-last-edited, fetch page blocks for body (exercises the chunk + gloss path):

Now register both connectors:

Now wire event-mode ingestion. First extend the ingest task to handle a single webhook event:

Now add enqueueEvent to the enqueue helper:

Now the webhook route — HMAC-verified Slack Event ingestion that acks fast and enqueues to Trigger.dev:

Now the slim deletion pass. First the DB diff helper (with a guard against mass-deletion on an empty fetch):

Now the scheduled slim trigger job:

Let me verify the source_connection table has the metadata column my webhook tenant-resolver reads, and that types.ts reflects it:

metadata column exists — good. Now let me typecheck. The worktree needs node_modules; let me check what's available.

No node_modules in the worktree. Since I added no dependencies, I'll symlink the primary checkout's node_modules to typecheck/test against:

One typecheck error — noUncheckedIndexedAccess on rows[0]. Let me fix:

Let me check the existing test pattern before adding Phase 2 tests:

Pattern is clear. Let me add Phase 2 tests — Slack signature verification (pure), Slack connector mapping, and Notion connector mapping:

Now typecheck and run the full test suite:

Typecheck clean, 18/18 tests pass (added 9). Let me confirm Trigger.dev discovers the new slim schedule and that formatting matches conventions:

dirs: ['./trigger'] auto-discovers slim.ts — good. Three files have prettier issues; let me fix them:

Just the test file left. Let me format it and run the final full check:

The 6 prettier warnings are pre-existing files I never touched (e.g. assemble.ts, synthesize.ts) — not in scope. Let me confirm all my Phase 2 files are clean:

All Phase 2 files are clean. Let me verify one integration point: the query-understanding plan needs to know slack/notion are valid sources, or it won't route to them. Let me check:

The plan prompt already lists slack and notion as valid sources, and sources is an open array — so retrieval routes to them with zero changes. The whole answer path works for the new sources automatically.

Let me review the full diff before committing:

Clean diff — 13 files, all net-new Phase 2 work, nothing unrelated touched. Let me commit and push to a properly-named feature branch (not main — respecting your Greptile-5/5-before-main gate):

Phase 2 is built, green, and pushed. Here's the summary.

Where we stood

Phase 0 — DONE (migrations, RLS, hybrid_search, typed clients).
Phase 1 — DONE end to end (auth, Gmail/Calendar/Linear connectors, Trigger.dev ingest, full cited-answer retrieval path, Ask UI). Typecheck clean, tests green, live Composio path already verified in commit 3dbde66.

Confirmed clear to move on — so I cooked Phase 2.

Phase 2 — Source breadth + sync robustness (you picked Notion as the 2nd source)

Built on top of the Greptile-fixed ingestion core, in worktree phase2-source-breadth, pushed to feature/source-breadth (not main — your 5/5 Greptile gate stays intact):

lib/connectors/slack.ts — channel-walk load/poll/slim + handleEvent for the Event API. Skips join/leave noise; channel-qualified external ids.
lib/connectors/notion.ts — search-by-last_edited_time load/poll/slim, assembles block text into the body (drives the chunk + gloss path; triple-extraction-eligible for Phase 3).
app/api/webhooks/[source]/route.ts + lib/webhooks/slack.ts — HMAC signing-secret verification with a replay window, url_verification handshake, fast ack then enqueue to Trigger.dev (ingestion never runs inline in a route).
trigger/ingest.ts — new event mode (single item; does not advance the poll cursor).
trigger/slim.ts + lib/db/slim.ts — periodic id-only reconcile that flips is_deleted on vanished ids, with an empty-live-set guard that refuses mass-deletion. rollup.ts:34 already excludes is_deleted, so flips drop items from answers with no retrieval change.
lib/connectors/util.ts — under-collection guard for paged fetches.
Registered both in the registry; the query-understanding prompt already enumerates slack/notion, so the answer path routes to them with zero changes. No new migration needed (is_deleted and metadata already exist).

Checks: tsc --noEmit clean, 18/18 tests pass (added Slack/Notion mapping + Slack signature tests), all new files Prettier-clean.

Honest caveat: the three Phase 2 acceptance gates (deleted item disappears, Slack webhook ingests in seconds, "what happened across the company" spans 5 sources) are code-complete and unit-tested but not yet live-verified — that needs the Slack + Notion accounts connected, a 90-day load run, and the Slack app's event subscription pointed at the deployed webhook URL with WEBHOOK_SECRET_SLACK set. The Composio slugs carry the same "verify against live data" caveat your Phase 1 Gmail connector did (which you cleared in 3dbde66).

result: Phase 0/1 confirmed done; Phase 2 (Slack + Notion connectors, HMAC webhook event-mode, slim deletion sync, under-collection guard) built, 18/18 tests green, pushed to feature/source-breadth — pending live Composio/Slack verification.

let's work on the live composios/Slack verification and make sure it properly works, and once we get through that, let's continue cooking

Let me check what credentials/connections are actually live before planning verification. I'll look at .env.local, the existing verify scripts, and whether Slack/Notion are connected in Composio.

The .env.local lives in the main checkout, not the worktree. Let me check what's configured there:

COMPOSIO_API_KEY and TRIGGER_SECRET_KEY are set, but I notice the COMPOSIO_*_AUTH_CONFIG_ID entries aren't showing and WEBHOOK_SECRET_SLACK is empty. Let me see the actual Composio config lines and how Phase 1 was live-verified:

Key finding: .env.local has COMPOSIO_API_KEY but no COMPOSIO_*_AUTH_CONFIG_ID entries — and WEBHOOK_SECRET_SLACK is empty. Phase 1's live verification used run-ingest.ts, which calls executeTool(slug, userId, …) directly against an already-connected Composio account (the auth-config IDs are only needed for the in-app OAuth initiate flow, not for fetching).

So the real question for live verification: what's actually connected in Composio, and under which userId? Let me probe the Composio account directly.

Module resolution needs to run from the worktree (where node_modules is symlinked). Let me put the probe there:

There it is — the blocker. Both Phase 2 sources are EXPIRED:

slack — EXPIRED
notion — EXPIRED
(gmail, calendar, linear, sentry, github all ACTIVE)

So live fetch will fail until they're reconnected via OAuth. Before I ask you to reconnect (browser consent — only you can do that), let me do everything headless first: get the userId/auth-config so I can hand you a clean reconnect link, and verify my connector's tool slugs actually exist in the Composio toolkit (catch slug drift now, not after you reconnect).

Caught real slug drift before it bit us — 2 of my 4 tool slugs are wrong:

❌ SLACK_LIST_ALL_SLACK_TEAM_CHANNELS_WITH_VARIOUS_FILTERS (miss)
❌ NOTION_FETCH_NOTION_BLOCK_CONTENTS (miss)
✅ SLACK_FETCH_CONVERSATION_HISTORY, NOTION_SEARCH_NOTION_PAGE (exist)

Let me find the correct slugs by listing the actual Slack and Notion tools in the toolkit:

Param name is toolkits (plural). Let me fix:

Now I have the real slugs. Let me inspect the exact input/output schemas for the candidates so I map fields correctly (not guess): Slack channel-list + history, Notion search + the best body-fetch tool.

Now I have ground truth on the schemas, and I can see more drift in my Notion connector: NOTION_SEARCH_NOTION_PAGE takes flat params (filter_property, filter_value, direction, timestamp) — not the nested filter/sort objects I wrote. And NOTION_GET_PAGE_MARKDOWN gives the whole page body in one call — much better than block-walking.

Let me get the userId (needed for reconnect + ingest) and confirm reconnect is possible:

The userId isn't exposed on the account object directly. Let me find the tenant from the DB instead — the Phase 1 verification already ingested under some user_id, and source_connection maps accounts to users:

Need the ws polyfill (same as the repo scripts). Let me add it:

The real tenant is 4847c952-0bef-40d3-a62b-da339552dfd2 — 515 Gmail items, 4 Linear, 1 Calendar from Phase 1's live run. source_connection is empty because that run used run-ingest.ts directly with the Composio userId. Let me confirm that userId works live against an ACTIVE account, then confirm Slack/Notion fail (proving reconnect is the blocker):

Even ACTIVE Gmail threw — so either the userId is wrong or there's an error detail I'm masking. Let me dump the full error and the full account object to find the real Composio userId:

No userId on the account object, and the error is wrapped. Let me extract the real error detail and check the account's state/params for the entity binding:

Error 1811 confirms a userId is required and the account object doesn't expose which one. Let me capture the full error for the userId-based call and dump a raw list item to find where Composio stores the entity binding:

Now the picture is complete. Two findings:

The Slack/Notion connections never completed — statusReason: "Connection initiation did not complete within 10 minutes". They're dead, not refreshable. Fresh OAuth needed.
The Composio userId is not the bare uuid — No connected account found for user ID 4847c952-… for toolkit gmail. The SDK strips the userId from list/get responses, and the live tenant's Composio user is some other handle. But that's fine: run-ingest.ts separates the fetch userId from the DB userId, and the 515 Gmail items already live under DB user 4847c952-…. So if I reconnect Slack/Notion under a known Composio userId and ingest into that same DB tenant, all sources unify.

Plan: (1) fix the verified slug/param drift in code, (2) hand you fresh OAuth links under a known userId, (3) you authorize, (4) I live-probe output shapes + ingest + verify.

Let me fix the connector drift first. Updating Slack channel-list slug and params:

Now the Notion connector — fix the flat search params and switch body-fetch to NOTION_GET_PAGE_MARKDOWN (one call, whole page). Let me read the current version:

Now update the fetchAll and slim to use the shared searchArgs:

The body-fetch changed, so the Notion test needs updating to mock NOTION_GET_PAGE_MARKDOWN instead of block results:

Now typecheck and re-run tests to confirm the slug/param fixes are clean:

Slug/param drift fixed, typecheck clean, 18/18 tests pass. Now I need fresh OAuth connections. Let me generate Slack + Notion authorize links under the canonical tenant userId (4847c952-…, matching the existing DB data):

The initiate endpoint is deprecated for Composio-managed OAuth. Let me get the full message and find the current method:

There's a link method. Let me use it:

I have everything wired and fresh OAuth links generated. The connector slug/param drift is fixed (caught 2 wrong slugs before they bit us), typecheck clean, 18/18 tests. The one thing I genuinely can't do for you is click through OAuth consent in a browser — that's your step.

Please authorize both (open each link, complete the consent):

Slack → https://connect.composio.dev/link/lk_X1-Zj2y4ncN3
Notion → https://connect.composio.dev/link/lk_QCHNZ6N6jsTj

Both are linked under the canonical tenant 4847c952-… so the new Slack/Notion data lands in the same DB tenant as your existing 515 Gmail + Linear + Calendar items (so the cross-source question works).

Two things that matter for data to actually flow:

Slack: after authorizing, the bot only reads channels it's in. Add it to a couple of real channels (/invite @<appname>) so conversations.history returns messages. Without that, channels list will be empty.
Notion: the Notion consent screen lets you select which pages/databases to share. Pick the ones you want indexed, or it'll have nothing to read.

Once you've done both, tell me and I'll: probe the live output shapes (confirm my field mappings against real Slack/Notion payloads), run the 90-day ingest for both, verify the webhook HMAC path, and then run the cross-source demo question. I'll fix any remaining field drift live.

needs input: Authorize the Slack and Notion OAuth links above (browser consent), then tell me when done so I can run live ingestion + verification.

I've given you access to both Slack and Notion now; you're all good to roll and continue cooking; lmk if you have any other blockers

Let me verify both connections are now ACTIVE, then live-probe the actual payload shapes before ingesting.

Both ACTIVE. Now let me live-probe the real payload shapes for Slack channel-list, Slack history, Notion search, and Notion markdown — to confirm my field mappings against actual data:

Slack mappings confirmed correct (data.channels[] with id/name/is_member, data.messages[] with ts/text/user/subtype, has_more, empty next_cursor→undefined handled). Note the test workspace is sparse — the sampled channel had only a join message (correctly skipped). Now let me probe Notion shapes and check which Slack channels actually have content:

Both confirmed against live data:

Slack: ai-agent channel has 7 real content messages (general/random are empty/join-only). My mappings match exactly.
Notion: data.results[], has_more, next_cursor, properties.title, and data.markdown (a string) — all match my extractMarkdown first-key check.

Every field mapping is verified. Now let me run the real 90-day ingest. Slack first (small), capping Notion to bound cost:

Slack: 17 items, 0 failures. Now Notion (capped at 25 to bound cost on template pages):

Notion: 11 items, 0 failures. Let me verify the stored data is sane (titles extracted, chunk content well-formed, embeddings present) before the answer test:

Data quality is clean — provenance lines correct, titles extracted, embeddings present, slack=17, notion=11. Now the answer-path test. Let me check the ask CLI signature and run the cross-source demo question:

Important: the connectors work (data ingested, embedded), but the cross-source question surfaced only Gmail — the 515-item inbox swamps the 17 Slack + 11 Notion items in ranking, so the model correctly refused rather than inventing. That's a retrieval-diversity concern, not a connector bug. Let me first prove Slack and Notion are genuinely retrievable + citable with targeted queries:

Both Slack and Notion are fully verified end-to-end — connector → ingest → embed → retrieve → synthesize → cited answer:

Slack: grounded answer citing slack/message (captured the actual content: 429 rate limits, "sleep 60s then retry", "3AM debugging session").
Notion: cited notion/doc, honestly flagging the workspace holds only default-template pages (which is the truth of that account).

The connectors are working. The only gap is acceptance gate #3 (one answer spanning 4+ sources) — the 515-item Gmail inbox dominates ranking so low-volume sources never become candidates. Let me confirm whether it's candidate-generation vs rollup by probing hybrid_search directly:

Confirmed: it's candidate-generation, not rollup — the top-60 for that broad query is 49 gmail + 1 slack, 0 from notion/linear/calendar. Gmail's volume crowds everything out before rollup even runs. A per-source cap in rollup can't fix what was never retrieved.

The right fix is source-stratified retrieval for broad intents (cross_source / company_summary / daily_briefing): pull each connected source's top-k so every source is represented, then round-robin the rollup so the final items span sources. This is well-motivated by the explicit acceptance gate and leaves single-source queries (blocker_scan→linear, lookup) untouched. Let me read the retrieval wiring to implement it surgically:

Clear structure. SearchHit carries no source, but stratified retrieval queries per-source so I can tag them. Let me implement: a source-diversity path in search.ts for broad intents, and a round-robin mode in rollup.ts. First, search.ts:

Loading more messages…