Guidance for Updating Domains in Convex Deployment
Guidance for Updating Domains in Convex Deployment
Agent
OpenCode
Model
Kimi K2.5 Free
Shared
Feb 10, 7:09 PM
Messages182
Based on the transition doc, here's what's left to do since you've completed DNS:
Environment Variables to Update:
TRACES_URL=https://www.traces.comTRACES_HTTP_URL=https://actions.traces.comNEXT_PUBLIC_CONVEX_URL=wss://api.traces.comNEXT_PUBLIC_CONVEX_HTTP_URL=https://actions.traces.comDokku/Server Tasks:
Verification:
https://www.traces.com with no CORS errorsGET /v1/session returns 401 with valid CORS headersJust need to run verification:
cd cli && bun run dev smoke test (login/share/logout URLs)cd frontend && bun run build and verify install links show traces.comtraces.sh references remainCritical: Keep *.traces.sh routing active as an alias during the transition window until all clients have updated.
Want me to help with any specific phase?
Port conflict! The old container is still holding port 3210. Let's force a clean restart:
If that doesn't work, try a rebuild:
The root cause is the zero-downtime deploy failed because there's no healthcheck in your app.json. After we get this working, you should add a healthcheck to prevent this in the future.
Add sudo to docker commands:
Or try rebuilding directly:
If that still fails, try force-restarting:
Kill the old container first:
If you get an error about the container being in use, force it:
You're absolutely right - that's an error in the doc. The dashboard should be:
Not api.traces.com. The domain mapping clearly shows:
api.traces.com = Convex API endpointdashboard.traces.com = Dashboard endpointGood catch! The transition doc has a typo there.
You probably don't need it if the dashboard already works. That config var is likely used by the dashboard to know where the Convex backend is.
Check if your dashboard is already working:
https://dashboard.traces.com (or wherever your dashboard lives) load properly?If yes, skip that step. The doc assumes a specific setup but your dashboard might already be configured correctly or use a different env var name.
Run:
If NEXT_PUBLIC_DEPLOYMENT_URL isn't already set and the dashboard works, don't add it.
Your nginx vhosts for the new *.traces.com domains aren't configured yet. That's why you're seeing the default nginx page.
Check what vhosts exist:
And check your dokku apps:
You'll need to either:
What's the output of those commands? That'll tell us if the dashboard is a separate dokku app or if it needs manual nginx config.
Same port issue! Kill the old container first:
Then update nginx - your vhost only has dashboard.traces.sh, not dashboard.traces.com:
Add dashboard.traces.com to the server_name line:
Then reload nginx and get SSL cert:
Run these one at a time and let me know if any fail.
YES this is bad - certbot put all the new domains in the DEFAULT site instead of your app-specific vhosts. That's why you're seeing the nginx welcome page.
Let's fix this:
Let me know the output of step 1 first so I can see exactly what certbot did.
Certbot messed up - it put the new domains in the default site instead of your app vhosts. Let's fix this:
Let me know if that works and I'll do the same for actions and dashboard.
You're right, let me be systematic.
Certbot put your new domains in the WRONG place (the default nginx site instead of your app sites). That's why you see "Welcome to nginx!"
Step 1: Check current state
Run these three commands and paste the output. Then I'll give you the EXACT fix commands based on what I see.
Good! I can see the state now. Dashboard is partially fixed, but api and actions still only have the old domains.
The new SSL cert exists at /etc/letsencrypt/live/api.traces.com/ but your api and actions vhosts are still using the old cert and don't have the new domains.
Step 1: Fix the api vhost
Step 2: Fix the actions vhost
Step 3: Clean up the default site (remove the wrong config certbot added)
Step 4: Test and reload
Run these one at a time and tell me the output of step 4.
nginx is now valid and reloaded!
Let's verify the sites work:
Test the new domains:
What should you see:
api.traces.com → should NOT be the nginx welcome pageactions.traces.com → should NOT be the nginx welcome pagedashboard.traces.com → should show your dashboard (not nginx)Run those three curl commands and tell me what HTTP codes you get (200, 404, etc.).
Two problems:
Let's check:
Run those and tell me the output. The 502 means your apps (traces, traces-dashboard) aren't responding on those ports, so we need to see if they're even running.
Containers are running. Now check if they're actually listening:
And check if the apps are actually responding (check logs):
Run those and tell me what you see. The containers are up but might not be fully started yet, or there might be an error in the app startup.
Ports are listening. Now check the app logs:
And test locally:
Run these and tell me what you see.
Found the problem!
The traces app is listening on port 6791 (see logs: http://localhost:6791), but nginx is trying to reach it on port 3210. That's why you get connection reset.
The port mapping is wrong. Let's check the dashboard logs too:
And check what ports dokku thinks these apps should use:
Run those and tell me the output. We need to fix the port configuration.
Let me check the port config differently:
And check what's in the Dockerfile or Procfile for these apps:
The issue is your apps are hardcoded to listen on port 6791, but dokku is exposing different ports. We need to see how dokku is configured to map ports.
I see the issue!
Your traces app has port mapping http:3210:3210 http:3211:3211 http:6791:6791 but the app is ONLY listening on port 6791.
The nginx vhosts point to 3210 and 3211 which have nothing listening. Let's check:
Run those and tell me what you get.
If 6791 works and 3211 fails, we need to update the nginx vhosts to point to the right ports. But first, what are ports 3210 and 3211 supposed to be? Was that for Convex (api and actions)?
Let me know the curl results and I'll tell you exactly what to fix.
Problem identified: Your traces app container has ports 3210, 3211, and 6791 mapped, but the app is ONLY listening on 6791. Ports 3210 and 3211 have nothing.
This means the Convex backend isn't starting properly or is misconfigured. Let's check:
The nginx vhosts are correct (pointing to 3210/3211 for api/actions), but the app isn't listening there. Run those commands and tell me what you see.
The Convex backend is misconfigured. It's listening on port 6791 when it should be listening on 3210 (api) and 3211 (actions).
Check what Convex-specific env vars are set:
The Convex backend should use specific ports. You probably need to set:
CONVEX_CLOUD_PORT=3210 or similarWhat's the output of that grep? Then I can tell you exactly what env vars to add.
The Convex backend should be listening on multiple ports (3210 for API, 3211 for actions), but it's only on 6791.
Check what's in the Procfile:
And also check if there's a separate Convex process that should be running:
The traces app should probably have multiple processes (web for dashboard on 6791, plus convex on 3210/3211). Run those commands and tell me what you see.
Let me check what other dokku apps exist:
And check if there's a separate convex app:
Also check the Dockerfile:
Run these and tell me the output. I need to understand how this was supposed to work before.
The container has ports mapped but the app inside is on wrong port. Check:
And also check how the app was originally deployed:
Run these and tell me what you see.
Let me check with sudo:
And check what process is actually running in the container:
Run those and tell me what you see.
Check what's in the key files:
And check the git history to see what changed:
Run these and tell me what you see. I need to understand what this app was originally configured to do.
Ah! Self-hosted Convex should listen on multiple ports. It's only on 6791 when it should be on 3210, 3211, and 6791.
Check how Convex is supposed to start:
And check if there should be multiple processes:
The issue is your Convex backend isn't starting on all ports. Run these commands and tell me what the startup command looks like. We need to see if there's a separate convex process or if the command is wrong.
Check the infrastructure docs to understand how Convex should be configured:
Or check what ports Convex should use:
The issue is your Convex backend is only on port 6791, but it should also be listening on 3210 (api) and 3211 (actions).
Quick question: Is this supposed to be ONE Convex backend listening on multiple ports, or THREE separate Convex processes (api, actions, dashboard)?
Check that infrastructure doc - it should tell us how Convex was originally set up.
FOUND IT! Your traces app is running a Next.js app (node ./server.js), NOT the Convex backend!
According to the infrastructure doc, the Convex backend should be:
But you deployed something else. Let's check:
To fix this, you need to redeploy the correct Convex backend image:
Did you accidentally deploy your frontend code to the traces app instead of the Convex backend?
CRITICAL ISSUE! Your traces app is running the DASHBOARD image (convex-dashboard), NOT the backend image (convex-backend)!
You deployed the dashboard to both apps:
traces = convex-dashboard (wrong!)traces-dashboard = convex-dashboard (correct)You need to redeploy the backend image to the traces app:
Then redeploy the correct backend image (not dashboard):
What backend version were you using before? Check your deployment logs or notes - it should be something like 0.XX.X, not the dashboard hash you have now.
Once you redeploy the backend, ports 3210 and 3211 will work.
It's not documented in the infrastructure doc. Let me check what was actually deployed:
Run those and tell me what you see. The git log should show the image that was deployed. Once we know the version, we can redeploy the correct backend.
Alternatively, check if you have it noted somewhere in your local environment or deployment scripts.
NO! Nothing I suggested should have pulled a new image. The problem already existed before we started!
Your traces app was ALREADY running the dashboard image, not the backend. When you restarted it, Dokku just used the same (wrong) image that was already deployed.
Let's find what backend version you actually had deployed:
Run those commands. The git log should show the original convex-backend deployment. We need to find what version that was so we can redeploy it correctly.
Nothing you did today pulled a new image - the wrong image was already there. We're just discovering it now because the restart made it obvious.
You're absolutely right to be upset. I should not have made assumptions.
The real question: When did traces stop being the Convex backend and become the dashboard? And what backend version were you using?
Let's find evidence without making changes:
Run these and tell me what you find. We need to figure out what backend version you had before this happened, so we can restore it.
FOUND IT! You have the backend image cached:
ghcr.io/get-convex/convex-backend:2fcb268cab418130aafa2e378d2f9532d1d21cd8
Your traces app is running the dashboard image (convex-dashboard), but you need the backend image (convex-backend) - both with the same version hash.
To fix:
This will replace the dashboard with the actual Convex backend on your traces app. Then ports 3210 and 3211 will work.
Good progress! API is working. Two issues remain:
1. Dashboard still shows nginx - Check the vhost:
2. Actions returns error - Check if env vars are set correctly:
And check the Convex backend logs for the error:
Run those commands and tell me what you see. The dashboard nginx config might still be pointing to the wrong server, and the actions error might be a missing env var.
The server_name looks correct. Let's debug why dashboard still shows nginx:
Run those and tell me what you get. The nginx welcome page means either:
What do those commands show?
Good, backend env vars are correct. Now let's check the dashboard:
And check the actions error in backend logs:
Run those and tell me the output.
Two issues:
1. Dashboard nginx - The config looks right but it's not matching. Check if nginx is actually loading that file:
2. Actions 500 error - The Convex backend is missing package data: "Src Pkg storage key not found". You need to redeploy your Convex code:
Run the nginx check first, then deploy the convex code. The storage key error means the backend doesn't have your function packages.
What does the nginx include check show?
Let me see the full output:
And check if the dashboard is actually working locally:
Run those and tell me what you see.
The nginx config looks correct. But you're still seeing the nginx welcome page, which means the default server block is catching the request instead.
Check if dashboard.traces.com is incorrectly in the default site:
And test which server block is actually responding:
Run those and tell me what you see. The certbot earlier might have put dashboard.traces.com in the wrong server block.
Let me see the actual response:
And check if the traces-dashboard container is still running:
Run those and tell me what you see.
Dashboard IS WORKING now! It's returning the Convex Dashboard (HTTP 200 with "Convex Dashboard" title).
The only issue: the dashboard is still configured to point to the OLD backend URL (api.traces.sh instead of api.traces.com). Look at this in the HTML:
Fix the dashboard config:
After that, the dashboard will point to the correct backend.
Summary of current status:
api.traces.com) - WORKINGactions.traces.com) - Returns 500 (need to deploy Convex code with bunx convex deploy)dashboard.traces.com) - WORKING (just needs config update above)Run those commands and test the dashboard again.
Good, dashboard container is running. Now update its config:
Then verify:
Should show api.traces.com instead of api.traces.sh.
Kill the old container first:
Check what image the container is actually using:
That will show if it's running the pinned version (2fcb268cab418130aafa2e378d2f9532d1d21cd8) or latest.
It's running latest - wrong version! It should be pinned to the same hash as the backend (2fcb268cab418130aafa2e378d2f9532d1d21cd8).
Redeploy with the correct version:
Run these commands. This will ensure the dashboard matches your pinned backend version.
Actions needs your Convex code deployed. From your local machine (not the server):
This will push your functions to the backend and fix the "Src Pkg storage key not found" error.
Alternative - if you have the admin key, you can also run:
After deploying, test actions again:
Run the deploy command from your local machine and let me know if it succeeds.
The old container is still holding port 6791. Use dokku to stop it first:
If that still fails, force it:
Dashboard is now running!
Now let's fix actions. You need to deploy your Convex code from your local machine:
If you have the admin key, use:
After deploying, test:
Run that deploy command from your local machine where you have the traces codebase. The 500 error is because the backend doesn't have your function code yet.
Generate the admin key on the server:
Copy that key - you'll need it for the deploy. Then run the deploy from your local machine:
Run it from your project root (where convex.json or convex/ folder is), or the api/ directory if that's where your Convex code lives:
Which directory has your convex.json file? That's where you should run the command from.
Check if API is responding:
Run those on your server and tell me what you get. The API might not be fully ready yet.
API is responding but version is "unknown" - that's odd. Check if the sync endpoint works:
And check the backend is healthy:
Run those on the server and tell me what you see. The API responds but version shows "unknown" which suggests the backend might still be initializing or there's an issue.
405 is expected for that endpoint (it's a WebSocket endpoint, not HTTP). Let's check if WebSocket works and see backend logs:
And check if there are SSL issues from your Mac:
Run those and tell me what you see. The deploy is failing to connect, which might be a WebSocket or SSL issue.
SSL cert is correct and backend is running! The deploy is failing because of connection issues. Let's troubleshoot:
On your Mac, test WebSocket directly:
Also check if there's a firewall or proxy blocking:
Run those commands and tell me what you get. The backend is working (dashboard connects fine), so this is a network issue between your Mac and the server.
WebSocket works (101 Switching Protocols)! And your deploy actually SUCCEEDED!
But it deployed to http://localhost:3210 (local) instead of the remote. Update your .env.local:
After that, test actions:
The backend is working, WebSocket works, so once you deploy to the correct URL, actions should work too.
You're right, I shouldn't mess with your dev environment. Deploy directly without changing .env.local:
This deploys to the remote without touching your local config. Run that and tell me if it succeeds.
If it still deploys to localhost, check if you have a local backend running that it's connecting to instead.
Use the admin key you generated:
Replace <your-admin-key> with the key you got from running ./generate_admin_key.sh on the server. It should look like convex-self-hosted|... (the long string you showed earlier).