Guidance for Updating Domains in Convex Deployment
Guidance for Updating Domains in Convex Deployment
Agent
OpenCode
Model
Kimi K2.5 Free
Shared
Feb 10, 7:09 PM
Messages182
Downloads15
Based on the transition doc, here's what's left to do since you've completed DNS:
Environment Variables to Update:
TRACES_URL=https://www.traces.comTRACES_HTTP_URL=https://actions.traces.comNEXT_PUBLIC_CONVEX_URL=wss://api.traces.comNEXT_PUBLIC_CONVEX_HTTP_URL=https://actions.traces.comDokku/Server Tasks:
Verification:
https://www.traces.com with no CORS errorsGET /v1/session returns 401 with valid CORS headersJust need to run verification:
cd cli && bun run dev smoke test (login/share/logout URLs)cd frontend && bun run build and verify install links show traces.comtraces.sh references remainCritical: Keep *.traces.sh routing active as an alias during the transition window until all clients have updated.
Want me to help with any specific phase?
Port conflict! The old container is still holding port 3210. Let's force a clean restart:
If that doesn't work, try a rebuild:
The root cause is the zero-downtime deploy failed because there's no healthcheck in your app.json. After we get this working, you should add a healthcheck to prevent this in the future.
Add sudo to docker commands:
Or try rebuilding directly:
If that still fails, try force-restarting:
Kill the old container first:
If you get an error about the container being in use, force it:
You're absolutely right - that's an error in the doc. The dashboard should be:
Not api.traces.com. The domain mapping clearly shows:
api.traces.com = Convex API endpointdashboard.traces.com = Dashboard endpointGood catch! The transition doc has a typo there.
You probably don't need it if the dashboard already works. That config var is likely used by the dashboard to know where the Convex backend is.
Check if your dashboard is already working:
https://dashboard.traces.com (or wherever your dashboard lives) load properly?If yes, skip that step. The doc assumes a specific setup but your dashboard might already be configured correctly or use a different env var name.
Run:
If NEXT_PUBLIC_DEPLOYMENT_URL isn't already set and the dashboard works, don't add it.
Your nginx vhosts for the new *.traces.com domains aren't configured yet. That's why you're seeing the default nginx page.
Check what vhosts exist:
And check your dokku apps:
You'll need to either:
What's the output of those commands? That'll tell us if the dashboard is a separate dokku app or if it needs manual nginx config.
Same port issue! Kill the old container first:
Then update nginx - your vhost only has dashboard.traces.sh, not dashboard.traces.com:
Add dashboard.traces.com to the server_name line:
Then reload nginx and get SSL cert:
Run these one at a time and let me know if any fail.
YES this is bad - certbot put all the new domains in the DEFAULT site instead of your app-specific vhosts. That's why you're seeing the nginx welcome page.
Let's fix this:
Let me know the output of step 1 first so I can see exactly what certbot did.
Certbot messed up - it put the new domains in the default site instead of your app vhosts. Let's fix this:
Let me know if that works and I'll do the same for actions and dashboard.
You're right, let me be systematic.
Certbot put your new domains in the WRONG place (the default nginx site instead of your app sites). That's why you see "Welcome to nginx!"
Step 1: Check current state
Run these three commands and paste the output. Then I'll give you the EXACT fix commands based on what I see.
Good! I can see the state now. Dashboard is partially fixed, but api and actions still only have the old domains.
The new SSL cert exists at /etc/letsencrypt/live/api.traces.com/ but your api and actions vhosts are still using the old cert and don't have the new domains.
Step 1: Fix the api vhost
Step 2: Fix the actions vhost
Step 3: Clean up the default site (remove the wrong config certbot added)
Step 4: Test and reload
Run these one at a time and tell me the output of step 4.
nginx is now valid and reloaded!
Let's verify the sites work:
Test the new domains:
What should you see:
api.traces.com → should NOT be the nginx welcome pageactions.traces.com → should NOT be the nginx welcome pagedashboard.traces.com → should show your dashboard (not nginx)Run those three curl commands and tell me what HTTP codes you get (200, 404, etc.).
Two problems:
Let's check:
Run those and tell me the output. The 502 means your apps (traces, traces-dashboard) aren't responding on those ports, so we need to see if they're even running.
Containers are running. Now check if they're actually listening:
And check if the apps are actually responding (check logs):
Run those and tell me what you see. The containers are up but might not be fully started yet, or there might be an error in the app startup.
Ports are listening. Now check the app logs:
And test locally:
Run these and tell me what you see.
Found the problem!
The traces app is listening on port 6791 (see logs: http://localhost:6791), but nginx is trying to reach it on port 3210. That's why you get connection reset.
The port mapping is wrong. Let's check the dashboard logs too:
And check what ports dokku thinks these apps should use:
Run those and tell me the output. We need to fix the port configuration.
Let me check the port config differently:
And check what's in the Dockerfile or Procfile for these apps:
The issue is your apps are hardcoded to listen on port 6791, but dokku is exposing different ports. We need to see how dokku is configured to map ports.
I see the issue!
Your traces app has port mapping http:3210:3210 http:3211:3211 http:6791:6791 but the app is ONLY listening on port 6791.
The nginx vhosts point to 3210 and 3211 which have nothing listening. Let's check:
Run those and tell me what you get.
If 6791 works and 3211 fails, we need to update the nginx vhosts to point to the right ports. But first, what are ports 3210 and 3211 supposed to be? Was that for Convex (api and actions)?
Let me know the curl results and I'll tell you exactly what to fix.
Problem identified: Your traces app container has ports 3210, 3211, and 6791 mapped, but the app is ONLY listening on 6791. Ports 3210 and 3211 have nothing.
This means the Convex backend isn't starting properly or is misconfigured. Let's check:
The nginx vhosts are correct (pointing to 3210/3211 for api/actions), but the app isn't listening there. Run those commands and tell me what you see.
The Convex backend is misconfigured. It's listening on port 6791 when it should be listening on 3210 (api) and 3211 (actions).
Check what Convex-specific env vars are set:
The Convex backend should use specific ports. You probably need to set:
CONVEX_CLOUD_PORT=3210 or similarWhat's the output of that grep? Then I can tell you exactly what env vars to add.
The Convex backend should be listening on multiple ports (3210 for API, 3211 for actions), but it's only on 6791.
Check what's in the Procfile:
And also check if there's a separate Convex process that should be running:
The traces app should probably have multiple processes (web for dashboard on 6791, plus convex on 3210/3211). Run those commands and tell me what you see.
Let me check what other dokku apps exist:
And check if there's a separate convex app:
Also check the Dockerfile:
Run these and tell me the output. I need to understand how this was supposed to work before.
The container has ports mapped but the app inside is on wrong port. Check:
And also check how the app was originally deployed:
Run these and tell me what you see.
Let me check with sudo:
And check what process is actually running in the container:
Run those and tell me what you see.
Check what's in the key files:
And check the git history to see what changed:
Run these and tell me what you see. I need to understand what this app was originally configured to do.
Ah! Self-hosted Convex should listen on multiple ports. It's only on 6791 when it should be on 3210, 3211, and 6791.
Check how Convex is supposed to start:
And check if there should be multiple processes:
The issue is your Convex backend isn't starting on all ports. Run these commands and tell me what the startup command looks like. We need to see if there's a separate convex process or if the command is wrong.
Check the infrastructure docs to understand how Convex should be configured:
Or check what ports Convex should use:
The issue is your Convex backend is only on port 6791, but it should also be listening on 3210 (api) and 3211 (actions).
Quick question: Is this supposed to be ONE Convex backend listening on multiple ports, or THREE separate Convex processes (api, actions, dashboard)?
Check that infrastructure doc - it should tell us how Convex was originally set up.
FOUND IT! Your traces app is running a Next.js app (node ./server.js), NOT the Convex backend!
According to the infrastructure doc, the Convex backend should be:
But you deployed something else. Let's check:
To fix this, you need to redeploy the correct Convex backend image:
Did you accidentally deploy your frontend code to the traces app instead of the Convex backend?
CRITICAL ISSUE! Your traces app is running the DASHBOARD image (convex-dashboard), NOT the backend image (convex-backend)!
You deployed the dashboard to both apps:
traces = convex-dashboard (wrong!)traces-dashboard = convex-dashboard (correct)You need to redeploy the backend image to the traces app:
Then redeploy the correct backend image (not dashboard):
What backend version were you using before? Check your deployment logs or notes - it should be something like 0.XX.X, not the dashboard hash you have now.
Once you redeploy the backend, ports 3210 and 3211 will work.
It's not documented in the infrastructure doc. Let me check what was actually deployed:
Run those and tell me what you see. The git log should show the image that was deployed. Once we know the version, we can redeploy the correct backend.
Alternatively, check if you have it noted somewhere in your local environment or deployment scripts.
NO! Nothing I suggested should have pulled a new image. The problem already existed before we started!
Your traces app was ALREADY running the dashboard image, not the backend. When you restarted it, Dokku just used the same (wrong) image that was already deployed.
Let's find what backend version you actually had deployed:
Run those commands. The git log should show the original convex-backend deployment. We need to find what version that was so we can redeploy it correctly.
Nothing you did today pulled a new image - the wrong image was already there. We're just discovering it now because the restart made it obvious.
You're absolutely right to be upset. I should not have made assumptions.
The real question: When did traces stop being the Convex backend and become the dashboard? And what backend version were you using?
Let's find evidence without making changes:
Run these and tell me what you find. We need to figure out what backend version you had before this happened, so we can restore it.
FOUND IT! You have the backend image cached:
Your traces app is running the dashboard image (convex-dashboard), but you need the backend image (convex-backend) - both with the same version hash.
To fix:
This will replace the dashboard with the actual Convex backend on your traces app. Then ports 3210 and 3211 will work.
Good progress! API is working. Two issues remain:
1. Dashboard still shows nginx - Check the vhost:
2. Actions returns error - Check if env vars are set correctly:
And check the Convex backend logs for the error:
Run those commands and tell me what you see. The dashboard nginx config might still be pointing to the wrong server, and the actions error might be a missing env var.
The server_name looks correct. Let's debug why dashboard still shows nginx:
Run those and tell me what you get. The nginx welcome page means either:
What do those commands show?
Good, backend env vars are correct. Now let's check the dashboard:
And check the actions error in backend logs:
Run those and tell me the output.
Two issues:
1. Dashboard nginx - The config looks right but it's not matching. Check if nginx is actually loading that file:
2. Actions 500 error - The Convex backend is missing package data: "Src Pkg storage key not found". You need to redeploy your Convex code:
Run the nginx check first, then deploy the convex code. The storage key error means the backend doesn't have your function packages.
What does the nginx include check show?
Let me see the full output:
And check if the dashboard is actually working locally:
Run those and tell me what you see.
The nginx config looks correct. But you're still seeing the nginx welcome page, which means the default server block is catching the request instead.
Check if dashboard.traces.com is incorrectly in the default site:
And test which server block is actually responding:
Run those and tell me what you see. The certbot earlier might have put dashboard.traces.com in the wrong server block.
Let me see the actual response:
And check if the traces-dashboard container is still running:
Run those and tell me what you see.
Dashboard IS WORKING now! It's returning the Convex Dashboard (HTTP 200 with "Convex Dashboard" title).
The only issue: the dashboard is still configured to point to the OLD backend URL (api.traces.sh instead of api.traces.com). Look at this in the HTML:
Fix the dashboard config:
After that, the dashboard will point to the correct backend.
Summary of current status:
api.traces.com) - WORKINGactions.traces.com) - Returns 500 (need to deploy Convex code with bunx convex deploy)dashboard.traces.com) - WORKING (just needs config update above)Run those commands and test the dashboard again.
Good, dashboard container is running. Now update its config:
Then verify:
Should show api.traces.com instead of api.traces.sh.
Kill the old container first:
Check what image the container is actually using:
That will show if it's running the pinned version (2fcb268cab418130aafa2e378d2f9532d1d21cd8) or latest.
It's running latest - wrong version! It should be pinned to the same hash as the backend (2fcb268cab418130aafa2e378d2f9532d1d21cd8).
Redeploy with the correct version:
Run these commands. This will ensure the dashboard matches your pinned backend version.
Actions needs your Convex code deployed. From your local machine (not the server):
This will push your functions to the backend and fix the "Src Pkg storage key not found" error.
Alternative - if you have the admin key, you can also run:
After deploying, test actions again:
Run the deploy command from your local machine and let me know if it succeeds.
The old container is still holding port 6791. Use dokku to stop it first:
If that still fails, force it:
Dashboard is now running!
Now let's fix actions. You need to deploy your Convex code from your local machine:
If you have the admin key, use:
After deploying, test:
Run that deploy command from your local machine where you have the traces codebase. The 500 error is because the backend doesn't have your function code yet.
Generate the admin key on the server:
Copy that key - you'll need it for the deploy. Then run the deploy from your local machine:
Run it from your project root (where convex.json or convex/ folder is), or the api/ directory if that's where your Convex code lives:
Which directory has your convex.json file? That's where you should run the command from.
Check if API is responding:
Run those on your server and tell me what you get. The API might not be fully ready yet.
API is responding but version is "unknown" - that's odd. Check if the sync endpoint works:
And check the backend is healthy:
Run those on the server and tell me what you see. The API responds but version shows "unknown" which suggests the backend might still be initializing or there's an issue.
405 is expected for that endpoint (it's a WebSocket endpoint, not HTTP). Let's check if WebSocket works and see backend logs:
And check if there are SSL issues from your Mac:
Run those and tell me what you see. The deploy is failing to connect, which might be a WebSocket or SSL issue.
SSL cert is correct and backend is running! The deploy is failing because of connection issues. Let's troubleshoot:
On your Mac, test WebSocket directly:
Also check if there's a firewall or proxy blocking:
Run those commands and tell me what you get. The backend is working (dashboard connects fine), so this is a network issue between your Mac and the server.
WebSocket works (101 Switching Protocols)! And your deploy actually SUCCEEDED!
But it deployed to http://localhost:3210 (local) instead of the remote. Update your .env.local:
After that, test actions:
The backend is working, WebSocket works, so once you deploy to the correct URL, actions should work too.
You're right, I shouldn't mess with your dev environment. Deploy directly without changing .env.local:
This deploys to the remote without touching your local config. Run that and tell me if it succeeds.
If it still deploys to localhost, check if you have a local backend running that it's connecting to instead.
Use the admin key you generated:
Replace <your-admin-key> with the key you got from running ./generate_admin_key.sh on the server. It should look like convex-self-hosted|... (the long string you showed earlier).
# 1. Update Convex backend originsdokku config:set traces CONVEX_CLOUD_ORIGIN="https://api.traces.com" CONVEX_SITE_ORIGIN="https://actions.traces.com"dokku ps:restart traces
# 2. Issue TLS certs for new domainssudo certbot --nginx -d api.traces.com -d actions.traces.com -d dashboard.traces.com
# 3. Update dashboard appdokku config:set traces-dashboard NEXT_PUBLIC_DEPLOYMENT_URL="https://api.traces.com"dokku ps:restart traces-dashboardError response from daemon: failed to set up container networking: driver failed programming external connectivity on endpoint traces.web.1.upcoming-9276 ... Bind for 127.0.0.1:3210 failed: port is already allocated# Kill any existing containers holding the portdocker ps | grep traces.webdocker stop <container_id_from_above>docker rm <container_id_from_above>
# Or force kill all traces containersdocker ps -q --filter "name=traces" | xargs -r docker stopdocker ps -aq --filter "name=traces" | xargs -r docker rm
# Then restart the appdokku ps:restart tracesdokku ps:rebuild tracessudo docker ps | grep traces.websudo docker stop $(sudo docker ps -q --filter "name=traces")sudo docker ps -aq --filter "name=traces" | xargs -r sudo docker rmdokku ps:restart tracesdokku ps:rebuild tracesdokku ps:stop tracesdokku ps:start tracessudo docker stop ba4c89b647d6sudo docker rm ba4c89b647d6dokku ps:restart tracessudo docker rm -f ba4c89b647d6dokku ps:restart traces- [ ] Dokku (dashboard app): update deployment URL and redeploy/restart: - `dokku config:set traces-dashboard NEXT_PUBLIC_DEPLOYMENT_URL="https://api.traces.com"`- [ ] Dokku (dashboard app): update deployment URL and redeploy/restart: - `dokku config:set traces-dashboard NEXT_PUBLIC_DEPLOYMENT_URL="https://api.traces.com"`dokku config:set traces-dashboard NEXT_PUBLIC_DEPLOYMENT_URL="https://dashboard.traces.com"- [ ] Dokku (dashboard app): update deployment URL and redeploy/restart: - `dokku config:set traces-dashboard NEXT_PUBLIC_DEPLOYMENT_URL="https://api.traces.com"`dokku config traces-dashboardls /etc/nginx/sites-enabled/cat /etc/nginx/sites-enabled/traces-dashboarddokku apps:listlisten 443 ssl; # managed by Certbotssl_certificate /etc/letsencrypt/live/api.traces.sh/fullchain.pem; # managed by Certbotssl_certificate_key /etc/letsencrypt/live/api.traces.sh/privkey.pem; # managed by Certbotinclude /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbotssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbotsudo docker ps | grep dashboardsudo docker stop <container_id>sudo docker rm <container_id>dokku ps:restart traces-dashboardsudo nano /etc/nginx/sites-enabled/traces-dashboardserver_name dashboard.traces.sh dashboard.traces.com;sudo nginx -tsudo systemctl reload nginxsudo certbot --nginx -d dashboard.traces.com# 1. Kill old dashboard containersudo docker stop 5766f0e69df6sudo docker rm 5766f0e69df6
# 2. Restart dashboard appdokku ps:restart traces-dashboard
# 3. Update nginx vhost to add dashboard.traces.comsudo sed -i 's/server_name dashboard.traces.sh;/server_name dashboard.traces.sh dashboard.traces.com;/' /etc/nginx/sites-enabled/traces-dashboard
# 4. Test and reload nginxsudo nginx -t && sudo systemctl reload nginx
# 5. Get SSL cert for new domainsudo certbot --nginx -d dashboard.traces.com# 1. Check what certbot did to defaultcat /etc/nginx/sites-enabled/default
# 2. Remove the domains from default site (they don't belong there)sudo sed -i '/server_name api.traces.com/d' /etc/nginx/sites-enabled/defaultsudo sed -i '/server_name actions.traces.com/d' /etc/nginx/sites-enabled/defaultsudo sed -i '/server_name dashboard.traces.com/d' /etc/nginx/sites-enabled/default
# 3. Now add the new domains to the correct vhostssudo sed -i 's/server_name api.traces.sh;/server_name api.traces.sh api.traces.com;/' /etc/nginx/sites-enabled/traces-apisudo sed -i 's/server_name actions.traces.sh;/server_name actions.traces.sh actions.traces.com;/' /etc/nginx/sites-enabled/traces-actionssudo sed -i 's/server_name dashboard.traces.sh;/server_name dashboard.traces.sh dashboard.traces.com;/' /etc/nginx/sites-enabled/traces-dashboard
# 4. Test and reloadsudo nginx -t && sudo systemctl reload nginx # SSL configuration # # listen 443 ssl default_server; # listen [::]:443 ssl default_server; # # Note: You should disable gzip for SSL traffic. # See: https://bugs.debian.org/773332 # # Read up on ssl_ciphers to ensure a secure configuration. # See: https://bugs.debian.org/765782 # # Self signed certs generated by the ssl-cert package # Don't use them in a production server! # # include snippets/snakeoil.conf;
root /var/www/html;
# Add index.php to the list if you are using PHP index index.html index.htm index.nginx-debian.html;
server_name _;
location / { # First attempt to serve request as file, then # as directory, then fall back to displaying a 404. try_files $uri $uri/ =404; }
# pass PHP scripts to FastCGI server # #location ~ \.php$ { # include snippets/fastcgi-php.conf; # # # With php-fpm (or other unix sockets): # fastcgi_pass unix:/run/php/php7.4-fpm.sock; # # With php-cgi (or other tcp sockets): # fastcgi_pass 127.0.0.1:9000; #}
# deny access to .htaccess files, if Apache's document root # concurs with nginx's one # #location ~ /\.ht { # deny all; #} # SSL configuration # # listen 443 ssl default_server; # listen [::]:443 ssl default_server; # # Note: You should disable gzip for SSL traffic. # See: https://bugs.debian.org/773332 # # Read up on ssl_ciphers to ensure a secure configuration. # See: https://bugs.debian.org/765782 # # Self signed certs generated by the ssl-cert package # Don't use them in a production server! # # include snippets/snakeoil.conf;
root /var/www/html;
# Add index.php to the list if you are using PHP index index.html index.htm index.nginx-debian.html;server_name dashboard.traces.com api.traces.com actions.traces.com; # managed by Certbot
location / { # First attempt to serve request as file, then # as directory, then fall back to displaying a 404. try_files $uri $uri/ =404; }
# pass PHP scripts to FastCGI server # #location ~ \.php$ { # include snippets/fastcgi-php.conf; # # # With php-fpm (or other unix sockets): # fastcgi_pass unix:/run/php/php7.4-fpm.sock; # # With php-cgi (or other tcp sockets): # fastcgi_pass 127.0.0.1:9000; #}
# deny access to .htaccess files, if Apache's document root # concurs with nginx's one # #location ~ /\.ht { # deny all; #}
listen [::]:443 ssl ipv6only=on; # managed by Certbotlisten 443 ssl; # managed by Certbotssl_certificate /etc/letsencrypt/live/api.traces.com/fullchain.pem; # managed by Certbotssl_certificate_key /etc/letsencrypt/live/api.traces.com/privkey.pem; # managed by Certbotinclude /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbotssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbotif ($host = actions.traces.com) { return 301 https://$host$request_uri;} # managed by Certbot
if ($host = api.traces.com) { return 301 https://$host$request_uri;} # managed by Certbot
listen 80 ; listen [::]:80 ;server_name dashboard.traces.com api.traces.com actions.traces.com;return 404; # managed by Certbot# 1. Remove the certbot-added server blocks from default (keep only the first default server block)sudo head -n 89 /etc/nginx/sites-enabled/default > /tmp/default_backup && sudo mv /tmp/default_backup /etc/nginx/sites-enabled/default
# 2. Update traces-api vhost to add api.traces.comsudo bash -c 'cat > /etc/nginx/sites-enabled/traces-api << "EOF"server { server_name api.traces.sh api.traces.com; location / { proxy_pass http://127.0.0.1:3210; proxy_set_header Host $host; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; } listen 443 ssl; ssl_certificate /etc/letsencrypt/live/api.traces.com/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/api.traces.com/privkey.pem; include /etc/letsencrypt/options-ssl-nginx.conf; ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;}server { if ($host = api.traces.sh) { return 301 https://$host$request_uri; } if ($host = api.traces.com) { return 301 https://$host$request_uri; } server_name api.traces.sh api.traces.com; listen 80; return 404;}EOF'
# 3. Test and reloadsudo nginx -t && sudo systemctl reload nginxcat /etc/nginx/sites-enabled/traces-api | head -20cat /etc/nginx/sites-enabled/traces-actions | head -20cat /etc/nginx/sites-enabled/traces-dashboard | head -20listen 443 ssl; # managed by Certbotssl_certificate /etc/letsencrypt/live/api.traces.sh/fullchain.pem; # managed by Certbotssl_certificate_key /etc/letsencrypt/live/api.traces.sh/privkey.pem; # managed by Certbotinclude /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbotssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbotlisten 443 ssl; # managed by Certbotssl_certificate /etc/letsencrypt/live/api.traces.sh/fullchain.pem; # managed by Certbotssl_certificate_key /etc/letsencrypt/live/api.traces.sh/privkey.pem; # managed by Certbotinclude /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbotssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbotlisten 443 ssl; # managed by Certbotssl_certificate /etc/letsencrypt/live/dashboard.traces.com/fullchain.pem; # managed by Certbotssl_certificate_key /etc/letsencrypt/live/dashboard.traces.com/privkey.pem; # managed by Certbotinclude /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbotssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbotsudo sed -i 's/server_name api.traces.sh;/server_name api.traces.sh api.traces.com;/' /etc/nginx/sites-enabled/traces-apisudo sed -i 's|/etc/letsencrypt/live/api.traces.sh/|/etc/letsencrypt/live/api.traces.com/|g' /etc/nginx/sites-enabled/traces-apisudo sed -i 's/server_name actions.traces.sh;/server_name actions.traces.sh actions.traces.com;/' /etc/nginx/sites-enabled/traces-actionssudo sed -i 's|/etc/letsencrypt/live/api.traces.sh/|/etc/letsencrypt/live/api.traces.com/|g' /etc/nginx/sites-enabled/traces-actionssudo head -n 89 /etc/nginx/sites-enabled/default > /tmp/default_clean && sudo mv /tmp/default_clean /etc/nginx/sites-enabled/defaultsudo nginx -t && sudo systemctl reload nginxcurl -I https://api.traces.comcurl -I https://actions.traces.comcurl -I https://dashboard.traces.com# Check if containers are runningsudo docker ps | grep traces
# Check what ports they're actually listening onsudo netstat -tlnp | grep -E '3210|3211|6791'sudo netstat -tlnp | grep -E '3210|3211|6791'dokku logs traces --tail 20dokku logs traces-dashboard --tail 20dokku logs traces --tail 20dokku logs traces-dashboard --tail 20curl -I http://127.0.0.1:3210curl -I http://127.0.0.1:6791dokku logs traces-dashboard --tail 10dokku proxy:ports tracesdokku proxy:ports traces-dashboarddokku config traces | grep PORTdokku config traces-dashboard | grep PORTdokku ps:report tracesdokku ps:report traces-dashboard# Check what's actually on those portscurl -I http://127.0.0.1:6791curl -I http://127.0.0.1:3211# Check full logs for errorsdokku logs traces --tail 50 | head -50
# Check what port the app thinks it should usedokku config traces | grep -E 'PORT|URL'db695a0f7407 dokku/traces:latest "node ./server.js" 15 minutes ago Up 15 minutes 127.0.0.1:3210-3211->3210-3211/tcp, 6791/tcp traces.web.1dokku config traces | grep -i convexdokku ps:report traces | grep -A5 "Procfile"cat /home/dokku/traces/Procfile 2>/dev/null || echo "No Procfile found"dokku ps:scale tracesdb695a0f7407 dokku/traces:latest "node ./server.js" 15 minutes ago Up 15 minutes 127.0.0.1:3210-3211->3210-3211/tcp, 6791/tcp traces.web.1dokku apps:listsudo docker ps | grep -E '3210|3211'cat /home/dokku/traces/Dockerfile 2>/dev/null | head -30# Check what's in the app directoryls -la /home/dokku/traces/
# Check the Dockerfilecat /home/dokku/traces/Dockerfile 2>/dev/null || echo "No Dockerfile"
# Check all dokku appsdokku apps:list# Check git remotecd /home/dokku/traces && git log --oneline -5 2>/dev/null || echo "Not a git repo"sudo ls -la /home/dokku/traces/sudo cat /home/dokku/traces/DOKKU_SCALE 2>/dev/nullsudo docker exec traces.web.1 ps auxsudo docker inspect traces.web.1 | grep -A5 '"ExposedPorts"'sudo cat /home/dokku/traces/PORT.web.1sudo cat /home/dokku/traces/ENVsudo cat /home/dokku/traces/VHOSTsudo git -C /home/dokku/traces log --oneline -10# Check what command starts the appsudo cat /home/dokku/traces/ENV | grep -i entrysudo docker inspect traces.web.1 | grep -A3 '"Cmd"'# Check the deployment configsudo cat /home/dokku/traces/DOCKER_OPTIONS_DEPLOYsudo cat /home/dokku/traces/DOKKU_SCALE 2>/dev/null || echo "No scale file"127.0.0.1:3210-3211->3210-3211/tcp, 6791/tcpcat /home/dokku/traces/docs/deployment/infrastructure.md 2>/dev/null || cat ~/docs/deployment/infrastructure.md 2>/dev/null# Check if there's a convex-specific configfind ~ -name "convex*" -type f 2>/dev/null | head -5db695a0f7407 dokku/traces:latest "node ./server.js" 19 minutes ago Up 19 minutes 127.0.0.1:3210-3211->3210-3211/tcp, 6791/tcp traces.web.1dokku git:from-image traces ghcr.io/get-convex/convex-backend:latest# Check what image is actually runningsudo docker inspect traces.web.1 | grep -i image# 1. Stop the current wrong appdokku ps:stop traces
# 2. Deploy the CORRECT Convex backend imagedokku git:from-image traces ghcr.io/get-convex/convex-backend:latest
# 3. Make sure the env vars are still setdokku config:set traces \ CONVEX_CLOUD_ORIGIN="https://api.traces.com" \ CONVEX_SITE_ORIGIN="https://actions.traces.com""dokku/traces:latest""ghcr.io/get-convex/convex-dashboard:2fcb268cab418130aafa2e378d2f9532d1d21cd8"# Find what version you were using beforedokku config traces | grep -i image
# Or check git historysudo git -C /home/dokku/traces log --oneline -5# Replace VERSION with the actual backend version you were usingdokku git:from-image traces ghcr.io/get-convex/convex-backend:VERSION"com.dokku.docker-image-labeler/alternate-tags": "[\"ghcr.io/get-convex/convex-dashboard:2fcb268cab418130aafa2e378d2f9532d1d21cd8\"]"# Check the git history to see what image was deployedsudo git -C /home/dokku/traces log --oneline -10
# Or check if there's a version filels /home/dokku/traces/ | grep -i versioncat /home/dokku/traces/TAG 2>/dev/null || echo "No TAG file"db695a0f7407 dokku/traces:latest "node ./server.js" 15 minutes ago Up 15 minutes 127.0.0.1:3210-3211->3210-3211/tcp, 6791/tcp traces.web.1"com.dokku.docker-image-labeler/alternate-tags": "[\"ghcr.io/get-convex/convex-dashboard:2fcb268cab418130aafa2e378d2f9532d1d21cd8\"]"# Check git history to see what image was originally deployedsudo git -C /home/dokku/traces log --all --oneline -20
# Check if there's a refs file with the old imagels -la /home/dokku/traces/refs/cat /home/dokku/traces/refs/heads/master 2>/dev/null || cat /home/dokku/traces/refs/heads/main 2>/dev/null# Check docker image history to see what was running beforesudo docker images | grep convex
# Check if there are old containers that might show what version workedsudo docker ps -a | grep traces
# Check logs from before today - do they show backend startup?sudo ls -la /home/dokku/traces/logs/sudo cat /home/dokku/traces/logs/web-1/*.log 2>/dev/null | head -100sudo docker inspect traces.web.1 | grep -i image"com.dokku.docker-image-labeler/alternate-tags": "[\"ghcr.io/get-convex/convex-dashboard:2fcb268cab418130aafa2e378d2f9532d1d21cd8\"]"ghcr.io/get-convex/convex-backend:2fcb268cab418130aafa2e378d2f9532d1d21cd8# Deploy the CORRECT backend image to tracesdokku git:from-image traces ghcr.io/get-convex/convex-backend:2fcb268cab418130aafa2e378d2f9532d1d21cd8cat /etc/nginx/sites-enabled/traces-dashboard | grep server_namedokku config traces | grep -E 'CONVEX|ORIGIN'dokku logs traces --tail 50 | grep -i error# Test locally firstcurl -I http://127.0.0.1:6791
# Check if nginx is using the right configsudo nginx -T | grep -A10 "server_name dashboard"
# Check which server block is actually matchingcurl -I -H "Host: dashboard.traces.com" http://127.0.0.1# Test the dashboard container directlycurl -I http://127.0.0.1:6791
# Check which nginx server block is catching the requestcurl -I -H "Host: dashboard.traces.com" http://127.0.0.1
# Check nginx config testsudo nginx -T 2>/dev/null | grep -B5 -A10 "dashboard.traces.com"dokku logs traces --tail 20listen 443 ssl; # managed by Certbotssl_certificate /etc/letsencrypt/live/dashboard.traces.com/fullchain.pem; # managed by Certbotssl_certificate_key /etc/letsencrypt/live/dashboard.traces.com/privkey.pem; # managed by Certbotinclude /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbotssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbotif ($host = dashboard.traces.sh) { return 301 https://$host$request_uri;} # managed by Certbotsudo nginx -T 2>/dev/null | grep "sites-enabled" | head -5ls -la /etc/nginx/sites-enabled/traces-dashboard# From your local machine, deploy the convex codebunx convex deployls -la /etc/nginx/sites-enabled/traces-dashboardcat /etc/nginx/sites-enabled/traces-dashboardcurl -I http://127.0.0.1:6791curl -H "Host: dashboard.traces.com" http://127.0.0.1listen 443 ssl; # managed by Certbotssl_certificate /etc/letsencrypt/live/dashboard.traces.com/fullchain.pem; # managed by Certbotssl_certificate_key /etc/letsencrypt/live/dashboard.traces.com/privkey.pem; # managed by Certbotinclude /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbotssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbotif ($host = dashboard.traces.sh) { return 301 https://$host$request_uri;} # managed by Certbot}# configuration file /etc/nginx/sites-enabled/traces-dashboard:server { server_name dashboard.traces.sh dashboard.traces.com;sudo certbot --nginx -d api.traces.com -d actions.traces.com -d dashboard.traces.comgrep -r "dashboard.traces.com" /etc/nginx/sites-enabled/curl -k -v https://dashboard.traces.com 2>&1 | grep -E "(Server:|Connected to)"curl -k -I https://dashboard.traces.comcurl -k -s https://dashboard.traces.com | head -20sudo docker ps | grep dashboard<title data-next-head="">Convex Dashboard</title><meta name="description" content="Manage your Convex apps" data-next-head=""/>"deploymentUrl":"https://api.traces.sh""deploymentUrl":"https://api.traces.sh"dokku config:set traces-dashboard NEXT_PUBLIC_DEPLOYMENT_URL="https://api.traces.com"dokku ps:restart traces-dashboarddokku config:set traces-dashboard NEXT_PUBLIC_DEPLOYMENT_URL="https://api.traces.com"dokku ps:restart traces-dashboardcurl -k -s https://dashboard.traces.com | grep -o 'deploymentUrl":"[^"]*'sudo docker stop 5b5710e62cd8sudo docker rm 5b5710e62cd8dokku ps:restart traces-dashboarddokku ps:stop traces-dashboarddokku ps:start traces-dashboardtarunsachdeva@sd-183383:~$ sudo docker ps | grep dashboard5b5710e62cd8 dokku/traces-dashboard:latest "node ./server.js" 40 minutes ago Up 40 minutes 127.0.0.1:6791->6791/tcp traces-dashboard.web.1ghcr.io/get-convex/convex-dashboard:2fcb268cab418130aafa2e378d2f9532d1d21cd8 214a19b51a06 1.65GB 350MBghcr.io/get-convex/convex-dashboard:latest 14668ff7ce36 1.65GB 349MBsudo docker inspect 5b5710e62cd8 | grep -A2 '"Image"'sudo docker images | grep traces-dashboard "Image": "dokku/traces-dashboard:latest", "Volumes": null, "WorkingDir": "/app",# 1. Stop current dashboarddokku ps:stop traces-dashboard
# 2. Deploy the pinned version (same as backend)dokku git:from-image traces-dashboard ghcr.io/get-convex/convex-dashboard:2fcb268cab418130aafa2e378d2f9532d1d21cd8
# 3. Set the correct env vardokku config:set traces-dashboard NEXT_PUBLIC_DEPLOYMENT_URL="https://api.traces.com"# 1. Update your local .env.local to point to new domainecho 'CONVEX_SELF_HOSTED_URL=https://api.traces.com' > .env.local
# 2. Deploy your Convex codebunx convex deploybunx convex deploy --admin-key YOUR_ADMIN_KEY --url https://api.traces.comcurl -I https://actions.traces.comdokku ps:stop traces-dashboarddokku ps:start traces-dashboarddokku ps:rebuild traces-dashboard# On your LOCAL machine (not the server):cd /path/to/your/traces/project
# Update env to point to new domainecho 'CONVEX_SELF_HOSTED_URL=https://api.traces.com' > .env.local
# Deploy your Convex functionsbunx convex deploybunx convex deploy --admin-key YOUR_ADMIN_KEY --url https://api.traces.comcurl -I https://actions.traces.comdokku enter traces./generate_admin_key.shexitdokku enter traces./generate_admin_key.shexitbunx convex deploy --admin-key <the-key-you-just-copied> --url https://api.traces.comtraces/├── api/ # Convex backend (see api/AGENTS.md)# From the root of your traces project:bunx convex deploy --admin-key <key> --url https://api.traces.com
# Or if convex.json is in api/:cd apibunx convex deploy --admin-key <key> --url https://api.traces.comcurl -I https://api.traces.comcurl https://api.traces.com/versioncurl -I https://api.traces.com/api/1.31.4/syncdokku logs traces --tail 20# On server - check backend healthdokku logs traces --tail 30
# Test WebSocket connectioncurl -i -N -H "Connection: Upgrade" -H "Upgrade: websocket" -H "Host: api.traces.com" -H "Origin: https://api.traces.com" https://api.traces.com/api/1.31.4/sync 2>&1 | head -5# On your Mac - test SSLopenssl s_client -connect api.traces.com:443 -servername api.traces.com </dev/null 2>/dev/null | openssl x509 -noout -text | grep -A1 "Subject Alternative Name"# Test if WebSocket upgrade workscurl -i -N \ -H "Connection: Upgrade" \ -H "Upgrade: websocket" \ -H "Host: api.traces.com" \ -H "Origin: https://api.traces.com" \ -H "Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==" \ -H "Sec-WebSocket-Version: 13" \ https://api.traces.com/api/1.31.4/sync 2>&1 | head -10# Test basic connectivityping api.traces.com
# Test port 443nc -zv api.traces.com 443
# Try with verbose loggingbunx convex deploy --admin-key <your-key> --url https://api.traces.com --verbose 2>&1 | head -50# On your Macecho 'CONVEX_SELF_HOSTED_URL=https://api.traces.com' > .env.local
# Then deploy again to the remotebunx convex deploycurl -I https://actions.traces.combunx convex deploy --url https://api.traces.combunx convex deploy --admin-key convex-self-hosted|0139185e4215482be39a374b5fa8f64fc304ca33188457d5a3d0f61f0b6fe8b28854a641d5 --url https://api.traces.com --verbose 2>&1 | head -50bunx convex deploy --url https://api.traces.comdokku enter traces./generate_admin_key.shbunx convex deploy --admin-key <your-admin-key> --url https://api.traces.com