# Troubleshooting

Start with two things:

- **Admin → System health** (as platform staff). It shows workers, queues, integrations and configuration checks.
- **The logs.** Use `docker compose logs -f api worker` or `journalctl -u devwebs-api -u devwebs-worker -f`.

Every error page and 500 response shows a **reference**: the request id. Search the logs for it to find the exact
failure.

## Starting up

| Symptom | Cause and fix |
|---|---|
| API exits with `Refusing to start: configuration errors` | The preflight found an unsafe production setting; the log lists each one. Usual causes: `APP_ENCRYPTION_KEY` missing or not 32 bytes base64; `PUBLIC_URL` not `https://`; `COOKIE_SECURE` not `true`; `DB_CLIENT` not `mysql2`. |
| Migration fails with `You do not have the SUPER privilege and binary logging is enabled` (MySQL error 1419) | MySQL needs permission to create the tenancy triggers. Run `SET PERSIST log_bin_trust_function_creators = 1;` as root (managed MySQL: set it in the parameter group), then migrate again. Docker Compose sets this already. |
| Migration fails with `TRIGGER command denied` | Grant the app user `ALL PRIVILEGES` on its database (includes `TRIGGER`). |
| `/api/ready` returns 503 with `migrations: pending` | Run `node dist/cli.js migrate` (Compose: `docker compose run --rm migrate`), or keep `MIGRATE_ON_START=true`. |
| `/api/ready` shows `redis: unreachable` | `REDIS_URL` is wrong or Redis is down. With Redis configured, the API won't report ready without it. Fix Redis, or remove `REDIS_URL` if you run a single API process. |
| `better-sqlite3` fails to build during `npm ci` | It is only for local development and tests, and it is optional. On servers use MySQL; the install continues without it. |
| Web app shows "Could not reach the server" everywhere | The web app was built with the wrong `API_URL`. Rebuild it with the API's internal address (`http://api:4000` in Compose, `http://127.0.0.1:4000` on one server). |

## Signing in

| Symptom | Cause and fix |
|---|---|
| Signed in, but sent straight back to the sign-in page | The cookie is `Secure` but the site is served over plain HTTP, or the domain doesn't match. Serve over HTTPS, or set `COOKIE_SECURE=false` only for local testing. |
| "Too many failed sign-in attempts" | The email is locked for `LOGIN_LOCKOUT_MINUTES` after `LOGIN_LOCKOUT_THRESHOLD` failures. Wait, or use **Forgot password**. Each lockout is in the audit log. |
| "Too many attempts. Try again in a few minutes." | The IP rate limit (30 sign-in requests per 15 minutes). Behind a proxy, check `TRUST_PROXY`: if it is too low, every user shares the proxy's IP. |
| Two-step code is always "not right" | The phone's or server's clock is off. Turn on automatic time on the phone, and NTP on the server. Codes are accepted 30 seconds either side. |
| Lost the authenticator app | Use a recovery code ("Use a recovery code" on the code step), then turn two-step sign-in off and on again with the new phone, or generate new codes. |
| Lost the authenticator app **and** every recovery code | Someone with server access runs `node dist/cli.js 2fa:reset --email user@company.com` (Compose: `docker compose run --rm api node dist/cli.js 2fa:reset --email ...`). This turns two-step sign-in off, signs the user out everywhere and is audited. Confirm the person's identity first. Staff are asked to set it up again at their next sign-in. |
| Staff get `two_factor_setup_required` | Expected: staff must turn on two-step sign-in (`REQUIRE_2FA_FOR_PLATFORM`). They are sent to Account security. |
| Password reset email never arrives | SMTP isn't configured or is failing. System health → Email shows the status and last error, and **Check login** tests it. Also check spam and the sender domain's SPF/DKIM. |

## WhatsApp

| Symptom | Cause and fix |
|---|---|
| Connecting fails with "token rejected" or "permission" | The token is temporary or missing permissions. Create a system user token with `business_management`, `whatsapp_business_messaging` and `whatsapp_business_management`, and assign the WhatsApp account to the system user. |
| Meta's "Verify and save" fails | `PUBLIC_URL` is not the public HTTPS address, the certificate is invalid, or the Verify token was mistyped. The callback URL must be reachable from the internet. |
| Messages send, but replies and ticks never arrive | The webhook fields aren't subscribed (`messages`), or another app or override is receiving the events. Check WhatsApp → Configuration in the Meta app, and the number's **Events** tab in Devwebs. With one app for several numbers, see INTEGRATIONS.md §1.3. |
| Number's Events tab shows `invalid signature` | The App secret saved in Devwebs doesn't belong to the app that sends the webhooks. Update it under **Update credentials**. |
| "Outside the 24-hour window" | WhatsApp rule: free-form messages are allowed only within 24 hours of the customer's last message. Send an approved template instead. |
| Template send refused for a contact | The contact opted out (marketing templates), the template isn't approved, or the wallet can't cover the estimated price. The error says which. |
| Campaign is slow | Sending is capped at `CAMPAIGN_MESSAGES_PER_SECOND` per number, matching Meta's throughput. Add worker processes only if System health shows the campaign queue backing up. |
| Number shows "Token rejected" | The token was revoked or expired in Meta. Paste a new one; sending resumes after the next health check. |
| "Simulation mode" banner in production | `WHATSAPP_PROVIDER=fake`. Set it to `cloud` (or remove it; `cloud` is the production default) and restart. |

## Payments

| Symptom | Cause and fix |
|---|---|
| "Online payments aren't set up yet" / "are not configured yet" | No Razorpay keys. Add them in Admin → Billing settings. |
| Checkout doesn't open (CSP error in the browser console) | Razorpay loaded from a host the Content Security Policy doesn't allow. Add it in `frontend/next.config.ts` and rebuild. See INTEGRATIONS.md §2.3. |
| Customer paid, but the wallet isn't credited | The browser closed before verification and the webhook isn't arriving. Check the Razorpay webhook URL, secret and events, and its delivery log. Admin → Payments → the payment shows its status; once the webhook arrives the wallet is credited automatically, exactly once. |
| Razorpay webhook deliveries show 400 `invalid signature` | The webhook secret in Razorpay and in Devwebs differ. |
| Reconciliation shows a difference | Should not happen. Don't edit balances; contact support with the reference. Every change is in the ledger and audit log. |

## Inbox and real time

| Symptom | Cause and fix |
|---|---|
| New messages appear only after a few seconds, or on refresh | A proxy is buffering `/api/inbox/stream`. Use the supplied Caddy or Nginx config (buffering off). The inbox falls back to polling, so nothing is lost. |
| With several API servers, some agents miss live updates | `REDIS_URL` isn't set, so events stay inside one process. Set it on every API process. |
| Old messages show "Content erased under this workspace's message retention setting" | The workspace's message retention period erased them. Change it in Settings → Workspace; erased content can't be recovered. |

## Webhooks to customer systems

| Symptom | Cause and fix |
|---|---|
| Endpoint rejected as "private or local", or for not using https | SSRF protection. The target must be a public HTTPS URL. Only for testing, outside production: `WEBHOOKS_ALLOW_PRIVATE_TARGETS=true`, `WEBHOOKS_REQUIRE_HTTPS=false`. |
| Endpoint switched off | It returned `410 Gone`, or failed 50 times in a row. Fix the receiver, turn the endpoint back on, and resend failed deliveries from its log. |
| Receiver says the signature is invalid | Verify over the raw body, using the base64-decoded secret after `whsec_`, and the `id.timestamp.body` string. See README "Outgoing webhooks". |

## Workers

| Symptom | Cause and fix |
|---|---|
| System health shows a worker as "Late" or "Not run yet" | No process with `PROCESS_ROLE=worker` (or `all`) is running, or it crashed. Check `docker compose ps worker` and its logs. Workers resume from the database where they stopped. |
| Emails, renewals or campaigns pile up | The same as above, or SMTP/Meta is failing. Each job's last error is shown in System health and in its own page. |

## Getting help

When asking for support, include:

- the **reference** (request id) and the time;
- your Devwebs version (the zip or commit);
- the relevant log lines. They contain no secrets, which are redacted before writing.