# Deploying Devwebs

This guide takes Devwebs from a fresh Linux server to a live, HTTPS-protected installation with real WhatsApp,
Razorpay and email, then covers upgrades, backups, scaling and monitoring.

There are two supported layouts:

- **A. Docker Compose** (recommended): one server runs MySQL, Redis, the API, the background workers, the web app
  and Caddy (automatic HTTPS) from [`docker-compose.yml`](../docker-compose.yml).
- **B. Without Docker**: Node.js, MySQL and Redis installed on the server, systemd units from
  [`deploy/systemd/`](../deploy/systemd/), and Nginx or Caddy in front.

Related documents: [INTEGRATIONS.md](INTEGRATIONS.md) (Meta, Razorpay, SMTP setup), [SECURITY.md](SECURITY.md)
(security model and checklist), [TROUBLESHOOTING.md](TROUBLESHOOTING.md), [TENANCY.md](TENANCY.md).

> **What was verified before release.** The production pieces were run in the build environment:
> - the API and workers in production mode on MySQL-compatible MariaDB 10.11 with Redis;
> - the image build steps from both Dockerfiles, run by hand (install, build, prune, then run the result);
> - the Caddyfile (`caddy validate`, and the WhatsApp and inbox end-to-end tests through it);
> - the Nginx file (`nginx -t`);
> - a backup and a restore into a new database (triggers and files intact, isolation check passed).
>
> The Docker images themselves and `docker compose` could not be run there (no Docker daemon). Build them once on
> your server, then follow [the go-live checklist](#6-go-live-checklist).

---

## 1. Architecture in production

```
Internet ──HTTPS──▶ Caddy / Nginx ──┬── /api/*  ──▶ API (Node, PROCESS_ROLE=api)  ──┐
                                    └── /*      ──▶ Web (Next.js standalone)       │
                                                                                    ├──▶ MySQL 8 (data, queues)
                     Workers (Node, PROCESS_ROLE=worker) ───────────────────────────┤
                                                                                    └──▶ Redis (rate limits, live inbox)
                     Uploaded files: STORAGE_DIR (a volume; shared by API and workers)
```

- `/api/*` goes **straight to the API**. This keeps webhook bodies byte-exact for signature checks, and lets the
  inbox's live event stream through without buffering. Everything else goes to the web app.
- **Queues live in MySQL** and are claimed atomically: campaigns, flow delays, webhook deliveries, emails,
  renewals. You can run any number of API and worker processes.
- **Redis** shares rate-limit counters and live inbox events between processes. Without it, run exactly one API
  process.
- **Files** (WhatsApp media, bot assets) are on local disk under `STORAGE_DIR`. With several servers, put it on
  shared storage (NFS, EFS); object storage is a planned driver.

## 2. Server requirements

| | Minimum | Comfortable |
|---|---|---|
| CPU / RAM | 2 vCPU, 4 GB | 4 vCPU, 8 GB |
| Disk | 40 GB SSD | 100 GB SSD (media grows with chat volume) |
| OS | Ubuntu 22.04/24.04 LTS or Debian 12 | |
| Software (A) | Docker Engine 24+ with the Compose plugin | |
| Software (B) | Node.js 22 LTS, MySQL 8.0/8.4, Redis 7, Caddy 2 or Nginx | |

Other requirements:

- **DNS:** an `A`/`AAAA` record for your domain (e.g. `crm.example.com`) pointing at the server.
- **Firewall:** open 80 and 443 only; MySQL and Redis must not be reachable from the internet.
- **Clock:** keep it in sync (NTP / `systemd-timesyncd`). Two-step sign-in codes and webhook timestamps depend on
  it.

## 3. Install with Docker Compose (A)

```bash
git clone <your repository> /opt/devwebs && cd /opt/devwebs
cp deploy/production.env.example .env
nano .env                           # domain, passwords, APP_ENCRYPTION_KEY, SMTP, Razorpay (see section 5)
openssl rand -base64 32             # paste as APP_ENCRYPTION_KEY, and store it in your password manager
docker compose up -d --build        # builds the images, migrates the database, starts everything
docker compose ps                   # all services "running"/"healthy"; migrate "exited (0)"
docker compose run --rm api node dist/cli.js admin:create --email you@company.com --name "Your Name"
```

`admin:create` prints a one-time password. Then:

1. Open `https://<your domain>/login` and sign in with it.
2. Devwebs sends you straight to **Account security**: platform staff must turn on two-step sign-in. Do it, and
   store the recovery codes.
3. Change the one-time password on the same page.
4. Work through [the go-live checklist](#6-go-live-checklist).

What `docker compose up` runs:

| Service | What it is |
|---|---|
| `mysql` | MySQL 8.4 with `utf8mb4` and `log_bin_trust_function_creators=1` (needed to create the tenancy triggers) |
| `redis` | Redis 7 with append-only persistence |
| `migrate` | Runs `node dist/cli.js migrate` once per deploy, then exits |
| `api` | `PROCESS_ROLE=api`; the HTTP API on port 4000 (internal only) |
| `worker` | `PROCESS_ROLE=worker`; campaigns, flows, billing, webhooks, email, retention, watchdog |
| `web` | The Next.js standalone server on port 3000 (internal only) |
| `caddy` | Ports 80/443, Let's Encrypt certificates, routing per [`deploy/Caddyfile`](../deploy/Caddyfile) |

Volumes:

- `mysql-data` and `storage` are the ones you must back up.
- `redis-data` holds only counters and can be lost.
- `caddy-data` holds the certificates.

## 4. Install without Docker (B)

```bash
# As root: packages
apt install -y mysql-server redis-server caddy        # or nginx + certbot
curl -fsSL https://deb.nodesource.com/setup_22.x | bash - && apt install -y nodejs
adduser --system --group --home /opt/devwebs devwebs

# Database (MySQL 8): utf8mb4, an app user, and permission to create triggers with binary logging on
mysql -e "CREATE DATABASE devwebs CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
          CREATE USER 'devwebs'@'localhost' IDENTIFIED BY '<long random password>';
          GRANT ALL PRIVILEGES ON devwebs.* TO 'devwebs'@'localhost';
          SET PERSIST log_bin_trust_function_creators = 1;"

# App
sudo -u devwebs git clone <your repository> /opt/devwebs
cd /opt/devwebs/backend
cp .env.example .env && nano .env           # see section 5; NODE_ENV=production, DB_CLIENT=mysql2, ...
npm ci && npm run build && npm prune --omit=dev
node dist/cli.js migrate
node dist/cli.js admin:create --email you@company.com --name "Your Name"

cd /opt/devwebs/frontend
npm ci
API_URL=http://127.0.0.1:4000 PUBLIC_URL=https://crm.example.com npm run build
cp -r .next/static .next/standalone/.next/static && cp -r public .next/standalone/public

# Services
cp /opt/devwebs/deploy/systemd/*.service /etc/systemd/system/
systemctl daemon-reload && systemctl enable --now devwebs-api devwebs-worker devwebs-web
```

The web app reads `API_URL` and `PUBLIC_URL` **at build time**: they are compiled into the proxy rules and security
headers, so rebuild the web app if either changes. The unit files expect `node` at `/usr/bin/node` (NodeSource);
adjust `ExecStart` if yours is elsewhere.

**Reverse proxy.**

- **Caddy:** use [`deploy/Caddyfile`](../deploy/Caddyfile) with `DOMAIN`, `ACME_EMAIL`, `API_UPSTREAM=127.0.0.1:4000`
  and `WEB_UPSTREAM=127.0.0.1:3000` set in `/etc/default/caddy`.
- **Nginx:** use [`deploy/nginx.conf`](../deploy/nginx.conf) and get a certificate with `certbot --nginx`. It turns
  buffering off for the inbox stream.

## 5. Environment reference

Settings go in `.env`: the repository root for Compose, `backend/.env` without Docker. Blank values mean "use the
default". In production the API **refuses to start** when a setting marked **required** is missing or unsafe, and
logs a warning for each one marked **warned**; Admin → System health shows the same list.

| Variable | Default | Notes |
|---|---|---|
| `NODE_ENV` | `development` | `production` on servers |
| `PUBLIC_URL` | `http://localhost:3000` | **Required https in production.** Used for webhook URLs, email links and (web, at build time) HSTS |
| `APP_ENCRYPTION_KEY` | dev key | **Required in production**: 32 bytes base64 (`openssl rand -base64 32`). Encrypts WhatsApp tokens, app secrets, Razorpay and webhook secrets. Losing it means re-entering every integration secret |
| `COOKIE_SECURE` | `false` | **Required `true` in production** |
| `DB_CLIENT` | `better-sqlite3` | **Required `mysql2` in production** |
| `DB_HOST` `DB_PORT` `DB_USER` `DB_PASSWORD` `DB_NAME` | `127.0.0.1` `3306` `devwebs` – `devwebs` | MySQL connection |
| `PORT` | `4000` | API port |
| `PROCESS_ROLE` | `all` | `api`, `worker` or `all` |
| `MIGRATE_ON_START` | `true` | Set `false` when a release step runs `cli.js migrate` |
| `TRUST_PROXY` | `1` | Number of proxies in front (for client IPs in rate limits and logs) |
| `REDIS_URL` `REDIS_PREFIX` | – `devwebs:` | **Warned** if unset in production. Needed for more than one API process |
| `STORAGE_DIR` `MAX_UPLOAD_MB` | `./storage` `16` | Uploaded and received files |
| `WHATSAPP_PROVIDER` | `cloud` in production | `fake` = simulation (**warned** in production) |
| `WHATSAPP_GRAPH_VERSION` | `v26.0` | Meta Graph API version (v26.0 was released 29 July 2026) |
| `WHATSAPP_APP_SECRET` `WHATSAPP_VERIFY_TOKEN` | – | Only for numbers onboarded through Devwebs' own Meta app |
| `WHATSAPP_HEALTH_CHECK_MINUTES` `WHATSAPP_TIMEOUT_MS` | `360` `15000` | |
| `CAMPAIGN_MESSAGES_PER_SECOND` `CAMPAIGN_MAX_ATTEMPTS` | `10` `3` | Per number |
| `PAYMENTS_PROVIDER` | `razorpay` in production | `fake` = test mode (**warned** in production) |
| `RAZORPAY_KEY_ID` `RAZORPAY_KEY_SECRET` `RAZORPAY_WEBHOOK_SECRET` | – | Or enter them in Admin → Billing settings (stored encrypted; they take precedence) |
| `SMTP_HOST` `SMTP_PORT` `SMTP_SECURE` `SMTP_USER` `SMTP_PASSWORD` `MAIL_FROM` | – `587` `false` | **Warned** if unset: password resets and notification emails need it |
| `REQUIRE_2FA_FOR_PLATFORM` | `true` in production | Staff must use two-step sign-in (**warned** if off) |
| `SESSION_TTL_HOURS` `PLATFORM_SESSION_TTL_HOURS` | `168` `12` | Absolute session lifetime; staff sessions are shorter |
| `LOGIN_LOCKOUT_THRESHOLD` `LOGIN_LOCKOUT_MINUTES` | `5` `15` | Per email address |
| `PASSWORD_RESET_MINUTES` | `60` | Reset link lifetime |
| `WEBHOOKS_REQUIRE_HTTPS` `WEBHOOKS_ALLOW_PRIVATE_TARGETS` | `true` `false` in production | Customer webhook targets (SSRF protection; **warned** if loosened) |
| `RETENTION_*_DAYS` | 30 / 30 / 90 / 90 | API logs, delivered webhook events, incoming WhatsApp webhooks, sent emails |
| `LOG_LEVEL` `LOG_FORMAT` | `info` `json` in production | Logs go to stderr; secrets are redacted |

## 6. Go-live checklist

1. [ ] `https://<domain>/api/ready` returns `{"ready":true,...}` with `redis: ok`.
2. [ ] Admin → **System health**: status *All systems running*, no configuration errors, every worker *Running*
       within a few minutes.
3. [ ] Your staff account has two-step sign-in on; recovery codes are stored; the one-time password is changed.
4. [ ] **Email:** System health → Email → *Check login* succeeds. *Forgot password* on the sign-in page delivers
       an email.
5. [ ] **Razorpay:** live keys and webhook secret in Admin → Billing settings. Webhook created in the Razorpay
       Dashboard (see [INTEGRATIONS.md](INTEGRATIONS.md)). A ₹10 recharge succeeds and is invoiced; refund it
       from Admin → Payments.
6. [ ] **Seller details:** legal name, GSTIN, state code, address and invoice prefix in Admin → Billing settings.
       Plans and message prices reviewed against Meta's current rate card.
7. [ ] **WhatsApp:** connect a real number in a test workspace. Send a template test message, receive a reply
       (the webhook works), and check delivery ticks.
8. [ ] **Backups** scheduled and a restore tested (section 8).
9. [ ] **Security headers:** `curl -I https://<domain>/login` shows `Strict-Transport-Security`,
       `Content-Security-Policy`, `X-Frame-Options: DENY`.
10. [ ] Remove any demo accounts. `db:seed` refuses to run in production, so there should be none.

## 7. Upgrades

```bash
cd /opt/devwebs && git pull
deploy/scripts/backup.sh                          # always back up first (section 8)
docker compose up -d --build                      # migrate runs first; api/worker restart after it succeeds
```

- Without Docker: build the backend and web app as in section 4, run `node dist/cli.js migrate`, then
  `systemctl restart devwebs-api devwebs-worker devwebs-web`.
- Processes stop gracefully on SIGTERM. The API stops accepting requests, finishes open ones, and closes live
  inbox streams after 10 s (browsers reconnect). Workers finish their current batch. Both exit within 25 s.
- Migrations are forward-only in production. Each was also tested to roll back in development, but restoring the
  pre-upgrade backup is the supported way back.

## 8. Backups and restore

**What to back up:**

- the MySQL database, **including triggers** (they enforce tenant isolation);
- `STORAGE_DIR`;
- your `.env`;
- `APP_ENCRYPTION_KEY`, in a password manager (it is deliberately not in the backup).

[`deploy/scripts/backup.sh`](../deploy/scripts/backup.sh) makes a consistent dump (`--single-transaction`,
triggers, routines), a tarball of the files and a SHA-256 manifest, and deletes sets older than `KEEP_DAYS`:

```bash
# Without Docker, nightly at 02:30 (crontab -e as the devwebs user)
30 2 * * * cd /opt/devwebs && set -a && . backend/.env && set +a && BACKUP_DIR=/var/backups/devwebs \
  STORAGE_DIR=/opt/devwebs/backend/storage deploy/scripts/backup.sh >> /var/log/devwebs-backup.log 2>&1

# With Docker
docker compose exec -T mysql sh -c 'mysqldump -u root -p"$MYSQL_ROOT_PASSWORD" --single-transaction --routines \
  --triggers --events --hex-blob --no-tablespaces --set-gtid-purged=OFF "$MYSQL_DATABASE"' | gzip > devwebs-$(date -u +%F).sql.gz
docker run --rm -v devwebs_storage:/s -v "$PWD":/b alpine tar -C /s -czf /b/devwebs-storage-$(date -u +%F).tar.gz .
```

Copy backups off the server (object storage or another machine), and **test a restore** at least monthly.

**Restore** into a new, empty database, check it, then switch the app to it:

```bash
mysql -e "CREATE DATABASE devwebs_restored CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;"
DB_USER=devwebs DB_PASSWORD=... DB_NAME=devwebs_restored STORAGE_DIR=/opt/devwebs/backend/storage \
  deploy/scripts/restore.sh /var/backups/devwebs/devwebs-20261015T023000Z
DB_NAME=devwebs_restored node backend/dist/cli.js tenancy:check
```

`restore.sh` checks the manifest, refuses a non-empty database, and fails if no triggers were restored.

## 9. Scaling

- **More API capacity:** run more `api` containers (`docker compose up -d --scale api=3`, with Caddy load-balancing
  `api:4000`), or more `devwebs-api` units on more servers. You need `REDIS_URL` and shared `STORAGE_DIR`.
- **More background throughput:** add `worker` processes. Each queue row is claimed by one process only. Campaign
  speed is bounded per WhatsApp number by `CAMPAIGN_MESSAGES_PER_SECOND` (Meta's throughput tier).
- **Database:** the first bottleneck. Use a managed MySQL 8 (RDS, Cloud SQL, Azure) with automated backups. On RDS,
  set `log_bin_trust_function_creators=1` in the parameter group (the migration user creates triggers).
- **Dedicated databases** for selected enterprise customers are designed for but not switched on; see TENANCY.md
  section 7.

## 10. Monitoring and logs

- **Liveness** `GET /api/health`, **readiness** `GET /api/ready` (database, migrations, Redis). Both images have a
  Docker `HEALTHCHECK`.
- **Admin → System health** shows:
  - every worker's heartbeat and last error;
  - queue depths;
  - integrations;
  - configuration checks.

  A watchdog notifies platform staff when a worker falls behind.
- **Logs:**
  - One JSON object per line on stderr (`docker compose logs -f api worker`, or `journalctl -u devwebs-api`).
  - Each line carries `request_id`. The same id is returned in the `X-Request-Id` header and in every 500
    response, so a customer's "reference" finds the log line.
  - Tokens, secrets, passwords, cookies, signatures and SQL values are redacted before writing.
- Ship logs to your platform of choice (Loki, CloudWatch, Datadog). Alert on `level=error` and on `/api/ready`
  returning 503.