Public relay runbook

The public relay is the server anyone's app can use as its relay: accounts, devices, profile cards, the queue of what other servers send, the request inbox, and the forward path to the servers that host people's workspaces. It runs the stock image with RIM_ROLE=relay from deploy/compose.yml with the deploy/compose.relay.yml override (deploy/README.md brings it up). The project's relay is relay.rim-platform.org; read it as "the relay's domain" wherever this page says it.

Everything marked FILL is a decision or a fact the operator records here before the relay opens to the public.

The commands below assume, on the host, in the repository checkout:

export COMPOSE_FILE=deploy/compose.yml:deploy/compose.relay.yml
alias dc='docker compose'
alias rim-relay='docker compose exec rim /app/rim-server'
alias rim-sql='docker compose exec -T postgres psql -U rim -d rim'

COMPOSE_FILE points docker compose and the backup scripts at the relay's two files.

What lives where #

Keys #

The server recovery key #

The recovery key replaces a lost or stolen server key. Whoever holds it can make any key this server's, so it never stays on the host: the first start writes it to /data/server-recovery-key.txt and warns at every start until it is gone (deploy/README.md, "Start").

The file is one line, rim-server-recovery:1:<base64url>. Check a copy without using it: the public half it names is recovery key in rim-relay server-key show.

Yearly rotation #

Rotate the server key once a year (FILL: the month) and when someone with host access leaves.

rim-relay server-key rotate
rim-relay server-key show          # key seq went up by one

The running server switches within a minute. Peers accept the previous key for 24 hours after they see the rotation, so nothing in flight fails. Take a backup right after: an older backup of /data with a newer database (or the reverse) does not start, "Restore".

Compromise: recover #

When the server key may be stolen (host broken into, /data or a backup of it leaked), recover with the recovery key. The new key is signed by the recovery key, and peers stop accepting the old one as soon as they see it.

dc cp ./recovery.txt rim:/tmp/recovery.txt        # the offline copy, briefly
rim-relay server-key recover --recovery-key-file /tmp/recovery.txt --new-recovery-key
dc exec rim rm /tmp/recovery.txt
dc exec rim cat /data/server-recovery-key.txt     # the new recovery key: store it as above
dc exec rim rm /data/server-recovery-key.txt
rim-relay franking-key rotate                     # /data leaked with the key

Replace the old recovery key's copies with the new one. If the recovery key itself may be known to someone else, --new-recovery-key is what takes it away; do it before they do.

The franking key file #

/data/franking-key.json signs the receipts that let users report encrypted messages. Rotation adds a key and keeps every older one, since old receipts must still verify:

rim-relay franking-key show
rim-relay franking-key rotate

Rotate when /data may have leaked. Losing the file loses the ability to verify reports of messages sent before: back it up with everything else in /data.

Backups #

deploy/backup.sh writes one archive of the database dump, the /data volume and deploy/.env; docs/self-hosting.md, "Backups", says what is in it. It never includes the recovery key file.

Nightly (FILL: the time and the cron or timer that runs it), before every update and right after every key rotation; encrypted with age to FILL: the recipient, whose identity is held by FILL; copied off the host to FILL: the destination; kept for FILL days.

RIM_BACKUP_AGE_RECIPIENT=age1… deploy/backup.sh       # or set it in deploy/.env
# copy the archive it names off the host (FILL: destination)

Cron, with RIM_BACKUP_AGE_RECIPIENT in deploy/.env:

15 3 * * * cd /srv/rim-server && COMPOSE_FILE=deploy/compose.yml:deploy/compose.relay.yml deploy/backup.sh >>/var/log/rim-backup.log 2>&1

The dump and the data archive are one set: the key file must match the head of the key log in the database. The scripts restore both from one archive; never mix sets.

Restore #

On the new host, with the repository checked out, deploy/.env taken from the archive's env file, and the image pulled (dc pull rim):

dc stop rim                                            # restore.sh refuses while it runs
RIM_BACKUP_AGE_IDENTITY=/path/to/identity deploy/restore.sh rim-backup-rim-relay-<time>.tar.age
rim-relay server-key show

It drops and recreates the database from the dump, empties /data and refills it, starts the stack and waits for /readyz. The server refuses to start when the key file and the log head disagree: that is a mixed set, or a rotation after the backup. Use a matching set, or server-key recover if the key file of the newest log is gone.

Restore drill #

Every quarter (FILL: the months; record each drill's date and result below), restore the latest set:

RIM_BACKUP_AGE_IDENTITY=/path/to/identity deploy/restore-drill.sh rim-backup-rim-relay-<time>.tar.age

The drill runs in its own project, rim-drill, with server-to-server traffic and mail off and no port published, so the copy never speaks for the relay; it removes the project and its volumes when it ends. Check its output:

• server-key show names the production server id and key seq;
• the row counts (user, session, device_key, relay_item, …) are close to production's;
• the recovery key copies read back (see above).

For abuse-reports list and principal list against the copy, keep it running with RIM_DRILL_KEEP=1, run docker compose -p rim-drill exec rim /app/rim-server principal list, and remove it with docker compose -p rim-drill down -v: two running servers with one key are one server to every peer.

Quotas #

Per local user, set in deploy/.env; the compose file has the public relay's values.

The public relay keeps the DM defaults, RIM_DM_INBOUND=requests and RIM_DM_OUTBOUND=allowed: a stranger's first message is a request, from anywhere ("Who May Start A DM" in docs/messaging.md).

To change one, edit deploy/.env and run dc up -d: the server restarts in seconds and apps reconnect. The per-minute and per-day counters are in memory, so a restart starts them afresh. Each extra bit of proof-of-work doubles a signup's cost; the descriptor tells apps the new value at once.

To let people in without the proof-of-work, give them a signup link: rim-relay invite create --max-uses <1-50> --expires-in-days <1-30> --public-url https://<domain> prints it on its first line (the token is shown once); rim-relay invite list shows each live link's use count and rim-relay invite revoke <id> ends one. For one person, rim-relay invite create --email <their address> --public-url https://<domain> gives a single-use link bound to that address: the account it signs up gets that address.

Monitoring #

Logs are JSON on stdout (dc logs rim); audit events ("target":"audit") also go to /data/audit.log. The database answers what logs do not. Alert thresholds are FILL until a month of normal traffic shows the baseline.

dc logs --since 1h rim | grep -c '"event":"relay.registration_pow"'
dc logs --since 1h rim | grep 'refused a server-to-server signature' | grep -o '"reason":"[a-z_]*"' | sort | uniq -c
rim-sql <<'SQL'
SELECT o.peer_server_id, count(*) AS rows, min(o.created_at) AS oldest,
       d.failures, d.next_attempt_at, d.last_success_at
FROM s2s_outbox o LEFT JOIN s2s_peer_delivery d USING (peer_server_id)
WHERE o.id > COALESCE(d.delivered_through, 0)
GROUP BY o.peer_server_id, d.failures, d.next_attempt_at, d.last_success_at
ORDER BY rows DESC;
SQL

Per-status counts of every client request come from the proxy. Caddy keeps no access log by default, since it would hold users' IP addresses; during an incident add log to the site block of deploy/Caddyfile.relay, dc restart caddy, and remove it afterwards.

Incidents #

Record each incident: start, what was seen, what was done, end (FILL: where the record goes).

A spam wave #

Signs: relay.registration_pow far above normal, many reports, DM requests from few senders.

1. Find the senders: rim-sql -c "SELECT from_principal_id, count(*) FROM relay_dm_request WHERE created_at > now() - interval '1 hour' GROUP BY 1 ORDER BY 2 DESC LIMIT 20" and rim-relay abuse-reports list --open.
2. Look at each: rim-relay principal show <principalId> prints the account, who invited it, the accounts its invites created and the reports naming it.
3. Block each: rim-relay principal block <principalId> --reason "spam wave <date>". Their sessions end, their invites that have uses left are revoked, their email can neither sign up nor sign in, their pending requests are ignored for everyone here, and their signups, logins and deliveries are refused from then on. The command prints the accounts the blocked one's invites created (username, principal id, created at).
4. When those accounts are part of the wave, block them all at once: rim-relay principal block <principalId> --reason "spam wave <date>" --cascade blocks the account and every account its invites created, recursively, in one transaction, each with the reason followed by "(invited by <username>)". When the wave came in through an invite, cascade from the account that gave it out. Accounts without a key identity (no key identity in the output) are reached only this way. Resolve the reports that named them.
5. Make the next account dearer: raise RIM_REGISTRATION_POW_BITS by 2 (four times the work) and, if the spam goes to people here, lower RIM_RELAY_DM_REQUESTS_PER_DAY; dc up -d.
6. When the senders are on another server, block that server (next section) and write to its operator (its discovery document names its origin: rim-sql -c "SELECT origin FROM peer_server WHERE server_id = '<id>'").
7. Lower the settings again a day after the wave ends.

rim-relay principal unblock <principalId> lifts a block and its email block after a review. It does not bring back the sessions or the invites the block ended, and it lifts one account: unblock each account a cascade blocked.

A hostile peer #

Signs: one peer_server_id in many signature refusals, replays or rate limits; deliveries refused as invalid_envelope; reports naming one server.

1. rim-relay peer block <serverId>: refused both ways at once, its requests get 403 peer_not_allowed, nothing is sent to it. Audited as s2s.peer_blocked.
2. A server never pinned here cannot be blocked that way (the command says so): add it to RIM_S2S_DENIED_PEERS in deploy/.env and dc up -d.
3. rim-relay peer unblock <serverId> when it is resolved. A peer refused as forked is not unblocked: check with its operator out of band, then rim-relay peer reset <serverId>.

Traffic from one address #

Signs: rim_rate_limit_rejections_total{limiter="api_ip"} climbing, one address over and over in Caddy's access log, signup or login refusals from one range.

1. rim-relay ip block <address or range> --reason "<what> <date>" --expires-in-days 7. Every route but /healthz answers it 403 within 30 seconds; audited as ip.block.
2. A block is per address, so a client with many addresses comes back: block its range (a /24 or a /48) and set RIM_RATE_REQUESTS_PER_MIN_PER_IP if it is not set yet.
3. rim-relay ip list shows what is blocked until when; rim-relay ip unblock <range> lifts one early. An expired block is ignored at once and deleted within 30 seconds.

A stolen server key #

Signs: a peer's operator reports requests signed by this server that it did not send; the host or a backup of /data was exposed.

1. "Compromise: recover", including --new-recovery-key and the franking key rotation.
2. Rotate POSTGRES_PASSWORD (ALTER USER rim PASSWORD … in rim-sql, then deploy/.env, then dc up -d).
3. Find how it left: host logins, who had the backups. Rebuild the host if it was broken into.
4. Tell users what the thief could do until the recovery (sign as this server towards peers and receive what peers delivered for its users; never sign as a user or read their DMs) and when it stopped. FILL: the channel for such notices.

A database leak #

What a dump holds: accounts and who invited them, key logs, devices and cards (public anyway), session token hashes, password hashes of the few password accounts, contacts, DM requests and their senders' cards, queued host events (plaintext channel events are readable, DMs are MLS-encrypted), abuse reports, blocked emails. No private key: the server and franking keys are files in /data.

1. End every session: rim-sql -c 'UPDATE session SET revoked_at = now() WHERE revoked_at IS NULL'.
2. Rotate POSTGRES_PASSWORD as above, and find and close the way out.
3. If /data or a backup set went too: "A stolen server key".
4. Notify users and, where the law asks for it, the authority (FILL: who decides, the deadline that applies, e.g. 72 hours under the GDPR, and the contact).

Abuse policy #

The text to publish (FILL: where, e.g. https://<domain>/abuse), filled in:

Abuse on <relay name>

This relay carries messages for anyone who signs up. We block an account (its key identity) on this relay when it sends spam or unsolicited bulk requests, harasses or threatens people, impersonates someone, spreads malware or phishing, tries to break or overload the service, or uses it for anything illegal where the relay operates (FILL: jurisdiction). We block another server when it sends such traffic and its operator does not stop it, or when it attacks this relay.

To report: use "Report" on a message request in the app, or write to FILL: abuse address with the principal id or server id, what happened and when. We cannot read encrypted messages; a report from the app carries what the sender's own request showed.

We answer reports within FILL (e.g. one working day) and act on clear spam or attacks within FILL (e.g. four hours). We keep reports for FILL and the blocked identity until it is unblocked. A blocked person may ask for a review at the same address; we answer within FILL.

Reports from the abuse address go into the same list as those from the app:

rim-relay abuse-reports add --principal rimp1_… --reason "mail of <date>: phishing links"
rim-relay abuse-reports add --server rims1_… --reason "mail of <date>: spam from its users"
rim-relay abuse-reports list --open
rim-relay abuse-reports resolve <report id>

Command reference #

rim-relay server-key show | rotate | recover --recovery-key-file F [--new-recovery-key]
rim-relay franking-key show | rotate
rim-relay principal block <principalId> --reason <text> [--cascade] | unblock <principalId> | show <principalId> | list
rim-relay peer block <serverId> | unblock <serverId> | reset <serverId>
rim-relay abuse-reports list [--open] | add [--principal P] [--server S] --reason <text> | resolve <id>
rim-relay invite create [--email A] [--max-uses N] [--expires-in-days D] [--public-url https://<domain>] | list | revoke <id>
rim-relay audit purge [--dry-run] [--older-than-days N]
rim-relay ip block <cidr> --reason <text> [--expires-in-days D] | unblock <cidr> | list
rim-relay operator grant <username> | revoke <username> | list

operator grant makes an account an instance operator: it reads the abuse reports, principal blocks and IP blocks in the app and changes them as the abuse-reports, principal and ip commands do (docs/api.md, "Instance moderation"). A blocked account or one scheduled for deletion cannot be granted the role; operator revoke takes it back and operator list prints the operators.

Every one is audited (target audit): relay.principal_blocked (scope server, one per account a block reaches, cascade true for those after the first), relay.principal_unblocked, s2s.peer_blocked, s2s.peer_unblocked, s2s.peer_reset, relay.abuse_report_filed, relay.abuse_report_resolved, server_key.rotated, server_key.recovered, instance_invite_issued, instance_invite_revoked (issued_by or actor operator), ip.block and ip.unblock (scope server, with the cidr), operator.granted and operator.revoked (scope server, with the user_id). The principal, report and IP lines carry actor_id: cli for the command line, the operator's user id for a change made in the app. audit purge writes audit.retention.purged (trigger cli), the same event the daily purge writes; the purge, its export files and the logrotate setup for /data/audit.log are in docs/operations/audit-retention.md.