bring on more tildes hopefully

docs/shellserver.md

@@ -1,47 +1,165 @@
-# Setting up the tilde.club shell server (user host)
+# Running the shell host
 
-We want to document the ins and outs of setting up the server so others who are interested can learn (and help!).
+Use [the setup guide](how-to-set-up-a-tilde.md) for a new machine. This chapter
+covers the work after people start using it.
 
-## System setup
+## Accounts and groups
 
-Start with [Build your own tilde-style server](./how-to-set-up-a-tilde.md) for a modern baseline. Historical package notes still live in [docs/server.org](./server.org).
+Give each person a private primary group. Use supplementary groups for specific
+shared resources. The production `makeuser` also adds a common group by numeric
+ID; resolve that ID locally before copying the policy to another host.
 
+Keep the list of privileged groups short. Membership in `wheel`, Docker's
+administration group, or a service configuration group can be equivalent to
+root access. Shared web backends need their own identities and narrowly scoped
+permissions; see [web publishing](web.md).
 
-## Quick priorities for new operators
+To suspend a compromised account, disable its actual login paths. Locking the
+Unix password alone does not necessarily prevent public-key SSH or existing
+sessions. Check keys, SSH account rules, active sessions, cron, user services,
+ZNC, web applications, and application-specific credentials. Preserve the data
+needed to understand the incident before deleting anything.
 
-If you are bringing up a new host, focus on these first:
+Use [account provisioning](accounts.md) as a checklist. Provisioning a Unix
+account and provisioning its external services are separate operations.
 
-1. Lock down SSH and require keys.
-2. Get `/etc/skel` right before creating many users.
-3. Verify `~/public_html` publishing works.
-4. Document onboarding, moderation, and backup/restore workflows.
+## Disk quotas
 
-These four steps prevent many common early-stage tilde problems.
+The collected host uses XFS with `usrquota,grpquota` on the root filesystem.
+`/home` is a directory on that filesystem. The collector's attempt to query
+`xfs_quota` with `/home` as a mount path failed even though the command returned
+zero, so inspect its output as well as its exit status.
 
-## /etc/skel directory
-
-This is the directory that's used as the basis for all newly-created users' home directories, so it's good to get it right before adding new users to a shell server. We've created [a separate repo for the contents of the directory itself](https://github.com/tildeclub/tilde.etcskel), but since it's impossible to check in the appropriate file and directory permissions, [they're documented here](https://github.com/tildeclub/tilde.club/blob/master/docs/etc-skel-permissions.md).
-
-## Email
-
-The default MTA on CentOS is `postfix`. Our goal was to have a `localhost`-only mail service, which required that we configure `postfix` to listen only to `localhost`, and to bounce any email which local users try to send off-server. Both configuration changes are handled in `/etc/postfix/main.cf`.
-
-* the `inet_interfaces` value should just be `localhost` (`inet_interfaces = localhost`)
-* the `default_transport` parameter should be the bounce message we want (so add `default_transport = error: outside mail is not deliverable` to the bottom of the file)
-
-### pine
-
-Pine is sort of brain-dead about creating its `.addressbook` file in a user's home directory with `744` permissions; there doesn't appear to be an option to fix this. Instead, it's probably important to work around it before adding any new users:
-
-1. Add `~/mail/` to `/etc/skel` with permissions `700` so that there's a user-accessible-only place for the file to live.
-2. Create an `/etc/pine.conf` file that includes the config directive `address-book=mail/.addressbook` to put that file into this new home.
-
-## identd
-
-Users will connect from their shell account to an IRC server, so it is *very* handy to have an `identd` server. For us that just meant installing the standard CentOS `identd` server and configuring it to start automatically:
+First discover the mount:
 
+```bash
+findmnt --target /home --output TARGET,FSTYPE,OPTIONS
 ```
-sudo yum install oidentd
-sudo /etc/init.d/oidentd start
-sudo chkconfig oidentd on
+
+On the recorded layout, query and set limits using `/`:
+
+```bash
+xfs_quota -x -c 'state' /
+xfs_quota -x -c 'report -h -u' /
+xfs_quota -x -c 'limit -u bsoft=1G bhard=3G member' /
+xfs_quota -x -c 'quota -h -u member' /
 ```
+
+These match the block limits requested by the current provisioning script.
+The soft limit can be exceeded during a grace period; the hard limit cannot.
+Check the grace policy and add inode limits if small-file counts are a concern.
+If `/home` is a separate XFS mount on your host, use that mount instead.
+
+Enabling XFS quotas can require mount or boot configuration work. Follow the
+filesystem's documentation, take a backup, and arrange a maintenance window.
+Do not remount a busy filesystem using an unreviewed command copied from here.
+For ext4 or another filesystem, use its own quota tools and activation procedure.
+
+Mail and user applications can live in the same quota as the home. Decide what
+happens when someone reaches the hard limit: mail delivery and saving an editor
+buffer are both affected.
+
+## Process, memory, and CPU limits
+
+The system record did not establish per-user limits. It reported an unlimited
+aggregate `user.slice`, which alone says nothing about individual user slices
+or PAM limits.
+
+For a new host using systemd user slices, a starting example is
+`/etc/systemd/system/user-.slice.d/50-tilde-limits.conf`:
+
+```ini
+[Slice]
+TasksMax=512
+MemoryHigh=1G
+MemoryMax=2G
+CPUQuota=200%
+```
+
+Those values are examples to test, not the existing server's measured policy.
+The drop-in applies to `user-UID.slice` units. Apply configuration changes with
+`systemctl daemon-reload`, then check a fresh test session:
+
+```bash
+id -u member
+systemctl show user-UID.slice -p TasksMax -p MemoryHigh -p MemoryMax -p CPUQuotaPerSecUSec
+loginctl user-status member
+```
+
+Replace `UID` with the number printed by `id`. Verify SSH sessions are actually
+placed in that slice. A hard memory limit can terminate a user's work. Tell
+people what the limits are and watch for repeated limit hits before tightening
+them. PAM `limits.d` settings are useful for session limits but are not a
+replacement for aggregate cgroup limits.
+
+Run long-lived community daemons as separate service users with their own
+limits. Enable user lingering only where persistent user services are intended.
+Do not quietly turn every interactive account into an unrestricted boot-time
+service account.
+
+## Ports and firewall policy
+
+The existing host uses CSF, with IPv4 and IPv6 rules and a number of historical
+exceptions. Its broad port ranges are not a minimum firewall template. A new
+host should allow the services it actually serves.
+
+| Service | Typical inbound port | When it is needed |
+| --- | --- | --- |
+| SSH | TCP 22 | Member and admin shell access. |
+| HTTP and HTTPS | TCP 80, 443 | Publishing and certificate validation. |
+| SMTP delivery | TCP 25 | Receiving Internet mail. |
+| Authenticated submission | TCP 587 or 465 | Sending mail from remote clients with TLS. |
+| IMAPS | TCP 993 | Remote mailbox access. |
+| ManageSieve | TCP 4190 | Remote filter editing, if enabled with appropriate TLS/auth. |
+| Gopher | TCP 70 | Public Gopher service. |
+| Gemini | TCP 1965 | Public Gemini service. |
+| Finger | TCP 79 | A deliberately enabled finger service. |
+| Ident | TCP 113 | Trusted IRC servers querying user identity. |
+| NNTP/NNTP over TLS | TCP 119, 563 | Remote newsreaders, if intentionally allowed. |
+| Mosh | UDP chosen from 60000–61000 | An explicit Mosh port allocation; the range is not a universal default. |
+
+IRC client connections are outbound. You do not need an inbound IRC daemon port
+just because members run an IRC client. Database, cache, PHP, CGI, and mail
+filter sockets should remain local unless there is a separately designed reason
+for remote access.
+
+Check `ss -lntup`, the configured bind addresses, host rules, provider rules,
+and an outside connection test. A listening socket is not proof of public
+reachability, and a firewall listing is not proof of a working service.
+
+## Updates and local programs
+
+Install supported OS updates, read service migration notes, and schedule reboots.
+For a shared host, announce downtime through a public status/contact path and
+inside the shell. People may have a compiler or editor running when you reboot.
+
+Keep an inventory of programs in `/usr/local`, services built from source,
+Python virtual environments, Ruby gems, and application dependency lockfiles.
+An RPM update will not update a different executable earlier in PATH. The
+recorded borgmatic installation is a concrete example: cron uses
+`/usr/bin/borgmatic` 2.1.9 while `/usr/local/bin/borgmatic` reports 1.5.21.
+
+Do not update every language ecosystem globally as root on behalf of all users.
+Separate OS packages, operator applications, and user-managed dependencies.
+Keep root-owned service code outside user-writable directories.
+
+## Routine checks
+
+```bash
+systemctl --failed
+systemctl list-timers --all
+ss -lntup
+df -h
+df -i
+journalctl -p err --since today
+```
+
+Review mail queues, backup completion, certificate expiry, disk and inode use,
+resource-limit events, and pending account requests. Keep logs rotated and
+retain enough history to investigate failures without logging passwords or
+private user content unnecessarily. See [accounting](accounting.md) for what
+session and process accounting can actually tell you.
+
+Document how to recover from a failed update, restore an account, rotate a
+credential, handle an abuse report, and reach someone when the host is down.
+See [backup and recovery](backups.md) for the restore procedure.