From d002e6864b1527000ffc3dc92a01cffdb09ecb38 Mon Sep 17 00:00:00 2001 From: Mario Fetka Date: Wed, 22 Jul 2026 17:57:39 +0200 Subject: [PATCH] Complete protocol and service documentation --- docs/README.md | 5 ++++ docs/architecture.md | 4 +++ docs/dav.md | 44 +++++++++++++++++++++++++++++ docs/pop3.md | 66 ++++++++++++++++++++++++++++++++++++++++++++ docs/protocols.md | 10 ++++++- docs/queue.md | 51 ++++++++++++++++++++++++++++++++++ docs/sieve.md | 61 ++++++++++++++++++++++++++++++++++++++++ docs/web.md | 56 +++++++++++++++++++++++++++++++++++++ 8 files changed, 296 insertions(+), 1 deletion(-) create mode 100644 docs/dav.md create mode 100644 docs/pop3.md create mode 100644 docs/queue.md create mode 100644 docs/sieve.md create mode 100644 docs/web.md diff --git a/docs/README.md b/docs/README.md index 932acdd..e2b4847 100644 --- a/docs/README.md +++ b/docs/README.md @@ -39,6 +39,11 @@ partial, and planned work. - [Authentication, TOTP, LDAP, and app passwords](authentication.md) - [SMTP, submission, LMTP, and trusted relay](smtp.md) - [IMAP compatibility](imap.md) +- [POP3 and POP3S](pop3.md) +- [Sieve filtering and ManageSieve](sieve.md) +- [Webmail and user administration](web.md) +- [CalDAV and CardDAV](dav.md) +- [Queue, routing, and delivery](queue.md) - [Mail storage quotas](quotas.md) - [SPF, DKIM, DMARC, and SRS](mail-authentication.md) - [SpamAssassin and ClamAV](antispam.md) diff --git a/docs/architecture.md b/docs/architecture.md index fc8b137..a49015f 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -42,6 +42,10 @@ handled by the authenticated outbound SMTP path. | Bongo Web | Python 3 Webmail, account settings, CalDAV, and CardDAV. | | `bongo-setup` / `bongo-admin` | Guided initial setup and validated administration. | +Protocol and component details are maintained in [smtp.md](smtp.md), +[imap.md](imap.md), [pop3.md](pop3.md), [sieve.md](sieve.md), +[queue.md](queue.md), [web.md](web.md), and [dav.md](dav.md). + Agents use common C libraries for configuration, logging, authentication, TLS, protocol I/O, Store access, and Queue access. GNU GSASL is shared by the mail protocols; its direct Bongo verification callback does not create or diff --git a/docs/dav.md b/docs/dav.md new file mode 100644 index 0000000..8b42029 --- /dev/null +++ b/docs/dav.md @@ -0,0 +1,44 @@ +# CalDAV and CardDAV + +Bongo exposes a user's calendar and address book through the Web service so +Thunderbird and other DAV clients see the same objects as Bongo Webmail. +Calendar resources are iCalendar documents; contact resources are vCards. + +## Discovery and paths + +Clients may use `/.well-known/caldav` and `/.well-known/carddav` discovery. +Authenticated homes use these forms: + +```text +/calendars/USER/personal/ +/addressbooks/USER/personal/ +``` + +Calendar objects end in `.ics`; address-book objects end in `.vcf`. The +authenticated user may access only that user's home. Resource names are +decoded and validated before being mapped to Store paths, preventing encoded +slashes, traversal, NUL bytes, or another account name from escaping the +personal collection. + +## HTTP and DAV behaviour + +The service supports `OPTIONS`, `GET`, `HEAD`, `PUT`, `DELETE`, `PROPFIND`, +and `REPORT` for the implemented calendar and address-book resources. +Depth-aware multistatus responses describe the principal, home, personal +collection, and objects. Calendar multiget/query and address-book multiget +read the requested resources. ETags and `If-Match`/`If-None-Match` protect +updates against accidental overwrite. + +Incoming iCalendar and vCard payloads are parsed strictly and bounded by the +Web request and object-size limits. Bongo preserves the canonical source data +needed for lossless DAV round trips while also mapping useful fields into the +Store model used by Webmail. Calendar events include recurrence and the +standard date/time data supported by the converter; contacts include +structured names, multiple email addresses and telephone numbers, +organization, birthday/anniversary, URL, note, and postal address fields. + +DAV authentication follows the Web master/app-password scope and external +TLS policy. When a reverse proxy terminates TLS, it must pass DAV methods and +bodies unchanged and use the same trusted-proxy configuration as Webmail. +The release gate tests discovery, CRUD, reports, ETags, isolation, and +TLS/proxy policy under DAV-01 through DAV-03. diff --git a/docs/pop3.md b/docs/pop3.md new file mode 100644 index 0000000..115982d --- /dev/null +++ b/docs/pop3.md @@ -0,0 +1,66 @@ +# POP3 and POP3S + +Bongo keeps POP3 for simple and older mail clients while using the same +accounts, authentication policy, message files, flags, and per-user mail quota +as IMAP and Webmail. POP3 is an access protocol only: SMTP remains responsible +for message submission. + +## Listeners and TLS + +The suffixless `pop3` configuration document controls the two listeners: + +```json +{ + "version": 1, + "port": 110, + "port_ssl": 995, + "allow_legacy_tls": false, + "proxy_protocol_enabled": false, + "proxy_protocol_networks": [] +} +``` + +Port 110 offers `STLS` before authentication. Port 995 uses implicit TLS. +Password and GNU GSASL mechanisms are made available only on a protected +connection under the secure default. Enabling `allow_legacy_tls` is an +explicit compatibility exception for obsolete clients; current TLS is still +negotiated when both peers support it. + +PROXY protocol must remain disabled unless a trusted TCP proxy is actually in +front of POP3. Enabling it requires at least one exact trusted IPv4 address or +CIDR in `proxy_protocol_networks`; ordinary clients must never be included. + +## Commands and capabilities + +The server implements the normal authorization and transaction commands: +`USER`, `PASS`, `AUTH`, `APOP`, `STAT`, `LIST`, `UIDL`, `RETR`, `TOP`, `DELE`, +`RSET`, `NOOP`, `CAPA`, `UTF8`, `STLS`, and `QUIT`. Multiline replies apply +POP3 dot transparency and canonical CRLF termination even when a stored +message was read in chunks. + +`CAPA` reports the implemented `TOP`, `RESP-CODES`, `AUTH-RESP-CODE`, +`PIPELINING`, `EXPIRE NEVER`, `UIDL`, and `UTF8` features. `STLS`, `USER`, and +the `SASL` mechanism list are state-dependent. After `UTF8` succeeds, `STLS` +is no longer offered and is rejected, preventing already-buffered cleartext +UTF-8 session state from crossing the TLS boundary. + +## Deletion and quota semantics + +`DELE` marks a message only for the current POP3 transaction. `RSET` or a +disconnect leaves it in the mailbox; a successful `QUIT` commits the marked +deletions. Storage is therefore released only after that commit. POP3 has no +standard quota query command, but delivery to the same account is still +limited by the shared Store quota described in [quotas.md](quotas.md). + +Use IMAP or Webmail when a client needs server-side folders, flags, searching, +or concurrent mailbox changes. POP3 clients which delete downloaded mail +should be tested carefully before enabling that option for a large account. + +## Operational checks + +After changing the listener or TLS policy, validate and synchronize the +configuration, restart the agent through `bongo-manager`, and test both the +clear listener's `STLS` transition and a direct POP3S login. Also verify +`STAT`, `LIST`, `RETR`, `TOP`, `UIDL`, deletion rollback on disconnect, and +deletion commit on `QUIT`. The release gate is recorded under POP-01 through +POP-06 in [release-testing-0.7.md](release-testing-0.7.md). diff --git a/docs/protocols.md b/docs/protocols.md index 12cb2b5..e4390fa 100644 --- a/docs/protocols.md +++ b/docs/protocols.md @@ -43,7 +43,7 @@ Mail-storage limits are enforced by the shared Store for SMTP delivery, IMAP, POP3, Web access, and external collection. IMAP exposes RFC 9208 quota queries; POP3 has no standardized quota command and releases charged storage only when its `QUIT` update transaction successfully commits marked deletions. See -[Mail storage quotas](quotas.md). +[POP3 and POP3S](pop3.md) and [Mail storage quotas](quotas.md). ## Server-side filtering @@ -59,6 +59,9 @@ without depending on the Web interface. Later interfaces, including a future JMAP implementation, must operate on the same user filter model rather than inventing incompatible rules. +See [Sieve filtering and ManageSieve](sieve.md) for listener, command, +storage, and vacation behaviour. + ## Web, calendars, and contacts The Python 3 Web service provides Webmail and user settings. CalDAV exposes @@ -66,6 +69,9 @@ calendars using iCalendar data, and CardDAV exposes address books using vCard data. External HTTPS is normally supplied by a reverse proxy; the Bongo Web listener itself defaults to HTTP on loopback. +See [Webmail and user administration](web.md) and +[CalDAV and CardDAV](dav.md) for the current 0.7 interfaces. + JMAP and Exchange/MAPI protocols are not part of Bongo 0.7. JMAP is planned for 0.9 after the mail protocols and Store API are stable. Native Exchange/Outlook compatibility remains research beyond that milestone. @@ -75,6 +81,8 @@ Exchange/Outlook compatibility remains research beyond that milestone. The Store and Queue protocols coordinate local Bongo agents. They are not public client protocols and have no Internet compatibility promise. Bind them to a private interface and firewall ports 689 and 8670 from clients. +Queue durability and retry policy are described in +[Queue, routing, and delivery](queue.md). ## Authentication and message security diff --git a/docs/queue.md b/docs/queue.md new file mode 100644 index 0000000..3a20576 --- /dev/null +++ b/docs/queue.md @@ -0,0 +1,51 @@ +# Queue, routing, and delivery + +`bongoqueue` durably owns messages after SMTP or the external collector has +accepted them and before every recipient has reached a final destination. It +coordinates filtering, local Store delivery, remote SMTP, and LMTP; it is not +a public submission protocol. + +## Message flow and durability + +Queue data lives below `/var/lib/bongo/spool`. An accepted message remains +there while a temporary scanner, Store, DNS, SMTP, LMTP, or quota-dependent +delivery condition is retried. Permanent failures produce the configured +delivery-status response or bounce. The Queue must not acknowledge a message +to its upstream path before the payload and recipient state are durable. + +Enabled filtering agents register for Queue stages. Antivirus, +SpamAssassin, rules/Sieve, and final local or remote delivery therefore see a +common envelope and message rather than independent copies. External POP3 or +IMAP collection enters this same path and may delete a source message only +after Bongo has committed it successfully. + +The internal Queue protocol defaults to TCP 8670. Bind and firewall it as a +private service interface. It does not authenticate arbitrary Internet +clients and must never be exposed as an SMTP alternative. + +## Limits and failure handling + +The suffixless `queue` document controls spool free-space reservation, +message lifetime, retry interval, worker concurrency, trusted/hosted domains, +and bounce policy. `minimumfreespace` prevents normal acceptance from +consuming the filesystem's final space. Per-user Store quota is separate: a +full mailbox defers or rejects that recipient without corrupting other +recipient state. + +A scanner outage is a delivery-policy decision, not permission to silently +bypass inspection. ClamAV and SpamAssassin failures leave the message queued +for the configured retry path. Likewise, a per-recipient LMTP temporary error +must preserve only that recipient for retry and must not redeliver recipients +which already returned a final success. + +## Administration and shutdown + +Use `bongo-queuetool` to inspect and perform documented queue operations; do +not edit spool files manually. A service stop first prevents new work and asks +the managed agents to finish or abandon their current transaction safely. +Queue and Store state must be included in a consistent backup. + +The 0.7 release matrix exercises byte preservation, disk exhaustion, retry, +expiry, bounce, hold/release, concurrency, invalid commands, quota mapping, +and collector floods under STQ-02 through STQ-12. SMTP and LMTP rows add their +protocol-specific recipient and retry cases. diff --git a/docs/sieve.md b/docs/sieve.md new file mode 100644 index 0000000..48c4a05 --- /dev/null +++ b/docs/sieve.md @@ -0,0 +1,61 @@ +# Sieve filtering and ManageSieve + +Bongo stores each user's server-side mail rules independently from the mail +payloads. `bongosieve` provides the ManageSieve management protocol, while +`bongorules` evaluates the active script as mail passes through the Queue. +Thunderbird and other ManageSieve clients can therefore maintain filters +without editing server files. + +## Listener and authentication + +ManageSieve is disabled by the base template until it is required. Its +suffixless `sieve` document has conservative loopback defaults: + +```json +{ + "enabled": false, + "listen_address": "127.0.0.1", + "port": 4190, + "allow_legacy_tls": false, + "proxy_protocol_enabled": false, + "proxy_protocol_networks": [], + "maximum_script_size": 1048576, + "maximum_connections": 64 +} +``` + +The greeting offers `STARTTLS`; GNU GSASL mechanisms are advertised only +after TLS. Authentication uses the same Bongo user and app-password policy as +the mail protocols, with the Sieve service scope. Cleartext received after a +`STARTTLS` command is discarded before negotiation. PROXY protocol has the +same source-restricted rules as the other mail listeners. + +## Management protocol + +The RFC 5804 service implements `CAPABILITY`, `STARTTLS`, `AUTHENTICATE`, +`HAVESPACE`, `PUTSCRIPT`, `CHECKSCRIPT`, `LISTSCRIPTS`, `GETSCRIPT`, +`SETACTIVE`, `RENAMESCRIPT`, `DELETESCRIPT`, `NOOP`, and `LOGOUT`. Script +names and literals are bounded, syntax is validated before activation, an +active script cannot be deleted accidentally, and scripts are isolated by +authenticated user. + +The implementation advertises `fileinto`, `envelope`, `body`, `variables`, +`vacation`, and `vacation-seconds`. Actions include keep, discard, file into a +folder, redirect, reject, and vacation response. Vacation history is durable +so repeated delivery does not generate an immediate reply loop. Scripts and +vacation state live in `/var/lib/bongo/dbf/sieve.sqlite`; include that database +and its SQLite sidecars in backups. + +## Filters in Webmail + +The Web filter editor and ManageSieve are two interfaces to the same intended +user rule model. A filter may match senders or other message properties and +deliver matching messages to a folder such as `Family`. Mailing-list +recognition is one possible filter condition, not a separate replacement for +the filter system. Future JMAP support must map to the same rules rather than +creating a third incompatible store. + +Test script upload, syntax rejection, activation, rename, retrieval, deletion, +folder delivery, redirect, reject, vacation deduplication, and quota limits +before exposing the service. The 0.7 release cases are SIEVE-01 through +SIEVE-05 in [release-testing-0.7.md](release-testing-0.7.md). diff --git a/docs/web.md b/docs/web.md new file mode 100644 index 0000000..d678f19 --- /dev/null +++ b/docs/web.md @@ -0,0 +1,56 @@ +# Webmail and user administration + +The Python 3 `bongo-web` service supplies the classic Bongo Webmail layout, +user settings, tasks, contacts, calendars, and the restricted administrator +view. It talks to the same Store used by IMAP and the delivery agents; it is +not a second mailbox database. + +## Deployment + +The suffixless `web` configuration document is the only Web listener +configuration. A normal deployment listens on loopback port 8080 behind +Apache, nginx, nginx-proxy-manager, or HAProxy. Set `public_base_url`, +`external_https`, `trusted_proxies`, and `client_ip_header` to describe the +real external path. Forwarded client addresses are trusted only from the +configured proxies. + +Bongo can instead serve HTTPS directly. It loads the configured certificate +and private key, binds the privileged HTTPS and redirect ports, then +permanently changes to the `bongo` UID/GID before accepting requests. The HTTP +listener redirects to the configured HTTPS base URL except for the narrowly +scoped ACME HTTP-01 challenge path. Reverse-proxy examples are installed +below `/usr/share/bongo/examples/reverse-proxy`. + +## Sessions and security + +Login creates a server-side session with an HttpOnly cookie. State-changing +API requests require the session's CSRF token. Login attempt windows, session +lifetime, second-factor lifetime, and second-factor attempt limits are +configurable. TOTP fields are shown only when the account has enabled that +optional feature; recovery codes and app passwords are handled by the shared +authentication subsystem. + +HTML mail is sanitized before display. Safe inline or explicitly permitted +images can remain useful for signatures and normal content, while active +markup and unwanted remote tracking are blocked. Administrator actions are +both role-checked and limited to a fixed command set; selecting the +administration view does not create a second login session. + +## User functions + +The current API and interface cover mailbox folders and conversations, +reading and sending mail, contacts, calendar events, filters, actionable +tasks, external account collection, sending identities, preferences, and +signatures. A task is a workflow item: it can be opened for its details and +then dismissed, retried, ignored, or completed where the task type permits. +English, German, and every installed gettext catalogue use the same API +instead of hard-coded interface text. + +The administrator view exposes health and a restricted set of validated +actions. Full server configuration remains the responsibility of +`bongo-admin` and the configuration TUI; Webmail must not become an arbitrary +root command interface. + +The current classic interface is the 0.7 baseline. The responsive PWA redesign +is scheduled for 0.8 and documented separately in [web-pwa.md](web-pwa.md). +JMAP and background Web Push belong to 0.9.