bongo/docs/queue.md

Queue, routing, and delivery

bongoqueue durably owns messages after SMTP or the external collector has accepted them and before every recipient has reached a final destination. It coordinates filtering, local Store delivery, remote SMTP, and LMTP; it is not a public submission protocol.

Message flow and durability

Queue data lives below /var/lib/bongo/spool. An accepted message remains there while a temporary scanner, Store, DNS, SMTP, LMTP, or quota-dependent delivery condition is retried. Permanent failures produce the configured delivery-status response or bounce. The Queue must not acknowledge a message to its upstream path before the payload and recipient state are durable.

Enabled filtering agents register for Queue stages. Antivirus, SpamAssassin, rules/Sieve, and final local or remote delivery therefore see a common envelope and message rather than independent copies. External POP3 or IMAP collection enters this same path and may delete a source message only after Bongo has committed it successfully.

The internal Queue protocol defaults to TCP 8670. Bind and firewall it as a private service interface. It does not authenticate arbitrary Internet clients and must never be exposed as an SMTP alternative.

Limits and failure handling

The suffixless queue document controls spool free-space reservation, message lifetime, retry interval, worker concurrency, trusted/hosted domains, and bounce policy. minimumfreespace prevents normal acceptance from consuming the filesystem's final space. Per-user Store quota is separate: a full mailbox defers or rejects that recipient without corrupting other recipient state.

Temporary delivery failures use one persistent exponential retry policy, regardless of whether the next hop is a directly resolved Internet MX, a configured relayhost, or an internal LMTP destination. The retry counter is part of the durable Queue envelope and the next eligible attempt survives a daemon or host restart. retry_initial_seconds defaults to 300 seconds and the delay doubles up to retry_maximum_seconds, which defaults to 4000 seconds. queueinterval controls how often the monitor looks for eligible work; it does not shorten a message's persisted backoff. An administrator can request an immediate attempt with bongo-queuetool retry or flush.

Normal messages expire after queuetimeout, while delivery-status messages and other null-envelope-sender traffic expire after bounce_queue_timeout. Both defaults are 432000 seconds (five days). On normal-message expiry Bongo uses its existing RFC delivery-status path to return a failure for the recipients still pending. An expired null-sender message is discarded after the configured bounce lifetime, preventing a bounce loop. Held queue entries do not run or expire until released.

Collector backpressure is also separate from the byte reserve. The collector document limits globally pending and per-user pending external messages. Queue counts only entries carrying its durable collected-message flag, including entries currently owned by a Queue worker. Once a limit is reached, the collector does not download or delete the next source message. bongo-queuetool collected-count [user] exposes the global count and, optionally, the selected user's count.

JSON integral configuration values and the generic agent loader preserve signed 64-bit values. In particular, a large minimumfreespace value is not truncated into a negative 32-bit number while Store configuration is mirrored to disk.

A scanner outage is a delivery-policy decision, not permission to silently bypass inspection. ClamAV and SpamAssassin failures leave the message queued for the configured retry path. Likewise, a per-recipient LMTP temporary error must preserve only that recipient for retry and must not redeliver recipients which already returned a final success.

Queue control-file rewrites use a same-directory work file and an atomic rename. The original is retained unless every buffered write and close succeeds. A failed submission flush or final rename is rejected and its partial entry discarded, so the submitting SMTP peer remains responsible for retrying the message instead of receiving a false success.

Administration and shutdown

Use bongo-queuetool to inspect and perform documented queue operations; do not edit spool files manually. A service stop first prevents new work and asks the managed agents to finish or abandon their current transaction safely. Queue and Store state must be included in a consistent backup.

bongo-queuetool message exports the exact Queue-owned bytes. The hold-local, deliver-local, and delete commands provide a bounded recovery and diagnostic path: target 999 is not part of normal routing, deliver-local retains its source entry, and delete must finish the operation explicitly. The STQ-02 fixture uses these commands to compare the Queue payload against the Store document, including the intentional Queue trace header.

The 0.7 release matrix exercises byte preservation, disk exhaustion, retry, expiry, bounce, hold/release, concurrency, invalid commands, quota mapping, and collector floods under STQ-02 through STQ-12. SMTP and LMTP rows add their protocol-specific recipient and retry cases.