Mario Fetka
97 lines
5.2 KiB
Markdown
97 lines
5.2 KiB
Markdown
# Queue, routing, and delivery
|
|
|
|
`bongoqueue` durably owns messages after SMTP or the external collector has
|
|
accepted them and before every recipient has reached a final destination. It
|
|
coordinates filtering, local Store delivery, remote SMTP, and LMTP; it is not
|
|
a public submission protocol.
|
|
|
|
## Message flow and durability
|
|
|
|
Queue data lives below `/var/lib/bongo/spool`. An accepted message remains
|
|
there while a temporary scanner, Store, DNS, SMTP, LMTP, or quota-dependent
|
|
delivery condition is retried. Permanent failures produce the configured
|
|
delivery-status response or bounce. The Queue must not acknowledge a message
|
|
to its upstream path before the payload and recipient state are durable.
|
|
|
|
Enabled filtering agents register for Queue stages. Antivirus,
|
|
SpamAssassin, rules/Sieve, and final local or remote delivery therefore see a
|
|
common envelope and message rather than independent copies. External POP3 or
|
|
IMAP collection enters this same path and may delete a source message only
|
|
after Bongo has committed it successfully.
|
|
|
|
The internal Queue protocol defaults to TCP 8670. Bind and firewall it as a
|
|
private service interface. It does not authenticate arbitrary Internet
|
|
clients and must never be exposed as an SMTP alternative.
|
|
|
|
## Limits and failure handling
|
|
|
|
The suffixless `queue` document controls spool free-space reservation,
|
|
message lifetime, retry interval, worker concurrency, trusted/hosted domains,
|
|
and bounce policy. `minimumfreespace` prevents normal acceptance from
|
|
consuming the filesystem's final space. Per-user Store quota is separate: a
|
|
full mailbox defers or rejects that recipient without corrupting other
|
|
recipient state.
|
|
|
|
Temporary delivery failures use one persistent exponential retry policy,
|
|
regardless of whether the next hop is a directly resolved Internet MX, a
|
|
configured relayhost, or an internal LMTP destination. The retry counter is
|
|
part of the durable Queue envelope and the next eligible attempt survives a
|
|
daemon or host restart. `retry_initial_seconds` defaults to 300 seconds and
|
|
the delay doubles up to `retry_maximum_seconds`, which defaults to 4000
|
|
seconds. `queueinterval` controls how often the monitor looks for eligible
|
|
work; it does not shorten a message's persisted backoff. An administrator can
|
|
request an immediate attempt with `bongo-queuetool retry` or `flush`.
|
|
|
|
Normal messages expire after `queuetimeout`, while delivery-status messages
|
|
and other null-envelope-sender traffic expire after
|
|
`bounce_queue_timeout`. Both defaults are 432000 seconds (five days). On
|
|
normal-message expiry Bongo uses its existing RFC delivery-status path to
|
|
return a failure for the recipients still pending. An expired null-sender
|
|
message is discarded after the configured bounce lifetime, preventing a
|
|
bounce loop. Held queue entries do not run or expire until released.
|
|
|
|
Collector backpressure is also separate from the byte reserve. The
|
|
`collector` document limits globally pending and per-user pending external
|
|
messages. Queue counts only entries carrying its durable collected-message
|
|
flag, including entries currently owned by a Queue worker. Once a limit is
|
|
reached, the collector does not download or delete the next source message.
|
|
`bongo-queuetool collected-count [user]` exposes the global count and,
|
|
optionally, the selected user's count.
|
|
|
|
JSON integral configuration values and the generic agent loader preserve
|
|
signed 64-bit values. In particular, a large `minimumfreespace` value is not
|
|
truncated into a negative 32-bit number while Store configuration is mirrored
|
|
to disk.
|
|
|
|
A scanner outage is a delivery-policy decision, not permission to silently
|
|
bypass inspection. ClamAV and SpamAssassin failures leave the message queued
|
|
for the configured retry path. Likewise, a per-recipient LMTP temporary error
|
|
must preserve only that recipient for retry and must not redeliver recipients
|
|
which already returned a final success.
|
|
|
|
Queue control-file rewrites use a same-directory work file and an atomic
|
|
rename. The original is retained unless every buffered write and close
|
|
succeeds. A failed submission flush or final rename is rejected and its
|
|
partial entry discarded, so the submitting SMTP peer remains responsible for
|
|
retrying the message instead of receiving a false success.
|
|
|
|
## Administration and shutdown
|
|
|
|
Use `bongo-queuetool` to inspect and perform documented queue operations; do
|
|
not edit spool files manually. A service stop first prevents new work and asks
|
|
the managed agents to finish or abandon their current transaction safely.
|
|
Queue and Store state must be included in a consistent backup.
|
|
|
|
`bongo-queuetool message` exports the exact Queue-owned bytes. The
|
|
`hold-local`, `deliver-local`, and `delete` commands provide a bounded recovery
|
|
and diagnostic path: target 999 is not part of normal routing,
|
|
`deliver-local` retains its source entry, and `delete` must finish the
|
|
operation explicitly. The STQ-02 fixture uses these commands to compare the
|
|
Queue payload against the Store document, including the intentional Queue
|
|
trace header.
|
|
|
|
The 0.7 release matrix exercises byte preservation, disk exhaustion, retry,
|
|
expiry, bounce, hold/release, concurrency, invalid commands, quota mapping,
|
|
and collector floods under STQ-02 through STQ-12. SMTP and LMTP rows add their
|
|
protocol-specific recipient and retry cases.
|