Mario Fetka
23 KiB
Bongo 0.7.0 release test matrix
This matrix is the release gate for the 0.7.0 tag. A feature is not treated as working merely because it builds, appears in documentation, or is listed in a protocol capability response.
Test-run rules
Three ordered passes are required. Use PASS, FAIL, or N/A: reason in
every result cell; an empty cell is untested. N/A is allowed only for a
genuinely optional deployment mode and never for a feature advertised by the
tested configuration.
R1 is the implementation and repair pass. Test every row in order. When a row fails, fix it, add automated regression coverage where practical, reinstall, and repeat that row until it passes. Earlier R1 results remain useful; R1 is complete only after every row has produced at least one passing result.
R2 starts from the first row only after R1 is entirely green. Its purpose is to find regressions introduced while fixing later R1 rows. A failure is fixed and retested, then R2 continues. R3 starts only after R2 is entirely green and is the final complete regression pass. The 0.7.0 tag requires every applicable R3 row to pass.
Each result links to evidence recording the Git commit, package build, configuration checksum, compiler, machine, and date used for that individual test. Therefore fixes during R1 or R2 remain traceable without pretending the whole pass used one immutable source snapshot.
| Pass | Started | Finished | Evidence index | Result |
|---|---|---|---|---|
| R1 implementation/repair | 2026-07-19 | R1 evidence | In progress | |
| R2 regression | ||||
| R3 final regression |
The disposable test domain, users, passwords, private keys, captured messages, and scanner samples must not be reused in production. Secret material belongs only in the pass evidence directory, which must not be committed.
Build, portability, and packages
| ID | Test | R1 | R2 | R3 |
|---|---|---|---|---|
| BLD-01 | Clean GCC debug build with actionable warnings enabled | PASS | ||
| BLD-02 | Complete GCC CTest suite | PASS | ||
| BLD-03 | Clean Clang debug build with the same feature set | PASS | ||
| BLD-04 | Complete Clang CTest suite | PASS | ||
| BLD-05 | ASan and UBSan build plus unit and protocol sessions | PASS | ||
| BLD-06 | LeakSanitizer run over startup, delivery, access, and shutdown | PASS | ||
| BLD-07 | Separate ThreadSanitizer concurrency run | PASS | ||
| BLD-08 | 32-bit x86 build and tests without pointer/size truncation | PASS | ||
| BLD-09 | ARMv5 build with the supported baseline toolchain | PASS | ||
| BLD-10 | ARMv7 hard-float build and tests | PASS | ||
| BLD-11 | AArch64 build and tests | PASS | ||
| BLD-12 | CPack binary packages install into a clean staging root | PASS | ||
| BLD-13 | CPack source archive has no -linux, VCS, build, or private files |
PASS | ||
| BLD-14 | Gentoo live ebuild builds, tests, installs, and passes package QA | PASS | ||
| BLD-15 | Debian package builds and installs on the target Debian release | PASS | ||
| BLD-16 | /usr, /usr/local, lib/lib64, sysconf, state, and runtime paths follow FHS |
PASS | ||
| BLD-17 | Install, upgrade, reinstall, and uninstall preserve/remove documented state | PASS | ||
| BLD-18 | Installed shared libraries have complete dependencies and valid SONAMEs | PASS | ||
| BLD-19 | Man pages, translations, Web assets, docs, units, and examples are installed | PASS |
Debian Trixie release bundle
The Gitea release workflow must not silently use an unpatched distribution
Mailutils or an arbitrary system copy of libsrs2. It must fetch the source for
both dependencies, apply the maintained Mailutils Sieve repair, and rebuild
both as native Debian Trixie source and binary packages. In particular, the
archived historical libsrs2 .deb files must never be copied into the release
bundle. They are input references for compatibility testing only; the release
packages must be newly produced from the archived libsrs2 source on Trixie.
The workflow then installs those freshly built development packages before
configuring Bongo and builds Bongo against those exact headers and libraries.
The Trixie release ZIP must contain the resulting Mailutils and libsrs2 binary
packages as well as their complete Debian source-package artifacts (.dsc,
upstream source archive, Debian patch archive, and .changes/.buildinfo where
produced). BLD-15 fails if an old libsrs2 binary package is bundled, if either
dependency was obtained only from the runner filesystem, if the Bongo link
step resolves a different copy, or if any of these redistributable build
inputs and outputs is absent from the ZIP.
Setup, configuration, and service lifecycle
| ID | Test | R1 | R2 | R3 |
|---|---|---|---|---|
| CFG-01 | Fresh curses setup with sane defaults and correctly placed prompts | PASS | ||
| CFG-02 | Setup is safe and idempotent when repeated | PASS | ||
| CFG-03 | bongo-admin config check, edit, sync, rollback, and manager reload |
PASS | ||
| CFG-04 | Suffixless config, alias, SSL, DKIM, DMARC, and ACME directory layout | PASS | ||
| CFG-05 | One-shot migration recognises historical names and visibly moves old files | PASS | ||
| CFG-06 | Alias fragments support aliases only, domains only, or both | PASS | ||
| CFG-07 | Scanner editor preserves unrelated settings and restores on failure | PASS | ||
| CFG-08 | Reverse-proxy, HAProxy, direct HTTP, and direct HTTPS validation paths | PASS | ||
| CFG-09 | Self-signed GnuTLS certificate/key generation and permission checks | PASS | ||
| CFG-10 | Existing certificate/key validation and safe rejection of broken pairs | PASS | ||
| CFG-11 | DKIM/SPF/DMARC discovery, key import/generation, DNS hints, and task | PASS | ||
| CFG-12 | Manager start, reload, restart, clean stop, stale PID, and crash recovery | PASS | ||
| CFG-13 | Every agent starts as intended; permanent/effective privilege drops and supervisor exceptions are bounded | PASS | ||
| CFG-14 | tmpfiles creates runtime paths; package contains no runtime PID directories | PASS | ||
| CFG-15 | systemd ordering, hardening, scanner timer, and worker scheduling | PASS | ||
| CFG-16 | Timezone cache rebuild uses the installed checked-in snapshot | PASS |
Store and Queue
| ID | Test | R1 | R2 | R3 |
|---|---|---|---|---|
| STQ-01 | New users receive independent stores and standard folders | PASS | ||
| STQ-02 | Local message survives Queue-to-Store delivery byte-for-byte | PASS | ||
| STQ-03 | Queue free-space calculation is correct on 32/64-bit and reserves space | PASS | ||
| STQ-04 | Full-disk and unreadable-spool failures defer safely without mail loss | PASS | ||
| STQ-05 | Retry, expiry, bounce, listing, hold/release, and deletion | PASS | ||
| STQ-06 | Concurrent delivery/read/move/delete/restart preserves integrity | PASS | ||
| STQ-07 | Backup/export and restore reproduce mail, metadata, contacts, calendars | PASS | ||
| STQ-08 | Invalid commands, oversized lines, and unauthorised access fail safely | PASS | ||
| STQ-09 | Per-user mail quota excludes contacts/calendars and atomically blocks concurrent growth | PASS | ||
| STQ-10 | Queue/SMTP and external collection map Store 5220 to quota-exceeded delivery without data loss | PASS | ||
| STQ-11 | Lowered/unlimited limits, large sizes, upgrade migration, backup and restore preserve quota state | PASS | ||
| STQ-12 | Bounded 128-KiB collector flood reaches a small test quota without exceeding it, duplicating mail, or deleting the first message that was not durably imported | PASS |
SMTP receive, submission, and delivery
| ID | Test | R1 | R2 | R3 |
|---|---|---|---|---|
| SMTP-01 | Port 25 greeting, HELO, EHLO, NOOP, RSET, HELP, VRFY policy, QUIT | PASS | ||
| SMTP-02 | Port 587 STARTTLS submission with shared GSASL authentication | PASS | ||
| SMTP-03 | Port 465 implicit TLS submission | PASS | ||
| SMTP-04 | Trusted port 26 accepts only configured networks and applies limits | PASS | ||
| SMTP-05 | Internal sender/domain normalisation uses peer DNS/IP mapping safely | PASS | ||
| SMTP-06 | Local authenticated delivery from test1 to test2 | PASS | ||
| SMTP-30 | curl, swaks, Xeams, GNU Mailutils, and paced Telnet delivery over authenticated submission and unauthenticated port 25/26 | PASS | ||
| SMTP-31 | Unprivileged sendmail-compatible local submission without the Bongo system credential | PASS | ||
| SMTP-32 | Telnet activity resets the configured per-operation timeout; abandoned port 25/26 sessions close | PASS | ||
| SMTP-33 | Load threshold gives newly accepted inbound, trusted-device, and submission sessions the short stress timeout | PASS | ||
| SMTP-07 | Internet delivery prefers STARTTLS and follows opportunistic fallback | PASS | ||
| SMTP-08 | Required TLS and REQUIRETLS defer rather than downgrade |
PASS | ||
| SMTP-27 | DANE takes precedence; MTA-STS enforce/testing/cache and MX wildcard policy | |||
| SMTP-09 | Relayhost hostname, port, TLS, authentication, failure, and recovery | |||
| SMTP-10 | Open-relay tests reject unauthenticated/untrusted third-party relay | |||
| SMTP-11 | SIZE exact limit, over-limit, absent size, and disk-space responses |
|||
| SMTP-12 | PIPELINING ordering and error recovery |
|||
| SMTP-13 | 8BITMIME remains byte-correct |
|||
| SMTP-14 | DSN RET, ENVID, NOTIFY, and ORCPT cases |
|||
| SMTP-15 | CHUNKING/bounded BDAT, including zero/last and malformed chunks |
|||
| SMTP-16 | BINARYMIME remains unadvertised and rejected |
|||
| SMTP-17 | SMTPUTF8 delivery and invalid UTF-8 rejection |
|||
| SMTP-26 | IDNA2008 domains use one canonical A-label in config, aliases, DNS lookup, SMTP and Web compose | |||
| SMTP-18 | Recipient validation, multiple recipients, partial refusal, aliases | |||
| SMTP-19 | Domain/address aliases, forwarding loops, and SRS forwarding | |||
| SMTP-20 | LMTP client uses a disposable status-aware LMTP test daemon | |||
| SMTP-21 | LMTP per-recipient 2xx, 4xx, 5xx, retry, and duplicate prevention | |||
| SMTP-22 | XCLIENT/XFORWARD negotiation only to trusted destinations | |||
| SMTP-23 | Fallback client-address header only for configured trusted peers | |||
| SMTP-24 | Incoming PROXY v1/v2 only from configured proxy networks | |||
| SMTP-25 | Malformed commands, long lines, floods, timeout, and disconnect | |||
| SMTP-28 | TLSRPT preserves the exact DNS policy, aggregates completed UTC days, and emits valid RFC 8460 JSON | |||
| SMTP-29 | TLSRPT gzip mail/HTTPS delivery, DKIM requirement, random delay, durable retry, expiry, and report-loop suppression |
IMAP4rev2, IMAP4rev1, and legacy IMAP4
| ID | Test | R1 | R2 | R3 |
|---|---|---|---|---|
| IMAP-01 | Capability list matches implemented rev2/rev1/legacy behaviour | |||
| IMAP-02 | Port 143 STARTTLS, pre-TLS LOGINDISABLED, tagged NO for plaintext LOGIN, repeated STARTTLS rejection |
|||
| IMAP-03 | Port 993 TLS, LOGIN, SASL PLAIN, and SASL LOGIN | |||
| IMAP-04 | CAPABILITY, NOOP, LOGOUT, ID, and NAMESPACE | |||
| IMAP-05 | LIST, LSUB, extended LIST, LIST-STATUS, hierarchy quoting | |||
| IMAP-06 | CREATE, DELETE, RENAME, SUBSCRIBE, UNSUBSCRIBE | |||
| IMAP-07 | SELECT, EXAMINE, STATUS/SIZE, CHECK, CLOSE, UNSELECT | |||
| IMAP-08 | APPEND, LITERAL+, APPENDLIMIT, flags/date, oversized literals | |||
| IMAP-09 | FETCH/UID FETCH body, headers, partials, flags, envelope, binary | |||
| IMAP-10 | STORE/UID STORE add/remove/replace flags and silent variants | |||
| IMAP-11 | COPY/UID COPY/MOVE/UID MOVE and UIDPLUS response codes | |||
| IMAP-12 | EXPUNGE/UID EXPUNGE preserve sequence and UID semantics | |||
| IMAP-13 | SEARCH/UID SEARCH sets, flags, dates, sizes, HEADER/SUBJECT/FROM/TO | |||
| IMAP-14 | SEARCH BODY/TEXT/NOT/OR/CHARSET, ESEARCH RETURN, saved $ set |
|||
| IMAP-15 | IDLE notifications and DONE with concurrent delivery/disconnect | |||
| IMAP-16 | ENABLE UTF8=ACCEPT and UTF-8 mailbox/message names | |||
| IMAP-17 | SPECIAL-USE and CREATE-SPECIAL-USE folders | |||
| IMAP-18 | Old-client plain IMAP4 subset without rev1/rev2 assumptions | |||
| IMAP-19 | Invalid states, malformed sets/literals, oversized input, and STARTTLS plaintext-pipeline injection resistance | |||
| IMAP-20 | Reconnect, simultaneous clients, restart, mailbox consistency | |||
| IMAP-21 | RFC 9208 capability, GETQUOTA/GETQUOTAROOT, storage units and unknown mailbox handling | |||
| IMAP-22 | DELETED-STORAGE and [OVERQUOTA] for APPEND/COPY/MOVE; no unadvertised QUOTASET |
POP3 and ManageSieve
| ID | Test | R1 | R2 | R3 |
|---|---|---|---|---|
| POP-01 | Port 110 CAPA/STLS; credentials unavailable before TLS; buffered plaintext discarded | |||
| POP-02 | Port 995 TLS with USER/PASS and shared GSASL mechanisms | |||
| POP-03 | STAT, LIST, UIDL, RETR, TOP, and dot transparency | |||
| POP-04 | DELE, RSET, NOOP, QUIT commit, and disconnect rollback | |||
| POP-05 | UTF8, STLS rejection after UTF8, PIPELINING, response codes, malformed input, timeout | |||
| POP-06 | Shared Store quota: DELE reserves space until QUIT, RSET/disconnect release none, committed delete frees space | |||
| SIEVE-01 | Port 4190 greeting, STARTTLS, and GSASL authentication | |||
| SIEVE-02 | CAPABILITY, HAVESPACE, PUT/CHECK/LIST/GETSCRIPT | |||
| SIEVE-03 | SETACTIVE, RENAME, DELETE, LOGOUT, and persistence | |||
| SIEVE-04 | fileinto, envelope, body, variables, vacation extensions | |||
| SIEVE-05 | Invalid syntax, quota/size, duplicate names, traversal, auth |
Authentication and identity
| ID | Test | R1 | R2 | R3 |
|---|---|---|---|---|
| AUTH-01 | New Argon2id and successful PBKDF2/legacy upgrade | |||
| AUTH-02 | Wrong password, disabled user, throttling, constant-time failure | |||
| AUTH-03 | Optional TOTP enable/login/disable; field hidden when disabled | |||
| AUTH-04 | Valid in-memory QR data never written or logged | |||
| AUTH-05 | Recovery copy-all, one-time atomic use, regeneration | |||
| AUTH-06 | App-password expiry/revocation/service/network scope | |||
| AUTH-07 | App-password requirement cannot be enabled while TOTP is off | |||
| AUTH-08 | Local/LDAP master password follows documented protocol policy | |||
| AUTH-09 | LDAP/AD lookup, group provisioning, adoption, rejection | |||
| AUTH-10 | LDAP StartTLS/LDAPS validation and secure bind-secret file | |||
| AUTH-11 | LDAP outage cache only for outages and expires as configured | |||
| AUTH-12 | Admin cache status/extend-all audits every user safely | |||
| AUTH-13 | Rename, aliases, forced logout, and old-name warning task | |||
| AUTH-14 | Client inventory records useful metadata without secrets |
Mail authentication, filtering, antivirus, and antispam
| ID | Test | R1 | R2 | R3 |
|---|---|---|---|---|
| MAILAUTH-01 | SPF pass/fail/softfail/neutral/none/temp/permerror | |||
| MAILAUTH-02 | Incoming DKIM valid/invalid/multiple/modified/DNS error | |||
| MAILAUTH-03 | DMARC alignment and none/quarantine/reject/temp policies | |||
| MAILAUTH-04 | Forged Authentication-Results removed; local result added | |||
| MAILAUTH-05 | Temporary key signs outbound; independent verification passes | |||
| MAILAUTH-06 | Missing/unreadable/mismatched/rotated DKIM key safety | |||
| MAILAUTH-07 | SRS encode/decode/tamper/expiry/bounce/loop handling | |||
| SCAN-01 | ClamAV clean and EICAR paths | |||
| SCAN-02 | SpamAssassin ham/spam and score/header handling | |||
| SCAN-03 | External POP3 and IMAP collection traverse the same ClamAV, SpamAssassin, rules, and final Store quota path as SMTP mail | |||
| SCAN-04 | Runtime-generated EICAR is rejected/quarantined according to policy and GTUBE receives the configured spam result; neither test silently bypasses a disabled/unreachable scanner | |||
| SCAN-03 | Scanner unavailable/timeout/malformed/restart policy | |||
| SCAN-04 | Official/KAM/Heinlein channels, signatures, lint, timer | |||
| SCAN-05 | Worker detects ClamAV/SA data faults without SA-age false alarm | |||
| SCAN-06 | Filters, mailing-list detection, folders, vacation, loop prevention | |||
| SCAN-07 | Quarantine/task creation, ignore, retry, delete, audit |
Web, DAV, and external accounts
| ID | Test | R1 | R2 | R3 |
|---|---|---|---|---|
| WEB-01 | Classic login/assets render in current Firefox/Chromium | |||
| WEB-02 | Login/logout/session/CSRF/cookies/brute-force limiting | |||
| WEB-03 | Trusted proxy/header chain resists client-address spoofing | |||
| WEB-04 | Folders/read/compose/reply/forward/attachments/move/delete | |||
| WEB-05 | HTML sanitisation keeps safe images and blocks active tracking | |||
| WEB-06 | Identities, external senders, signatures, SMTP selection | |||
| WEB-07 | Filters/tasks/quarantine/external accounts/app passwords/2FA | |||
| WEB-08 | English, German, and every installed gettext catalogue load | |||
| WEB-09 | ICS file import and bounded public-HTTPS holiday subscription into personal/named calendars | PASS | ||
| DAV-01 | CalDAV discovery, CRUD, sync, ETag, and iCalendar | |||
| DAV-02 | CardDAV discovery, CRUD, sync, ETag, and vCard | |||
| DAV-03 | DAV master/app-password scope and TLS/proxy policy | |||
| EXT-01 | Provider presets and manual configuration validation | |||
| EXT-02 | Multiple LMTP-seeded POP3 source accounts per Bongo user, multiple messages, successful-store-before-delete, empty Cyrus sources | |||
| EXT-03 | Multiple LMTP-seeded IMAP source accounts per Bongo user, multiple messages, per-account UID state, successful-store-before-delete, empty Cyrus sources | |||
| EXT-04 | Multiple accounts and identities stay isolated per user | |||
| EXT-05 | Matching external SMTP identity from Web and SMTP client | |||
| EXT-06 | Collector TLS/failure/retry/deduplication/secret-safe logs |
ACME, administration, observability, and security
| ID | Test | R1 | R2 | R3 |
|---|---|---|---|---|
| ACME-01 | HTTP-01 order/challenge/renew/deploy/agent reload | |||
| ACME-02 | DNS-01 nsupdate with TSIG and cleanup | |||
| ACME-03 | DNS-01 Cloudflare and generic provider contract | |||
| ACME-04 | Multi-domain/wildcard, CA retry limits, failure/rollback | |||
| ADM-01 | Admin/config/queue/store tools, TUI, help, man pages agree | |||
| ADM-02 | User/domain/alias/password/rename/config/scanner validation | |||
| ADM-03 | Structured Web/SMTP/IMAP/POP3/Sieve auth-failure logs | |||
| ADM-04 | Fail2ban matches failures and not successful logins | |||
| SEC-01 | No secrets in argv, diagnostics, normal logs, Web responses | |||
| SEC-02 | Safe modes for auth/TLS/DKIM/SRS/ACME/provider secrets | |||
| SEC-03 | Secure TLS default; explicit legacy mode isolated/logged | |||
| SEC-04 | Malformed corpus, limits, traversal, injection, smuggling | |||
| SEC-05 | Connection/auth/message rate limits resist floods | |||
| SEC-06 | Restarts during pipeline cause no loss or duplicates | |||
| SEC-07 | Dependency/CVE/source review has no 0.7 blockers |
Required integration fixtures
- two disposable local users and at least two hosted/alias domains;
- local DNS able to publish temporary SPF, DKIM, and DMARC records;
- temporary GnuTLS certificates and a temporary OpenDKIM signing key;
- ClamAV, SpamAssassin, EICAR, and deterministic ham/spam mail;
- Xeams DevNull SMTP for captured outbound relay messages;
- the loopback-only smtp4dev fixture in
contrib/testing, providing authenticated STARTTLS and implicit-TLS provider submission plus a capture API for relayhost and external-identity tests; - the loopback-only Cyrus fixture in
contrib/testing, providing real IMAP, POP3, and LMTP delivery for external collection and forwarding tests; - a status-aware LMTP fault fixture which records envelope/message and can return per-recipient 2xx, 4xx, and 5xx responses;
- disposable LDAP/AD-compatible and ACME/DNS challenge fixtures;
- old and current client probes, including a basic legacy IMAP4 path.
DevNull and LMTP captures are checked for exact envelope, headers, body, DKIM signature, SRS address, client metadata, and duplicate or missing deliveries. Merely receiving a TCP connection is not a pass.
Web test delivery safety
The automated HTTP-level web suite replaces the queue submission function with
an in-process fake, so it cannot send mail outside the test process. Live web
mail tests may use only recipients in the configured local test domain. Before
using any non-local recipient, configure Bongo's relay host to the Xeams DevNull
SMTP listener and verify that direct Internet delivery is disabled. The DevNull
capture must then be inspected for the envelope recipients (including Bcc) and
for absence of a Bcc message header.
The web release matrix covers session login/logout, optional TOTP, mail folder listing/opening/submission, contact and calendar create/list/update/delete, filters, tasks, external accounts, sender identities and signatures, server administration, and CalDAV/CardDAV CRUD and discovery.