bongo/docs/release-testing-0.7.md

Bongo 0.7.0 release test matrix

This matrix is the release gate for the 0.7.0 tag. A feature is not treated as working merely because it builds, appears in documentation, or is listed in a protocol capability response.

Test-run rules

Three ordered passes are required. Use PASS, FAIL, or N/A: reason in every result cell; an empty cell is untested. N/A is allowed only for a genuinely optional deployment mode and never for a feature advertised by the tested configuration.

R1 is the implementation and repair pass. Test every row in order. When a row fails, fix it, add automated regression coverage where practical, reinstall, and repeat that row until it passes. Earlier R1 results remain useful; R1 is complete only after every row has produced at least one passing result.

R2 starts from the first row only after R1 is entirely green. Its purpose is to find regressions introduced while fixing later R1 rows. A failure is fixed and retested, then R2 continues. R3 starts only after R2 is entirely green and is the final complete regression pass. The 0.7.0 tag requires every applicable R3 row to pass.

Each result links to evidence recording the Git commit, package build, configuration checksum, compiler, machine, and date used for that individual test. Therefore fixes during R1 or R2 remain traceable without pretending the whole pass used one immutable source snapshot.

Pass	Started	Finished	Evidence index	Result
R1 implementation/repair	2026-07-19		R1 evidence	In progress
R2 regression
R3 final regression

The disposable test domain, users, passwords, private keys, captured messages, and scanner samples must not be reused in production. Secret material belongs only in the pass evidence directory, which must not be committed.

Build, portability, and packages

ID	Test	R1	R2	R3
BLD-01	Clean GCC debug build with actionable warnings enabled	PASS
BLD-02	Complete GCC CTest suite	PASS
BLD-03	Clean Clang debug build with the same feature set	PASS
BLD-04	Complete Clang CTest suite	PASS
BLD-05	ASan and UBSan build plus unit and protocol sessions	PASS
BLD-06	LeakSanitizer run over startup, delivery, access, and shutdown	PASS
BLD-07	Separate ThreadSanitizer concurrency run	PASS
BLD-08	32-bit x86 build and tests without pointer/size truncation	PASS
BLD-09	ARMv5 build with the supported baseline toolchain	PASS
BLD-10	ARMv7 hard-float build and tests	PASS
BLD-11	AArch64 build and tests	PASS
BLD-12	CPack binary packages install into a clean staging root	PASS
BLD-13	CPack source archive has no -linux, VCS, build, or private files	PASS
BLD-14	Gentoo live ebuild builds, tests, installs, and passes package QA	PASS
BLD-15	Debian package builds and installs on the target Debian release
BLD-16	/usr, /usr/local, lib/lib64, sysconf, state, and runtime paths follow FHS
BLD-17	Install, upgrade, reinstall, and uninstall preserve/remove documented state
BLD-18	Installed shared libraries have complete dependencies and valid SONAMEs
BLD-19	Man pages, translations, Web assets, docs, units, and examples are installed

Debian Trixie release bundle

The Gitea release workflow must not silently use an unpatched distribution Mailutils or an arbitrary system copy of libsrs2. It must fetch the source for both dependencies, apply the maintained Mailutils Sieve repair, and rebuild both as native Debian Trixie source and binary packages. In particular, the archived historical libsrs2 .deb files must never be copied into the release bundle. They are input references for compatibility testing only; the release packages must be newly produced from the archived libsrs2 source on Trixie. The workflow then installs those freshly built development packages before configuring Bongo and builds Bongo against those exact headers and libraries.

The Trixie release ZIP must contain the resulting Mailutils and libsrs2 binary packages as well as their complete Debian source-package artifacts (.dsc, upstream source archive, Debian patch archive, and .changes/.buildinfo where produced). BLD-15 fails if an old libsrs2 binary package is bundled, if either dependency was obtained only from the runner filesystem, if the Bongo link step resolves a different copy, or if any of these redistributable build inputs and outputs is absent from the ZIP.

Setup, configuration, and service lifecycle

ID	Test	R1	R2	R3
CFG-01	Fresh curses setup with sane defaults and correctly placed prompts
CFG-02	Setup is safe and idempotent when repeated
CFG-03	bongo-admin config check, edit, sync, rollback, and manager reload
CFG-04	Suffixless config, alias, SSL, DKIM, DMARC, and ACME directory layout
CFG-05	One-shot migration recognises historical names and visibly moves old files
CFG-06	Alias fragments support aliases only, domains only, or both
CFG-07	Scanner editor preserves unrelated settings and restores on failure
CFG-08	Reverse-proxy, HAProxy, direct HTTP, and direct HTTPS validation paths
CFG-09	Self-signed GnuTLS certificate/key generation and permission checks
CFG-10	Existing certificate/key validation and safe rejection of broken pairs
CFG-11	DKIM/SPF/DMARC discovery, key import/generation, DNS hints, and task
CFG-12	Manager start, reload, restart, clean stop, stale PID, and crash recovery
CFG-13	Every agent starts as intended and drops privileges
CFG-14	tmpfiles creates runtime paths; package contains no runtime PID directories
CFG-15	systemd ordering, hardening, scanner timer, and worker scheduling
CFG-16	Timezone cache rebuild uses the installed checked-in snapshot

Store and Queue

ID	Test	R1	R2	R3
STQ-01	New users receive independent stores and standard folders
STQ-02	Local message survives Queue-to-Store delivery byte-for-byte
STQ-03	Queue free-space calculation is correct on 32/64-bit and reserves space
STQ-04	Full-disk and unreadable-spool failures defer safely without mail loss
STQ-05	Retry, expiry, bounce, listing, hold/release, and deletion
STQ-06	Concurrent delivery/read/move/delete/restart preserves integrity
STQ-07	Backup/export and restore reproduce mail, metadata, contacts, calendars
STQ-08	Invalid commands, oversized lines, and unauthorised access fail safely

SMTP receive, submission, and delivery

ID	Test	R1	R2	R3
SMTP-01	Port 25 greeting, HELO, EHLO, NOOP, RSET, HELP, VRFY policy, QUIT
SMTP-02	Port 587 STARTTLS submission with shared GSASL authentication
SMTP-03	Port 465 implicit TLS submission
SMTP-04	Trusted port 26 accepts only configured networks and applies limits
SMTP-05	Internal sender/domain normalisation uses peer DNS/IP mapping safely
SMTP-06	Local authenticated delivery from test1 to test2
SMTP-07	Internet delivery prefers STARTTLS and follows opportunistic fallback
SMTP-08	Required TLS and REQUIRETLS defer rather than downgrade
SMTP-09	Relayhost hostname, port, TLS, authentication, failure, and recovery
SMTP-10	Open-relay tests reject unauthenticated/untrusted third-party relay
SMTP-11	SIZE exact limit, over-limit, absent size, and disk-space responses
SMTP-12	PIPELINING ordering and error recovery
SMTP-13	8BITMIME remains byte-correct
SMTP-14	DSN RET, ENVID, NOTIFY, and ORCPT cases
SMTP-15	CHUNKING/bounded BDAT, including zero/last and malformed chunks
SMTP-16	BINARYMIME remains unadvertised and rejected
SMTP-17	SMTPUTF8 delivery and invalid UTF-8 rejection
SMTP-18	Recipient validation, multiple recipients, partial refusal, aliases
SMTP-19	Domain/address aliases, forwarding loops, and SRS forwarding
SMTP-20	LMTP client uses a disposable status-aware LMTP test daemon
SMTP-21	LMTP per-recipient 2xx, 4xx, 5xx, retry, and duplicate prevention
SMTP-22	XCLIENT/XFORWARD negotiation only to trusted destinations
SMTP-23	Fallback client-address header only for configured trusted peers
SMTP-24	Incoming PROXY v1/v2 only from configured proxy networks
SMTP-25	Malformed commands, long lines, floods, timeout, and disconnect

IMAP4rev2, IMAP4rev1, and legacy IMAP4

ID	Test	R1	R2	R3
IMAP-01	Capability list matches implemented rev2/rev1/legacy behaviour
IMAP-02	Port 143 STARTTLS and pre-TLS LOGINDISABLED
IMAP-03	Port 993 TLS, LOGIN, SASL PLAIN, and SASL LOGIN
IMAP-04	CAPABILITY, NOOP, LOGOUT, ID, and NAMESPACE
IMAP-05	LIST, LSUB, extended LIST, LIST-STATUS, hierarchy quoting
IMAP-06	CREATE, DELETE, RENAME, SUBSCRIBE, UNSUBSCRIBE
IMAP-07	SELECT, EXAMINE, STATUS/SIZE, CHECK, CLOSE, UNSELECT
IMAP-08	APPEND, LITERAL+, APPENDLIMIT, flags/date, oversized literals
IMAP-09	FETCH/UID FETCH body, headers, partials, flags, envelope, binary
IMAP-10	STORE/UID STORE add/remove/replace flags and silent variants
IMAP-11	COPY/UID COPY/MOVE/UID MOVE and UIDPLUS response codes
IMAP-12	EXPUNGE/UID EXPUNGE preserve sequence and UID semantics
IMAP-13	SEARCH/UID SEARCH sets, flags, dates, sizes, HEADER/SUBJECT/FROM/TO
IMAP-14	SEARCH BODY/TEXT/NOT/OR/CHARSET, ESEARCH RETURN, saved $ set
IMAP-15	IDLE notifications and DONE with concurrent delivery/disconnect
IMAP-16	ENABLE UTF8=ACCEPT and UTF-8 mailbox/message names
IMAP-17	SPECIAL-USE and CREATE-SPECIAL-USE folders
IMAP-18	Old-client plain IMAP4 subset without rev1/rev2 assumptions
IMAP-19	Invalid states, malformed sets/literals, oversized/pipelined input
IMAP-20	Reconnect, simultaneous clients, restart, mailbox consistency

POP3 and ManageSieve

ID	Test	R1	R2	R3
POP-01	Port 110 CAPA/STLS; credentials unavailable before TLS
POP-02	Port 995 TLS with USER/PASS and shared GSASL mechanisms
POP-03	STAT, LIST, UIDL, RETR, TOP, and dot transparency
POP-04	DELE, RSET, NOOP, QUIT commit, and disconnect rollback
POP-05	UTF8, PIPELINING, response codes, malformed input, timeout
SIEVE-01	Port 4190 greeting, STARTTLS, and GSASL authentication
SIEVE-02	CAPABILITY, HAVESPACE, PUT/CHECK/LIST/GETSCRIPT
SIEVE-03	SETACTIVE, RENAME, DELETE, LOGOUT, and persistence
SIEVE-04	fileinto, envelope, body, variables, vacation extensions
SIEVE-05	Invalid syntax, quota/size, duplicate names, traversal, auth

Authentication and identity

ID	Test	R1	R2	R3
AUTH-01	New Argon2id and successful PBKDF2/legacy upgrade
AUTH-02	Wrong password, disabled user, throttling, constant-time failure
AUTH-03	Optional TOTP enable/login/disable; field hidden when disabled
AUTH-04	Valid in-memory QR data never written or logged
AUTH-05	Recovery copy-all, one-time atomic use, regeneration
AUTH-06	App-password expiry/revocation/service/network scope
AUTH-07	App-password requirement cannot be enabled while TOTP is off
AUTH-08	Local/LDAP master password follows documented protocol policy
AUTH-09	LDAP/AD lookup, group provisioning, adoption, rejection
AUTH-10	LDAP StartTLS/LDAPS validation and secure bind-secret file
AUTH-11	LDAP outage cache only for outages and expires as configured
AUTH-12	Admin cache status/extend-all audits every user safely
AUTH-13	Rename, aliases, forced logout, and old-name warning task
AUTH-14	Client inventory records useful metadata without secrets

Mail authentication, filtering, antivirus, and antispam

ID	Test	R1	R2	R3
MAILAUTH-01	SPF pass/fail/softfail/neutral/none/temp/permerror
MAILAUTH-02	Incoming DKIM valid/invalid/multiple/modified/DNS error
MAILAUTH-03	DMARC alignment and none/quarantine/reject/temp policies
MAILAUTH-04	Forged Authentication-Results removed; local result added
MAILAUTH-05	Temporary key signs outbound; independent verification passes
MAILAUTH-06	Missing/unreadable/mismatched/rotated DKIM key safety
MAILAUTH-07	SRS encode/decode/tamper/expiry/bounce/loop handling
SCAN-01	ClamAV clean and EICAR paths
SCAN-02	SpamAssassin ham/spam and score/header handling
SCAN-03	Scanner unavailable/timeout/malformed/restart policy
SCAN-04	Official/KAM/Heinlein channels, signatures, lint, timer
SCAN-05	Worker detects ClamAV/SA data faults without SA-age false alarm
SCAN-06	Filters, mailing-list detection, folders, vacation, loop prevention
SCAN-07	Quarantine/task creation, ignore, retry, delete, audit

Web, DAV, and external accounts

ID	Test	R1	R2	R3
WEB-01	Classic login/assets render in current Firefox/Chromium
WEB-02	Login/logout/session/CSRF/cookies/brute-force limiting
WEB-03	Trusted proxy/header chain resists client-address spoofing
WEB-04	Folders/read/compose/reply/forward/attachments/move/delete
WEB-05	HTML sanitisation keeps safe images and blocks active tracking
WEB-06	Identities, external senders, signatures, SMTP selection
WEB-07	Filters/tasks/quarantine/external accounts/app passwords/2FA
WEB-08	English, German, and every installed gettext catalogue load
DAV-01	CalDAV discovery, CRUD, sync, ETag, and iCalendar
DAV-02	CardDAV discovery, CRUD, sync, ETag, and vCard
DAV-03	DAV master/app-password scope and TLS/proxy policy
EXT-01	Provider presets and manual configuration validation
EXT-02	POP3 collection keep/delete and INBOX/custom target
EXT-03	IMAP collection keep/delete and INBOX/custom target
EXT-04	Multiple accounts and identities stay isolated per user
EXT-05	Matching external SMTP identity from Web and SMTP client
EXT-06	Collector TLS/failure/retry/deduplication/secret-safe logs

ACME, administration, observability, and security

ID	Test	R1	R2	R3
ACME-01	HTTP-01 order/challenge/renew/deploy/agent reload
ACME-02	DNS-01 nsupdate with TSIG and cleanup
ACME-03	DNS-01 Cloudflare and generic provider contract
ACME-04	Multi-domain/wildcard, CA retry limits, failure/rollback
ADM-01	Admin/config/queue/store tools, TUI, help, man pages agree
ADM-02	User/domain/alias/password/rename/config/scanner validation
ADM-03	Structured Web/SMTP/IMAP/POP3/Sieve auth-failure logs
ADM-04	Fail2ban matches failures and not successful logins
SEC-01	No secrets in argv, diagnostics, normal logs, Web responses
SEC-02	Safe modes for auth/TLS/DKIM/SRS/ACME/provider secrets
SEC-03	Secure TLS default; explicit legacy mode isolated/logged
SEC-04	Malformed corpus, limits, traversal, injection, smuggling
SEC-05	Connection/auth/message rate limits resist floods
SEC-06	Restarts during pipeline cause no loss or duplicates
SEC-07	Dependency/CVE/source review has no 0.7 blockers

Required integration fixtures

• two disposable local users and at least two hosted/alias domains;
• local DNS able to publish temporary SPF, DKIM, and DMARC records;
• temporary GnuTLS certificates and a temporary OpenDKIM signing key;
• ClamAV, SpamAssassin, EICAR, and deterministic ham/spam mail;
• Xeams DevNull SMTP for captured outbound relay messages;
• a status-aware LMTP daemon which records envelope/message and can return per-recipient 2xx, 4xx, and 5xx responses;
• vtestmail for external POP3/IMAP collection tests;
• disposable LDAP/AD-compatible and ACME/DNS challenge fixtures;
• old and current client probes, including a basic legacy IMAP4 path.

DevNull and LMTP captures are checked for exact envelope, headers, body, DKIM signature, SRS address, client metadata, and duplicate or missing deliveries. Merely receiving a TCP connection is not a pass.