Upstream mail-server test suites

Bongo uses current Cyrus IMAP, Dovecot, and Postfix sources as protocol comparison points. Their tests are valuable, but they are not interchangeable and must not replace Bongo's own Store, Queue, authentication, scanner, and security tests.

Dovecot

The Dovecot source tree contains broad unit coverage for its internal IMAP, POP3, SMTP, MIME, parser, sequence-set, date, envelope, body-structure, and UTF-7 implementations. Those tests link to Dovecot internals and cannot test a Bongo process directly. Their input and expected-output vectors may be independently expressed in Bongo regression tests when the corresponding protocol feature is implemented.

The separate Dovecot ImapTest project is a generic black-box server tester. It provides:

scripted IMAP compliance cases;
state-tracked, concurrent and randomized IMAP sessions;
deterministic seeds and raw protocol logs;
POP3 client profiles;
benchmarking and optional LMTP-generated deliveries.

contrib/testing/upstream-imaptest-check.py runs it against Bongo without installing files or exposing the test password in the process command line. The scripted mode stages only upstream test data in a temporary directory. The stress mode uses a unique disposable mailbox, deterministic random seed, state checkpoints, message ownership, and flag ownership.

Build ImapTest against a completed Dovecot build as described by its upstream build documentation, then run:

export BONGO_IMAPTEST_BIN=/tmp/dovecot-imaptest/src/imaptest
export BONGO_IMAPTEST_SOURCE=/tmp/dovecot-imaptest
export BONGO_TEST_PASSWORD='a-disposable-password-of-at-least-12-characters'

./contrib/testing/upstream-imaptest-check.py scripted --suite core
./contrib/testing/upstream-imaptest-check.py scripted --suite search
./contrib/testing/upstream-imaptest-check.py stress --imap4rev2 \
  --clients 4 --seconds 30 --seed 700
./contrib/testing/upstream-imaptest-check.py pop --seconds 30

The complete upstream suite can be selected with --suite all. Missing optional capabilities are reported as skips by ImapTest; a failure in the base protocol or an advertised extension fails the Bongo test. For diagnosis, --test-group append selects one named upstream file and may be repeated. The adapter does not enable ImapTest's raw logging because a raw IMAP trace can contain the decoded test password; the captured test summary is the release artifact.

The upstream scripted corpus uses IMAP4rev1 response rules, including RECENT. It must not be combined with --imap4rev2. Bongo's dedicated live tests verify the revision boundary, while ImapTest's randomized stress mode can exercise an enabled IMAP4rev2 session.

Cyrus IMAP

Cyrus keeps its Cassandane integration suite inside the current Cyrus source tree. Cassandane provisions and controls a Cyrus instance and relies on Cyrus administrative commands, configuration, storage layout, and test-only utilities. It therefore cannot be pointed wholesale at Bongo.

The protocol-facing Cassandane cases remain a required comparison source. Bongo tests independently reproduce applicable cases from these groups:

Append, Create, Delete, Expunge, Fetch, Flags, List, Move, Rename, Search, Simple, Subscriptions, and UIDonly;
IMAP4rev2, IMAPLimits, StartTLS, UTF8Accept, Quota, and Pop3;
Sieve when the ManageSieve rows of the release matrix are reached.

Cyrus-specific replication, database, administration, JMAP, annotation, Murder, and search-backend cases are not claimed as Bongo coverage unless the matching Bongo feature exists.

Postfix

Postfix Ptest and most make tests targets exercise Postfix internal APIs, mocks, parsers, maps, policy code, and queue formats. They are reference vectors for Bongo's independently implemented SMTP tests, not external test programs.

Two Postfix smtpstone tools are directly reusable:

smtp-source is a parallel SMTP/LMTP message generator;
smtp-sink is a parallel SMTP/LMTP receiver which can capture transactions and inject delays, command failures, premature DATA failures, and capability variations.

They complement, but do not replace, Bongo's exact SMTP tests. In particular, they are suitable for load, retry, disconnect, LMTP, command-failure, and outbound-client interoperability rows. Protocol correctness, open-relay policy, Queue durability, DKIM/SRS, DANE, MTA-STS, REQUIRETLS, TLSRPT, proxy metadata, and Store delivery remain assertions made by Bongo's own fixtures.

Additional independent implementations

The release matrix also uses these independently maintained tools where they add a different protocol implementation rather than duplicating a Bongo fixture:

GreenMail standalone provides a sandboxed SMTP, SMTPS, IMAP, IMAPS, POP3, and POP3S peer. It is a second provider implementation for Collector and outbound interoperability after the Cyrus and smtp4dev fixtures.
caldav-tester exercises CalDAV and CardDAV discovery and operations against a live server. litmus independently checks the WebDAV base methods on which both protocols depend.
Dovecot Pigeonhole's Sieve test corpus supplies language and action vectors. Its sieve-test binary tests Pigeonhole's engine, so Bongo re-expresses applicable vectors in its own engine tests. Cyrus Cassandane remains the external comparison for ManageSieve protocol behavior.
GNU SASL's client and Swaks remain independent authentication and SMTP clients. OpenDKIM, libspf2, OpenDMARC, and libsrs2 verification is performed on captured messages and controlled DNS data, without sending test mail to a public service.

Adapters must keep all delivery local, create disposable users and mailboxes, avoid passwords in process arguments and logs, and restore configuration after both success and failure.

Version and evidence rules

Every upstream-assisted release run records:

the Bongo commit and installed package build;
the upstream project, version, and commit;
the exact Bongo adapter command and deterministic seed where applicable;
the resulting raw log or summary;
which advertised capability or base-protocol behavior failed.

Upstream source is not copied into the Bongo tree. Test vectors are reimplemented where needed, with the upstream file and version recorded in the release evidence.

6.2 KiB Raw Blame History