bongo/docs/storage.md

Storage and backup

Bongo does not put every message in one global SQLite database. The Store uses a per-user layout so a large account does not lock or enlarge the metadata database of every other account.

Store layout

Each account below /var/lib/bongo/users has its own store.db SQLite metadata database and Maildir-like collection directories containing the document payloads. Mail, calendar objects, and contacts use the Store's common object and collection model. The SQLite database tracks collections, UIDs, flags, properties, and indexes; message bodies remain separate files.

On first access a new account receives INBOX, Drafts, Archive, Sent, Trash and Junk mail collections with their standard IMAP SPECIAL-USE roles, plus a personal calendar and personal and collected address books. These objects and their payload directories belong only to that account's Store.

Optional per-user mail quotas are stored and enforced in each Store database. They count logical mail-message sizes across all mailboxes, but not calendar, contact, configuration, or authentication data. See Mail storage quotas for administration and protocol behaviour.

The NMAP system account is /var/lib/bongo/users/_system/store.db. It contains server configuration documents and new server-wide tables such as TLSRPT events and deliveries and the worker scheduler. The historical /var/lib/bongo/system directory is a different facility and still contains the original alarm database. The Queue spool below /var/lib/bongo/spool contains messages which have been accepted but not yet fully delivered.

Not every separate SQLite database is accidental. The following layouts predate the 0.7 work and remain supported:

• /var/lib/bongo/dbf/userdb.sqlite stores local authentication records, Argon2id password hashes, encrypted TOTP secrets, recovery codes, app passwords, login aliases, and LDAP cache metadata.
• /var/lib/bongo/dbf/cookies.db stores the original Store authentication cookies.
• /var/lib/bongo/system/alarms.db stores the original calendar alarm state.

These databases are not migration targets merely because they are separate. The per-user Store, authentication, cookie, alarm, and historical MsgAPI SQL designs were present in the archived Bongo history.

By contrast, standalone databases first added by the 0.7 development work must be moved behind bongostore and its NMAP interface:

• /var/lib/bongo/dbf/sieve.sqlite;
• /var/lib/bongo/dbf/external-accounts.sqlite;

TLSRPT and the worker scheduler originally joined that list during development, but have already moved into the _system Store and no longer own separate databases. Because these paths existed only in unreleased 0.7 development builds, no production migration is required.

Sieve scripts and vacation state belong to the Store of the user who owns them. External account definitions, encrypted credentials, destination folder, deletion policy, linked sending identity, remote UID state, and duplicate-detection state likewise belong to that user's Store. The _system Store holds only a credential-free due index containing the owner, account ID, and next collection time, so bongocollector need not scan every account. The two remaining standalone paths are transitional until their Store protocol migrations have landed.

Web tasks, sender identities, and signatures are ordinary user Store documents below /preferences. Login sessions exist only in bongoweb memory and are invalidated whenever that service restarts.

SQLite write-ahead-log and sidecar files are part of a live database state; copying only the main .db file while Bongo is writing is not a reliable backup.

Backup scope

A complete backup includes at least:

• /etc/bongo;
• all of /var/lib/bongo, including Store payloads, metadata, Queue spool, account databases and user preference objects;
• TLS, DKIM, SRS, external-account encryption, TOTP encryption, and ACME hook keys wherever the configuration points to them;
• distribution service overrides and reverse-proxy configuration needed to reproduce the deployment.

For a simple consistent filesystem backup, stop Bongo, confirm that its agents have exited, snapshot or copy the entire state and configuration tree, then restart it. A storage snapshot is acceptable only when the filesystem and SQLite state are captured atomically. Online backup tooling must use the SQLite backup API or checkpoint/locking rules and coordinate Store payload creation; a file-by-file copy is insufficient.

bongo-storetool can import and export mailbox data in mbox or Maildir form. That is useful for migration and recovery of mail content, but it is not a replacement for a complete server backup because it does not preserve every account, key, configuration, calendar, contact, task, and protocol state.

Restore testing

Restore into an isolated host with the same or a newer compatible Bongo build. Check database integrity, ownership and permissions, then validate the configuration before starting the agents. Test login, message counts, flags, folder hierarchy, calendar/contact access, queued delivery, app passwords, and a signed outbound message.

Large-store and migration benchmarks are part of the 1.0 release work. The per-user metadata design avoids a single database bottleneck, but it does not remove the need to test filesystem inode usage, indexes, backup duration, search performance, and recovery with realistically large mailboxes.

The future JMAP-capable generation and its restartable conversion procedure are specified in JMAP and the Bongo Store.