Mario Fetka
106 lines
5.3 KiB
Markdown
106 lines
5.3 KiB
Markdown
# Storage and backup
|
|
|
|
Bongo does not put every message in one global SQLite database. The Store
|
|
uses a per-user layout so a large account does not lock or enlarge the
|
|
metadata database of every other account.
|
|
|
|
## Store layout
|
|
|
|
Each account below `/var/lib/bongo/users` has its own `store.db` SQLite
|
|
metadata database and Maildir-like collection directories containing the
|
|
document payloads. Mail, calendar objects, and contacts use the Store's common
|
|
object and collection model. The SQLite database tracks collections, UIDs,
|
|
flags, properties, and indexes; message bodies remain separate files.
|
|
|
|
On first access a new account receives INBOX, Drafts, Archive, Sent, Trash and
|
|
Junk mail collections with their standard IMAP SPECIAL-USE roles, plus a
|
|
personal calendar and personal and collected address books. These objects and
|
|
their payload directories belong only to that account's Store.
|
|
|
|
Optional per-user mail quotas are stored and enforced in each Store database.
|
|
They count logical mail-message sizes across all mailboxes, but not calendar,
|
|
contact, configuration, or authentication data. See
|
|
[Mail storage quotas](quotas.md) for administration and protocol behaviour.
|
|
|
|
The NMAP system account is `/var/lib/bongo/users/_system/store.db`. It contains
|
|
server configuration documents and new server-wide tables such as TLSRPT
|
|
events and deliveries and the worker scheduler. The historical
|
|
`/var/lib/bongo/system` directory is a different facility and still contains
|
|
the original alarm database. The Queue spool below `/var/lib/bongo/spool`
|
|
contains messages which have been accepted but not yet fully delivered.
|
|
|
|
Not every separate SQLite database is accidental. The following layouts
|
|
predate the 0.7 work and remain supported:
|
|
|
|
- `/var/lib/bongo/dbf/userdb.sqlite` stores local authentication records,
|
|
Argon2id password hashes, encrypted TOTP secrets, recovery codes, app
|
|
passwords, login aliases, and LDAP cache metadata.
|
|
- `/var/lib/bongo/dbf/cookies.db` stores the original Store authentication
|
|
cookies.
|
|
- `/var/lib/bongo/system/alarms.db` stores the original calendar alarm state.
|
|
|
|
These databases are not migration targets merely because they are separate.
|
|
The per-user Store, authentication, cookie, alarm, and historical MsgAPI SQL
|
|
designs were present in the archived Bongo history.
|
|
|
|
TLSRPT, the worker scheduler, Sieve state, and external-account state were
|
|
initially implemented with standalone databases during 0.7 development. They
|
|
now live behind `bongostore` and NMAP. Those development layouts were never
|
|
released, so Bongo neither opens nor migrates their former database paths.
|
|
|
|
Sieve scripts and vacation state now belong to the Store of the user who owns
|
|
them; `bongosieve` and `bongorules` access them only through NMAP. External
|
|
account definitions, encrypted credentials, destination
|
|
folder, deletion policy, linked sending identity, remote UID state, and
|
|
duplicate-detection state likewise belong to that user's Store. The `_system`
|
|
Store holds only a credential-free due index containing the owner, account ID,
|
|
and next collection time, so `bongocollector` need not scan every account.
|
|
|
|
Web tasks, sender identities, and signatures are ordinary user Store
|
|
documents below `/preferences`. Login sessions exist only in `bongoweb`
|
|
memory and are invalidated whenever that service restarts.
|
|
|
|
SQLite write-ahead-log and sidecar files are part of a live database state;
|
|
copying only the main `.db` file while Bongo is writing is not a reliable
|
|
backup.
|
|
|
|
## Backup scope
|
|
|
|
A complete backup includes at least:
|
|
|
|
- `/etc/bongo`;
|
|
- all of `/var/lib/bongo`, including Store payloads, metadata, Queue spool,
|
|
account databases and user preference objects;
|
|
- TLS, DKIM, SRS, external-account encryption, TOTP encryption, and ACME hook
|
|
keys wherever the configuration points to them;
|
|
- distribution service overrides and reverse-proxy configuration needed to
|
|
reproduce the deployment.
|
|
|
|
For a simple consistent filesystem backup, stop Bongo, confirm that its
|
|
agents have exited, snapshot or copy the entire state and configuration tree,
|
|
then restart it. A storage snapshot is acceptable only when the filesystem
|
|
and SQLite state are captured atomically. Online backup tooling must use the
|
|
SQLite backup API or checkpoint/locking rules and coordinate Store payload
|
|
creation; a file-by-file copy is insufficient.
|
|
|
|
`bongo-storetool` can import and export mailbox data in mbox or Maildir form.
|
|
That is useful for migration and recovery of mail content, but it is not a
|
|
replacement for a complete server backup because it does not preserve every
|
|
account, key, configuration, calendar, contact, task, and protocol state.
|
|
|
|
## Restore testing
|
|
|
|
Restore into an isolated host with the same or a newer compatible Bongo build.
|
|
Check database integrity, ownership and permissions, then validate the
|
|
configuration before starting the agents. Test login, message counts, flags,
|
|
folder hierarchy, calendar/contact access, queued delivery, app passwords,
|
|
and a signed outbound message.
|
|
|
|
Large-store and migration benchmarks are part of the 1.0 release work. The
|
|
per-user metadata design avoids a single database bottleneck, but it does not
|
|
remove the need to test filesystem inode usage, indexes, backup duration,
|
|
search performance, and recovery with realistically large mailboxes.
|
|
|
|
The future JMAP-capable generation and its restartable conversion procedure
|
|
are specified in [JMAP and the Bongo Store](jmap.md).
|