diff --git a/AI.md b/AI.md index 2b8c3ad..f6a99c1 100644 --- a/AI.md +++ b/AI.md @@ -414,6 +414,108 @@ Desired future server log format: - Do not invent a parallel logger casually. Reuse existing mars_nwe logging functions/macros and normalize message format gradually. + +## 2026-06-05 - current superbuild / storage / TLS handoff + +This chat switched from documentation-only endpoint audit work back to build and +integration work around the mars-nwe superbuild. The old patch-number guidance +below remains useful historical context for the endpoint-audit series, but the +current accepted work in this chat is a separate functional/build patch stack. +Do not assume the next patch after this point is an endpoint-audit `0269` patch +unless the user explicitly returns to that series. + +Current root/superbuild direction implemented or in progress: + +- `update-submodules.sh` is now the normal helper for keeping mars-nwe's + submodules reproducible. Private mars-owned submodules are updated to their + configured latest branch, while external upstream snapshots are pinned by a + single editable `EXTERNAL_TAG_PINS` block near the top of the script. +- Current external pins are `third_party/yyjson=0.12.0`, + `third_party/zlog=1.2.18`, and + `third_party/libsodium/libsodium=1.0.20-FINAL`. +- The helper must not run a root-level `git submodule update --init --recursive` + after updating top-level private submodules, because that can reset updated + gitlinks such as `third_party/matrixssl` back to the parent commit. Nested + submodules are initialized inside the owning top-level submodule instead. +- The helper prints recursive submodule status on success and failure and can + auto-commit root gitlink/script changes. Root gitlinks must only reference + commits that exist in the respective submodule remotes; a parent gitlink to an + unpushed submodule commit will make later `git submodule update` fail with + `upload-pack: not our ref`. +- `yyjson` is being folded into the mars-nwe core library path. Consumers should + include it through the `nwcore` include namespace and link `mars_nwe::core`, + not link an independent public yyjson target directly. +- FLAIM is currently only needed when the directory service is enabled. The root + CMake should therefore add/build `third_party/flaim` only when + `ENABLE_DIRECTORY=ON`. + +Current `libnwssl` / MatrixSSL / OpenSSL-compat direction: + +- MatrixSSL owns only the renamed low-level backend library (`libnwmatrixssl`). + Temporary OpenSSL-compat test headers/sources that were placed in the + MatrixSSL fork must move out of MatrixSSL. +- `libnwssl` owns the mars-nwe crypto/TLS facade plus the narrow compatibility + surfaces needed by FLAIM: OpenSSL-style FTK/network headers and NICI/CCS-style + private headers. +- OpenSSL-compat headers should live under the `nwssl` include subtree, e.g. + `include/nwssl/openssl/*.h`, not as a root-level `include/openssl` directory + that could conflict with system OpenSSL headers. +- Private FLAIM/NICI compatibility headers should also live under the `nwssl` + subtree, e.g. `include/nwssl/private/nici/...`. They are private build + compatibility headers, not the public TLS API for normal mars-nwe code. +- `smart`, `directory`, and FLAIM should link against `libnwssl` / the + `mars_nwe::ssl` target when they need TLS/crypto compatibility. They should + not include MatrixSSL headers directly and should not link OpenSSL directly. + +Current FLAIM import/build direction: + +- The imported FLAIM tree is kept under `third_party/flaim` and should remain as + close to upstream source as practical. Prefer CMake/build glue, include paths, + and tiny compile fixes over broad C/C++ rewrites. +- The mars-nwe build currently wants `libnwflaimtk`, `libnwflaim`, + `libnwflaimsql`, and `libnwxflaim`. XFLAIM is now included even though it is + not immediately required, so it stays build-covered. +- FLAIM library versions should come from the public headers when those disagree + with `configure.ac`, because the headers are the ABI-facing version source in + this import. Current expected shared-object versions are: + `libnwflaimtk.so.1.2`, `libnwflaim.so.4.62`, + `libnwflaimsql.so.6.00`, and `libnwxflaim.so.5.12`. +- All installed FLAIM public headers should go below one mars-nwe namespace + directory: `include/nwflaim/`. Do not make `xflaim.h` a special + `include/nwxflaim/` exception. +- FLAIM command-line utilities should be built and installed with `nw`-prefixed + binary names, for example `nwflmcheckdb`, `nwflmrebuild`, `nwflmview`, + `nwflmdbshell`, `nwflmgigatest`, and the matching `nwxflm...` utilities. +- FLAIM tools require curses/ncurses. CMake should report clearly whether + curses/ncurses was found and whether the curses-backed tools will be built. +- The uploaded `ncurses-stable.tar.gz` is a valid current upstream ncurses/stable + source snapshot for local build testing even if the top-level extracted name + does not look like a conventional ncurses release tarball. +- The CMake conversion should keep translating the old `Makefile.am` source + inventories. Recent build fixes added missing result-set sources and fixed + XFLAIM tool include ordering so `xflaim/util` code includes the XFLAIM + `flaimsys.h` rather than the classic FLAIM one. +- Minimal FLAIM C++ compile fixes are acceptable when required by modern + compilers, for example replacing pointer assignments/comparisons using + character `\0` with `NULL`/null-pointer checks. Keep those patches small and + separate from build-system changes where possible. + +Current local dependency policy for build checks: + +- Build GDBM locally from the uploaded `gdbm-1.26.tar.gz` when testing in an + isolated prefix. +- Build ncurses locally from the uploaded `ncurses-stable.tar.gz` when testing + FLAIM tools in an isolated prefix. +- Use the uploaded `Linux-PAM-1.7.2.tar.xz` for PAM headers if needed, but link + against the system PAM library. Do not vendor PAM as a mars-nwe library. +- These local dependency builds are for verification; they are not new vendored + submodules unless a later explicit import decision says otherwise. + +When continuing this work, expect more incremental compile/build patches rather +than one large redesign patch. Build after each FLAIM/CMake change, record the +next concrete compiler/linker error, and keep generated patches per repository +or submodule so the user can apply them with `git am` at the correct path. + ## Build and test notes Dependencies used during local checks in this conversation: diff --git a/REDESIGN.md b/REDESIGN.md index 7844fd1..29efd81 100644 --- a/REDESIGN.md +++ b/REDESIGN.md @@ -1557,6 +1557,63 @@ mars-tinyldap/ later wired to libdirectory/libflaim instead of tinyldap's original flat files ``` + +### Current superbuild integration status + +The current implementation work has moved several items from planning into an +initial mars-nwe superbuild shape. Keep this status separate from the older +endpoint-audit patch-number notes: these are functional/build integration facts, +not a promise that every compatibility layer is complete. + +Current source layout decisions: + +- `third_party/yyjson` and `third_party/zlog` remain external upstream snapshots + pinned by release tag in `update-submodules.sh`. +- `third_party/yyjson` is being compiled into `libnwcore`; consumers should use + the `nwcore` include namespace and link the core target instead of exposing a + standalone yyjson API as a mars-nwe public dependency. +- `third_party/libsodium/libsodium` remains a nested external upstream snapshot + pinned to `1.0.20-FINAL` inside the mars-libsodium wrapper submodule. +- `third_party/matrixssl` is now the mars-maintained MatrixSSL fork producing + the renamed backend library `libnwmatrixssl`. It should not contain the + temporary OpenSSL-compat shim. +- `libnwssl` in the mars-nwe root owns the SSL/crypto facade plus FLAIM + compatibility headers. Its compatibility header layout should stay under the + `nwssl` include subtree, for example `include/nwssl/openssl/*.h` and + `include/nwssl/private/nici/...`. +- `third_party/flaim` is the current FLAIM import path used by the working tree. + It provides renamed mars-nwe libraries and tools; future prose may still refer + to `libflaim` as the logical storage engine, but the concrete submodule path + is `third_party/flaim` unless the user explicitly renames it. +- `third_party/flaim` is currently gated by `ENABLE_DIRECTORY`. The normal build + should not configure or build FLAIM when the directory service is disabled. + +Current FLAIM CMake import decisions: + +- Build `libnwflaimtk`, `libnwflaim`, `libnwflaimsql`, and `libnwxflaim` with + mars-nwe names so they do not collide with any system FLAIM installation. +- Build and install the FLAIM/XFLAIM utilities with `nw`-prefixed executable + names such as `nwflmcheckdb` and `nwxflmcheckdb` when tools are enabled. +- Use the ABI-facing version values from the public headers when they disagree + with `configure.ac`: `libnwflaimtk.so.1.2`, `libnwflaim.so.4.62`, + `libnwflaimsql.so.6.00`, and `libnwxflaim.so.5.12`. +- Install all FLAIM public headers under one namespace directory, + `include/nwflaim/`, including `xflaim.h`. Do not install a separate + `include/nwxflaim/` tree. +- Keep CMake messages explicit about curses/ncurses detection so it is obvious + whether curses-backed FLAIM tools will be built. +- Continue to prefer build glue and include-path fixes over invasive FLAIM source + edits. Small modern-compiler fixes are acceptable when necessary to compile, + but keep them as small, reviewable patches. + +Current local dependency-test policy: + +- GDBM and ncurses may be built locally from the uploaded release tarballs for + verification and passed to CMake through an isolated prefix. +- PAM may use headers from the uploaded Linux-PAM tarball for compilation checks, + but mars-nwe should link to the system PAM library rather than vendoring PAM. +- These local builds are test dependencies, not new third-party submodules. + ### libowfat dependency rule `libowfat` should be a hard bundled dependency, initially for the