mars-nwe/AI.md

# AI working notes for mars-nwe

This file is the handoff and assistant-rule document.  It should keep the next
chat on the current track without forcing it to reconstruct old patch history.
Keep it short enough to read before touching code.

## Start of a new chat

1. Treat `mars-nwe-master` as the root/superproject.  The other `mars-*` bundles
   are submodules or sibling component sources needed by the root build.
2. Read the root `*.md` files and `doc/*.md` before changing namespace, salvage,
   transport or dependency code.  The most important active files are
   `TODO.md`, `REDESIGN.md`, `ENDPOINTS.md`, this file, and:
   - `doc/NSS_NAMESPACE_AUDIT.md`
   - `doc/NSS_PUBLIC_CORE_AUDIT.md`
   - `doc/NWFS_SALVAGE_COMPRESSION_TOOLS.md`
   - `doc/HANDOFF_AUDIT.md`
   - `doc/TUI_TOOLBOX_PLAN.md`
3. Do not assume an older bundle is current.  Use the latest applied user state
   or the latest bundle/patch explicitly accepted by the user.
4. For a fresh build in a new chat, check whether `prepare-local-deps.sh` is
   present in the root and run/use it before trying to build; it stages the
   uploaded offline third-party tarballs and local-only headers needed by the
   current dependency layout.
5. Default language with the user is German; repository documentation remains
   English unless the existing file is German.

## Documentation ownership

- `TODO.md` is the active backlog.  The top implementation dashboard is allowed
  and should stay.  Unfinished long-running tracks stay in `TODO.md` until real
  implementation/test work closes them.
- `REDESIGN.md` is the durable architecture/design record.  Put stable plans,
  rationale and completed design decisions there, not patch chronology.
- `AI.md` is for working rules, current handoff state, rejected patches and next
  action guidance.  Keep it concise; do not append duplicate historic handoff
  dumps forever.
- `ENDPOINTS.md` is the decimal/hex NCP audit table.
- `doc/*.md` files are focused topic audits/roadmaps.  Keep namespace, NSS
  public-core and salvage/compression/tool details in their matching doc files.

When a TODO is finished, remove it from `TODO.md` in the same patch and record
its stable outcome in `REDESIGN.md` or the focused `doc/*.md` file.  Do not move
unfinished work out of `TODO.md` merely because its architecture is documented.

## Current accepted patch line

Latest patch marker expected in an up-to-date bundle:

- `0554 nwnss: import COMN connection structural headers`

When a later chat receives a new `mars-nwe-master` bundle, compare `git log -1`
with this marker.  If the uploaded bundle already contains this commit subject,
the last patch was applied.  If the bundle is older and the old chat download no
longer works, recreate the missing patch instead of assuming it was applied.

The active line is expected to include:

- Quota work through the corrected Linuxquota/NWQUOTA split and DOS/live smokes.
- NSS low-level imports through the Unicode/codepage, UTC, GUID/ID, xCtype and
  xString helper line.
- `0434 build: record offline dependency bootstrap layout`.
- `0435 docs: record transport config and nwfs watch plan`.
- `0436 docs: keep unfinished NSS and salvage work in TODO`.

## Current patch handoff block

Last generated patch:

- `0554 nwnss: import COMN connection structural headers`

Purpose of that patch:

- Reject the previous shortcut direction for namespace registration: do not
  disable the AdminVolume/Beast path just to make `nameSpace.c` build.
- Import the next real COMN connection structure headers used by the
  AdminVolume/Beast path from `shared/sdk/comnSA` and `shared/sdk/include`.
- Keep this as a header-only bottom-up step before importing
  `public_core/comn/common/beastClass.c`, `pssConnection.c`, `adminVolume.c` or
  `public_core/comn/namespace/nameSpace.c`.
- Add only a narrow Linux-version macro compatibility fix around the imported
  `fileHandle.h` connection-key conditional; no NSS semantics are replaced.

## 0490 core: import NSS stdio formatter runtime

Imported the real NSS bounded formatter before parse/FSM/latch code that uses
`LB_snprintf` and friends:

- `public_core/library/stdio/snprintf.c` -> `src/core/snprintf.c`
- `public_core/library/stdio/snprintf.h` -> `include/core/snprintf.h`

The Linux userspace port keeps the NSS formatter body and only adapts platform
includes, Unicode declarations, and the formatter-local stack allocation path.
Do not replace this with libc `snprintf`; later stdio/wio imports should build
on this real NSS formatter.



Directory/NDS work order before any FLAIM storage conversion:

- Do not combine LDAPv2/LDAPv3 compatibility work with the FLAIM storage-engine
  swap.  First keep TinyLDAP/nwdirectory on its existing flatfile/mmap/journal
  backend and add behaviour tests.
- Prefer existing TinyLDAP test/client code for LDAP CTests.  LDAP tests should
  start `nwdirectory` on `127.0.0.1:<free-port>` with a temporary flatfile
  workdir and test LDAPv2/LDAPv3 bind/search/unbind against test LDIF/schema
  data.
- Add `libnwds` and console-only `nwsetup` together after LDAPv2/schema flatfile
  behavior is testable.  Do not make the first setup path depend on the TUI or
  on acting as an LDAP client.
- Later, when the flatfile LDAP/setup path is stable, add FLAIM storage as
  parallel backend source files selected by CMake.  For example,
  `flaimstorage_add_bin.c` may provide the same exported `mstorage_add_bin()`
  function as the flatfile file when the FLAIM backend is selected.  Do not break
  upstreamable flatfile TinyLDAP code while adding the mars-nwe FLAIM backend.


Rejected or superseded patches that must not be reused as-is:

- `0380`
- `0398`
- `0403`

## Current immediate direction

Before starting namespace implementation, keep the documentation boundaries
clean:

- `TODO.md`: active work only, dashboard retained.
- `REDESIGN.md`: architecture and design separation clarified.
- `AI.md`: concise current state and rules only.

Before namespace code changes, keep the legacy logging audit in mind: old MARS
accepts numeric debug thresholds from `0` to `99`, but the actual source mostly
uses `1..5`.  The future `nwlog` facade should give those normal levels clear
semantic names instead of preserving the old overloaded buckets: `1=error`,
`2=warn`, `3=info`, `4=debug` and `5=trace`.  `debug` is local diagnostic detail; `trace` is packet/message/handoff path following across process or provider boundaries.  New config should expose the cumulative numeric masks `0`, `1`, `12`, `123`, `1234` and `12345` as synonyms for `off`, `error`, `warn`, `info`, `debug` and `trace`, and use one global `[logging] level` plus optional `[logging.process.<name>] level` overrides; the old numeric `100..106` process entries stay accepted for compatibility.  Useful legacy `6..99` traces should
map to `nwlog_detail()` later.  `nwlog_detail()` is not part of the normal
`1..5` threshold ladder: normal builds return `0`, while `MAINTAINER_BUILD`
builds may emit detail whenever such a call site is reached.  No INI option may
enable it in a normal build.
`nwlog` is the facade/backend plan.  Do not add zlog calls or reintroduce a
zlog build dependency; zlog is Apache-2.0 and is not compatible with the
repository's GPL-2.0-only core policy.  See `include/nwlog.h` and
`doc/LOG_LEVEL_AUDIT.md`.

Before namespace code changes, keep the terminal-tooling decision in mind when
touching FLAIM tests or admin tools: do not add new direct `curses.h` users.
Future interactive tools should go through the planned `nwtui`/`nwi18n` stack
and the multi-call `nwtoolbox` applet model documented in
`doc/TUI_TOOLBOX_PLAN.md`.  Configuration parsing/editing is separate: use the
planned shared `libnwcore` `nw_ini_*` reader/writer from
`doc/NWCORE_INI_PLAN.md` so server code and tools share one policy.  Production
daemons must not depend on the TUI stack.

Before namespace code changes, use the handoff audit rules in `doc/HANDOFF_AUDIT.md`.
New namespace/NWFS work must not add new magic `return(-1)` / `return(-2)` paths.
Use the provider names in `include/ncp_endpoint.h` for audits and future endpoint
tables, but do not wire the enum into runtime dispatch until the current
`nwconn` to `nwbind` handoff sites are annotated.

Then continue the NSS namespace track inside `libnwnss`:

1. `libnwnss` is the single userspace library for imported NSS functions and
   semantics.  Do not create a separate namespace/AdminVol NSS library.  The
   library should grow from the existing imported runtime into the real NSS
   components needed by Mars NWE: namespace, trustee/effective-rights helpers,
   XATTR/EA/metadata helpers and later AdminVol semantics.
2. Work bottom-up from the original NSS Linux sources.  Use the files in
   `nss.tar.bz2` and `nss-common.tar.bz2`; inspect NSS Makefiles/Kbuild-style
   source lists as well as includes.  If a compile or link error reports an
   unresolved NSS symbol, search for and import the matching real NSS `.c` and
   header before considering anything else.
3. Do not write semantic wrappers or local replacement models for NSS namespace,
   XATTR, trustee or AdminVol logic.  The only acceptable wrappers/adaptations
   are narrow userspace ports for kernel/OS boundaries such as spinlocks,
   allocation, wait/schedule hooks, WIO/console output and similar platform
   plumbing.
4. For namespace, target the real `public_core/comn/namespace` code and its real
   dependencies under `public_core/comn/common`, `public_core/nss/lib`,
   `shared/sdk/include`, `shared/sdk/internal`, `shared/sdk/public` and
   `shared/sdk/comnSA`.  Keep the `src/nwnss` layout aligned with the original
   `public_core` tree, and keep the `include/nwnss` layout aligned with the
   original `shared/sdk` tree.
5. AdminVol is a later target because `nameSpace.c` reaches AdminVol and AdminVol
   reaches Beast/common code.  Before AdminVol cleanup, import the XATTR/EA/
   metadata side so the current modified `nwfs` XATTR chaos can be untangled
   against real NSS structures and functions.
6. `libnwnss` must become a userspace library, not an NSS disk implementation.
   Do not import block-device, VFS or NSS-disk-layout I/O blindly.  If the first
   real disk-I/O dependency appears, stop and ask the user before designing the
   boundary.  The expected path is that NSS functions create/interpret NetWare-
   compatible metadata and `libnwfs` persists it to host XATTRs, sidecar state or
   Mars volume state.
7. `_ADMIN` is expected to become virtual/in-memory semantics in `libnwnss` and
   later a userspace `nwadminvol` process served via IPC from `nwconn`/`nwfs`,
   with the volume-numbering rule `SYS = 0`, `_ADMIN = 1`, further volumes from
   `2`.
8. Keep `src/namspace.c` as NCP glue over the future namespace engine.  Do not
   expand old MARS `namedos`/`nameos2` as the long-term solution; they are
   replacement targets.

## NCP notation rule

Always write NCP groups/selectors with both decimal and wire/code hex when
recording protocol details.  Examples:

- decimal 87 == wire/code `0x57`
- decimal 90/12 == wire/code `0x5a/0x0c`
- decimal 123/70 == wire/code `0x7b/0x46`

Use `ENDPOINTS.md`, the NDK Core Protocols PDF, WebSDK docs and SDK includes as
references.  NetWare 3.x/default compatibility has priority over broad 4.x/NDS
work.

## Patch workflow

- Make small, reviewable patches with one clear subject.
- Inspect current sources before editing; no speculative patching.
- Run at least `git diff --check` before exporting a patch.
- Also run syntax/build/test checks that match the touched files when feasible.
- Export only the `.patch` file by default when producing repository changes.
  Do not create or link a bundle unless the user explicitly asks for one.
- Every repository patch must update this `AI.md` with the new final commit
  subject in the current patch marker before exporting the patch.  If the user
  later uploads a bundle from their repository, the marker makes that bundle
  self-identifying even when old chat downloads have expired or are unavailable.
- Every repository patch must also replace the single `Current patch handoff
  block` in this file with a short description of what the patch was meant to
  do.  Replace the old block; do not append a new one for every patch.
- Always include a copy/paste `git am <patch-name>.patch` command in the user
  response next to the patch download link.  This is part of the handoff
  contract, not an optional nicety.
- Generate patch mail headers with `Mario Fetka <mario.fetka@disconnected-by-peer.at>`
  as the author identity, not an AI/OpenAI identity.
- Do not claim a build/test passed unless it actually ran in the current work
  tree.

## Source/layout rules

- Keep imported NSS sources in `libnwnss`, not `libnwcore`.  `libnwcore` is
  Mars-core only and must not link against `libnwnss` merely because NSS runtime
  code exists.  NSS tests should be named `nwnss.*`; core tests should only cover
  real Mars core helpers such as INI/logging.
- Preserve original NSS API names and original filenames wherever possible.
  Import real NSS files from their provenance paths and keep the new layout
  recognizable: `public_core/library/...` -> `src/nwnss/library/...`,
  `public_core/nss/...` -> `src/nwnss/nss/...`, `public_core/comn/...` ->
  `src/nwnss/comn/...`; `shared/sdk/include/...` ->
  `include/nwnss/include/...`, `shared/sdk/internal/...` ->
  `include/nwnss/internal/...`, `shared/sdk/public/...` ->
  `include/nwnss/public/...`, and `shared/sdk/comnSA/...` ->
  `include/nwnss/comnSA/...`.
- GPL-2.0-only source imports are allowed when the imported source permits GPLv2
  use and the original provenance/license headers are preserved.  For Linux
  kernel reference material, record that mars-nwe selects GPL-2.0-only.
- Clean NSS staging incrementally.  Staging/reference copies under `nwfs` should
  not remain just because they are confusing; keep only active Mars adaptation
  code there.  When a file is needed, import the real NSS source/header into
  `libnwnss` in the provenance-preserving layout.
- Do not import broad NSS/OES subsystems blindly, but also do not replace missing
  NSS symbols with local semantic wrappers.  Use unresolved symbols and original
  Makefile object lists to find the next real bottom-up dependency.  Only
  kernel, VFS, memory-tracking and platform hooks may get small userspace glue.
- `libnwfs` is for the Mars/NWFS bridge and persistence layer: namespace glue,
  file IDs, metadata, streams, EA, salvage, compression, host-side reconcile and
  `_ADMIN` virtual-volume filesystem access.  It should use `libnwnss` functions
  once those semantics are imported, then persist to host XATTRs, sidecar state
  or Mars volume state.
- `libnwbind` is for bindery identity/storage; `libnwnds` is for later
  eDirectory/NDS work.

## Namespace rules

- Do not expand old MARS `namedos`/`nameos2` as the long-term solution.  They are
  replacement targets.
- Import/adapt the NSS namespace engine directly in `libnwnss`, then retire old
  MARS namespace logic after tests cover DOS/LONG behaviour.  Missing namespace
  dependencies must be resolved from the original NSS sources, not by adding
  replacement wrappers.
- Stable namespace state belongs in `netware.metadata`: file ID, parent file ID,
  DOS name, LONG/OS2 name, MAC name, UNIX/backend name, casefold/hash fields and
  namespace flags.
- Existing Linux files created by Samba, rsync or local admin tools must be
  reconciled by libnwfs watcher/scanner work, not by a private side database.
- MAC namespace is a namespace/stream/metadata problem, not a transport problem.
  Resource forks and Finder info belong in streams/metadata later.

## Salvage/recycle rules

- Keep `.recycle` as the Samba-compatible deleted-payload backend.
- Make `netware.metadata` on the recycled payload authoritative for deleted
  state.
- Treat `.salvage` JSON as legacy migration/debug data only; do not make it a
  second long-term authority.
- Samba `vfs_recycle` normally uses rename, so xattrs remain attached to the
  recycled inode.  Manually copied files in `.recycle` without
  `netware.metadata` are not valid NetWare salvage objects unless an explicit
  admin repair tool marks them.
- If a compressed file is recycled, the `.recycle` payload should be a normal
  uncompressed Linux file; keep previous compression state in `netware.metadata`.

## Compression and stream rules

- Compression belongs in `libnwfs`, not `libnwcore`.
- Future private stream/compression data belongs under
  `.nwfs_streams/<stable-file-id>/...`.
- The stable file ID comes from MARS/NWFS/NSS-shaped metadata, not from Linux
  inode numbers and not from visible filenames.
- Do not encode compression state in visible names such as `compressed_*`.

## Transport rules

- TCP/IP is a transport split under `nwconn`/`nwserv`, not a new daemon.
- Planned files: `src/nwtransport.c`, `src/nwipx.c`, later `src/nwtcp.c`.
- NCP providers stay transport-neutral:
  `IPX/TCP -> nwtransport -> nwconn -> dispatcher -> providers`.
- Use audited libowfat API names from the bundled `mars-libowfat` source:
  `socket_tcp4()`, `socket_tcp6()`, `socket_bind4_reuse()`,
  `socket_bind6_reuse()`, `socket_listen()`, nonblocking accept helpers and
  `io_*` readiness helpers.
- IPX config sections use frame tokens without dots:
  `8022`, `8023`, `etherii`, `snap`, `tr8022`, `auto`.
- `[transport.ipx.local]` is the internal network; explicit routes use
  `[transport.ipx.route.<target>]`.
- If IPX is requested but `socket(AF_IPX, ...)` fails, report that AF_IPX is
  unavailable instead of silently falling back.
- A future L2 IPX backend is allowed, but it is an additional backend below
  `nwtransport`, not a replacement for the kernel path.  Start with Ethernet_II
  IPX, SAP/RIP, and NCP socket `0x0451`.
- Do not vendor a capability library for L2 IPX.  Open the packet socket and
  report missing `CAP_NET_RAW`/`CAP_NET_ADMIN`; root, setcap, or systemd
  capabilities are deployment policy.
- Do not copy GPL-3.0-or-later code from the Rust `nwserver` reference.  Use only
  the observed architecture: userland Ethernet/IPX packet handling, SAP/RIP, and
  NCP socket dispatch.
- SPX, if needed, is a later userland layer above IPX.  Do not rely on kernel
  SPX and do not make namespace work wait for SPX.
- Linux 2.4.37.9 SPX is imported only as reference material in
  `src/kernel/af_spx.c` and `include/kernel/spx.h`; do not build it as-is.

## Dependency/source bundle rules

- `mars-nwe-master` is the root repository.
- Uploaded `mars-*` bundles are submodules/sibling components needed in the
  configured locations.
- Offline point-release tarballs are staged by `prepare-local-deps.sh`.  In a
  new chat or clean checkout, use this script as the first dependency/bootstrap
  step before CMake/build attempts:
  - `yyjson-0.12.0` into `third_party/yyjson`
  - `libsodium-1.0.20` into the nested libsodium snapshot path
  - `gdbm-1.26` built locally under `.local-deps/prefix`
  - PAM and ncurses headers staged while linking to system libraries
- Do not vendor random system headers directly into endpoint/provider code.

## New reference inputs available for the next namespace work

The user supplied fresh reference archives for the next phase:

- `nss-common.tar.bz2`
- `nss.tar.bz2`
- `ncp__enu.pdf`
- `websdk.tar.gz`
- `include.tar.gz`

Use these for source audits and exact NCP structure/selector verification before
implementing namespace changes.


0488 core: import NSS mailbox runtime
0489 core: import NSS production debug header base
- Imported shared/sdk/include/mailbox.h as include/core/mailbox.h.
- Imported public_core/library/os/mailbox.c as src/core/mailbox.c.
- Added nwcore.mailbox CTest.

- 0491 core: imported NSS stdio sprintf/vsprintf from public_core/library/stdio as the next formatter prerequisite before parse/wio.

## 0492 core: import NSS Linux WIO output base

Imported the NSS Linux WIO output path as the next parse prerequisite:

- `shared/sdk/include/wio.h` -> `include/core/wio.h`
- `public_core/library/wio/wio.c` -> `src/core/wio.c`
- `public_core/library/wio/nssUI.c` -> `src/core/nssUI.c`
- `public_core/library/stdio/{printf,vprintf,aprintf,vaprintf}.c` -> `src/core/`
- `public_core/library/wio/{wprintf,wvprintf,waprintf}.c` -> `src/core/`

The port keeps the NSS Linux console path instead of adding local wrappers.
Kernel/procfs pieces in `nssUI.c` are reduced to a Linux-userspace stdout
console buffer while `LB__wioOutput()` and `NSS_UI_vprintf()` remain the real
NSS call path.  `wio.h` uses `wStdout == NULL` for the Linux userspace build,
matching the NSS Linux restriction that only the console output path is active
at this layer.
## 0493 core: route NSS production debug output through WIO

After WIO landed in 0492, removed the remaining production-debug header
placeholder by exposing the real NSS stdio/inlines declarations to `pssDebug.h`
unconditionally.  The heavy scheduler include remains gated until the real
scheduler/FSM layer is imported.  The `nwcore.pssdebug` test now exercises
`PRINT()` through the imported WIO/stdio stack.