mars-nwe/AI.md

AI working notes for mars-nwe

This file is the handoff and assistant-rule document. It should keep the next chat on the current track without forcing it to reconstruct old patch history. Keep it short enough to read before touching code.

Start of a new chat

1. Treat mars-nwe-master as the root/superproject. The other mars-* bundles are submodules or sibling component sources needed by the root build.
2. Read the root *.md files and doc/*.md before changing namespace, salvage, transport or dependency code. The most important active files are TODO.md, REDESIGN.md, ENDPOINTS.md, this file, and:
   • doc/NSS_IMPORT_NOTES.md
   • doc/NWFS_SALVAGE_COMPRESSION_TOOLS.md
   • doc/HANDOFF_AUDIT.md
   • doc/TOOLBOX_PLAN.md
3. Do not assume an older bundle is current. Use the latest applied user state or the latest bundle/patch explicitly accepted by the user.
4. For a fresh build in a new chat, check whether prepare-local-deps.sh is present in the root and run/use it before trying to build; it stages the uploaded offline third-party tarballs and local-only headers needed by the current dependency layout.
5. Default language with the user is German; repository documentation remains English unless the existing file is German.

Documentation ownership

• TODO.md is the active backlog. The top implementation dashboard is allowed and should stay. Unfinished long-running tracks stay in TODO.md until real implementation/test work closes them.
• REDESIGN.md is the durable architecture/design record. Put stable plans, rationale and completed design decisions there, not patch chronology.
• AI.md is for working rules, current handoff state, rejected patches and next action guidance. Keep it concise; do not append duplicate historic handoff dumps forever.
• ENDPOINTS.md is the decimal/hex NCP audit table.
• doc/*.md files are focused topic audits/roadmaps. Keep namespace, NSS public-core and salvage/compression/tool details in their matching doc files.

When a TODO is finished, remove it from TODO.md in the same patch and record its stable outcome in REDESIGN.md or the focused doc/*.md file. Do not move unfinished work out of TODO.md merely because its architecture is documented.

Current accepted patch line

Latest commit after this handoff refresh in an up-to-date tree:

• build: update warning-cleanup submodule pins

Compare git log -1 --format="%s" with this subject. If the tree is older, check which documentation/import-boundary commits are missing and replay them before starting new work.

Current patch handoff block

Last completed work block: Clean GCC 15 build warnings in maintained submodules

Three maintained mars-* submodules were updated and pushed, then the root gitlinks were advanced with update-submodules.sh:

• directory / mars-tinyldap: guard local _GNU_SOURCE definitions so the root CMake compile definition does not trigger duplicate macro warnings.
• third_party/flaim / mars-flaim: remove obsolete C++ register use, format dynamic printf buffers safely, and make static month string tables const.
• third_party/matrixssl / mars-matrixssl: remove misleading fixed-size array annotations from pointer-style TLS/crypto APIs that confused GCC 15, guard the LINUX platform macro, and fix ECDSA signature length byte packing.

The real GitHub-pinned external submodules (yyjson, termbox2, iniparser) were not modified.

Verification used the out-of-tree build directory /home/mario/mars/mars-nwe-build: final cmake --build log had no warning:/error: lines, and ctest --test-dir /home/mario/mars/mars-nwe-build -LE networked --output-on-failure passed all 9 offline tests.

Next import/use-case direction

The src/nwnss / include/nwnss / tests/nwnss tree has been removed. The next step is a root-level NSS re-import under nss/. The source-ordered import sequence remains:

1. root-level nss/ tree with provenance-preserving layout (public_core/library/…, public_core/comn/…, public_core/lsa/…, etc.);
2. block/file I/O boundary using pread/pwrite against an image or blockdevice, wired under the original zlssBioIO* calls;
3. ZLSS/ZIO/SBI startup and storage-image access;
4. LSA super/pool/volume binding;
5. COMN file, namespace and data-stream operations on that mounted context;
6. nwnssmount FUSE data-plane surface;
7. management/list APIs, then create/modify/RAID/snapshot/AdminFS features.

For each area, keep original NSS source semantics in the imported files and put Mars/NWFS/OtherFS/HostFS/FUSE/I/O adaptation in role-specific companion files. Use netware.metadata / netware.userquota.* NSS-shaped xattrs and Netatalk EAs for OtherFS where applicable.

FUSE is a presentation/mount layer: in real NSS mode it must sit above the explicit I/O backend; in OtherFS overlay mode it must store metadata on the lower host objects rather than becoming a private side database.

Directory/NDS work order

• Do not combine LDAPv2/LDAPv3 compatibility work with the FLAIM storage-engine swap. Keep TinyLDAP/nwdirectory on its existing flatfile/mmap/journal backend and add behaviour tests first.
• LDAP CTests should start nwdirectory on 127.0.0.1:<free-port> with a temporary flatfile workdir and test LDAPv2/LDAPv3 bind/search/unbind.
• Add libnwnds and console-only nwsetup together after LDAPv2/schema flatfile behaviour is testable. Do not make the first setup path depend on the TUI.
• Later, add FLAIM storage as CMake-selected parallel backend source files that export the same function names as the flatfile files. Do not break upstreamable flatfile TinyLDAP code.

Rejected or superseded patches that must not be reused as-is:

• 0380
• 0398
• 0403
• 0677 (ncpNdpUserspace.c collector — rejected; search real NSS sources instead)
• 0671 (superseded by 0673; virtualIOUserspace.c removed, real sharedsrc/virtualIO.c.h used)

Current immediate direction

Root-level NSS re-import. The former src/nwnss / include/nwnss / tests/nwnss tree has been removed. The next action is to re-import NSS sources under a root-level nss/ directory using the same provenance-preserving layout as before.

Import rules that carry forward unchanged:

• Original NSS files stay original-like; only compile fixes, modern casts, include path fixes and explicit NSS_USERSPACE guards are allowed inline.
• virtualIO is _ADMIN/management virtual-file and XML datastream I/O — not a disk/block backend. Do not route real NSS block/image I/O through it or through generic libc read/write macros.
• FUSE has two future roles (NSS_FUSE_VOLUME for real NSS volumes, OTHERFS_FUSE_OVERLAY for metadata-overlay over host FS) and must not be treated as the lower block I/O implementation. Each FUSE mode keeps an explicit *Fuse.c / *Fuse.h companion.
• For OtherFS quota/salvage/namespace, do not require FUSE: use NSS-shaped xattrs first, later sidecar metadata where host xattrs are unavailable.
• Quota, salvage and namespace ownership: the NSS library is the provider for semantics; libnwfs is the Mars wrapper/adapter. Existing nwfs quota and salvage code is an implementation inventory only.
• Salvage: .salvage JSON is legacy migration/debug state; .recycle is the OtherFS payload directory; valid salvage metadata comes from netware.metadata.
• Do not activate nwnssmount, nwnssadminvol, *Fuse.c callbacks or real NSS-volume I/O as a prerequisite for the MARS-NWE 3.x cleanup.

Before namespace code changes, keep the legacy logging audit in mind: old MARS accepts numeric debug thresholds from 0 to 99, but the actual source mostly uses 1..5. The future nwlog facade should give those normal levels clear semantic names instead of preserving the old overloaded buckets: 1=error, 2=warn, 3=info, 4=debug and 5=trace. debug is local diagnostic detail; trace is packet/message/handoff path following across process or provider boundaries. New config should expose the cumulative numeric masks 0, 1, 12, 123, 1234 and 12345 as synonyms for off, error, warn, info, debug and trace, and use one global [logging] level plus optional [logging.process.<name>] level overrides; the old numeric 100..106 process entries stay accepted for compatibility. Useful legacy 6..99 traces should map to nwlog_detail() later. nwlog_detail() is not part of the normal 1..5 threshold ladder: normal builds return 0, while MAINTAINER_BUILD builds may emit detail whenever such a call site is reached. No INI option may enable it in a normal build. nwlog is the facade/backend plan. Do not add zlog calls or reintroduce a zlog build dependency; zlog is Apache-2.0 and is not compatible with the repository's GPL-2.0-only core policy. See include/nwlog.h and doc/LOG_LEVEL_AUDIT.md.

Before namespace code changes, keep the terminal-tooling decision in mind when touching FLAIM tests or admin tools: do not add new direct curses.h users. Future interactive tools should go through the planned nwtui/nwi18n stack and the multi-call nwtoolbox applet model documented in doc/TOOLBOX_PLAN.md. Configuration parsing/editing is covered in the same file: use the planned shared libnwcore nw_ini_* reader/writer so server code and tools share one policy. Production daemons must not depend on the TUI stack.

Before namespace code changes, use the handoff audit rules in doc/HANDOFF_AUDIT.md. New namespace/NWFS work must not add new magic return(-1) / return(-2) paths. Use the provider names in include/ncp_endpoint.h for audits and future endpoint tables, but do not wire the enum into runtime dispatch until the current nwconn to nwbind handoff sites are annotated.

Then continue the NSS namespace track inside libnwnss:

1. libnwnss is the single userspace library for imported NSS functions and semantics. Do not create a separate namespace/AdminVol NSS library. The library should grow from the existing imported runtime into the real NSS components needed by Mars NWE: namespace, trustee/effective-rights helpers, XATTR/EA/metadata helpers and later AdminVol semantics.
2. Work bottom-up from the original NSS Linux sources. Use the files in nss.tar.bz2 and nss-common.tar.bz2; inspect NSS Makefiles/Kbuild-style source lists as well as includes. If a compile or link error reports an unresolved NSS symbol, search for and import the matching real NSS .c and header before considering anything else.
3. Do not write semantic wrappers or local replacement models for NSS namespace, XATTR, trustee or AdminVol logic. Trustees are serialized through netware.metadata / zNW_metadata_s.nwm_trustee[]; do not invent an active netware.trustees or user.netware.trustees xattr. The only acceptable wrappers/adaptations are narrow userspace ports for kernel/OS boundaries such as spinlocks, allocation, wait/schedule hooks, WIO/console output and similar platform plumbing.
4. For namespace, target the real public_core/comn/namespace code and its real dependencies under public_core/comn/common, public_core/nss/lib, shared/sdk/include, shared/sdk/internal, shared/sdk/public and shared/sdk/comnSA. Keep the src/nwnss layout aligned with the original public_core tree, and keep the include/nwnss layout aligned with the original shared/sdk tree.
5. AdminVol is a later target because nameSpace.c reaches AdminVol and AdminVol reaches Beast/common code. Before AdminVol cleanup, import the XATTR/EA/ metadata side into libnwnss from the original NSS archives so later nwfs can consume real NSS structures and functions instead of keeping local semantic copies.
6. libnwnss must become a userspace NSS semantics library, not a hidden NSS disk implementation. Do not import block-device, VFS or NSS-disk-layout I/O blindly. Preserve original disk/VFS-shaped code where possible, but place HostFS, OtherFS, sidecar, FUSE and block-I/O policy in explicit parallel *Userspace.c / *Userspace.h boundary files. Public userspace APIs should use libnwnss handles or backend contexts, not real Linux-kernel struct dentry, struct inode or struct super_block as ABI.
7. _ADMIN is expected to become virtual/in-memory semantics in libnwnss and later a userspace nwadminvol process served via IPC from nwconn/nwfs, with the volume-numbering rule SYS = 0, _ADMIN = 1, further volumes from 2.
8. Keep src/namspace.c as NCP glue over the future namespace engine. Do not expand old MARS namedos/nameos2 as the long-term solution; they are replacement targets.
9. Namespace pure helpers can stay provider-shaped in libnwnss: compare, wildcard match/replace, ASCII/Unicode conversion, legal-name checks and component scan for DOS, LONG/OS2, UNIX/NFS, MAC, DATASTREAM and EA. *_generateUniqueName() is not a pure string helper; it needs directory uniqueness lookup. Preserve the original NSS namespace files and put HostFS, OtherFS, sidecar or Mars directory-collision policy in a parallel nameSpaceUserspace.c / nameSpaceUserspace.h boundary when that path is implemented.

NCP notation rule

Always write NCP groups/selectors with both decimal and wire/code hex when recording protocol details. Examples:

• decimal 87 == wire/code 0x57
• decimal 90/12 == wire/code 0x5a/0x0c
• decimal 123/70 == wire/code 0x7b/0x46

Use ENDPOINTS.md, the NDK Core Protocols PDF, WebSDK docs and SDK includes as references. NetWare 3.x/default compatibility has priority over broad 4.x/NDS work.

Patch workflow

• Make small, reviewable patches with one clear subject.
• Inspect current sources before editing; no speculative patching.
• Run at least git diff --check before exporting a patch.
• Also run syntax/build/test checks that match the touched files when feasible.
• Export only the .patch file by default when producing repository changes. Do not create or link a bundle unless the user explicitly asks for one.
• Every repository patch must update this AI.md with the new final commit subject in the current patch marker before exporting the patch. If the user later uploads a bundle from their repository, the marker makes that bundle self-identifying even when old chat downloads have expired or are unavailable.
• Every repository patch must also replace the single Current patch handoff block in this file with a short description of what the patch was meant to do. Replace the old block; do not append a new one for every patch.
• Always include a copy/paste git am <patch-name>.patch command in the user response next to the patch download link. This is part of the handoff contract, not an optional nicety.
• Generate patch mail headers with Mario Fetka <mario.fetka@disconnected-by-peer.at> as the author identity, not an AI/OpenAI identity.
• Do not claim a build/test passed unless it actually ran in the current work tree.

Source/layout rules

• Keep imported NSS sources in the root-level nss/ tree, not in libnwcore. libnwcore is Mars-core only and must not link against the NSS library. NSS tests carry the nss.* prefix; core tests cover only real Mars core helpers.
• Preserve original NSS API names and original filenames wherever possible. Import real NSS files from their provenance paths: public_core/library/… → nss/library/…, public_core/nss/… → nss/nss/…, public_core/comn/… → nss/comn/…, public_core/lsa/… → nss/lsa/…; shared/sdk/include/… → nss/include/include/…, shared/sdk/internal/… → nss/include/internal/…, shared/sdk/public/… → nss/include/public/…, shared/sdk/comnSA/… → nss/include/comnSA/….
• GPL-2.0-only source imports are allowed when the imported source permits GPLv2 use and the original provenance/license headers are preserved.
• Do not import broad NSS/OES subsystems blindly; do not replace missing NSS symbols with local semantic wrappers. Use unresolved symbols and original Makefile object lists to find the next real bottom-up dependency.
• nss/include/library must not be exported as a PUBLIC CMake include directory (shadows host headers such as <stdlib.h> and <ctype.h>).
• nss/sharedsrc/*.c.h are NSS shared-source implementation fragments; keep them in the source tree, do not move them to include/.
• For every NSS function that needs a companion layer, keep imported original files focused on original control flow. Choose the companion suffix by role: *Userspace.c / *.h for libnwfs/Mars-NCP/OtherFS/HostFS-facing adapter boundaries; *Fuse.c / *.h for FUSE/nwnssmount-facing callbacks; *Io.c / *.h for concrete Linux pread/pwrite block/image I/O; *Runtime.c / *.h for NSS runtime semantics that replace former kernel helpers. Shared helper files use Nss* names (NssBlockIo.c, NssFuse.c, NssRuntime.c). Preprocessor/CMake switches use generic NSS_* names.
• *Userspace.c / *Userspace.h companion files are ALLOWED only when they sit alongside the matching imported original and carry libnwfs/Mars-NCP/OtherFS/ HostFS adaptation that must not go inline. Collector/stub-filler companions are FORBIDDEN — always search the original archives first.
• The NSS library may use: AdminVol, AdminFS, VirtualIO (management/template/ function data streams), in-memory/callback streams, and pure NSS management callbacks. Real NSS disk-layout I/O requires the NSS_BLOCK_IO guard and the pread/pwrite zlssBioIO* implementation.
• libnwfs is the Mars/NCP file compatibility wrapper; it translates Mars/NCP operations into NSS provider calls.
• libnwbind is for bindery identity/storage. NDS/eDirectory compatibility belongs in a future libnwnds backed by libnwdirectory. nwnssmount is the future FUSE mount consumer; nwnssmu is the future TUI management utility.

Service and provider ownership rules

• nwserv is the governor/control-plane supervisor for the Mars process tree. It starts and monitors processes such as nwconn, nwdirectory and later optional provider services. It is not the NCP server and should not become the normal data-plane router for decoded NCP payloads.

• nwconn is the NCP server process. It owns client NCP request parsing and NCP reply construction. For file/volume operations it should call libnwfs; for future NDS/eDirectory identity, authentication, trustee and security equivalence work it should call libnwnds.

• libnwfs is the Mars/NCP file wrapper around libnwnss. Keep Mars wire parsing, NCP reply packing, Mars volume options and HostFS/OtherFS adaptation there; keep NSS semantics in libnwnss.

• libnwnds is the planned NDS/eDirectory compatibility library. It translates NetWare concepts such as DN, GUID, UserID, Security Equivalence Vector, tree context, object/class/attribute mapping and trustee identity lookup to libnwdirectory. Do not hide LDAP/eDirectory policy in libnwnss or libnwfs.

• libnwdirectory is the library/client layer for nwdirectory; nwdirectory is the LDAP/TinyLDAP-based directory server.

• NSS headers such as eDir.h, ncpIDAPI.h and ndp_idbroker.h are boundary references for GUID/DN/UserID/SEV mapping. They are not a full eDirectory backend and should point toward libnwnds, not toward new directory policy in libnwnss.

• NSS encp.h is useful as an NCP wire/layout reference for Mars header convergence, especially namespace, quota, trustee and salvage layouts. Do not cast Mars byte buffers directly to imported NSS structs; use byte-safe parser/builder wrappers or compatibility aliases.

• _ADMIN / AdminVolume is a Mars-NWE/NCP server service, not a side effect of an individual data-volume mount. The planned nwnssadminvol service is started and owned by nwserv; it publishes the visible _ADMIN view for that Mars-NWE instance and collects provider/volume registrations from nwnssmount processes. A volume mounted from /etc/fstab must work before nwserv is running: nwnssmount records a runtime mount descriptor and optional control socket under /run/nwnss/, then nwnssadminvol discovers and binds it later when nwserv starts.

• There must be only one visible AdminVolume owner for one Mars-NWE/NCP server namespace. Do not publish a nwnss FUSE _ADMIN and an original Kernel-NSS _ADMIN as competing roots. A later nwnssadminvol may read the Kernel-NSS admin volume and expose those pools/volumes as imported read-only or explicit proxy records, but it must keep owner=KERNEL_NSS metadata and must not treat them as native nwnss volumes.

Namespace rules

• Do not expand old MARS namedos/nameos2 as the long-term solution. They are replacement targets.
• Import/adapt the NSS namespace engine directly in libnwnss, then retire old MARS namespace logic after tests cover DOS/LONG behaviour. Missing namespace dependencies must be resolved from the original NSS sources, not by adding replacement wrappers.
• Stable namespace state belongs in netware.metadata: file ID, parent file ID, DOS name, LONG/OS2 name, MAC name, UNIX/backend name, casefold/hash fields and namespace flags. *_generateUniqueName() must use the generic userspace companion-file pattern when it needs HostFS/OtherFS directory collision lookup or metadata/xattr-backed namespace state; do not put that lookup directly into the original NSS namespace source files.
• Existing Linux files created by Samba, rsync or local admin tools must be reconciled by libnwfs watcher/scanner work, not by a private side database.
• MAC namespace is a namespace/stream/metadata problem, not a transport problem. Resource forks and Finder info belong in streams/metadata later.

Salvage/recycle rules

• Keep .recycle as the Samba-compatible deleted-payload backend.
• Make netware.metadata on the recycled payload authoritative for deleted state.
• Treat .salvage JSON as legacy migration/debug data only; do not make it a second long-term authority.
• Samba vfs_recycle normally uses rename, so xattrs remain attached to the recycled inode. Manually copied files in .recycle without netware.metadata are not valid NetWare salvage objects unless an explicit admin repair tool marks them.
• If a compressed file is recycled, the .recycle payload should be a normal uncompressed Linux file; keep previous compression state in netware.metadata.

Compression and stream rules

• Compression belongs in libnwfs, not libnwcore.
• Future private stream/compression data belongs under .nwfs_streams/<stable-file-id>/....
• The stable file ID comes from MARS/NWFS/NSS-shaped metadata, not from Linux inode numbers and not from visible filenames.
• Do not encode compression state in visible names such as compressed_*.

Transport rules

• TCP/IP is implemented in libnwtransport, a library that nwconn links against; it is not a new daemon.
• Planned files: src/nwtransport.c, src/nwipx.c, later src/nwtcp.c.
• NCP providers stay transport-neutral: IPX/TCP -> nwtransport -> nwconn -> dispatcher -> providers.
• Use audited libowfat API names from the bundled mars-libowfat source: socket_tcp4(), socket_tcp6(), socket_bind4_reuse(), socket_bind6_reuse(), socket_listen(), nonblocking accept helpers and io_* readiness helpers.
• IPX config sections use frame tokens without dots: 8022, 8023, etherii, snap, tr8022, auto.
• [transport.ipx.local] is the internal network; explicit routes use [transport.ipx.route.<target>].
• If IPX is requested but socket(AF_IPX, ...) fails, report that AF_IPX is unavailable instead of silently falling back.
• A future L2 IPX backend is allowed, but it is an additional backend below nwtransport, not a replacement for the kernel path. Start with Ethernet_II IPX, SAP/RIP, and NCP socket 0x0451.
• Do not vendor a capability library for L2 IPX. Open the packet socket and report missing CAP_NET_RAW/CAP_NET_ADMIN; root, setcap, or systemd capabilities are deployment policy.
• Do not copy GPL-3.0-or-later code from the Rust nwserver reference. Use only the observed architecture: userland Ethernet/IPX packet handling, SAP/RIP, and NCP socket dispatch.
• SPX, if needed, is a later userland layer above IPX. Do not rely on kernel SPX and do not make namespace work wait for SPX.
• Linux 2.4.37.9 SPX is imported only as reference material in src/kernel/af_spx.c and include/kernel/spx.h; do not build it as-is.

Dependency/source bundle rules

• mars-nwe-master is the root repository.
• Uploaded mars-* bundles are submodules/sibling components needed in the configured locations.
• Offline point-release tarballs are staged by prepare-local-deps.sh. In a new chat or clean checkout, use this script as the first dependency/bootstrap step before CMake/build attempts:
  • yyjson-0.12.0 into third_party/yyjson
  • libsodium-1.0.20 into the nested libsodium snapshot path
  • gdbm-1.26 built locally under .local-deps/prefix
  • PAM and ncurses headers staged while linking to system libraries
• Do not vendor random system headers directly into endpoint/provider code.

Reference archives

The following archives were used for the NSS import/audit work and are available for further namespace implementation and NCP structure verification:

• nss-common.tar.bz2
• nss.tar.bz2
• ncp__enu.pdf
• websdk.tar.gz
• include.tar.gz

Networked CTest skip policy

Server-dependent NCPFS/AFP/salvage/quota smokes must be visible in CTest but must not fail ordinary offline builds. Register them with the networked;integration labels and SKIP_RETURN_CODE 77. The test launcher prints SKIP/WARN and exits 77 unless MARS_NWE_RUN_NETWORKED_TESTS=1 and the required MARS_NWE_TEST_* environment variables are present. Use ctest -LE networked for offline unit coverage and ctest -L networked for server-backed integration runs. When the TCP/IP transport test harness lands, these CTest entries become the path for real local execution.