mars-nwe/AI.md

AI working notes for mars-nwe

This file is the handoff and assistant-rule document. It should keep the next chat on the current track without forcing it to reconstruct old patch history. Keep it short enough to read before touching code.

Start of a new chat

1. Treat mars-nwe-master as the root/superproject. The other mars-* bundles are submodules or sibling component sources needed by the root build.
2. Read the root *.md files and doc/*.md before changing namespace, salvage, transport or dependency code. The most important active files are TODO.md, REDESIGN.md, ENDPOINTS.md, this file, and:
   • doc/NSS_NAMESPACE_AUDIT.md
   • doc/NSS_PUBLIC_CORE_AUDIT.md
   • doc/NWFS_SALVAGE_COMPRESSION_TOOLS.md
   • doc/HANDOFF_AUDIT.md
   • doc/TUI_TOOLBOX_PLAN.md
3. Do not assume an older bundle is current. Use the latest applied user state or the latest bundle/patch explicitly accepted by the user.
4. For a fresh build in a new chat, check whether prepare-local-deps.sh is present in the root and run/use it before trying to build; it stages the uploaded offline third-party tarballs and local-only headers needed by the current dependency layout.
5. Default language with the user is German; repository documentation remains English unless the existing file is German.

Documentation ownership

• TODO.md is the active backlog. The top implementation dashboard is allowed and should stay. Unfinished long-running tracks stay in TODO.md until real implementation/test work closes them.
• REDESIGN.md is the durable architecture/design record. Put stable plans, rationale and completed design decisions there, not patch chronology.
• AI.md is for working rules, current handoff state, rejected patches and next action guidance. Keep it concise; do not append duplicate historic handoff dumps forever.
• ENDPOINTS.md is the decimal/hex NCP audit table.
• doc/*.md files are focused topic audits/roadmaps. Keep namespace, NSS public-core and salvage/compression/tool details in their matching doc files.

When a TODO is finished, remove it from TODO.md in the same patch and record its stable outcome in REDESIGN.md or the focused doc/*.md file. Do not move unfinished work out of TODO.md merely because its architecture is documented.

Current accepted patch line

Latest patch marker expected in an up-to-date bundle:

• 0460 test: add core NSS helper unit coverage

When a later chat receives a new mars-nwe-master bundle, compare git log -1 with this marker. If the uploaded bundle already contains this commit subject, the last patch was applied. If the bundle is older and the old chat download no longer works, recreate the missing patch instead of assuming it was applied.

The active line is expected to include:

• Quota work through the corrected Linuxquota/NWQUOTA split and DOS/live smokes.
• NSS low-level imports through the Unicode/codepage, UTC, GUID/ID, xCtype and xString helper line.
• 0434 build: record offline dependency bootstrap layout.
• 0435 docs: record transport config and nwfs watch plan.
• 0436 docs: keep unfinished NSS and salvage work in TODO.

Current patch handoff block

Last generated patch:

• 0460 test: add core NSS helper unit coverage

Purpose of that patch:

• Add focused CTest coverage for the imported low-level NSS helper line in libnwcore: bit operations, bitmap allocation/search, CRC/hash behavior, xCtype classification/case conversion, xString compatibility helpers, and UTF-8/Unicode round-trips. Keep the tests in subsystem directories under tests/core/ so later codepage, byte-converter, GUID/ID, UTC, and Unicode table tests can be added without flattening the test tree.

Rejected or superseded patches that must not be reused as-is:

• 0380
• 0398
• 0403

Current immediate direction

Before starting namespace implementation, keep the documentation boundaries clean:

• TODO.md: active work only, dashboard retained.
• REDESIGN.md: architecture and design separation clarified.
• AI.md: concise current state and rules only.

Before namespace code changes, keep the legacy logging audit in mind: old MARS accepts numeric debug thresholds from 0 to 99, but the actual source mostly uses 1..5. The future nwlog facade should give those normal levels clear semantic names instead of preserving the old overloaded buckets: 1=error, 2=warn, 3=info, 4=debug and 5=trace. debug is local diagnostic detail; trace is packet/message/handoff path following across process or provider boundaries. New config should expose the cumulative numeric masks 0, 1, 12, 123, 1234 and 12345 as synonyms for off, error, warn, info, debug and trace, and use one global [logging] level plus optional [logging.process.<name>] level overrides; the old numeric 100..106 process entries stay accepted for compatibility. Useful legacy 6..99 traces should map to nwlog_detail() later. nwlog_detail() is not part of the normal 1..5 threshold ladder: normal builds return 0, while MAINTAINER_BUILD builds may emit detail whenever such a call site is reached. No INI option may enable it in a normal build. nwlog is the facade/backend plan. Do not add zlog calls or reintroduce a zlog build dependency; zlog is Apache-2.0 and is not compatible with the repository's GPL-2.0-only core policy. See include/nwlog.h and doc/LOG_LEVEL_AUDIT.md.

Before namespace code changes, keep the terminal-tooling decision in mind when touching FLAIM tests or admin tools: do not add new direct curses.h users. Future interactive tools should go through the planned nwtui/nwi18n stack and the multi-call nwtoolbox applet model documented in doc/TUI_TOOLBOX_PLAN.md. Configuration parsing/editing is separate: use the planned shared libnwcore nw_ini_* reader/writer from doc/NWCORE_INI_PLAN.md so server code and tools share one policy. Production daemons must not depend on the TUI stack.

Before namespace code changes, use the handoff audit rules in doc/HANDOFF_AUDIT.md. New namespace/NWFS work must not add new magic return(-1) / return(-2) paths. Use the provider names in include/ncp_endpoint.h for audits and future endpoint tables, but do not wire the enum into runtime dispatch until the current nwconn to nwbind handoff sites are annotated.

Then begin the namespace track:

1. Start with DOS namespace compatibility for NetWare 1.x/2.x/3.x clients.
2. Use the NSS namespace sources as the replacement/adaptation target, not a permanent wrapper over old MARS namedos/nameos2 code.
3. Keep src/namspace.c as NCP glue over the future namespace engine.
4. Move reusable filesystem/namespace/metadata helpers toward libnwfs.
5. Defer broad 4.x/NDS/OES runtime behaviour unless it is guarded or required by a concrete compatibility path.

NCP notation rule

Always write NCP groups/selectors with both decimal and wire/code hex when recording protocol details. Examples:

• decimal 87 == wire/code 0x57
• decimal 90/12 == wire/code 0x5a/0x0c
• decimal 123/70 == wire/code 0x7b/0x46

Use ENDPOINTS.md, the NDK Core Protocols PDF, WebSDK docs and SDK includes as references. NetWare 3.x/default compatibility has priority over broad 4.x/NDS work.

Patch workflow

• Make small, reviewable patches with one clear subject.
• Inspect current sources before editing; no speculative patching.
• Run at least git diff --check before exporting a patch.
• Also run syntax/build/test checks that match the touched files when feasible.
• Export only the .patch file by default when producing repository changes. Do not create or link a bundle unless the user explicitly asks for one.
• Every repository patch must update this AI.md with the new final commit subject in the current patch marker before exporting the patch. If the user later uploads a bundle from their repository, the marker makes that bundle self-identifying even when old chat downloads have expired or are unavailable.
• Every repository patch must also replace the single Current patch handoff block in this file with a short description of what the patch was meant to do. Replace the old block; do not append a new one for every patch.
• Always include a copy/paste git am <patch-name>.patch command in the user response next to the patch download link. This is part of the handoff contract, not an optional nicety.
• Generate patch mail headers with Mario Fetka <mario.fetka@disconnected-by-peer.at> as the author identity, not an AI/OpenAI identity.
• Do not claim a build/test passed unless it actually ran in the current work tree.

Source/layout rules

• Keep active NSS low-level helper imports directly under normal MARS paths such as src/core/<original>.c and include/core/<original>.h; do not create or keep a permanent src/core/nss/ wrapper/reference area for helpers already imported into normal core files.

• Preserve original NSS API names where that improves compatibility with later imported code.

• GPL-2.0-only source imports are allowed when the imported source permits GPLv2 use and the original provenance/license headers are preserved. For Linux kernel reference material, record that mars-nwe selects GPL-2.0-only.

• Clean NSS staging incrementally: once an NSS file has been imported, adapted or stubbed in a normal MARS path, remove the duplicate staging/reference copy unless an audit document names a still-open reason to keep it. Do not delete active src/nwfs/nss/ namespace/common/compression reference material before the matching libnwfs adaptation is complete.

• Do not import whole NSS/OES runtime subsystems just to satisfy helper dependencies. Replace kernel, VFS, memory-tracking and platform hooks with small userland glue when a helper is otherwise useful.

• libnwcore is for base helpers: Unicode/codepages, CRC/hash/bitmap, GUID/ID, UTC/time, allocation/string compatibility and similar low-level pieces.

• libnwfs is for namespace, file IDs, metadata, streams, EA, salvage, compression, host-side reconcile and _ADMIN virtual-volume filesystem work.

• libnwbind is for bindery identity/storage; libnwnds is for later eDirectory/NDS work.

Namespace rules

• Do not expand old MARS namedos/nameos2 as the long-term solution. They are replacement targets.
• Import/adapt the NSS namespace engine directly, then retire old MARS namespace logic after tests cover DOS/LONG behaviour.
• Stable namespace state belongs in netware.metadata: file ID, parent file ID, DOS name, LONG/OS2 name, MAC name, UNIX/backend name, casefold/hash fields and namespace flags.
• Existing Linux files created by Samba, rsync or local admin tools must be reconciled by libnwfs watcher/scanner work, not by a private side database.
• MAC namespace is a namespace/stream/metadata problem, not a transport problem. Resource forks and Finder info belong in streams/metadata later.

Salvage/recycle rules

• Keep .recycle as the Samba-compatible deleted-payload backend.
• Make netware.metadata on the recycled payload authoritative for deleted state.
• Treat .salvage JSON as legacy migration/debug data only; do not make it a second long-term authority.
• Samba vfs_recycle normally uses rename, so xattrs remain attached to the recycled inode. Manually copied files in .recycle without netware.metadata are not valid NetWare salvage objects unless an explicit admin repair tool marks them.
• If a compressed file is recycled, the .recycle payload should be a normal uncompressed Linux file; keep previous compression state in netware.metadata.

Compression and stream rules

• Compression belongs in libnwfs, not libnwcore.
• Future private stream/compression data belongs under .nwfs_streams/<stable-file-id>/....
• The stable file ID comes from MARS/NWFS/NSS-shaped metadata, not from Linux inode numbers and not from visible filenames.
• Do not encode compression state in visible names such as compressed_*.

Transport rules

• TCP/IP is a transport split under nwconn/nwserv, not a new daemon.
• Planned files: src/nwtransport.c, src/nwipx.c, later src/nwtcp.c.
• NCP providers stay transport-neutral: IPX/TCP -> nwtransport -> nwconn -> dispatcher -> providers.
• Use audited libowfat API names from the bundled mars-libowfat source: socket_tcp4(), socket_tcp6(), socket_bind4_reuse(), socket_bind6_reuse(), socket_listen(), nonblocking accept helpers and io_* readiness helpers.
• IPX config sections use frame tokens without dots: 8022, 8023, etherii, snap, tr8022, auto.
• [transport.ipx.local] is the internal network; explicit routes use [transport.ipx.route.<target>].
• If IPX is requested but socket(AF_IPX, ...) fails, report that AF_IPX is unavailable instead of silently falling back.
• A future L2 IPX backend is allowed, but it is an additional backend below nwtransport, not a replacement for the kernel path. Start with Ethernet_II IPX, SAP/RIP, and NCP socket 0x0451.
• Do not vendor a capability library for L2 IPX. Open the packet socket and report missing CAP_NET_RAW/CAP_NET_ADMIN; root, setcap, or systemd capabilities are deployment policy.
• Do not copy GPL-3.0-or-later code from the Rust nwserver reference. Use only the observed architecture: userland Ethernet/IPX packet handling, SAP/RIP, and NCP socket dispatch.
• SPX, if needed, is a later userland layer above IPX. Do not rely on kernel SPX and do not make namespace work wait for SPX.
• Linux 2.4.37.9 SPX is imported only as reference material in src/kernel/af_spx.c and include/kernel/spx.h; do not build it as-is.

Dependency/source bundle rules

• mars-nwe-master is the root repository.
• Uploaded mars-* bundles are submodules/sibling components needed in the configured locations.
• Offline point-release tarballs are staged by prepare-local-deps.sh. In a new chat or clean checkout, use this script as the first dependency/bootstrap step before CMake/build attempts:
  • yyjson-0.12.0 into third_party/yyjson
  • libsodium-1.0.20 into the nested libsodium snapshot path
  • gdbm-1.26 built locally under .local-deps/prefix
  • PAM and ncurses headers staged while linking to system libraries
• Do not vendor random system headers directly into endpoint/provider code.

New reference inputs available for the next namespace work

The user supplied fresh reference archives for the next phase:

• nss-common.tar.bz2
• nss.tar.bz2
• ncp__enu.pdf
• websdk.tar.gz
• include.tar.gz

Use these for source audits and exact NCP structure/selector verification before implementing namespace changes.