mars-nwe/AI.md

AI working notes for mars-nwe

This file is for future ChatGPT sessions. It records general working rules and local build/test notes only. It should not be used as the current project status log; the current patch stack and task context should be pasted into a new chat separately.

Start of a new chat

When the user says this is a new chat or asks to continue mars-nwe work, first read this file before proposing patches or making assumptions. Then ask for, or use, the current project status that the user pasted into the chat.

Current handoff status after patch 0171

The current accepted patch line for the endpoint-documentation work is expected to be:

• 0141-log-normalize-afp-get-file-info-mask-fields.patch through 0171-docs-document-ncp23-queue-core-layouts.patch

Do not apply or base work on these rejected/superseded patches:

• 0141-log-normalize-afp-file-information-bitmap-fields.patch
• 0144-docs-add-root-changes-summary.patch if it is the German version; use 0144-docs-add-root-changes-summary-en.patch instead
• 0152-docs-note-message-control-subfunction.patch because it documented a later NetWare 5.x/MOAB message-control subfunction outside the current default compatibility target
• 0160-docs-gate-netware4-directory-stubs.patch because it wrapped or moved already implemented endpoints; use 0160-docs-bucket-endpoint-audit-by-netware-generation.patch instead
• 0161-docs-recheck-ncp22-websdk-endpoint-coverage.patch because it mixed SDK decimal notation with wire hex cases and placed comments at the wrong spot; use 0161-docs-recheck-ncp22-websdk-coverage-corrected.patch instead

Next likely endpoint block: continue 0x2222/23 Queue Management after the core queue/job calls documented in 0171. The next patch should inspect the remaining queue/job-control subfunctions in nwbind.c and any nwconn.c prehandlers, compare them against WebSDK/PDF/includes, and record only real layout differences in TODO.md.

When handing off to a new chat, paste the latest patch name the user has actually applied. If the user says a patch was not applied, rebuild the next patch against the last confirmed applied patch, not against the rejected file.

Patch workflow

• Produce patches that apply with exactly:

  git am patchname.patch
  

• Assume the user has already applied and committed accepted earlier patches. Build every new patch against the current tree the user provides.

• Do not ask the user to apply a long patch chain unless they explicitly say earlier patches were not committed.

• Keep follow-up patches small and reviewable. Do not mix functional changes, cleanup, and logging refactors unless the user asks for that.

• If a patch is only documentation or test cleanup, keep it that way.

Current protocol audit scope

• The current endpoint documentation/audit pass is scoped to compatibility NCPs through NetWare 3.x by default, including NetWare 1.x/2.x legacy calls where they are documented. Bucket endpoints by the oldest NetWare generation that documents them: put 1.x/2.x legacy calls in their own sections, keep the remaining through-3.x compatibility calls in the 3.x/default section, and put endpoints introduced in NetWare 4.x or later in a separate planning/stub section.
• NetWare 4.x/OES/MOAB-only endpoints are not part of the default implementation target. Already implemented compatibility code must not be removed or wrapped just because it is 4.x-era; only new, not-yet-implemented 4.x stubs should be placed behind #if MARS_NWE_4. MARS_NWE_4 is currently hard-disabled in include/config.h.cmake and should stay 0 unless the user explicitly asks to start that work.
• When a 0x2222 group or subfunction is forwarded out of nwconn.c, follow the handoff before declaring the endpoint documented. nwconn.c should document the handoff and the exact header/payload bytes that are preserved or rewritten before forwarding; the destination file (for example nwbind.c) must document the concrete subfunction request/reply layout at the real handler. Do not stop at a comment such as nwbind must do prehandling, nwbind must do the rest, or handled by nwbind.
• For forwarded paths, document any nwconn-side payload mutation as part of the audit. Examples in the current tree include queue create path expansion, queue job file-handle insertion, quota bindery prehandling, and semaphore/message group forwarding. If a forwarded subfunction is not audited yet, record it as a target-file follow-up rather than only documenting the nwconn dispatcher.
• For documentation-only endpoint patches, do not change parser offsets, byte order, reply layout, or completion behavior. Always compare the code parser/reply layout against the applicable SDK/WebSDK/PDF request format and, when available, the uploaded SDK include prototypes. If the code differs from the SDK layout, document the concrete difference inline and mirror it in TODO.md for later testing. If it matches, say so in the patch summary so the audit trail is clear.
• When an SDK/WebSDK/PDF endpoint number is written in decimal notation, convert it carefully to the wire case value before adding inline documentation. Example: Directory Services 0x2222/22/12 in the PDF means SubFunctionCode decimal 12, i.e. wire case 0x0c; it is not the existing case 0x12 / decimal 18 Allocate Permanent Directory Handle. Place disabled stubs directly at the correct numeric slot inside the dispatcher, never appended at the end of the function. For implemented endpoints, keep the detailed documentation inside the relevant case block, immediately after the case label/opening brace, matching the local style; do not leave a large endpoint block before the case label.
• If a PDF/WebSDK page title and an internal table row disagree, prefer the endpoint title plus include/WebSDK cross-checks and record the mismatch instead of inventing a new wire case. Example: 0x2222/23 Verify Serialization is titled SDK decimal 23/12 / wire 0x0c, even though one PDF table row prints SubFunctionCode (212); do not add a wire 0xd4 case without a packet trace or include-level confirmation.
• In TODO.md and endpoint summaries, avoid ambiguous mixed notation for grouped subfunctions. Write SDK/PDF numbers as decimal and include the wire byte explicitly when it differs or could be confused, for example SDK 22/18 / wire 0x12 or SDK 22/12 / wire 0x0c. Do not write 22/12 for a wire case 0x12 unless the SDK number is actually decimal 12.
• Keep TODO.md endpoint audit notes grouped by endpoint family and NetWare generation instead of as one long flat list.
• Before starting the next detailed endpoint block, maintain a coverage index for SDK/WebSDK-listed 0x2222 groups that are not yet audited. Classify each group as present in code but not audited, missing a top-level handler, or likely later-generation/unclear. This index is only a planning aid: do not add active TODO work or source stubs until the specific block has been checked for handoffs and bucketed by oldest documented NetWare generation.
• For SDK-listed groups that appear missing from nwconn.c, also search destination files such as nwbind.c, queue helpers, salvage helpers, AFP/name-space dispatchers, and any prehandler path before declaring the endpoint absent.
• The rejected 0152-docs-note-message-control-subfunction.patch must not be applied: it documented 0x2222/21/0x0c Connection Message Control, which is outside the default NetWare 1.x/2.x/3.x MARS-NWE target scope.

mars-nwe coding style rules

• Prefer existing mars_nwe / NetWare functions over new helper code.
• Before adding a helper, search the tree for an existing equivalent.
• Do not introduce parallel mechanisms for paths, trustees, xattrs, AFP metadata, copy/write/restore, u16/u32 packing, or logging.
• Use existing integer and wire-format macros such as GET_16, GET_32, U16_TO_16, U32_TO_32, and related mars_nwe helpers instead of open-coded byte parsing/serialization.
• Use existing namespace/path conversion and basehandle logic instead of parsing NetWare paths by hand.
• For file restore/copy/write behavior, prefer the existing Novell/mars_nwe file functions over direct POSIX operations. Use POSIX only where there is no suitable internal mechanism, and keep it clearly isolated.
• Do not add a new trustee or xattr database. Salvage JSON is a snapshot; real restore should feed existing mars_nwe trustee/xattr/AFP mechanisms.

NCP path and hidden repository notes

• Normal NCP path resolution intentionally treats Unix dot path components as hidden/special. In the classic path resolver (build_dir_name() in connect.c), a component beginning with . is accepted only for ./.. semantics; a component such as .recycle or .salvage returns invalid path (0x899c).
• nwattrib.c also marks Unix dot files/directories hidden by default when no explicit NetWare attributes are stored.
• Therefore .recycle and .salvage are backend repositories, not user-visible NCP paths. Tests must not expect SYS:.recycle/... or SYS:.salvage/... to open through ordinary NCP file calls.
• Use the official salvage endpoints (87/16 scan, 87/17 recover, 87/18 purge, and old 22/27-22/29) to observe or operate on salvage entries. Verify recovered payload content by reading the restored live file through NCP, not by opening backend repository paths through NCP.

Salvage endpoint rules

• NCP 0x2222 / 87 / 16 is decimal 87/16, implemented as function 0x57, subfunction 0x10.
• NCP 0x2222 / 87 / 17 is decimal 87/17, function 0x57, subfunction 0x11.
• NCP 0x2222 / 87 / 18 is decimal 87/18, function 0x57, subfunction 0x12.
• Legacy salvage endpoints are old function 22 decimal / 0x16: 22/27 scan, 22/28 recover, and 22/29 purge. They should remain thin adapters over the same shared salvage backend, not a second implementation.
• Keep 0x57 subfunction dispatch in handle_func_0x57() / namespace code, not as a second subfunction switch in nwconn.c.
• Old 0x16 calls need a minimal bridge in namespace code because short directory handles must be resolved through existing build_base() / dir_base[] internals before reaching the shared backend.
• Versioned backend payload names follow Samba vfs_recycle literally: Copy #1 of NAME, Copy #2 of NAME, ... . Do not localize this string and do not run it through gettext; the NCP scan reply still reports the original deleted filename for every version.
• Versioned salvage entries may have different .recycle/.salvage names but 87/16 returns the original deleted filename for every version. Do not match recover/purge by display name alone.
• Scan must treat .salvage JSON as a sidecar for the matching .recycle payload. If an external tool such as Samba or an administrator removes the payload, 87/16 must not return the stale sidecar and should remove the JSON. The server log should contain a greppable line like WARN SALVAGE 87/16 STALE ... for this cleanup.
• Scan, recover, and purge should share the same scan/sequence/basehandle view so that a sequence returned by scan identifies the exact sidecar used later.
• The combined salvage smoke suite now covers NCP write/read payloads, 87/18 purge pre-clean, hidden backend repository behavior, stale sidecar cleanup with a manual payload-removal pause, three version captures, and recovering the oldest version via sequence 0.
• Append salvage endpoint tests to tests/salvage/salvage_smoke_suite.sh rather than creating unrelated top-level scripts, unless a helper binary is needed and then started by the suite.

AFP 0x13 deleted-file info notes

• AFP 0x13 Get Macintosh Info On Deleted File is NCP 0x2222 / 35 / 19 (wire subfunction byte 0x13). The Micro Focus / Novell WebSDK request is VolumeNumber plus DOSDirectoryNumber; the reply is FinderInfo[32], ProDOSInfo[6], ResourceForkSize, FileNameLen, FileName.
• Implement it only as an adapter over the shared mars_nwe salvage/deleted-entry record. Do not expose or normally open .recycle or .salvage through AFP code; those remain hidden backend repositories.
• The implementation returns FinderInfo[32], ProDOSInfo[6], resource fork size, and deleted original name from the Salvage JSON snapshot. FinderInfo and ProDOSInfo are captured through the existing nwatalk xattr-backed AFP metadata store, not through a parallel AFP metadata database.
• The AFP smoke suite has a dedicated afp_deleted_info_smoke helper. It pre-cleans salvage entries in the tested directory through NCP purge, creates a temporary AFP file, writes FinderInfo and ProDOSInfo, deletes it, verifies AFP 0x13, and purges the tested deleted entry afterwards.
• Verified AFP smoke status: the full suite completed with failures=0 after AFP 35/19 and ProDOSInfo work. It verifies live FinderInfo and ProDOSInfo xattrs on SYS:PUBLIC/pmdflts.ini, verifies AFP 35/19 returns prodos=010203040506 from the deleted-file Salvage snapshot, and leaves normal AFP-only attributes absent when Hidden/System/Archive map through the NetWare attribute path.
• Reuse existing AFP/nwatalk metadata mechanisms for FinderInfo, AFP attributes, entry ids, resource fork state, and related restore/lookup behavior. Do not add a parallel AFP metadata database.

Logging rules

Desired future server log format:

<LVL4> <AREA> <DEC-CODE> <EVENT> key=value ...

• LVL4 is exactly four characters: INFO, DBUG, WARN, ERRR.

• AREA examples: NCP, SALVAGE, AFP, MAP, BIND, TRUST, AUTH, CONN, FILE, QUEUE.

• The front code should be human/protocol decimal where applicable, for example 87/16, 87/17, 87/18.

• Exact wire values should still be logged later as key/value hex fields, for example fn=0x57 sub=0x10 seq=0x00000000 base=0x00000004 result=0x89ff.

• Unknown or unimplemented endpoints should be easy to grep, for example:

  INFO NCP 87/18 UNKNOWN fn=0x57 sub=0x12 msg="not implemented"
  INFO NCP 87/255 UNKNOWN fn=0x57 sub=0xff msg="unknown subfunction"
  INFO NCP 136 UNKNOWN fn=0x88 msg="unknown function"
  

• Do not invent a parallel logger casually. Reuse existing mars_nwe logging functions/macros and normalize message format gradually.

Build and test notes

Dependencies used during local checks in this conversation:

• gdbm-1.26.tar.gz
• Linux-PAM-1.7.2.tar.xz for PAM headers; link against system PAM if present
• ncpfs-master.zip for the salvage smoke helper client build
• yyjson under third_party/yyjson

If CMake finds GDBM but a target still cannot see gdbm.h, pass include paths explicitly for local verification, for example:

CFLAGS="-I/path/to/gdbm/include -I/path/to/Linux-PAM-1.7.2/libpam/include" \
cmake -S . -B build
cmake --build build --target nwconn ncp_salvage_scan_smoke ncp_salvage_recover_smoke

Useful quick checks:

bash -n tests/salvage/salvage_smoke_suite.sh
cc -DLINUX -fsyntax-only -Iinclude -Isrc -Ithird_party/yyjson/src src/nwsalvage.c src/namspace.c

When server-side code or smoke helper clients change, rebuild both the server and the helper targets so the runtime test is not using stale binaries:

cmake --build build --target nwserv ncpserv
cmake --build build --target \
  ncp_delete_smoke \
  ncp_read_smoke \
  ncp_salvage_scan_smoke \
  ncp_salvage_recover_smoke \
  ncp_salvage_purge_smoke \
  afp_entry_id_smoke \
  afp_file_info_smoke \
  afp_scan_info_smoke \
  afp_set_file_info_smoke \
  afp_deleted_info_smoke

Runtime smoke suites:

tests/salvage/salvage_smoke_suite.sh --out /tmp/mars-salvage-report.txt
tests/afp/afp_smoke_suite.sh --out /tmp/mars-afp-smoke.txt

The suite streams the report to --out while running, so a failure before the end should still leave useful output. It has a manual stale-payload pause: the script prints a sudo rm -f .../.recycle/... command; remove that payload in a second shell and press Enter. The next scan should remove the stale sidecar and grep /var/log/mars_nwe/nw.log for WARN SALVAGE 87/16 STALE.

Normal NCP reads of .recycle or .salvage are expected to fail with invalid path. Verify payload data through the visible live file after NCP write or recover, using ncp_read_smoke. Treat the final summary (failures=0, ncp_warnings=0) as the important signal.

AFP ProDOSInfo storage

ProDOSInfo is AFP/NCP per-entry metadata. Store it in the existing nwatalk AFP metadata layer, not in nwarchive/nwxattr directly and not in a parallel DB. The xattr key is user.org.mars-nwe.afp.prodos-info via the mars_nwe xattr wrapper name org.mars-nwe.afp.prodos-info; it is a raw 6-byte value, analogous to FinderInfo's 32-byte org.mars-nwe.afp.finder-info.

Salvage captures this as prodos_info_hex (12 hex characters) beside finder_info_hex. AFP 35/19 Get Macintosh Info On Deleted File returns FinderInfo[32] followed by ProDOSInfo[6] from the Salvage snapshot. The verified smoke value is 010203040506 and the Linux xattr dump should show:

user.org.mars-nwe.afp.prodos-info=0x010203040506