diff --git a/TODO.md b/TODO.md index 391d60e..33131a2 100644 --- a/TODO.md +++ b/TODO.md @@ -187,6 +187,33 @@ Follow-up: the WebSDK TTS calls include begin/end/abort transaction, status, threshold, and control/statistics operations. +### Salvage / deleted-entry backend + +Current status: + +- The shared salvage backend lives in `src/nwsalvage.c` / + `include/nwsalvage.h` and is hooked into the normal server delete path. +- Deleted payloads are moved to `.recycle` and per-object JSON sidecars are + written below `.salvage`. +- The sidecar schema preserves NetWare archive/fileinfo xattrs, AFP metadata + when present, inherited rights mask, and explicit trustee entries. +- The runtime configuration uses low `nw.ini` sections `51-59`. Section `53` + maps Samba-style behaviour flags (`k`, `v`, `t`, `m`), and sections + `55-58` cover size filters, exclude, exclude_dir, and noversions lists. +- NCP delete and Samba-compatible history naming are covered by smoke tests in + `tests/salvage/`. + +Follow-up: + +- Implement the normal NetWare salvage family on top of the shared backend: + - `NCP 0x2222 / 87 / 16` - Scan Salvageable Files + - `NCP 0x2222 / 87 / 17` - Recover Salvageable File + - `NCP 0x2222 / 87 / 18` - Purge Salvageable File + - optional legacy `NCP 0x2222 / 22 / 27` - Scan Salvageable File (old) +- Add endpoint-level tests for scan/recover/purge under `tests/salvage/`. +- Add cleanup coverage for stale `.salvage` JSON when the matching `.recycle` + payload has been manually removed. + ### AFP / Mac namespace backend Current status: @@ -195,19 +222,15 @@ Current status: tests under `tests/afp/`. Endpoint inventory, WebSDK audit notes, and AFP implementation history live in that directory instead of this project-level TODO file. -- The only AFP endpoint still tracked here is `0x13 AFP Get Macintosh Info On - Deleted Files`. It depends on a real mars_nwe salvage/deleted-entry backend - and must not be implemented as an AFP-local deleted-file scan. +- AFP `0x13 Get Macintosh Info On Deleted Files` remains unsupported until the + NCP salvage scan/recover/purge endpoints expose the shared deleted-entry + backend semantics. Follow-up: -- Implement or verify the normal NetWare salvage family first: - - `NCP 0x2222 / 87 / 16` - Scan Salvageable Files - - `NCP 0x2222 / 87 / 17` - Recover Salvageable File - - `NCP 0x2222 / 87 / 18` - Purge Salvageable File - - optional legacy `NCP 0x2222 / 22 / 27` - Scan Salvageable File (old) -- After that backend exists, implement AFP `0x13` as an adapter over the - mars_nwe salvage/deleted-entry record. +- After the NetWare salvage endpoints exist, implement AFP `0x13` as an adapter + over the mars_nwe salvage/deleted-entry record. +- AFP must not scan `.recycle` or `.salvage` directly. - Keep the detailed AFP TODO, inventory, and audit notes in `tests/afp/`. ## Deferred / optional protocol work diff --git a/tests/README.md b/tests/README.md new file mode 100644 index 0000000..f87c2e8 --- /dev/null +++ b/tests/README.md @@ -0,0 +1,29 @@ +# mars_nwe test helpers + +The test tree contains small smoke helpers for protocol areas that are easier to +exercise outside the normal install flow. + +## AFP + +`tests/afp` contains the current AFP endpoint smoke suite and AFP-specific +notes. AFP deleted-file endpoint `0x13` is intentionally still unsupported +until the shared salvage backend exposes scan/recover/purge semantics. + +## Salvage + +`tests/salvage` contains the shared NetWare salvage coverage. These tests are +kept outside `tests/afp` because AFP `0x13` must become a thin adapter over the +same backend that serves the NetWare NCP salvage calls. + +The current salvage tests cover: + +- local layout contract for `.recycle` payloads and `.salvage` JSON metadata, +- NCP create/delete capture through the normal server delete path, +- Samba-compatible history naming with `Copy #1 of NAME`, +- report-file generation with `--out FILE`. + +The NCP smoke scripts are intended to run as the same Unix user that normally +runs the test client, not necessarily as root. Pre-clean of old `.recycle` or +`.salvage` artifacts is therefore best-effort: permission failures are reported +as warnings and do not by themselves fail the smoke. The actual pass/fail check +is based on artifacts created by the NCP delete path. diff --git a/tests/salvage/README.md b/tests/salvage/README.md index 3bd7306..0f16c13 100644 --- a/tests/salvage/README.md +++ b/tests/salvage/README.md @@ -1,103 +1,134 @@ # Salvage smoke tests -This directory is reserved for NetWare salvage/deleted-entry coverage. The -backend is intentionally shared by NCP salvage calls and the later AFP `0x13` +This directory contains the shared mars_nwe NetWare salvage/deleted-entry tests. +The backend is shared by NetWare NCP salvage calls and the later AFP `0x13` `Get Macintosh Info On Deleted Files` adapter; AFP must not grow a parallel local deleted-file scanner. -The first executable coverage is layout-only. It records the repository -contract before `src/nwsalvage.c` exists: +## Configuration covered by the tests -- deleted file payloads live below the recycle repository, -- per-object JSON metadata lives below the salvage repository, -- JSON metadata preserves mars_nwe xattr-backed NetWare/AFP fields - from `nwarchive` and `nwatalk`, -- there is no large per-directory index file, -- stale JSON metadata is detectable when the matching recycle payload is gone. - -Default repository names come from the `nwserv.conf`/`nw.ini` template options: +The active salvage configuration block is intentionally in a low `nwserv.conf` / +`nw.ini` range: ```text 51 1 52 .recycle .salvage 53 kv +54 0700 0700 55 0 0 56 - 57 - 58 - +59 - ``` -Section `53` is a compact behaviour-flag string inspired by Samba -`vfs_recycle` options. Flags can be combined in one token: `k` = keeptree, -`v` = versions, `t` = touch, and `m` = touch_mtime. Section `55` -contains min/max file sizes and accepts raw bytes or case-insensitive `kb`, -`mb`, and `gb` suffixes. Sections `56`, `57`, and `58` mirror Samba's -`exclude`, `exclude_dir`, and `noversions` pattern lists. +Section meanings: -Expected layout for a deleted `SYS:PUBLIC/PMDFLTS.INI` owned by `SUPERVISOR`. -The sidecar JSON must carry the server-only metadata needed for exact recover, -including NetWare archive/fileinfo xattrs and AFP FinderInfo/entry-id/attribute -xattrs: +- `51` enables/disables salvage. +- `52` names the recycle payload repository and the `.salvage` metadata + repository. +- `53` is a compact Samba-`vfs_recycle`-style behaviour flag string: + - `k` = keeptree, + - `v` = versions, + - `t` = touch, + - `m` = touch_mtime. +- `54` reserves directory/subdirectory modes for generated repositories. +- `55` is `minsize maxsize`. Values may be raw bytes or case-insensitive + `kb`, `mb`, or `gb` units, for example `55 1kb 100MB`. +- `56` is the `exclude` pattern list. Matching files are deleted directly. +- `57` is the `exclude_dir` pattern list. Matching directories bypass salvage. +- `58` is the `noversions` pattern list. Matching files are recycled, but old + recycle names are replaced instead of getting `Copy #x of NAME` history. +- `59` is reserved for cleanup/history policy. + +## Repository layout + +With `k`/keeptree enabled, a deleted `SYS:PUBLIC/PMDFLTS.INI` owned by +`SUPERVISOR` is expected to produce: ```text SYS/.recycle/SUPERVISOR/PUBLIC/PMDFLTS.INI SYS/.salvage/SUPERVISOR/PUBLIC/PMDFLTS.INI.json ``` -Future runtime tests should add direct coverage for: +The `.recycle` tree contains deleted payload files. The `.salvage` tree +contains one JSON sidecar per deleted object. There is no large per-directory +index file. -- `NCP 0x2222 / 22 / 27` Scan Salvageable Files (old), -- `NCP 0x2222 / 87 / 16` Scan Salvageable Files, -- `NCP 0x2222 / 87 / 17` Recover Salvageable File, -- `NCP 0x2222 / 87 / 18` Purge Salvageable File, -- AFP `0x13` as a thin adapter over the shared salvage backend. +The sidecar JSON must preserve mars_nwe server metadata needed for exact +recover, including: -Trustee metadata ----------------- +- NetWare archive/fileinfo xattrs from `nwarchive`, +- AFP metadata from `nwatalk` when present, +- inherited rights mask and explicit trustee object/right pairs, +- selected recycle/salvage relative paths. -NetWare trustee rights are stored by mars_nwe in the trustee store, keyed by -volume/dev/inode, rather than in the file payload itself. Salvage metadata -therefore needs to preserve the inherited rights mask and the explicit trustee -object/right pairs so recovery can recreate equivalent trustee records for the -restored object. +`finder_info_hex` should only be present when FinderInfo metadata actually +exists. Missing FinderInfo must not be represented as synthetic all-zero data. -## NCP delete capture smoke +## Samba-compatible version naming -`salvage_ncp_delete_smoke.sh` is the integration check for the first real -server-side salvage hook. It uses `ncp_delete_smoke`, a small libncp client, to -create and delete a file through the classic NetWare NCP file functions: - -- `0x43` create file -- `0x42` close file -- `0x44` delete file - -The script does not call local `rm`/`unlink` for the tested live path. Local -filesystem access is used only after the NCP delete to inspect the expected -recycle payload and JSON sidecar: +With the `v` flag enabled, mars_nwe follows Samba `vfs_recycle` naming on +collision: ```text SYS/.recycle/SUPERVISOR/PUBLIC/SLVGCHK.TXT -SYS/.salvage/SUPERVISOR/PUBLIC/SLVGCHK.TXT.json +SYS/.recycle/SUPERVISOR/PUBLIC/Copy #1 of SLVGCHK.TXT +SYS/.recycle/SUPERVISOR/PUBLIC/Copy #2 of SLVGCHK.TXT ``` +The `.salvage` sidecar uses the same selected payload name plus `.json`: + +```text +SYS/.salvage/SUPERVISOR/PUBLIC/SLVGCHK.TXT.json +SYS/.salvage/SUPERVISOR/PUBLIC/Copy #1 of SLVGCHK.TXT.json +``` + +If `v` is not enabled, or a file matches the `noversions` list, the old recycle +payload/metadata is replaced instead of creating a `Copy #x of ...` history +entry. + +## Layout smoke + +`salvage_layout_smoke.sh` is a local filesystem-only contract test. It does not +require a running mars_nwe server and does not use NCP. + +```sh +./tests/salvage/salvage_layout_smoke.sh +``` + +It verifies the basic `.recycle`/`.salvage` directory contract and stale JSON +detection. + +## NCP delete capture smoke + +`salvage_ncp_delete_smoke.sh` is the integration check for the server-side +delete hook. It uses `ncp_delete_smoke`, a small libncp client, to create and +delete a file through classic NetWare NCP file functions. The script does not +call local `rm`/`unlink` for the tested live path; local filesystem access is +used only after the NCP delete to inspect the expected recycle payload and JSON +sidecar. + Example: ```sh ./tests/salvage/salvage_ncp_delete_smoke.sh \ -S MARS -U SUPERVISOR -P secret \ --path SYS:PUBLIC/SLVGCHK.TXT \ - --unix-path /var/mars_nwe/SYS/public/SLVGCHK.TXT + --unix-path /var/mars_nwe/SYS/public/SLVGCHK.TXT \ + --volume-root /var/mars_nwe/SYS \ + --out /tmp/mars-salvage-report.txt ``` -This check lives under `tests/salvage` rather than `tests/afp` because later -salvage scan/recover/purge and versioning tests should share the same fixture. -AFP can use the same backend later as an adapter for `0x13`. +The script can run as a normal user. Best-effort pre-clean of stale test +artifacts may warn when existing `.recycle`/`.salvage` files are owned by the +server user or root; those warnings do not fail the smoke by themselves. ## NCP history/version smoke -`salvage_ncp_history_smoke.sh` deletes the same NetWare path twice through -NCP and verifies Samba-compatible version naming. With the `v` flag enabled, -the first recycled payload keeps its original name and the second is written as +`salvage_ncp_history_smoke.sh` deletes the same NetWare path twice through NCP +and verifies Samba-compatible version naming. With the `v` flag enabled, the +first recycled payload keeps its original name and the second is written as `Copy #1 of NAME` in the same recycle directory. The `.salvage` sidecar uses the same selected payload name with a `.json` suffix. @@ -111,3 +142,17 @@ Example: --volume-root /var/mars_nwe/SYS \ --out /tmp/mars-salvage-history-report.txt ``` + +Like the delete smoke, stale artifact cleanup is best-effort and warning-only. +This matters when the script is run as a normal user while `.recycle` and +`.salvage` entries are owned by the mars_nwe server account. + +## Still to implement + +Runtime endpoint tests should be added as the NCP salvage calls are implemented: + +- `NCP 0x2222 / 22 / 27` Scan Salvageable Files (old), +- `NCP 0x2222 / 87 / 16` Scan Salvageable Files, +- `NCP 0x2222 / 87 / 17` Recover Salvageable File, +- `NCP 0x2222 / 87 / 18` Purge Salvageable File, +- AFP `0x13` as a thin adapter over the shared salvage backend. diff --git a/tests/salvage/SALVAGE_BACKEND_DESIGN.md b/tests/salvage/SALVAGE_BACKEND_DESIGN.md index 713d212..18ba3e3 100644 --- a/tests/salvage/SALVAGE_BACKEND_DESIGN.md +++ b/tests/salvage/SALVAGE_BACKEND_DESIGN.md @@ -33,14 +33,21 @@ SYS/.recycle/SUPERVISOR/PUBLIC/PMDFLTS.INI SYS/.salvage/SUPERVISOR/PUBLIC/PMDFLTS.INI.json ``` -When a name collides, mars_nwe should version the recycle name in a -Samba-like way and use the same versioned name for the sidecar JSON: +When a name collides and the `versions` behaviour is enabled, mars_nwe uses +Samba `vfs_recycle` compatible names and the same selected name for the sidecar +JSON: ```text -SYS/.recycle/SUPERVISOR/PUBLIC/PMDFLTS.~1~.INI -SYS/.salvage/SUPERVISOR/PUBLIC/PMDFLTS.~1~.INI.json +SYS/.recycle/SUPERVISOR/PUBLIC/PMDFLTS.INI +SYS/.recycle/SUPERVISOR/PUBLIC/Copy #1 of PMDFLTS.INI +SYS/.salvage/SUPERVISOR/PUBLIC/PMDFLTS.INI.json +SYS/.salvage/SUPERVISOR/PUBLIC/Copy #1 of PMDFLTS.INI.json ``` +If versioning is disabled, or a file matches the `noversions` list, the previous +recycled payload/metadata is replaced instead of creating a `Copy #x of ...` +entry. + The `.recycle` tree contains the actual deleted file. The `.salvage` tree contains only mars_nwe metadata and indexes. This keeps the recycle area usable for administrators and compatible with Samba-style recycle setups, while @@ -94,7 +101,6 @@ Minimal example: "ctime": 1700000000, "backup_time": 1700000000, - "finder_info_hex": "0000000000000000000000000000000000000000000000000000000000000000", "afp_entry_id": "0x2cc1243d", "resource_fork_size": 0 } @@ -102,6 +108,9 @@ Minimal example: The JSON should store paths relative to the volume root wherever possible. That makes the records easier to move with the volume and easier to read manually. +Optional AFP/Finder fields should only be present when the corresponding +mars_nwe-xattr metadata exists; missing FinderInfo must not be serialized as +synthetic all-zero data. ## Cleanup of stale metadata @@ -196,27 +205,38 @@ Create a separate test group first: tests/salvage/ ``` -Initial tests should cover the backend without AFP: +Initial tests cover the backend without AFP: -- create sample file, -- delete through the salvage helper path, +- local layout contract, +- create/delete through the normal NCP server path, - verify the file is moved to `.recycle`, - verify the JSON exists in `.salvage`, -- verify cleanup removes stale JSON when the recycle file is deleted, -- verify recover restores the file, -- verify purge removes both file and JSON. +- verify Samba-compatible `Copy #x of NAME` history naming. + +Follow-up tests should cover: + +- cleanup removes stale JSON when the recycle file is deleted, +- recover restores the file, +- purge removes both file and JSON, +- scan endpoints enumerate the JSON records in NetWare-compatible order. Only after those tests are stable should `tests/afp` gain an AFP `0x13` smoke. ## Configuration direction -The first implementation may use fixed per-volume defaults: +The active low-numbered configuration block is: ```text -recycle repository: .recycle -salvage metadata: .salvage +51 1 # enable salvage +52 .recycle .salvage # payload repository, metadata repository +53 kv # k=keeptree, v=versions, t=touch, m=touch_mtime +54 0700 0700 # repository/subdirectory modes +55 0 0 # minsize maxsize; bytes or kb/mb/gb suffixes +56 - # exclude patterns +57 - # exclude_dir patterns +58 - # noversions patterns +59 - # cleanup/history policy, reserved ``` -Later configuration can expose those names if needed. The default should remain -compatible with a Samba-style recycle layout and should keep `.salvage` as -metadata-only state. +The defaults should remain compatible with a Samba-style recycle layout and +should keep `.salvage` as metadata-only state. diff --git a/tests/salvage/salvage_ncp_history_smoke.sh b/tests/salvage/salvage_ncp_history_smoke.sh index 7b7e302..b2feb5d 100755 --- a/tests/salvage/salvage_ncp_history_smoke.sh +++ b/tests/salvage/salvage_ncp_history_smoke.sh @@ -166,7 +166,16 @@ emit "first_metadata=$FIRST_META" emit "second_metadata=$SECOND_META" section "pre-clean test history artifacts" -rm -f -- "$FIRST_META" "$SECOND_META" "$FIRST_RECYCLE" "$SECOND_RECYCLE" 2>&1 | tee -a "$REPORT_TMP" +for artifact in "$FIRST_META" "$SECOND_META" "$FIRST_RECYCLE" "$SECOND_RECYCLE"; do + if [ -e "$artifact" ]; then + emit "rm '$artifact'" + if rm -f -- "$artifact" 2>>"$REPORT_TMP"; then + : + else + emit "warning: could not remove stale test artifact: $artifact" + fi + fi +done run_delete "NCP create/delete #1" run_delete "NCP create/delete #2"