SAN transport support

README.md

@@ -13,21 +13,30 @@ Python naming).
 
 VIM login and inventory use [pyVmomi](https://github.com/vmware/pyvmomi).
 The NFC ticket, ESXi authd handshake, and disk I/O were reverse-engineered
-from VDDK 8 NBD traffic; see `docs/`.
+from VDDK 8 NBD traffic; see `docs/`. Linux HotAdd uses the public
+vSphere `ReconfigureVM` API (see `docs/hotadd.md`). Linux SAN reads a
+shared VMFS LUN locally (see `docs/san.md`).
 
 ## Status
 
 Implemented against vCenter 8 / ESXi 8. Default transport is `nbdssl`
-(`nbd` is still available):
+(`nbd` is still available). Linux hosts that see the VMFS LUN can use
+`san`. Linux guests can also use `hotadd`:
 
 - `VixDiskLib_ConnectEx` (UID credentials)
 - `VixDiskLib_Open` (datastore path, read-only or read-write)
 - `VixDiskLib_Read` (optional ``skip_decompression`` packs FastLZ extras)
 - `VixDiskLib_Write`
+- HotAdd on a Linux VMware guest (SCSI, NVMe, or SATA source disks,
+  attached onto a proxy SCSI controller)
+- SAN on a Linux host that sees the same VMFS LUN as ESXi (NAA match,
+  local `pread` / `pwrite` of the flat extent)
 
 Not implemented: compression open flags other than FastLZ, CBT /
 allocated-block queries, disk geometry (`DDB_GET`), encrypted disks,
-and direct ESXi `ha-nfc` without vCenter `vpxa-nfc`.
+direct ESXi `ha-nfc` without vCenter `vpxa-nfc`, file transport,
+snapshot / SESPARSE SAN chains, Windows HotAdd, and HotAdd onto a
+proxy NVMe controller.
 
 Requires Python 3.10 or later.
 
@@ -73,6 +82,8 @@ VDDK-shaped handle.
 | `openvixdisklib/openvixdisklib.py` | Drop-in handle (`connect` / `open` / `read` / `write`) |
 | `openvixdisklib/nfc_auth.py`       | VIM login, NFC ticket, authd on 902                    |
 | `openvixdisklib/nfc_open.py`       | Classic NFC handshake, AIO open, sector read/write     |
+| `openvixdisklib/hotadd.py`         | Linux-guest SCSI HotAdd attach, local block I/O        |
+| `openvixdisklib/san.py`            | Linux SAN: NAA match, VMFS map, local block I/O        |
 | `openvixdisklib/fastlz.py`         | FastLZ NFC adapter (pip `pyfastlz`)                    |
 | `tests/integration/`               | Live pytest suite against a lab vCenter                |
 | `tests/perf/`                      | Throughput comparison of OpenVixDiskLib vs VDDK        |
@@ -96,11 +107,20 @@ password: secret
 allow_untrusted: true
 datacenter: Datacenter
 datastore: datastore0
+hotadd_proxy:
+  host: hotadd-proxy.example.com
+  user: root
+iscsi_san:
+  portal: 192.0.2.10
 ```
 
 A session-scoped pytest fixture creates an empty VM with a 10 GiB thin
 disk on that datastore and tears it down when the session ends. Tests
-write known patterns and read them back.
+write known patterns and read them back. HotAdd tests SSH into
+`hotadd_proxy` (a Linux guest on the same datastore) and skip if SSH
+fails. SAN tests bring up a loop-backed iSCSI LUN, create a VMFS
+datastore, and skip if LIO, `iscsiadm`, or software iSCSI cannot be
+used. The session lab VM is not placed on that LUN.
 
 ```bash
 tox -e integration
@@ -117,7 +137,10 @@ tox -e integration -- --runslow
 Compare write/read throughput of OpenVixDiskLib and native VDDK
 (`64KiB`, 129-sector, and `32MiB` transfers; `nbdssl` and `nbd`;
 plain, FastLZ, and OpenVixDiskLib FastLZ ``skip_decompression``;
-AIO sessions 64 KiB×1, 1 MiB×1, 2 MiB×1, and 2 MiB×4).
+AIO sessions 64 KiB×1, 1 MiB×1, 2 MiB×1, and 2 MiB×4). The same sizes
+are also timed over Linux-guest ``hotadd`` (plain OpenVixDiskLib I/O
+on `hotadd_proxy`; FastLZ and NFC AIO do not apply) and skipped if
+SSH to the proxy fails.
 
 ```bash
 tox -e perf
@@ -145,5 +168,7 @@ Lint and typecheck: `tox -e pep8`, `tox -e mypy`.
 | `docs/nfc_open.md`                      | Classic NFC and AIO open         |
 | `docs/nfc_read.md`                      | AIO IO / `VixDiskLib_Read`       |
 | `docs/nfc_write.md`                     | AIO IO / `VixDiskLib_Write`      |
+| `docs/hotadd.md`                        | Linux-guest SCSI HotAdd          |
+| `docs/san.md`                           | Linux SAN / VMFS LUN I/O         |
 | `docs/ssl_hook.md`                      | TLS intercept used for capture   |
 | `docs/reverse_engineering_procedure.md` | How the protocol was recovered   |

docs/hotadd.md

@@ -0,0 +1,81 @@
+# HotAdd transport
+
+OpenVixDiskLib can SCSI-HotAdd a VMDK onto the Linux guest that is
+running the library, then read and write it as a local block device.
+This is not an NFC protocol: it uses public VIM `ReconfigureVM` plus
+guest SCSI I/O. There is no VixTransport linked clone and no VMDK
+parser; ESXi presents a single SCSI LUN.
+
+NBD and NBDSSL remain the default. `transport_modes=None` is still
+`nbdssl`. `hotadd` is advertised and selected only when the process is
+a VMware guest (`/sys/class/dmi/id/sys_vendor`).
+
+## Mapping from VDDK
+
+| VDDK behaviour | OpenVixDiskLib |
+| -------------- | -------------- |
+| Run inside a proxy VM | Same. DMI UUID is matched to `config.uuid`. |
+| SCSI HotAdd of the source VMDK | `ReconfigureVM` add of an existing backing onto a **SCSI** controller on the proxy |
+| Linked clone via VixTransport | Not implemented. The snapshot or base VMDK is attached directly. |
+| Open as a whole-disk VMDK | Open `/dev/sdX` with `pread` / `pwrite` |
+| IDE disks | Not supported (same as VDDK) |
+| NVMe / SATA source disks | Supported. The backing file is attached onto proxy SCSI; the guest sees `/dev/sdX`, not `/dev/nvme*`. |
+| HotAdd onto a proxy NVMe controller | Not implemented |
+
+Colon lists such as `file:san:hotadd:nbdssl:nbd` pick the first **usable**
+mode. On a bare-metal host that is `nbdssl`. Inside a guest it is
+`hotadd`. `"hotadd"` alone on bare metal raises `NotImplementedError`.
+
+## Attach and detach
+
+1. Find this guest in vCenter (`SearchIndex.FindByUuid`).
+2. Resolve `disk_path` on the source VM. SCSI, NVMe
+   (`VirtualNVMEController`), and SATA (`VirtualAHCIController`) are
+   accepted. IDE and RDM are rejected. A powered-on source VM requires
+   `snapshot_ref`; a powered-off VM may attach the base disk.
+3. Add the existing VMDK to a free SCSI unit on the proxy (unit 7 is
+   skipped). If every unit is taken, a PVSCSI controller is added.
+   Read-only opens use `independent_nonpersistent` (redo log, source
+   stays clean). Writable opens use `persistent`.
+4. Rescan SCSI hosts and wait for the device. Matching prefers sysfs
+   `bus:0:unit:0`, then `*:0:unit:0` when `unit != 0`.
+5. `close` detaches with `Operation.remove` and **no** `fileOperation`.
+   The source VMDK must not be deleted. Leftover attachments of the
+   same backing are detached before a new open.
+
+Never HotAdd the proxy's own boot disk. Never use “newest `sdX`” as the
+only match when a unique SCSI address exists.
+
+Do not remove the source VM or its snapshot while the disk is still
+attached. Independent-nonpersistent attaches create a redo log on the
+source datastore; detach is what cleans it up.
+
+## API
+
+`VixDiskLibHandle.connect(..., transport_modes="hotadd")` then
+`open` / `read` / `write` / `close` as for NBD. Compression open flags
+and NFC `skip_decompression` do not apply; FastLZ flags on a HotAdd
+open raise `NotImplementedError`. `readinto` returns a `ReadResult`
+with empty `fragments`.
+
+Implementation: `openvixdisklib.hotadd`.
+
+## Lab
+
+Live tests SSH into a Linux proxy that shares the lab datastore and
+run `tests/integration/hotadd_remote.py` there. Configure
+`.test_config.yaml`:
+
+```yaml
+hotadd_proxy:
+  host: hotadd-proxy.example.com
+  user: root
+  # identity_file: /home/user/.ssh/id_ed25519
+```
+
+Tests skip when SSH is unavailable. The session lab VM (PVSCSI) and a
+function-scoped NVMe VM are HotAdded onto the proxy, written, and
+checked again over `nbdssl` from the runner. `tox -e perf` times the
+same transfer sizes over HotAdd (plain I/O; FastLZ and NFC AIO do not
+apply). Dependencies on the proxy are installed into
+`/tmp/openvixdisklib-hotadd/.venv`, not the system Python.

docs/probing_samples/vddk_san_trace.py

@@ -0,0 +1,156 @@
+#!/usr/bin/env python3
+"""Trace native VDDK SAN open/read (device matching + VMFS, not NFC).
+
+SAN is not a wire protocol. Capture with::
+
+    strace -f -e openat,pread64,pwrite64,ioctl -o /tmp/vddk-san.strace \\
+        python3 docs/probing_samples/vddk_san_trace.py
+
+``InitEx`` must pass a libDir that contains ``lib64/libdiskLibPlugin.so``
+(the advanced transport plugin). VDDK then matches the VMFS LUN by NAA
+and reads the flat extent through its VMFS driver.
+
+Replace the ``<sanitized>`` fields with lab values. Do not commit
+credentials or live IPs.
+"""
+
+import ctypes
+import os
+
+REPO = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
+VDDK_DIR = os.path.join(REPO, ".vddk")
+lib = ctypes.CDLL(os.path.join(VDDK_DIR, "libvixDiskLib.so"))
+
+VIXDISKLIB_CRED_UID = 1
+VIXDISKLIB_FLAG_OPEN_READ_ONLY = 4
+SECTOR = 512
+
+
+class VixDiskLibUidPasswdCreds(ctypes.Structure):
+    _fields_ = [
+        ("userName", ctypes.c_char_p),
+        ("password", ctypes.c_char_p),
+    ]
+
+
+class VixDiskLibSessionIdCreds(ctypes.Structure):
+    _fields_ = [
+        ("cookie", ctypes.c_char_p),
+        ("userName", ctypes.c_char_p),
+        ("key", ctypes.c_char_p),
+    ]
+
+
+class VixDiskLibCreds(ctypes.Union):
+    _fields_ = [
+        ("uid", VixDiskLibUidPasswdCreds),
+        ("sessionId", VixDiskLibSessionIdCreds),
+    ]
+
+
+class VixDiskLibConnectParams(ctypes.Structure):
+    _fields_ = [
+        ("vmxSpec", ctypes.c_char_p),
+        ("serverName", ctypes.c_char_p),
+        ("thumbPrint", ctypes.c_char_p),
+        ("privateUse", ctypes.c_longlong),
+        ("credType", ctypes.c_uint32),
+        ("creds", VixDiskLibCreds),
+        ("port", ctypes.c_uint32),
+        ("nfcHostPort", ctypes.c_uint32),
+        ("vimApiVer", ctypes.c_char_p),
+    ]
+
+
+def check(err, what):
+    if err != 0:
+        lib.VixDiskLib_GetErrorText.restype = ctypes.c_void_p
+        lib.VixDiskLib_GetErrorText.argtypes = [ctypes.c_uint64, ctypes.c_char_p]
+        msg = lib.VixDiskLib_GetErrorText(err, None)
+        text = ctypes.cast(msg, ctypes.c_char_p).value
+        lib.VixDiskLib_FreeErrorText.argtypes = [ctypes.c_char_p]
+        lib.VixDiskLib_FreeErrorText(ctypes.cast(msg, ctypes.c_char_p))
+        raise SystemExit(f"{what} failed: {err} {text}")
+
+
+def main():
+    os.makedirs("/tmp/vddk-san-trace", exist_ok=True)
+    config_path = "/tmp/vddk-san-trace/vddk.config"
+    with open(config_path, "w") as f:
+        f.write("tmpDirectory=/tmp/vddk-san-trace\n")
+        f.write("log.fileName=/tmp/vddk-san-trace/vddk.log\n")
+        f.write("log.fileLevel=verbose\n")
+        f.write("vixDiskLib.transport.LogLevel=4\n")
+
+    plugin = os.path.join(VDDK_DIR, "lib64", "libdiskLibPlugin.so")
+    if not os.path.isfile(plugin):
+        raise SystemExit(
+            f"missing {plugin}; SAN needs libdiskLibPlugin under libDir/lib64"
+        )
+
+    lib.VixDiskLib_InitEx.argtypes = [
+        ctypes.c_uint32, ctypes.c_uint32, ctypes.c_void_p, ctypes.c_void_p,
+        ctypes.c_void_p, ctypes.c_char_p, ctypes.c_char_p]
+    lib.VixDiskLib_InitEx.restype = ctypes.c_uint64
+    check(lib.VixDiskLib_InitEx(
+        8, 0, None, None, None, VDDK_DIR.encode(), config_path.encode()),
+        "InitEx")
+
+    lib.VixDiskLib_ListTransportModes.restype = ctypes.c_char_p
+    print("ListTransportModes", lib.VixDiskLib_ListTransportModes(), flush=True)
+
+    params = VixDiskLibConnectParams()
+    params.vmxSpec = b"<sanitized>"
+    params.serverName = b"<sanitized>"
+    params.thumbPrint = b"<sanitized>"
+    params.credType = VIXDISKLIB_CRED_UID
+    params.creds.uid.userName = b"<sanitized>"
+    params.creds.uid.password = b"<sanitized>"
+    params.port = 443
+
+    lib.VixDiskLib_ConnectEx.argtypes = [
+        ctypes.POINTER(VixDiskLibConnectParams), ctypes.c_char,
+        ctypes.c_char_p, ctypes.c_char_p, ctypes.POINTER(ctypes.c_void_p)]
+    lib.VixDiskLib_ConnectEx.restype = ctypes.c_uint64
+    conn = ctypes.c_void_p()
+    check(lib.VixDiskLib_ConnectEx(
+        params, True, None, b"san", ctypes.byref(conn)),
+        "ConnectEx")
+    print("ConnectEx ok", flush=True)
+
+    lib.VixDiskLib_Open.argtypes = [
+        ctypes.c_void_p, ctypes.c_char_p, ctypes.c_uint32,
+        ctypes.POINTER(ctypes.c_void_p)]
+    lib.VixDiskLib_Open.restype = ctypes.c_uint64
+    disk = ctypes.c_void_p()
+    path = b"[ovdl-iscsi-<id>] <vm>/<vm>.vmdk"
+    check(lib.VixDiskLib_Open(conn, path, VIXDISKLIB_FLAG_OPEN_READ_ONLY, ctypes.byref(disk)),
+          "Open")
+    print("Open ok", flush=True)
+
+    lib.VixDiskLib_GetTransportMode.argtypes = [ctypes.c_void_p]
+    lib.VixDiskLib_GetTransportMode.restype = ctypes.c_char_p
+    print("transport", lib.VixDiskLib_GetTransportMode(disk), flush=True)
+
+    lib.VixDiskLib_Read.argtypes = [
+        ctypes.c_void_p, ctypes.c_uint64, ctypes.c_uint64, ctypes.c_char_p]
+    lib.VixDiskLib_Read.restype = ctypes.c_uint64
+    for start, n in ((0, 1), ((1024 * 1024 * 1024) // SECTOR, 1)):
+        buf = ctypes.create_string_buffer(n * SECTOR)
+        check(lib.VixDiskLib_Read(disk, start, n, buf), f"Read {start}+{n}")
+        print(
+            f"Read start={start} n={n} first16={buf.raw[:16].hex()}",
+            flush=True,
+        )
+
+    lib.VixDiskLib_Close.argtypes = [ctypes.c_void_p]
+    lib.VixDiskLib_Close.restype = ctypes.c_uint64
+    lib.VixDiskLib_Disconnect.argtypes = [ctypes.c_void_p]
+    lib.VixDiskLib_Disconnect.restype = ctypes.c_uint64
+    lib.VixDiskLib_Close(disk)
+    lib.VixDiskLib_Disconnect(conn)
+    lib.VixDiskLib_Exit()
+
+
+if __name__ == "__main__":
+    main()

docs/reverse_engineering_procedure.md

@@ -9,9 +9,12 @@ NFC work can follow the same loop instead of rediscovering it.
 
 Scope so far: `VixDiskLib_ConnectEx` + `VixDiskLib_Open` +
 `VixDiskLib_Read` + `VixDiskLib_Write` against lab vCenter 8.0.1 /
-ESXi 8, transports `nbd` and `nbdssl`. Validation method:
-`tests/integration/` (the session-scoped `lab` fixture creates a temporary
-empty VM with a 10 GiB disk and destroys it when the pytest session ends).
+ESXi 8, transports `nbd`, `nbdssl`, Linux-guest `hotadd`, and Linux
+`san`. Validation method: `tests/integration/` (the session-scoped `lab`
+fixture creates a temporary empty VM with a 10 GiB disk and destroys it
+when the pytest session ends). HotAdd live tests also SSH into a Linux
+proxy guest; see `docs/hotadd.md`. SAN live tests present a loop-backed
+iSCSI LUN; see `docs/san.md`.
 
 Rule from `AGENTS.md`: reuse pyVmomi for every public VIM operation.
 Only reimplement what pyVmomi does not expose.
@@ -61,6 +64,8 @@ names is in this table.
 | `strace -f -x` on `write` / `send*`      | First writable NFC capture without rebuilding the hook (Step 10)  | Noisy; TLS still opaque; `-s` truncates large extras               |
 | `pickle` of `LabEnv`                     | Create the temp VM unhooked, then load it under the hook          | `/tmp` only; never commit pickles (lab host and credentials)       |
 | ctypes drivers in `docs/probing_samples/` | Repeatable `ConnectEx` / `Open` / `Read` / `Write` under capture | Not library code                                                   |
+| in-kernel LIO + `losetup` / `iscsiadm`   | File-backed iSCSI LUN so ESXi and the runner share a NAA        | Not a VDDK protocol; lab-only (`tests/integration/iscsi_lab.py`)   |
+| `strace -e openat,pread64,pwrite64,ioctl` | Which `/dev/sd*` / `by-id` VDDK SAN opens and at which offsets  | No VMFS structure names; pair with `vixDiskLib.transport.LogLevel=4` |
 
 `ltrace` was considered for OpenSSL and libc `write`. It was not used:
 VDDK is stripped enough that `strace` on syscalls plus the `LD_PRELOAD`
@@ -409,3 +414,45 @@ Not yet reversed, same loop as above:
 - `VixDiskLib_GetInfo` capacity
 - Host-switch AIO messages
 - Direct ESXi `ha-nfc` without vCenter `vpxa-nfc`
+
+## HotAdd (not NFC)
+
+HotAdd does not use the capture loop above. VDDK SCSI-attaches the
+source VMDK to the proxy VM and opens a local whole disk. OpenVixDiskLib
+reuses pyVmomi `ReconfigureVM` for attach/detach and `pread`/`pwrite` on
+the Linux SCSI device. NVMe and SATA source disks are remapped onto a
+proxy SCSI controller. Details: `docs/hotadd.md`.
+
+## SAN (not NFC)
+
+SAN is also not a wire protocol. The backup host must see the **same
+SCSI LUN** ESXi uses for the VMFS datastore, match it by NAA, then read
+the VMDK data file through a VMFS driver. tcpdump of NFC is the wrong
+tool.
+
+Lab: a 20 GiB sparse file, `losetup`, in-kernel LIO iblock + iSCSI
+portal on the default-route IPv4, local `iscsiadm` login, pyVmomi
+`AddInternetScsiSendTargets` / `CreateVmfsDatastore`. Do not format or
+mount the LUN on Linux.
+
+Probe: `docs/probing_samples/vddk_san_trace.py` with
+`vixDiskLib.transport.LogLevel=4`. Native VDDK SAN needs
+`.vddk/lib64/libdiskLibPlugin.so`. Capture with
+`strace -e openat,pread64,pwrite64,ioctl`.
+
+Findings used by `openvixdisklib/san.py`:
+
+- GPT VMFS type GUID `2ae031aa-0f40-db11-9590-000c2911d1b8`, partition
+  LBA 2048 (1 MiB)
+- LVM magic `0xC001D00D` at partition + 1 MiB
+- FS magic `0x2fabf15e` version 24 at partition + 2 MiB or + 19 MiB
+- File descriptors (`fdmd`) hold 64-bit SFB/LFB pointers at the end of
+  a two-block descriptor. For large files the pointer array is in 64 KiB
+  sub-blocks. SFB
+  `((cluster * resourcesPerCluster) + resource) << fileBlockShift`
+  is relative to file-block 0, which sits after the LVM label and 16 ×
+  1 MiB heartbeats. Holes (address 0) read as zeros. Writes need an
+  allocated file block; SAN does not allocate.
+- First cut: powered-off persistent FlatVer2, no snapshot chain.
+
+Details: `docs/san.md`.