Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,18 @@
### Added

- Added `AnyStore`, a type-erased wrapper that selects any store backend (`FileStore`, `HeapStore`, or custom) at runtime and works with `TypedStore`.
- Added one path model shared by every filesystem adapter: a path is always relative to the adapter's root (`"/a"`, `"./a"` and `"a"` are the same file), and the base path says where that root is. See "Paths and base paths" in the user guide. Path resolution never allocates; its buffers are sized by the new `USTORE_MAX_PATH_LEN` (default 128) and `USTORE_MAX_BASEPATH_LEN` (default 32) macros.

### Changed

- **Behavior change (native `PosixFileSystem` / `StdioFileSystem` / `UniversalFileSystem`):** with the default (empty) base path, absolute paths now resolve under the current directory instead of the host's root, the same way relative paths already did. For example, `"/tmp/x"` now means `./tmp/x`. To use host-absolute paths, construct the adapter with `"/"` as the base path, e.g. `PosixFileSystem("/")`. Relative paths such as `"./kv/"` resolve exactly as before.
- **Behavior change (native `PosixFileSystem` / `StdioFileSystem`):** a non-empty base path is now honored. It was previously ignored, so every path now resolves under it.

### Fixed

- Fixed relative paths (`"./x"`) failing on the ESP32 `LittleFSFileSystem`, `SPIFFSFileSystem` and `SDFileSystem` adapters, whose Arduino FS layer only accepts paths that start with `/`. This also stops their `init()` write check (on `"./__init_test__"`) from failing, which reformatted the filesystem on every `init()` call when `reformatOnFail` was set (the default).
- Fixed the POSIX and stdio adapters ignoring their base path, which also left relative paths unresolvable on ESP32 with a non-empty mount point (#13).
- Fixed adapters keeping the caller's base-path pointer, which dangled if the string was temporary, and fixed the LittleFS mount point dangling after the adapter was destroyed, because the Arduino FS layer keeps that pointer (#13).
- Fixed `FileStore` compaction running with the active segment still open, which LittleFS and FatFS refuse to delete, and fixed the store being left without a writable segment (or with a stray out-of-range segment file) after a failed compaction (#6, #11).
- Fixed compaction recovery paths that could lose data: a power loss after the compacted segment was moved into place but before the journal was cleared deleted all data on the next boot; a failed rename while finalizing deleted all data and still reported success; recovery from a compaction interrupted mid-way kept a stale index, and a later compaction attempt could truncate the only copy of the interrupted compaction's records. A compaction interrupted by a write failure is now recovered immediately rather than at the next boot.

Expand Down
14 changes: 14 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,20 @@ microStore is a persistent key-value store for embedded systems, inspired by Bit

`File` accumulates a running CRC-32 on every read/write transparently; call `file.crc()` to retrieve it. To add a new platform, subclass `FileSystemImpl` and `FileImpl`.

### Paths

One path model for every adapter and platform, implemented in `include/microStore/Path.h`:

- **Paths are root-relative.** `/a`, `./a` and `a` name the same file under the adapter's root.
- **The base path is the root.** An empty base means the backend's default root: the cwd on native, the fallback mount on ESP32, and the filesystem root elsewhere. Native host-absolute paths need base `"/"`.
- **Adapters resolve every path argument** with `USTORE_RESOLVE_PATH(var, in, fail)`, which uses a stack buffer of `USTORE_MAX_PATH_LEN` and calls the adapter's private `resolve()`:
- ESP32 VFS and nRF52 adapters use `Path::normalize` (no copy when the path is already canonical); the framework adds the mount point.
- Posix/Stdio use `Path::join(base, p)`, with base `"."` when empty on native.
- **Keep the path code allocation-free:** no `std::string`.
- Base paths are copied into `char[USTORE_MAX_BASEPATH_LEN]` members with `Path::copy_base`.
- Mount points passed to `LittleFS.begin()` must come from `Path::persistent_mount()`, because Arduino's `FSImpl` keeps the raw pointer.
- **Don't resolve in the `FileSystem` wrapper:** the VFS adapters would get the mount point added twice.

### Platform Detection

- `PLATFORM_NATIVE`: uses `std::chrono` for timestamps
Expand Down
31 changes: 31 additions & 0 deletions docs/microstore_user_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,35 @@ Use it when the choice between persistent and volatile storage depends on runtim

Adding a new backend means subclassing `FileSystemImpl` and `FileImpl`. The `File` class accumulates a running CRC-32 on every read/write transparently — call `file.crc()` to retrieve it.

### 4.1 Paths and base paths

Every adapter, on every platform, treats paths the same way:

- **A path is always relative to the adapter's root.** `"/cfg"`, `"./cfg"` and `"cfg"` all name the same thing, `<root>/cfg`. Pick whichever spelling you like; the same string works unchanged on native hosts and on MCUs.
- **The base path says where the root is.** Leave it empty (the default) for the backend's natural root:

| Adapter | Root with the default (empty) base path | Root with base path `B` |
|---------|------------------------------------------|-------------------------|
| `PosixFileSystem`, `StdioFileSystem` (native) | the current working directory | host directory `B` (relative `B` is under the current directory) |
| `PosixFileSystem`, `StdioFileSystem` (ESP32) | LittleFS, mounted as ESP-IDF's fallback filesystem | LittleFS mounted at `B` (must start with `/`, e.g. `"/littlefs"`) |
| `LittleFSFileSystem` (ESP32) | LittleFS, mounted as ESP-IDF's fallback filesystem | LittleFS mounted at `B` |
| `SPIFFSFileSystem`, `SDFileSystem`, `InternalFSFileSystem`, `FlashFSFileSystem` | the filesystem's root | — (no base-path parameter) |

`".."` is passed through as-is: the base path is a root, not a security sandbox.

**Reaching the host's real root on native.** Give it as the base path:

```cpp
using namespace microStore;
FileSystem app { Adapters::PosixFileSystem() }; // "./kv/" and "/kv/" → <cwd>/kv/
FileSystem host{ Adapters::PosixFileSystem("/") }; // "/tmp/x" → /tmp/x
FileSystem data{ Adapters::PosixFileSystem("/var/lib/myapp") }; // "/kv/" → /var/lib/myapp/kv/
```

`FileSystem` objects are cheap handles, so use two of them when one part of a program stores data under its own root and another part needs absolute host paths.

**Limits and memory.** Path resolution never allocates. Each filesystem call resolves its path into a stack buffer of `USTORE_MAX_PATH_LEN` bytes (two for `rename()`), and the base path is copied into a fixed `USTORE_MAX_BASEPATH_LEN` buffer inside the adapter, so the caller's string needn't outlive the call. A resolved path that doesn't fit makes that call fail; it is never truncated. A base path that doesn't fit makes `init()` and every call fail. On ESP32, the LittleFS mount point lives in one static buffer for the life of the program, because the Arduino FS layer keeps the pointer it is given. LittleFS is a single shared filesystem, so the first adapter to `init()` fixes its mount point; a later adapter that asks for a different one logs a warning and uses the existing mount.

---

## 5. Quick Start
Expand Down Expand Up @@ -626,6 +655,8 @@ All have sane defaults; override only if you have a reason.
| `USTORE_MAX_KEY_LEN` | `64` | Maximum key length in bytes |
| `USTORE_MAX_VALUE_LEN` | `1024` | Maximum value length in bytes |
| `USTORE_MAX_FILENAME_LEN` | `64` | Maximum total length of generated filenames |
| `USTORE_MAX_PATH_LEN` | `128` | Maximum length of a resolved adapter path, base path included; one stack buffer of this size per filesystem call (see §4.1) |
| `USTORE_MAX_BASEPATH_LEN` | `32` | Maximum length of an adapter base path / mount point |
| `USTORE_COMPACT_RETRY_MS` | `60000` | Cooldown between failed compaction attempts |
| `USTORE_COMPACT_THRESHOLD` | `25` | % of dead records that triggers auto-compaction (0 to disable) |
| `USTORE_DEFAULT_TTL_SECS` | `0` | Initial global TTL (0 = disabled) |
Expand Down
35 changes: 27 additions & 8 deletions include/microStore/Adapters/FlashFSFileSystem.h
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
#include "../File.h"
#include "../FileSystem.h"
#include "../Log.h"
#include "../Path.h"

#include <Cached_SPIFlash.h>
#include <FlashFileSystem.h>
Expand Down Expand Up @@ -226,7 +227,8 @@ class FlashFSFileSystem : public microStore::FileSystem {
}


virtual microStore::File open(const char* path, microStore::File::Mode mode, const bool create = false) override {
virtual microStore::File open(const char* path_in, microStore::File::Mode mode, const bool create = false) override {
USTORE_RESOLVE_PATH(path, path_in, {});
int pmode;
switch (mode) {
// Read only. File must exist. ("r")
Expand Down Expand Up @@ -276,34 +278,41 @@ class FlashFSFileSystem : public microStore::FileSystem {
}


virtual bool exists(const char* path) override {
virtual bool exists(const char* path_in) override {
USTORE_RESOLVE_PATH(path, path_in, false);
return FlashFS.exists(path);
}

virtual bool remove(const char* path) override {
virtual bool remove(const char* path_in) override {
USTORE_RESOLVE_PATH(path, path_in, false);
return FlashFS.remove(path);
}

virtual bool rename(const char* from_path, const char* to_path) override {
virtual bool rename(const char* from_in, const char* to_in) override {
USTORE_RESOLVE_PATH(from_path, from_in, false);
USTORE_RESOLVE_PATH(to_path, to_in, false);
return FlashFS.rename(from_path, to_path);
}

virtual bool mkdir(const char* path) override {
virtual bool mkdir(const char* path_in) override {
USTORE_RESOLVE_PATH(path, path_in, false);
if (!FlashFS.mkdir(path)) {
return false;
}
return true;
}

virtual bool rmdir(const char* path) override {
virtual bool rmdir(const char* path_in) override {
USTORE_RESOLVE_PATH(path, path_in, false);
if (!FlashFS.rmdir_r(path)) {
return false;
}
return true;
}


virtual bool isDirectory(const char* path) override {
virtual bool isDirectory(const char* path_in) override {
USTORE_RESOLVE_PATH(path, path_in, false);
Adafruit_SPIFlash_LittleFS::File file(FlashFS);
if (file.open(path, Adafruit_SPIFlash_LittleFS::FILE_O_READ)) {
bool is_directory = file.isDirectory();
Expand All @@ -313,8 +322,9 @@ class FlashFSFileSystem : public microStore::FileSystem {
return false;
}

virtual std::list<std::string> listDirectory(const char* path, Callbacks::DirectoryListing callback = nullptr) override {
virtual std::list<std::string> listDirectory(const char* path_in, Callbacks::DirectoryListing callback = nullptr) override {
std::list<std::string> files;
USTORE_RESOLVE_PATH(path, path_in, files);
Adafruit_SPIFlash_LittleFS::File root = FlashFS.open(path);
if (!root) {
return files;
Expand Down Expand Up @@ -345,6 +355,15 @@ class FlashFSFileSystem : public microStore::FileSystem {
Adafruit_FlashTransport_SPI _transport;
Cached_SPIFlash _flash;
bool _mounted = false;

private:
// Paths are root-relative ("/a/b", "./a/b" and "a/b" are the same file);
// the framework adds the mount point.
inline const char* resolve(const char* path, char* buf, size_t n) const {
const char* resolved = Path::normalize(path, buf, n);
if (resolved == nullptr) USTORE_LOG("[ustore] FlashFSFileSystem: path too long: %s\n", path);
return resolved;
}
};

};
Expand Down
35 changes: 27 additions & 8 deletions include/microStore/Adapters/InternalFSFileSystem.h
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
#include "../File.h"
#include "../FileSystem.h"
#include "../Log.h"
#include "../Path.h"

#include <InternalFileSystem.h>

Expand Down Expand Up @@ -166,7 +167,8 @@ class InternalFSFileSystem : public microStore::FileSystem {
}


virtual microStore::File open(const char* path, microStore::File::Mode mode, const bool create = false) override {
virtual microStore::File open(const char* path_in, microStore::File::Mode mode, const bool create = false) override {
USTORE_RESOLVE_PATH(path, path_in, {});
int pmode;
switch (mode) {
// Read only. File must exist. ("r")
Expand Down Expand Up @@ -217,34 +219,41 @@ class InternalFSFileSystem : public microStore::FileSystem {
}


virtual bool exists(const char* path) override {
virtual bool exists(const char* path_in) override {
USTORE_RESOLVE_PATH(path, path_in, false);
return InternalFS.exists(path);
}

virtual bool remove(const char* path) override {
virtual bool remove(const char* path_in) override {
USTORE_RESOLVE_PATH(path, path_in, false);
return InternalFS.remove(path);
}

virtual bool rename(const char* from_path, const char* to_path) override {
virtual bool rename(const char* from_in, const char* to_in) override {
USTORE_RESOLVE_PATH(from_path, from_in, false);
USTORE_RESOLVE_PATH(to_path, to_in, false);
return InternalFS.rename(from_path, to_path);
}

virtual bool mkdir(const char* path) override {
virtual bool mkdir(const char* path_in) override {
USTORE_RESOLVE_PATH(path, path_in, false);
if (!InternalFS.mkdir(path)) {
return false;
}
return true;
}

virtual bool rmdir(const char* path) override {
virtual bool rmdir(const char* path_in) override {
USTORE_RESOLVE_PATH(path, path_in, false);
if (!InternalFS.rmdir_r(path)) {
return false;
}
return true;
}


virtual bool isDirectory(const char* path) override {
virtual bool isDirectory(const char* path_in) override {
USTORE_RESOLVE_PATH(path, path_in, false);
Adafruit_LittleFS_Namespace::File file(InternalFS);
if (file.open(path, Adafruit_LittleFS_Namespace::FILE_O_READ)) {
bool is_directory = file.isDirectory();
Expand All @@ -254,8 +263,9 @@ class InternalFSFileSystem : public microStore::FileSystem {
return false;
}

virtual std::list<std::string> listDirectory(const char* path, Callbacks::DirectoryListing callback = nullptr) override {
virtual std::list<std::string> listDirectory(const char* path_in, Callbacks::DirectoryListing callback = nullptr) override {
std::list<std::string> files;
USTORE_RESOLVE_PATH(path, path_in, files);
Adafruit_LittleFS_Namespace::File root = InternalFS.open(path);
if (!root) {
return files;
Expand Down Expand Up @@ -307,6 +317,15 @@ class InternalFSFileSystem : public microStore::FileSystem {
return (totalBytes() - usedBytes());
}


private:
// Paths are root-relative ("/a/b", "./a/b" and "a/b" are the same file);
// the framework adds the mount point.
inline const char* resolve(const char* path, char* buf, size_t n) const {
const char* resolved = Path::normalize(path, buf, n);
if (resolved == nullptr) USTORE_LOG("[ustore] InternalFSFileSystem: path too long: %s\n", path);
return resolved;
}
};

};
Expand Down
55 changes: 43 additions & 12 deletions include/microStore/Adapters/LittleFSFileSystem.h
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
#include "../File.h"
#include "../FileSystem.h"
#include "../Log.h"
#include "../Path.h"

#include <LittleFS.h>

Expand Down Expand Up @@ -85,7 +86,9 @@ class LittleFSFileSystem : public microStore::FileSystem {
class FileSystemImpl : public microStore::FileSystemImpl {

public:
FileSystemImpl(const char* basepath = "") : _basepath(basepath) {}
FileSystemImpl(const char* basepath = "") {
_basepath_ok = Path::copy_base(_basepath, sizeof(_basepath), basepath);
}
virtual ~FileSystemImpl() {}

public:
Expand All @@ -101,8 +104,18 @@ class LittleFSFileSystem : public microStore::FileSystem {

inline virtual bool init(bool reformatOnFail = true) override {
USTORE_LOG("[ustore] Initializing LittleFSFileSystem\n");
// Initialize LittleFS
if (!LittleFS.begin(true, _basepath)) {
if (!_basepath_ok) {
USTORE_LOG("[ustore] LittleFSFileSystem base path longer than %d characters!\n", USTORE_MAX_BASEPATH_LEN - 1);
return false;
}
// Initialize LittleFS. The base path is the mount point; Arduino
// keeps the pointer, so pass storage that outlives this adapter.
bool conflict = false;
const char* mount = Path::persistent_mount(_basepath, &conflict);
if (conflict) {
USTORE_LOG("[ustore] WARNING: LittleFS already mounted at \"%s\", ignoring base path \"%s\"\n", mount, _basepath);
}
if (!LittleFS.begin(true, mount)) {
USTORE_LOG("[ustore] Failed to initialize LittleFSFileSystem!\n");
return false;
}
Expand All @@ -129,7 +142,8 @@ class LittleFSFileSystem : public microStore::FileSystem {
}


virtual microStore::File open(const char* path, microStore::File::Mode mode, const bool create = false) override {
virtual microStore::File open(const char* path_in, microStore::File::Mode mode, const bool create = false) override {
USTORE_RESOLVE_PATH(path, path_in, {});
const char* pmode;
switch (mode) {
// Read only. File must exist. ("r")
Expand Down Expand Up @@ -165,30 +179,37 @@ class LittleFSFileSystem : public microStore::FileSystem {
}


inline virtual bool exists(const char* path) override {
inline virtual bool exists(const char* path_in) override {
USTORE_RESOLVE_PATH(path, path_in, false);
return LittleFS.exists(path);
}

inline virtual bool remove(const char* path) override {
inline virtual bool remove(const char* path_in) override {
USTORE_RESOLVE_PATH(path, path_in, false);
return LittleFS.remove(path);
}

inline virtual bool rename(const char* from_path, const char* to_path) override {
inline virtual bool rename(const char* from_in, const char* to_in) override {
USTORE_RESOLVE_PATH(from_path, from_in, false);
USTORE_RESOLVE_PATH(to_path, to_in, false);
return LittleFS.rename(from_path, to_path);
}

inline virtual bool mkdir(const char* path) override {
inline virtual bool mkdir(const char* path_in) override {
USTORE_RESOLVE_PATH(path, path_in, false);
if (!LittleFS.mkdir(path)) return false;
return true;
}

inline virtual bool rmdir(const char* path) override {
inline virtual bool rmdir(const char* path_in) override {
USTORE_RESOLVE_PATH(path, path_in, false);
if (!LittleFS.rmdir(path)) return false;
return true;
}


virtual bool isDirectory(const char* path) override {
virtual bool isDirectory(const char* path_in) override {
USTORE_RESOLVE_PATH(path, path_in, false);
fs::File file = LittleFS.open(path, FILE_READ);
if (file) {
bool is_directory = file.isDirectory();
Expand All @@ -198,8 +219,9 @@ class LittleFSFileSystem : public microStore::FileSystem {
return false;
}

virtual std::list<std::string> listDirectory(const char* path, Callbacks::DirectoryListing callback = nullptr) override {
virtual std::list<std::string> listDirectory(const char* path_in, Callbacks::DirectoryListing callback = nullptr) override {
std::list<std::string> files;
USTORE_RESOLVE_PATH(path, path_in, files);
fs::File root = LittleFS.open(path);
if (!root) {
return files;
Expand All @@ -226,7 +248,16 @@ class LittleFSFileSystem : public microStore::FileSystem {
}

private:
const char* _basepath = "";
char _basepath[USTORE_MAX_BASEPATH_LEN];
bool _basepath_ok = false;

// Paths are root-relative ("/a/b", "./a/b" and "a/b" are the same file);
// the framework adds the mount point.
inline const char* resolve(const char* path, char* buf, size_t n) const {
const char* resolved = Path::normalize(path, buf, n);
if (resolved == nullptr) USTORE_LOG("[ustore] LittleFSFileSystem: path too long: %s\n", path);
return resolved;
}
};

};
Expand Down
Loading
Loading