Skip to content

Stream stdEnumerator IEnumVARIANT sources without caching - #186

Merged
sancarn merged 3 commits into
masterfrom
feature/54-lazy-ienumvariant
Nov 1, 2026
Merged

sancarn merged 3 commits into
masterfrom
feature/54-lazy-ienumvariant

Conversation

@sancarn

@sancarn sancarn commented Nov 1, 2026 •

Copy link
Copy Markdown
Owner

CreateFromIEnumVariant currently reads the entire source twice during construction. Replace that eager snapshot with independent streaming traversals on Windows: each operation acquires its own IEnumVARIANT cursor and reads one item per Next call, without caching consumed source values.

Fixes #54.

  • countBy evaluates its predicate as items arrive and retains only the running count and current value. Length also traverses without retaining source values.
  • First(100) reads and retains only its returned prefix; non-positive counts return empty without opening the source.
  • Repeated operations reread the source. Nested/reentrant operations and procedural enumeration have separate cursors; enumRefresh releases its cursor and restarts on the next read.
  • Native For Each returns a fresh source cursor directly, including for nested loops and early exit.
  • Unique uses an independent lookahead traversal while preserving its existing comparison behavior.
  • No stdCOM dependency. Mac retains the eager fallback; Map, Filter, Sort and explicit conversions still materialise their results. This provides streaming source traversal, not a fully lazy transformation pipeline.
  • Changelog and generated docs.json updated.

Breaking changes

Windows construction previously captured an immediate snapshot. Operations now reread the live source, so mutations can affect later traversals and source errors occur during consumption. Callers needing stable replay, mutation isolation or eager error timing should capture immediately:
Set e = stdEnumerator.CreateFromArray(stdEnumerator.CreateFromIEnumVariant(o).AsArray())

First with a non-positive count previously returned the entire source and now returns empty. Callers relying on First(0) or First(-1) to copy all values should use stdEnumerator.CreateFromArray(e.AsArray()).

Public signatures, argument names/defaults, keys and callback contracts are unchanged.

Validation

  • Passed all 93 existing stdEnumerator assertions in isolated 64-bit Excel 16.0 / VBA 7.1 with a lightweight assertion recorder.
  • Instrumented COM-source tests prove zero constructor reads/cursors, exactly 100 reads for First(100), fresh reads on repeated operations, cursor release on early exit/failure/exhaustion/refresh, and independent nested/reentrant traversals.
  • A countBy over 3,000 freshly generated COM objects counts 1,500 matches, invokes the predicate between reads, keeps at most three generated values live, and leaves none retained afterward.
  • First retains exactly its 100 returned objects and releases them when the result is released. Native For Each exits after 100 reads and releases its cursor.
  • Tested source mutation visibility, procedural failures/recovery, nested native loops, mixed variants/object identity, dictionaries, Worksheets and a prefix of the full worksheet Cells collection.
  • Disposable hidden Excel instances with owned-dialog monitoring and bounded workers. Exact AccessVBOM/error-trapping registry values restored and verified.
  • Regenerated docs.json and compared against the PR base: only intended stdEnumerator content changed; no public members, parameters or return types lost. CRLF bytes and diff whitespace verified.

Windows 32-bit, legacy VBA and Mac runtime paths remain untested.

- Fetch Windows source values on demand and cache them for repeatable traversal
- Return empty enumerators for non-positive First counts
- Add Excel regression coverage and document the behavior
@sancarn

sancarn commented Nov 1, 2026

Copy link
Copy Markdown
Owner Author

Relates to #25 also

@sancarn sancarn left a comment •

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we make this sequential, fetching items one at a time rather than loading the full collection at once? That would keep memory use down and let operations like counting process items as they arrive, instead of materializing everything in RAM first. A fully lazy enumerator is a long-term goal, but this approach would be a step in that direction.

Comment thread src/stdEnumerator.cls Outdated
Select Case This.mode
Case FromIEnumVariant
#If Not Mac Then
Do While iCurrentIndex > This.Cache.Count And Not This.Exhausted

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do while loop here doesn't really make sense, and doesn't achieve the lazyness goals of the class. Sequential calls: the operation’s outer loop repeatedly calls NextItem; each NextItem should call COM’s Next once.

The better design is:

  • Each operation obtains its own source cursor.
  • Each NextItem retrieves one value and passes it to the operation.
  • Counting retains only the current value and running total.
  • .First(100) retains its 100 returned results.
  • Repeating an operation obtains a fresh cursor and enumerates again.

@sancarn sancarn changed the title Make stdEnumerator IEnumVARIANT sources lazy with cached replay Stream stdEnumerator IEnumVARIANT sources without caching Nov 1, 2026

@sancarn sancarn left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use enums rather than call indexes

Comment thread src/stdEnumerator.cls Outdated
.Started = True
End If
Dim value As Variant, fetched As Long, hr As Long
hr = enumCall(ObjPtr(.Cursor), 3, 1&, VarPtr(value), VarPtr(fetched))

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of calling a random index let's use an enum

@sancarn

sancarn commented Nov 1, 2026

Copy link
Copy Markdown
Owner Author

Looks good 👍

@sancarn
sancarn merged commit b720027 into master Nov 1, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

stdEnumerator - Better CreateFromIEnumVariant()

1 participant