Skip to content

fix(index): accept iterable data and explicit keys in load - #772

Open
ningyixi wants to merge 1 commit into
redis:mainfrom
ningyixi:fix/load-iterator-keys
Open

ningyixi wants to merge 1 commit into
redis:mainfrom
ningyixi:fix/load-iterator-keys

Conversation

@ningyixi

@ningyixi ningyixi commented Oct 18, 2026 •

Copy link
Copy Markdown

SearchIndex.load() and AsyncSearchIndex.load() accept iterable data and keys, but passing an iterator for either argument with explicit keys currently raises TypeError because storage calls len() before materializing the inputs. Explicit empty keys also bypass the length check and silently generate different keys for nonempty data.

Materialize each iterable once before checking lengths in both storage write paths, and distinguish omitted keys from an explicit empty iterable. This preserves the existing validation error and rejects mismatches before any pipeline writes.

The integration tests use Redis 8.4 via the repository's Testcontainers setup, exercise HASH and JSON storage, sync and async APIs, all list/iterator combinations, and shorter/longer/empty keys with no partial writes.

Validation (Python 3.12.14):

  • New regression file: 24 failed / 4 passed before the fix; 28 passed afterward.
  • Storage, sync/async search-index, bulk-operation and new integration tests: 164 passed.
  • make format, make check-types (120 source files), make check-format check-sort-imports, uv lock --check, and pre-commit on both changed files pass.
  • Full make test was attempted with the default locked development dependencies: 1,892 passed, 153 skipped, 2 xfailed; 195 failures and 154 setup errors reference missing optional packages (sentence_transformers, sql_redis, google, voyageai, pydantic_settings). This is not a passing full-suite result; provider/model tests were not run with credentials.
  • pre-commit run --all-files passes the code-quality hook but Codespell rewrites an existing Unparseable comment in untouched redisvl/redis/connection.py. That unrelated rewrite was reverted; the changed-file hooks pass.

Suggested release label: auto:patch.

Implemented and tested with OpenAI Codex assistance.


Note

Low Risk
Localized change to load/write input handling and validation order; behavior fix for iterables and explicit empty keys, covered by new integration tests.

Overview
Fixes SearchIndex.load / AsyncSearchIndex.load when data or explicit keys are single-pass iterators: storage now materializes both to lists once before length checks and preprocessing, so len() no longer raises TypeError and iterators are not consumed inconsistently.

The keys-vs-objects check now uses keys is not None instead of truthiness, so an explicit empty keys iterable is validated against nonempty data instead of being treated like omitted keys and auto-generating different keys.

The same logic applies to sync write and async awrite. New integration tests cover HASH/JSON, sync/async, list/iterator combinations, and mismatched key counts with no partial Redis writes before the error.

Reviewed by Cursor Bugbot for commit 5a4abe7. Bugbot is set up for automated code reviews on this repo. Configure here.

@vishal-bala

Copy link
Copy Markdown
Collaborator

Hi, thanks for your contribution! Can you rebase onto the latest origin/main and follow the new contribution process?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants