Skip to content

Fix nested directory include and exclude filters - #593

Open
wanghanyu654321-cell wants to merge 1 commit into
AsyncFuncAI:mainfrom
wanghanyu654321-cell:fix/nested-directory-filters
Open

wanghanyu654321-cell wants to merge 1 commit into
AsyncFuncAI:mainfrom
wanghanyu654321-cell:fix/nested-directory-filters

Conversation

@wanghanyu654321-cell

Copy link
Copy Markdown

Directory rules such as ./test/reference_data/ are currently compared to individual path components, so nested exclusions leak files into the wiki/RAG input and nested inclusions return no files. Match normalized slash-delimited directory paths instead, preserving directory boundaries and support for bare directory names.

Fixes #79. This changes directory selectors only; file glob matching in #430 and cached database invalidation in #495 are separate. Adds six regression cases covering include/exclude rules, forward/backslashes, nested children, and similarly named sibling directories.

Validation:

  • Before the fix: six new cases fail; all six existing test_config.py cases pass.
  • Windows: 74 tests pass (tests/backend, tests/unit/test_chat.py, tests/test_config.py).
  • Linux: 21 filtering/wiki-structure tests pass in the built application image with networking disabled.
  • Actual read_repo_file_tree and read_all_documents inclusion/exclusion checks fail on the base and pass after the fix; the Linux image also passes all four checks offline, without model calls.
  • The repository Dockerfile builds successfully, including the Next.js build/type/lint step; Python sources compile. Ruff 0.8.6 lint and backend format checks pass.
  • All applicable repository pre-commit hooks pass, including secret scanning.

Full-suite limitations: pytest tests -q gives 110 passed, 4 failed, 1 error after this change; the untouched base gives 104 passed with the same four failures and one error. These are the GitLab clone network test, three embedder factory tests requiring GOOGLE_API_KEY, and the streaming endpoint test's missing repo_url fixture. Additional mypy checking reports the same existing implicit-Optional error in get_embedder on both versions. No tests or check settings were weakened to address these.

AI assistance: this patch and regression tests were prepared with Codex on behalf of the account owner; the diff and the reported test outputs were reviewed by the agent.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

data_pipeline.py: excluded_dirs should be normalized using OS path separators

1 participant