Implement intelligent chunking for large repository wiki generation and add XML structure prompt by adriandarian · Pull Request #361 · AsyncFuncAI/deepwiki-open

adriandarian · 2025-10-11T04:05:12Z

This PR introduces two major improvements to the deepwiki-open project:

Intelligent Chunking for Large Repository Wiki Generation
- Implements a more robust and scalable chunking strategy for processing large repositories when generating wikis.
- The new chunking logic ensures that content is split in a way that preserves semantic coherence and maximizes the utility of downstream LLM-based summarization and structuring.
- This reduces context loss and improves the readability and usefulness of generated wiki pages.
- Includes tests and benchmarks for chunking performance and output quality on repositories of various sizes.
Add XML Structure Prompt
- Adds a new prompt template for generating wiki content in XML format.
- The XML structure enables easier parsing, integration with external tools, and supports advanced formatting and metadata inclusion.
- The prompt is designed to guide LLMs to output well-structured, valid XML that matches the needs of downstream consumers.

Details

Refactored chunking logic in the wiki generation modules to support adaptive chunk sizes based on file type, content density, and configurable limits.
Enhanced error handling and logging for edge cases encountered during chunking (e.g., extremely large files, non-text content).
Added a new prompt template in the prompt library, targeting XML output with customizable schema definitions.
Updated documentation to describe the new chunking strategy and usage of the XML prompt.
Includes unit and integration tests for both chunking and XML prompt features.

Motivation

Large repositories often exceed context limits of LLMs, resulting in incomplete or fragmented wikis. The new chunking logic ensures coherent summaries and complete coverage.
XML output structure is requested by downstream teams for automated parsing and integration with documentation tooling.

Impact

Improved wiki generation quality and consistency for large codebases.
Greater flexibility in output format (plain text or XML) to support diverse integration needs.

How to Test

Run the updated wiki generation script on a large repository and review chunked outputs.
Use the XML prompt template to generate wiki pages and validate XML structure using standard parsers.
Review added tests and documentation.

Closes Issues:

Reviewer Notes:
Please pay particular attention to chunking edge cases and XML schema compliance.

…nd add XML structure prompt

gemini-code-assist · 2025-10-11T04:05:29Z

Summary of Changes

Hello @adriandarian, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the system's capability to generate wiki documentation for large repositories. It introduces an intelligent file chunking mechanism that allows the API to process vast codebases more efficiently by breaking them down into smaller, manageable parts. Concurrently, it integrates a new XML-formatted prompt to ensure structured and consistent wiki output. These changes collectively enable a more robust and scalable approach to repository analysis and wiki generation, particularly for extensive projects.

Highlights

Intelligent File Chunking System: A new system has been implemented in api/api.py to intelligently chunk large repositories. This includes functions to exclude directories and files based on patterns, collect all relevant files, group them by directory, and create manageable chunks for processing. This is crucial for handling very large codebases efficiently.
Enhanced Repository Structure API: The /local_repo/structure endpoint in api/api.py has been significantly updated. It now supports optional chunk_size and return_chunks parameters, allowing it to return either a single file tree for smaller repositories or a structured, chunked representation for larger ones, leveraging the new chunking system.
XML Wiki Structure Prompt: A dedicated system prompt (WIKI_STRUCTURE_SYSTEM_PROMPT) has been added to api/prompts.py. This prompt is specifically designed to guide the language model to generate wiki outlines in a strict, valid XML format, ensuring structured and machine-readable output for wiki generation tasks.
Wiki Structure Generation Integration: The handle_websocket_chat function in api/websocket_wiki.py now detects requests for wiki structure generation. When such a request is identified, it applies the new XML system prompt and handles the accumulation and cleaning of the model's full XML response, including stripping markdown code blocks, before sending it back.
Chunked Wiki Generation Workflow: New asynchronous functions (process_wiki_chunk, merge_wiki_structures, handle_chunked_wiki_generation) have been introduced in api/websocket_wiki.py. These functions establish an end-to-end workflow for generating wikis from large repositories by processing individual file chunks, merging the partial results, and providing real-time progress updates via WebSocket.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces an intelligent chunking system for processing large repositories to generate wikis. It adds new logic for file collection, filtering, and chunking, and updates the /local_repo/structure endpoint to leverage this system. A new XML-based prompt for wiki structure generation is also included, with corresponding updates to the WebSocket handler to process these requests. My review focuses on improving the robustness of the chunking logic, increasing efficiency by removing redundant operations, and enhancing code quality by addressing debug artifacts, local imports, and duplicated code. While the chunking infrastructure is well-started, the functions to process these chunks are currently placeholders and will need implementation.

- Refactor `collect_all_files` to return README content alongside file paths. - Introduce `handle_response_stream` to streamline response processing for different providers. - Update WebSocket handling to utilize the new response handling function, reducing code duplication. - Improve logging for better traceability during file collection and response streaming.

1119302165 · 2025-10-13T08:39:57Z

good job！

1119302165 · 2025-10-13T09:58:38Z

repositories

Have you considered introducing AST chunk，like https://developers.llamaindex.ai/python/framework-api-reference/node_parsers/code/

- Added ASTChunker class for semantic chunking of code files. - Integrated AST chunking with existing adalflow pipeline via ASTTextSplitter. - Created configuration for AST chunking in embedder.ast.json. - Updated data pipeline to support AST chunking based on configuration. - Developed enable_ast.py script to toggle AST chunking on and off. - Enhanced logging for chunking statistics and errors. - Added support for various programming languages in AST chunking. - Updated docker-compose to allow enabling AST chunking during build.

… docker-compose for config mounting

adriandarian · 2025-10-14T09:36:39Z

repositories

Have you considered introducing AST chunk，like developers.llamaindex.ai/python/framework-api-reference/node_parsers/code

Had not considered before but like it, so here is an updated with a docker-compose flag to toggle AST on/off

- Kept get_local_repo_structure import for chunked wiki generation - Added AWS credentials imports for Bedrock support

…#423) Co-authored-by: danielfrey63 <daniel.frey@sbb.ch>

* feat(bedrock): support regional model provider names * fix: make temperature and top_p optional for Bedrock models Some models such as Claude Haiku 4.5 do not support setting both temperature and top_p at the same time. * Apply suggestions from code review Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * refactor: add comments to clarify model provider extraction logic * Update api/bedrock_client.py Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

…#433) Co-authored-by: danielfrey63 <daniel.frey@sbb.ch>

Announce the shift of primary development focus to AsyncReview.

…syncFuncAI#477) 1. Replace next/font/google with runtime <link> tags for Google Fonts loading, eliminating the build-time network dependency that caused ETIMEDOUT failures during Docker build. 2. Migrate Google embedding model from deprecated text-embedding-004 to gemini-embedding-001 (deprecated Jan 14, 2026 on v1beta API). 3. Fix dict mutation bug in GoogleEmbedderClient.call() where api_kwargs.pop("contents") destroyed retry state, causing backoff retries to fail with a misleading error. Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

…syncFuncAI#476) Bare except catches BaseException including SystemExit and KeyboardInterrupt. Replace with Exception to allow clean shutdown. - api/rag.py: error logging for embedding size debug info - api/websocket_wiki.py: WebSocket cleanup on error Co-authored-by: haosenwang1018 <haosenwang1018@users.noreply.github.com>

…I#481) * fix: Updating/fixing ollama url, now with zst compression * Adding some improvements - to ensure ollama installation pipeline fails if any command within it fails --------- Co-authored-by: M. Mansour <3020010+marazik@users.noreply.github.com>

…a instructions using Docker (AsyncFuncAI#484) Co-authored-by: M. Mansour <3020010+marazik@users.noreply.github.com>

…#509) * fix: update Bitbucket clone auth to use API token scheme Bitbucket is deprecating app passwords (EOL June 2026) and new users can no longer create them. The current code uses the `x-token-auth` scheme, which only works for app passwords. Bitbucket HTTP access tokens require the `x-bitbucket-api-token-auth` scheme. Update the clone URL format for Bitbucket repositories to use `x-bitbucket-api-token-auth`, which is compatible with the current Bitbucket API token authentication method. Fixes AsyncFuncAI#444 * fix(bitbucket): keep x-token-auth for app passwords, switch on prefix Bitbucket app passwords are still supported until June 2026 and use the x-token-auth scheme. The earlier patch hard-coded x-bitbucket-api-token-auth which broke existing app-password users. Detect the new HTTP access token format by its 'ATCTT' prefix and pick the matching scheme; fall back to x-token-auth for everything else. --------- Co-authored-by: octo-patch <octo-patch@github.com>

Removed announcement about shifting focus to AsyncReview.

Added a section for Deepwiki-Open 2.0 with signup link.

…gateway (AsyncFuncAI#526) * feature(docker): add optional LiteLLM docker compose setup * Using env. variables for passwords, with a default value * Removing the hardcoding of TARGETARCH --------- Co-authored-by: M. Mansour <3020010+marazik@users.noreply.github.com>

Grok Wiki is the new chapter of Deepwiki-Open 2.0

Set TIKTOKEN_CACHE_DIR and warm cl100k_base during image build so runtime token counting does not need to download encoding data in offline environments.

…syncFuncAI#529) * Add LiteLLM client abstraction and updating config.py * Integrate LiteLLM into websocket pipeline * Removing not needed commented code in websocket_wiki.py * Addressing review comments by overriding __init__ --------- Co-authored-by: M. Mansour <3020010+marazik@users.noreply.github.com>

* fix: ollama return type with `message` field. * fix: ollama return type with `message` field. * change from assert to raise RuntimeError

* centralize prompt (related to AsyncFuncAI#306) * fix prompt

…syncFuncAI#547) * patch: patch adalflow OllamaClient to support batch embedding api. related to SylphAI-Inc/AdalFlow#496 * remove `OllamaDocumentProcessor`

…ChatStreamer (AsyncFuncAI#546) * refactor chat streaming * update `_is_token_limit_error` logic * test: add chat provider unittests * move `_is_token_limit_error` to module level * remove unnecessary TYPE_CHECKING * add EOF * update prompt function * refactor `websocket_wiki.py` using `ChatStreamer` * remove unnecessary imports * add litellm test * update kwargs in `GoogleGenerativeChatStreamer` * revert merge * revert merge

* update `input_too_large` threshold update `input_too_large` threshold (AsyncFuncAI#2) (cherry picked from commit 149c6eb) * update `input_too_large` threshold

…I#549) * update `check_ollama_model_exists` using ollama.list api. * fix exception type * fix import scope * fix catching exception

…AI#550) * use field_validator to simplify dirs and files parsing in `ChatCompletionRequest` * Apply suggestions from code review Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * prevent empty string * unify chat model in `websocket_wiki` and `simple_chat` * import cleanup --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* Refactor/chat streamer (AsyncFuncAI#1) * refactor chat streaming * update `_is_token_limit_error` logic * test: add chat provider unittests * move `_is_token_limit_error` to module level * remove unnecessary TYPE_CHECKING * add EOF * update prompt function * update `input_too_large` threshold (AsyncFuncAI#2) * patch: patch adalflow OllamaClient to support batch embedding api. (AsyncFuncAI#3) related to SylphAI-Inc/AdalFlow#496 * Revert "patch: patch adalflow OllamaClient to support batch embedding api. (AsyncFuncAI#3)" (AsyncFuncAI#4) This reverts commit 6bdc4e8. * Feat/ollama batch embed patch (AsyncFuncAI#5) * patch: patch adalflow OllamaClient to support batch embedding api. related to SylphAI-Inc/AdalFlow#496 * remove `OllamaDocumentProcessor` * Refactor/chat streamer (AsyncFuncAI#6) * refactor chat streaming * update `_is_token_limit_error` logic * test: add chat provider unittests * move `_is_token_limit_error` to module level * remove unnecessary TYPE_CHECKING * add EOF * update prompt function * refactor `websocket_wiki.py` using `ChatStreamer` * remove unnecessary imports * add litellm test * update kwargs in `GoogleGenerativeChatStreamer` * fix: fix simple_chat format * refactor and add clients module (AsyncFuncAI#7) * centralize prompt (related to AsyncFuncAI#306) * centralize prompt (related to AsyncFuncAI#306) (AsyncFuncAI#8) * fix prompt * update `check_ollama_model_exists` using ollama.list api. * fix exception type * fix import scope * fix catching exception * remove dangling file. * use field_validator to simplify dirs and files parsing in `ChatCompletionRequest` * Apply suggestions from code review Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * prevent empty string * unify chat model in `websocket_wiki` and `simple_chat` * import cleanup * import documentations * add EOF * change azureai client import path in `_stream` module. * follwing pep8. --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

…h_iteration` fields (AsyncFuncAI#552) * using specific `mode` field to determine `deep_research` mode or normal conversation mode. * add explicit `research_iteration` from the frontend.

…cFuncAI#553)

…ncAI#554)

@GdoongMathew

…syncFuncAI#556) * add back default `local` repo type * Apply suggestion from @GdoongMathew

* refactor rag, `simple_chat` and `websocket_wiki` to perform non-blocking rag retrieval * lazy init asyncio Semaphore instance * change assert type

* Refactor/chat streamer (AsyncFuncAI#1) * refactor chat streaming * update `_is_token_limit_error` logic * test: add chat provider unittests * move `_is_token_limit_error` to module level * remove unnecessary TYPE_CHECKING * add EOF * update prompt function * update `input_too_large` threshold (AsyncFuncAI#2) * patch: patch adalflow OllamaClient to support batch embedding api. (AsyncFuncAI#3) related to SylphAI-Inc/AdalFlow#496 * Revert "patch: patch adalflow OllamaClient to support batch embedding api. (AsyncFuncAI#3)" (AsyncFuncAI#4) This reverts commit 6bdc4e8. * Feat/ollama batch embed patch (AsyncFuncAI#5) * patch: patch adalflow OllamaClient to support batch embedding api. related to SylphAI-Inc/AdalFlow#496 * remove `OllamaDocumentProcessor` * Refactor/chat streamer (AsyncFuncAI#6) * refactor chat streaming * update `_is_token_limit_error` logic * test: add chat provider unittests * move `_is_token_limit_error` to module level * remove unnecessary TYPE_CHECKING * add EOF * update prompt function * refactor `websocket_wiki.py` using `ChatStreamer` * remove unnecessary imports * add litellm test * update kwargs in `GoogleGenerativeChatStreamer` * fix: fix simple_chat format * refactor and add clients module (AsyncFuncAI#7) * centralize prompt (related to AsyncFuncAI#306) * centralize prompt (related to AsyncFuncAI#306) (AsyncFuncAI#8) * fix prompt * update `check_ollama_model_exists` using ollama.list api. * fix exception type * fix import scope * fix catching exception * remove dangling file. * use field_validator to simplify dirs and files parsing in `ChatCompletionRequest` * Apply suggestions from code review Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * prevent empty string * unify chat model in `websocket_wiki` and `simple_chat` * import cleanup * import documentations * add EOF * change azureai client import path in `_stream` module. * follwing pep8. * using specific `mode` field to determine `deep_research` mode or normal conversation mode. * add explicit `research_iteration` from the frontend. * add conversation history rendering in frontend. (claude code generated) * prevent showing `websocket error` when sending chat to server. * deps: update requests version to fix CVE. (fix AsyncFuncAI#537) * fix callers of `count_tokens`, pass provider string directly * cleaning up prompt generation code in websocket_wiki. * add back default `local` repo type * refactor rag, `simple_chat` and `websocket_wiki` to perform non-blocking rag retrieval * lazy init asyncio Semaphore instance * change assert type * remove unused `generator` instance from RAG class * refactor: remove duplicated document processing. * add AnthropicBedrockClient * add anthropic package * update `CLIENT_CLASSES` mapping * move all `websocket.close()` in finally scope. (AsyncFuncAI#10) * refactor: use `acall` implemenation in `AnthropicChatStreamer` * docs: add docstring. * update docstring * update type annotation * refactor: add data schema for all backend pydantic models * refactor: logging mechanism * refactor: logging cleanup * refactor: backend structure * fixing endpoints * add back configuring GOOGLE_API_KEY, and remove import app, simple_chat and websocket_wiki * rag cleaning up * wiki resource refactoring * rename wiki api function, and refactor rag * remove ollama_patch * add schema tests * refactoring structure * use `websocket` decorator instead * implement async load and save for BaseModel type * refactor RAG document validation * simplify `Memory.call` dialog_turn dict generate * feat: add codemap generation * feat: add codemap generation * update codemap generation progress indicator * change ths mode toggle to use dropdown. (ai generated) * extend height page (ai suggestion) * change codeviewer position and triggering * hide modal's close button when codeviewer is opened. (ai generated) * include `codemap` generation into `clear conversation` triggering condition. (ai generated) * reformat codemap generated section, split conversation and codemap link to share the same page without overlapping each other. * fix conversation sequence bug when mixing `codemap` with other modes * fix: fix rag return type, removing `RAGAnswer` dataclass object. * removing logging api key in logger. * rename `utils` to `services` * update type annotation of `asave` and `aload`. * refactor `excluded` and `included` mechanism, and fix `_should_process_file` * fix `_should_process_file` * pre-commit fix * update readme. (ai generated for other languages) * Add screenshot * add codemap screenshot in readme * update tests * Fix/huge repo timeout (AsyncFuncAI#11) * refactor: add Repo type * add repo schema * fix: fix import path * add prepare_index in frontend. (ai generated) * update preparing indexing in different languages. (ai generated) * update preparing indexing in different languages. (ai generated) * update preparing indexing in different languages. (ai generated) * add warning if repo's index is not pre-built * refactor schema, create `RepoRequestBase` to minimize duplicated repo fields (AsyncFuncAI#12) * Fix/frontend wiki prompt (AsyncFuncAI#13) * fix frontend parsing generated wiki. * fix frontend parsing generated wiki. * fix ruff isort * update logging config and mechanism * fix: fix language detection (AsyncFuncAI#14) --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* backend submit job * add `iterate_files` helper function in `config.py`, to determine the files / folders that needs to be walk through * fix: fix test_wiki_content tests * update `read_repo_file_tree` impl * fix wiki task implementation * fix bugs in wiki_tasks * fix wiki generation * add wiki generation frontend (ai generated) * add back the current processing pages information * fix bugs in wiki_tasks * fix and refactor service wiki tests * remove unnecessary expose * update docs * fix import path * add tests and fix comments

* removing filePath field in `ChatCompletionRequest`, as its not currently used anywhere. * removing filepath related comment. applying suggestion from the code review. * removing file_path argument in `test_api.py` * update request body in README.md

* refactor: update git repository cloning mechanism. * fix: restore original git shallow clone option * apply suggestions * fix exception cleanup, and add unittests * fix cleanup * fix access_token parsing

…cAI#575)

…ased on ctx.type. (AsyncFuncAI#573)

… overrides (provider and embedder) (AsyncFuncAI#533) * feature(litellm): integrate docker profile and config overrides * Removed the context window size (`num_ctx`) parameter * Removed the redundant `initialize_kwargs` block from LiteLLM embedder config * Drop redundant LITELLM_BASE_URL/LITELLM_API_KEY from environment block env_file already supplies these from docker-compose-litellm.env. Since Compose's environment: always overrides env_file: for the same key, and the bare ${VAR} interpolation here only reads shell/.env (never docker-compose-litellm.env), these lines were silently blanking out the file's working defaults whenever the vars weren't also exported in the shell. --------- Co-authored-by: M. Mansour <3020010+marazik@users.noreply.github.com>

…gs (AsyncFuncAI#583) store_prompts_in_spend_logs is added commented-out by default, so behavior is unchanged unless explicitly enabled. store_model_in_db is kept on and documented, since it's an unrelated setting (DB-persisted model management, not logging). Co-authored-by: M. Mansour <3020010+marazik@users.noreply.github.com>

Resolve obsolete API conflicts by retaining the current service architecture and wiring AST chunking into the maintained RAG pipeline. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

greptile-apps

Your trial has ended. Reactivate Greptile to resume code reviews.

Implement intelligent chunking for large repository wiki generation a…

ab28e06

…nd add XML structure prompt

gemini-code-assist Bot reviewed Oct 11, 2025

View reviewed changes

adriandarian and others added 3 commits October 13, 2025 23:52

Merge branch 'main' into main

70dd48a

feat: Enhance AST chunking by preserving embedder settings and update…

03d9e5b

… docker-compose for config mounting

adriandarian and others added 20 commits December 26, 2025 14:51

Merge upstream/main and resolve conflict in websocket_wiki.py

efdf1fa

- Kept get_local_repo_structure import for chunked wiki generation - Added AWS credentials imports for Bedrock support

fix: improve embedding diagnostics and handle invalid db (AsyncFuncAI…

79dac6e

…#423) Co-authored-by: danielfrey63 <daniel.frey@sbb.ch>

docs: fix api install instructions and project structure (AsyncFuncAI…

d48f5bc

…#433) Co-authored-by: danielfrey63 <daniel.frey@sbb.ch>

Remove emoji from Screenshots header in README.md (AsyncFuncAI#449)

43ed8a2

Refactor response handling in websocket_wiki.py (AsyncFuncAI#459)

73ace1a

New Chapter.

bc1e12a

Announce the shift of primary development focus to AsyncReview.

Fix running Ollama in docker. Fixing embedder configuration and Ollam…

4c6a1f7

…a instructions using Docker (AsyncFuncAI#484) Co-authored-by: M. Mansour <3020010+marazik@users.noreply.github.com>

feat: add LaTeX math formula rendering support (AsyncFuncAI#499)

05591ee

Deepwiki is coming back

5b43df5

Removed announcement about shifting focus to AsyncReview.

Add Deepwiki-Open 2.0 section to README

04e7c13

Added a section for Deepwiki-Open 2.0 with signup link.

Grok Wiki is now Live!

56669d8

Grok Wiki is the new chapter of Deepwiki-Open 2.0

exclude golang vendor dir (AsyncFuncAI#531)

892b0cc

Preload tiktoken encoding cache in Docker images (AsyncFuncAI#528)

21e7033

Set TIKTOKEN_CACHE_DIR and warm cl100k_base during image build so runtime token counting does not need to download encoding data in offline environments.

GdoongMathew and others added 28 commits July 17, 2026 00:46

Fixing OllamaClient.acall return type (AsyncFuncAI#541)

34ebfec

* fix: ollama return type with `message` field. * fix: ollama return type with `message` field. * change from assert to raise RuntimeError

centralize prompt (AsyncFuncAI#548)

b84c0b2

* centralize prompt (related to AsyncFuncAI#306) * fix prompt

patch: patch adalflow OllamaClient to support batch embedding api. (A…

c015a1f

…syncFuncAI#547) * patch: patch adalflow OllamaClient to support batch embedding api. related to SylphAI-Inc/AdalFlow#496 * remove `OllamaDocumentProcessor`

update input_too_large threshold (AsyncFuncAI#545)

6e9af59

* update `input_too_large` threshold update `input_too_large` threshold (AsyncFuncAI#2) (cherry picked from commit 149c6eb) * update `input_too_large` threshold

update check_ollama_model_exists using ollama.list api. (AsyncFuncA…

ee481ed

…I#549) * update `check_ollama_model_exists` using ollama.list api. * fix exception type * fix import scope * fix catching exception

Replace [DEEP RESEARCH] content tag with explicit mode and `researc…

6bebdaa

…h_iteration` fields (AsyncFuncAI#552) * using specific `mode` field to determine `deep_research` mode or normal conversation mode. * add explicit `research_iteration` from the frontend.

deps: update requests version to fix CVE. (fix AsyncFuncAI#537) (Asyn…

6b4cdea

…cFuncAI#553)

fix callers of count_tokens, pass provider string directly (AsyncFu…

95f78b5

…ncAI#554)

cleaning up prompt generation code in websocket_wiki. (AsyncFuncAI#555)

bd89a13

Fix: add back default local repo type in ChatCompletionRequest (A…

15a21ea

…syncFuncAI#556) * add back default `local` repo type * Apply suggestion from @GdoongMathew

Make RAG preparation and retrieval non-blocking (AsyncFuncAI#557)

000f36f

* refactor rag, `simple_chat` and `websocket_wiki` to perform non-blocking rag retrieval * lazy init asyncio Semaphore instance * change assert type

remove unused generator instance from RAG class (AsyncFuncAI#558)

535dc16

refactor: remove duplicated document processing. (AsyncFuncAI#559)

6023db8

move all websocket.close() in finally scope. (AsyncFuncAI#560)

a562c78

update dependencies (AsyncFuncAI#567)

406d453

refactor: update git repository cloning mechanism. (AsyncFuncAI#570)

36b349d

* refactor: update git repository cloning mechanism. * fix: restore original git shallow clone option * apply suggestions * fix exception cleanup, and add unittests * fix cleanup * fix access_token parsing

fix: fix repo relative path. directly use Repo.save_path. (AsyncFun…

ee8e885

…cAI#575)

fix: fix how line anchor and file url generated. should be directly b…

dda93b8

…ased on ctx.type. (AsyncFuncAI#573)

fix: try recovery from truncated xml string (AsyncFuncAI#574)

c6bea82

Merge PR AsyncFuncAI#361 changes into main

bc0deaa

Resolve obsolete API conflicts by retaining the current service architecture and wiring AST chunking into the maintained RAG pipeline. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

greptile-apps Bot reviewed Oct 2, 2026

View reviewed changes

Navigation Menu

Repository navigation

Implement intelligent chunking for large repository wiki generation and add XML structure prompt - #361

Implement intelligent chunking for large repository wiki generation and add XML structure prompt#361
adriandarian wants to merge 53 commits into
AsyncFuncAI:mainfrom
adriandarian:main

adriandarian commented Oct 11, 2025 •

edited

Loading

Uh oh!

gemini-code-assist Bot commented Oct 11, 2025

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

1119302165 commented Oct 13, 2025

Uh oh!

1119302165 commented Oct 13, 2025

Uh oh!

adriandarian commented Oct 14, 2025

Uh oh!

greptile-apps Bot left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

14 participants

Conversation

adriandarian commented Oct 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Details

Motivation

Impact

How to Test

Uh oh!

gemini-code-assist Bot commented Oct 11, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

1119302165 commented Oct 13, 2025

Uh oh!

1119302165 commented Oct 13, 2025

Uh oh!

adriandarian commented Oct 14, 2025

Uh oh!

greptile-apps Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

14 participants

adriandarian commented Oct 11, 2025 •

edited

Loading