Skip to content

Add langchain-flan example (Harithalakshmikt) - #22

Merged
Vedantgitbot merged 2 commits into
Vedantgitbot:mainfrom
Harithalakshmikt:add-langchain-flan-example
Oct 20, 2026
Merged

Vedantgitbot merged 2 commits into
Vedantgitbot:mainfrom
Harithalakshmikt:add-langchain-flan-example

Conversation

@Harithalakshmikt

Copy link
Copy Markdown

Adds a local RAG example using LangChain, FAISS and google/flan-t5-base,
with check_with_evidence() and an NLI-based compare_fn
(cross-encoder/nli-deberta-v3-base).

Tested model: google/flan-t5-base (local, no API key).
Includes a deliberately wrong answer (BLOCKED) and a correct control.
RESULTS.md reports the failures honestly: the compare_fn is over-strict
and blocks some correct answers.

Related to #2

@Harithalakshmikt

Copy link
Copy Markdown
Author

Hi @Vedantgitbot, I've opened a PR for this example: (#22)
)
It uses LangChain + FAISS + google/flan-t5-base (local, no API key) with check_with_evidence() and an NLI-based compare_fn. It includes a deliberately wrong answer (BLOCKED) and a correct control answer. RESULTS.md documents the failures honestly, including false blocks from the over-strict compare_fn.

Happy to make changes if you'd like any.

@Vedantgitbot

Copy link
Copy Markdown
Owner

Thanks @Harithalakshmikt Great writeup, especially the NLI comparator and false-block findings. Before merging, please fix the docstring mismatch and use pip install boothpy instead of -e .. Optional: try chunk_size=100 and add a default question for reproducibility. #18 on compare_fn examples could be a great next step!

@Harithalakshmikt

Copy link
Copy Markdown
Author

Thanks @Vedantgitbot! Pushed the changes: the docstring now matches the script (two hardcoded checks, one wrong and one correct), the install line is pip install boothpy, and I added a default question (press Enter). I left chunk_size unchanged because changing it would invalidate the recorded results; happy to try it as a follow-up.

@Vedantgitbot
Vedantgitbot merged commit 021f6a1 into Vedantgitbot:main Oct 20, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants