Skip to content

docs(ai): say what the docs assistant can't do and where to run the Celopedia install - #2359

Merged
GigaHierz merged 3 commits into
mainfrom
palango/assistant-and-celopedia-answers
Oct 7, 2026
Merged

GigaHierz merged 3 commits into
mainfrom
palango/assistant-and-celopedia-answers

Conversation

@palango

@palango palango commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Two topics keep showing up among the docs assistant's unanswered questions (#2320, #2324, #2343). People ask the assistant what it is and what it can do, and people pasted npx skills add celo-org/celopedia-skills into the chat three times (#2320, #2324), with two more requests in #2324 asking it to install the skill. Neither page answered either point. The widget calls itself "Celo Docs Assistant", but use-docs-with-ai.mdx never used that name. That page now names the assistant and says what it can't do. celopedia.mdx now says the install command has to run in a terminal.

The limits come from celo-org/docs-ai-assistant. In app/api/chat/route.ts, the only tools are the ones served by https://docs.celo.org/mcp, with submit_feedback denylisted, and the system prompt says to answer using only those tools and to refuse questions unrelated to Celo. lib/topic-gate.ts adds a classifier for off-topic questions. The widget's title comes from public/widget.js.

Refs #2320, refs #2324, refs #2343.

mint broken-links --check-redirects, mint validate and scripts/check-orphans.sh pass on the rebased head.

@palango
palango requested a review from a team as a code owner October 2, 2026 09:28
@palango
palango requested a review from GigaHierz October 2, 2026 09:29

@GigaHierz GigaHierz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

head eb22ee3 · setup gate matched API head (no STALE); branch is 7 commits behind main, 1 ahead, merge-tree clean, main did not touch either changed file; mint broken-links --check-redirects: success; check-orphans: none; gh pr checks: all pass (Mintlify Deployment skipping); mint dev render on desktop and 412x915: both pages load, 0 console errors, no horizontal overflow, anchors present

Verdict: REQUEST-CHANGES

Tier: rendered doc change with factual claims about a live system, full pass. Findings are all wording precision; no claim about the assistant is false and nothing is security-sensitive.

Findings (severity-ordered)

1. Medium · build-on-celo/build-with-ai/celopedia.mdx:28 and :35 · "in your terminal" does not say where, and one command does not install for Claude Code

Defect: the page says to run npx skills add celo-org/celopedia-skills "in your terminal" and then lists Claude Code as supported, but the skills tool installs project-scoped by default (./<agent>/skills/, global only with -g), and the Celopedia README says Claude Code only discovers .claude/skills/ or ~/.claude/skills/, so the bare command alone "is not enough" there (it needs a symlink or a manual copy).
Failure scenario: a reader opens any terminal (home directory, or a new empty folder), pastes the command, sees success, restarts Claude Code as the Note advises, and the skill never appears. The page the PR set out to clarify still leaves the "where" half unanswered, and the restart advice cannot fix the Claude Code case.
Fix shape: line 28, "Run this in a terminal, from the root of the project you want to use it in:"; add one sentence that Claude Code needs an extra step described in the repository README. Do not copy the symlink recipe in; keep one fact in one place.

2. Low · build-on-celo/build-with-ai/celopedia.mdx:35 · "Pasting it into a chat ... won't install it" is broader than what is true

Defect: the code only supports the narrower claim that the Ask AI assistant cannot run it (it has only the docs MCP tools, route.ts tool filter with submit_feedback denylisted, no shell). A coding-agent chat such as Claude Code or Codex has a shell and will run a pasted command if asked.
Failure scenario: a reader using a coding agent reads "pasting it into a chat won't install it" and does not try the one path that works for them.
Fix shape: "The command has to run in a terminal. The Ask AI assistant on these docs can't run it."

3. Low · build-on-celo/build-with-ai/use-docs-with-ai.mdx:206 · pronoun now points at the wrong thing

Defect: the new paragraph (line 204) ends on the link to the Celo MCP Server, so the next paragraph's "It searches the same content as the MCP server above" reads as if "It" is the Celo MCP Server, and "the MCP server above" is ambiguous between the docs MCP server and the Celo MCP Server linked one line earlier.
Failure scenario: a reader concludes the live-chain-data server searches the docs, the opposite of the point of the paragraph before.
Fix shape: line 206 "The assistant searches the same content as the docs MCP server above, ..." (or move the limits paragraph after it).

4. Low · build-on-celo/build-with-ai/use-docs-with-ai.mdx:247 vs :33 and :204 · feedback tool is an unlisted limit

Defect: the page lists submit_feedback as a docs MCP tool and ends with "Ask your assistant to use the submit_feedback tool", but the in-page assistant denylists it (route.ts TOOL_DENYLIST), and the new paragraph names the in-page assistant as a product while its limits list omits this. The text is pre-existing, but the PR makes "your assistant" ambiguous.
Failure scenario: a reader asks the Ask AI assistant to submit feedback, and it cannot.
Fix shape: line 247 "Ask your coding assistant (with the docs MCP server connected) to use ...". Do not document the denylist or its reason.

Verified good

Each limitation checked against celo-org/docs-ai-assistant main:

  • "can only search these docs": app/api/chat/route.ts builds tools from createMCPClient on https://docs.celo.org/mcp minus submit_feedback; the system prompt says "using ONLY the documentation tools". Accurate (the filesystem query tool reads pages, so "search" slightly understates).
  • "refuses questions unrelated to Celo": last system-prompt rule, plus lib/topic-gate.ts returning a refusal before retrieval. Caveat: the gate fails open on a classifier error (catch block), so "refuses" is intent plus a best-effort gate; the page does not promise more.
  • "can't see your wallet or account": the POST body reads only messages and escalate; no client or wallet context is sent. True.
  • "can't send transactions, install skills or run commands": no tool beyond the docs MCP tools is registered. True.
  • "doesn't read live chain data": same tool set. True.
  • "Celo Docs Assistant" name and Ask AI bottom-right: public/widget.js line 194 (panel title), line 191 (launcher "Ask AI"), .launcher { bottom: 20px; right: 20px }.
  • "Try a more thorough answer" re-runs on a stronger model (existing text): escalate selects the escalation tier.
  • Celopedia command: npx skills add celo-org/celopedia-skills is the first install line in the celopedia-skills README and appears in the agent-skills README; the skills tool's own docs give npx skills add <owner/repo> as the basic usage, run in a terminal. Real documented usage; project-scoped unless -g (finding 1).
  • Not published, correctly: rate limit, input cap, 12-message window, tool names the assistant uses, denylist, topic-gate behaviour and fail-open. Nothing on the changed pages helps an attacker beyond what the widget already shows; consistent with docs-ai-assistant issue #3 and docs#2302. Documenting the 12-message memory or input cap for users is optional; if added, keep the numbers out.
  • AGENTS.md: headings sentence case and unchanged, callout is a Note, no promotional wording or product comparison added, links root-relative and /build-on-celo/build-with-ai/mcp/celo-mcp exists, ## Related present, no new anchors added (#ask-a-question-in-the-docs and #install-celopedia resolve in the rendered pages).
  • PR title and body contain no private-source references; refs are public issues.

What the PR got right

  • Targets a real gap (repeated unanswered questions) with the smallest possible change: two files, five lines.
  • Every limit it states is backed by a line of code, and it avoids publishing numbers or internals.
  • Pointing to the Celo MCP Server for live chain data gives the reader the next step instead of a dead end.
  • Merges cleanly; CI green.

Pre-existing, not counted against this PR

  • use-docs-with-ai.mdx description ends with a period and is a long multi-clause sentence (AGENTS.md section 3 wants one sentence, no trailing period); the first paragraph does not name its reader. Candidates for the broader migration.

Merge-order hazards

None. Neither file changed on main since the merge base (the 7 main commits touch ANALYTICS.md, assistant.js, docs.json, wallets, operators, x402 and others). Squash merge; only the "branch up to date" gate applies.

@palango
palango force-pushed the palango/assistant-and-celopedia-answers branch from eb22ee3 to 584a330 Compare October 2, 2026 12:30
@palango

palango commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

All four are in 584a330, rebased onto current main.

  1. The install line now says to run the command in a terminal from the root of the project you want to use it in. The Note adds one sentence that Claude Code needs an extra step after the command, linking the Claude Code section of the celopedia-skills README. I checked both READMEs first: the skills CLI installs to the project by default, and the Celopedia README says npx skills add alone is not enough for Claude Code. The symlink steps stay in the README only.
  2. The Note now says the command has to run in a terminal and that the Ask AI assistant on these docs can't run it. I also dropped the old "see the GitHub repository for tool-specific install notes" clause, since the new README link covers it.
  3. The paragraph now starts "The assistant searches the same content as the docs MCP server above".
  4. The feedback Note now says "Ask your coding assistant (with the docs MCP server connected)". The denylist is not mentioned.

mint broken-links and scripts/check-orphans.sh both pass on the new head.

@palango
palango requested a review from GigaHierz October 2, 2026 12:30

@GigaHierz GigaHierz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

head 584a330 · API headRefOid == refs/pull/2359/head (matched after re-fetch); 2 commits on merge-base 92b8cc6, 4 behind origin/main (a12466b), git merge-tree clean; on the merge into main: mint broken-links --check-redirects "no broken links found", check-orphans "No orphan pages found"; CI "Check for broken links" pass; mint dev on 3422 at 1280 and 412x915: both pages render, zero console errors after warm-up, no horizontal overflow, anchors present

Verdict: REQUEST-CHANGES

Status of earlier findings

  1. MEDIUM, "in your terminal" does not say where / bare command does not install for Claude Code: FIXED. celopedia.mdx:28 now reads "Run this in a terminal, from the root of the project you want to use it in:", and the Note says "Claude Code needs an extra step after this command, described in the repository README". Verified at source: the celopedia-skills README has a ### Claude Code heading under Install that says npx skills add alone is not enough (symlink or copy under .claude/skills/), so the anchor resolves; the skills CLI README lists project scope (./<agent>/skills/) as the default and -g for global, so "from the root of the project" is correct. The symlink recipe is not copied in (one fact, one place).
  2. LOW, "pasting it into a chat won't install it" too broad: FIXED. Now "The command has to run in a terminal. The Ask AI assistant on these docs can't run it." That is the claim the assistant code supports (docs MCP tools only, no shell), and it no longer says anything about coding agents.
  3. LOW, pronoun at use-docs-with-ai.mdx now points at the Celo MCP Server: FIXED. The paragraph at :206 now begins "The assistant searches the same content as the docs MCP server above", so neither "It" nor "the MCP server" can bind to the Celo MCP Server linked in :204.
  4. LOW, submit_feedback told to "your assistant": FIXED. :247 now reads "Ask your coding assistant (with the docs MCP server connected) to use the submit_feedback tool". The tool is still listed in the docs MCP table at :33, which is accurate for the MCP server, and the denylist is not published.

New findings

  1. LOW · PR body, paragraph 1 · The body says "four people pasted npx skills add celo-org/celopedia-skills into the chat", and the three linked weekly issues do not reproduce that number: the exact command appears 3 times (#2320 once, #2324 twice) and a looser count of install-request variants is 5 (#2324 four rows, #2320 one row), with none in #2343.
    Failure scenario: the body is the squash-commit record and the only place the motivating evidence lives; a later reader re-counting from the issues gets a different number and stops trusting the other claims in it.
    Fix shape: edit the PR body to the number you can reproduce (for example "three people pasted the exact command, two more asked it to install the skill") or cite only the issue numbers. No commit needed. Clears the finding.

Verified good

Each limitation re-read at current celo-org/docs-ai-assistant main (last commit 2026-09-02, widget changes only):

  • "can only search these docs": chat route builds its tool set from the docs MCP endpoint at https://docs.celo.org/mcp, minus submit_feedback (TOOL_DENYLIST), and the system prompt says to answer using only the documentation tools. True.
  • "refuses questions unrelated to Celo": last system-prompt rule, plus the topic gate. The gate is best-effort, but the page states intent, not a guarantee. True as worded.
  • "can't see your wallet or account, send transactions, install skills or run commands, doesn't read live chain data": the request body reads only messages and escalate; no client or wallet context; no tool other than the docs MCP tools is registered. True.
  • "Celo Docs Assistant" and the bottom-right "Ask AI" launcher: public/widget.js:194 (panel title), :191 (launcher label), :29 (bottom: 20px; right: 20px).
  • Celo MCP Server link target /build-on-celo/build-with-ai/mcp/celo-mcp exists and its title is "Celo MCP Server"; the page's own sentence at :38 already makes the same split (docs vs live chain data).
  • npx skills add celo-org/celopedia-skills is the install line in the celopedia-skills README; it is the only occurrence of skills add in the repo's mdx, so no other page needs the same fix.
  • Nothing published that helps an attacker: no rate limit, input cap, message window, tool names beyond the existing docs MCP table, denylist or gate behaviour.
  • AGENTS.md: headings unchanged and sentence case, callouts are Note components, no promotional wording or product comparison added, links root-relative without extension, ## Related/closing structure untouched, #install-celopedia and #ask-a-question-in-the-docs ids exist in the rendered pages.
  • Other claims in the body: "the widget calls itself Celo Docs Assistant" true; refs are open public issues (#2320, #2324, #2343), and Refs not Closes is right for weekly-gap issues.

What the PR got right

Every one of the four earlier findings was fixed in a single follow-up commit with the wording I suggested or better; the author also checked both READMEs before writing the Claude Code sentence and kept the symlink recipe in the README rather than duplicating it. The PR is two files and a handful of lines, and each stated limit traces to a line of code.

Merge-order hazards

  • None against main: merges cleanly into a12466b, and the 4 main commits since the merge base do not touch either file.
  • #2368 (the Build tab merge) moves use-docs-with-ai.mdx to build/agents/use-docs-with-ai.mdx (it appears in that PR's file list), and presumably celopedia.mdx as well. Whichever merges second conflicts or needs a path re-apply: if this PR lands first, #2368 carries the new text across the rename (git handles it as a rename in most cases); if #2368 lands first, this PR must be rebased onto the new paths. The /build-on-celo/build-with-ai/mcp/celo-mcp link added here will also need the new build/ path then.
  • Pre-existing, not counted: use-docs-with-ai.mdx description ends with a period and runs to several clauses (AGENTS.md section 3), and the first paragraph does not name its reader.

@palango

palango commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

Fixed the count in the description. The issues show the exact command three times (#2320 once, #2324 twice) plus two looser install requests in #2324, and the body now says that.

@palango
palango enabled auto-merge (squash) October 6, 2026 12:29
@palango
palango requested a review from GigaHierz October 6, 2026 12:40

@GigaHierz GigaHierz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

head 584a330 · fetch matched headRefOid; main 7b8f97c (#2368 layout); merge-tree clean via rename detection; trial merge: broken-links --check-redirects pass, check-orphans pass, CI green; the merged use-docs-with-ai.mdx still links /build-on-celo/build-with-ai/mcp/celo-mcp (redirect only, rendered href observed); mint dev (smoke test) 0 console errors

Verdict: REQUEST-CHANGES

Status of earlier findings

  1. PR body "four people pasted the install command": FIXED. The body now reads "pasted ... three times (#2320, #2324), with two more requests in #2324 asking it to install the skill". Re-counted from the issue tables (data only): the exact command appears once in #2320 and twice in #2324 (one of those two carries a trailing install request), and two further looser variants appear in #2324. That is 3 exact plus 2 looser, which is what the body says. Its first paragraph also cites #2343 for the "what is it / what can it do" topic, and #2343 does contain such a question.
  2. Earlier wording findings (terminal location and Claude Code, "pasting into a chat", pronoun at the limits paragraph, submit_feedback told to "your assistant"): still fixed in the merged tree; the text is as in the diff below.
  3. CI: Check for broken links, GitGuardian and StepSecurity pass; Mintlify Deployment skipping. No other review threads or comments are open.

New findings

  1. LOW · build/agents/use-docs-with-ai.mdx:204 (the path after #2368; today build-on-celo/build-with-ai/use-docs-with-ai.mdx:204) · The new sentence links the Celo MCP Server as /build-on-celo/build-with-ai/mcp/celo-mcp. On current main that path exists only as a docs.json redirect (/build-on-celo/build-with-ai/:slug* to /build/agents/:slug*); the page itself is build/agents/mcp/celo-mcp.mdx. Failure scenario: after the PR merges, the link resolves through a redirect, mint broken-links --check-redirects still passes so nothing flags it, and it is the only internal /build-on-celo/ link left in any .mdx file; AGENTS.md section 2 asks for links to the final page and the next redirect clean-up would break it. Fix shape: one line, change the href to /build/agents/mcp/celo-mcp (done as part of the rebase below).

Verified good

  • Trial merge (git merge-tree --write-tree origin/main refs/remotes/pr/2359, exit 0): the edits follow the renames to build/agents/celopedia.mdx and build/agents/use-docs-with-ai.mdx; diff against main is 2 files, +7/-5, no leftovers under build-on-celo/.
  • Each stated limitation re-read at celo-org/docs-ai-assistant main (newest commit 2026-09-02, UI-only, nothing changed since the last review):
    • Only searches the docs: app/api/chat/route.ts builds its tools from the MCP client on https://docs.celo.org/mcp minus TOOL_DENYLIST (submit_feedback); system prompt says to answer using ONLY the documentation tools. True.
    • Refuses questions unrelated to Celo: final system-prompt rule plus a topic gate that fails open on a classifier error. The page states the intent, not a guarantee. True as worded.
    • Cannot see the wallet or account, send transactions, install skills, run commands, or read live chain data: the request body carries only messages and escalate, and no tool beyond the docs MCP tools is registered. True.
    • Name and launcher: public/widget.js panel title "Celo Docs Assistant" (:194), launcher "Ask AI" (:191), fixed bottom 20px / right 20px (:29). True.
    • Install line: npx skills add celo-org/celopedia-skills is the first install line in the celopedia-skills README, which also has a ### Claude Code heading saying that command alone is not enough there, so the #claude-code anchor in the Note resolves.
  • Not published, correctly: rate limit, input cap, message window, denylist reason, gate behaviour.
  • Rendered /build/agents/use-docs-with-ai on the trial merge with mint dev on :3532 (smoke test): "Celo Docs Assistant" present, 0 console errors, no horizontal scroll at 1280. Celopedia page serves the new "Run this in a terminal" line.
  • Style: sentence-case headings untouched, callout stays a Note, no promotional or comparative wording, links root-relative without extension (apart from finding 1), ## Related untouched.
  • PR title and body contain no private-source references; Refs (not Closes) is right for weekly-gap issues.

What the PR got right

Two files and a handful of lines, aimed at a repeated unanswered question; every limit it states traces to a line of code; no internals or numbers published; each earlier finding fixed with a commit or a body edit.

Merge-order hazards and rebase work

  • Main has the layout change (#2368): build-on-celo/build-with-ai/{celopedia,use-docs-with-ai}.mdx are now build/agents/{celopedia,use-docs-with-ai}.mdx. There is no textual conflict; git and GitHub follow the rename and the trial merge is clean.
  • Author's rebase work (one commit): rebase onto current main and change the href on build/agents/use-docs-with-ai.mdx:204 to /build/agents/mcp/celo-mcp; re-run mint broken-links --check-redirects and grep -rn 'build-on-celo' --include='*.mdx' . (expect no hits).
  • No other content hazard: neither page is touched by other PRs reviewed in this batch.

@palango
palango force-pushed the palango/assistant-and-celopedia-answers branch from 584a330 to b3f9b5c Compare October 7, 2026 05:13
@palango

palango commented Oct 7, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto current main. Git followed the rename, so both edits now sit in build/agents/celopedia.mdx and build/agents/use-docs-with-ai.mdx with no conflicts.

The Celo MCP Server link is fixed in b3f9b5c and now points at /build/agents/mcp/celo-mcp, which is in the nav. A search for build-on-celo across the .mdx files now finds nothing.

mint broken-links --check-redirects, mint validate and scripts/check-orphans.sh all pass on the new head.

@palango
palango requested a review from GigaHierz October 7, 2026 05:13

@GigaHierz GigaHierz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

head b3f9b5c · API headRefOid == refs/pull/2359/head (fetched, matched); merge base is current origin/main 7f5d9bb, git merge-tree --write-tree origin/main refs/remotes/pr/2359 exit 0 (no conflict, 3 commits, 2 files); git grep build-on-celo -- "*.mdx" returns nothing; mint broken-links --check-redirects "no broken links found", mint validate "build validation passed", check-orphans.sh "No orphan pages found"; gh pr checks: Check for broken links, GitGuardian, StepSecurity pass; mint dev on :3722 (own server, restarted once because the shared cache first served pre-PR text; then served the PR text): 0 console errors on /build/agents/use-docs-with-ai, no horizontal overflow

Verdict: APPROVE

Status of earlier findings

  1. LOW, link through /build-on-celo/build-with-ai/mcp/celo-mcp: FIXED. The sentence at build/agents/use-docs-with-ai.mdx now links /build/agents/mcp/celo-mcp; the served page renders that href, the page exists (build/agents/mcp/celo-mcp.mdx, title "Celo MCP Server", nav entry docs.json:195). The author's claim holds: git grep finds no build-on-celo string in any .mdx (it remains only in docs.json redirect sources).
  2. Earlier wording and body-count findings (terminal location and Claude Code, "docs MCP server" wording, submit_feedback told to a "coding assistant", install-command count in the body): still FIXED in the head diff.

New findings

None.

Verified good

  • Each stated assistant limitation, re-read at celo-org/docs-ai-assistant main (newest commit 2026-09-02, UI-only; no change since the last review):
    • Searches only the docs: app/api/chat/route.ts builds its tool set from the MCP client at https://docs.celo.org/mcp minus TOOL_DENYLIST (submit_feedback); the system prompt says to answer using ONLY the documentation tools. True.
    • Refuses unrelated questions: system-prompt rule "Refuse questions unrelated to Celo or its documentation", plus a topic gate. Stated as intent, not a guarantee; true as worded.
    • Cannot see a wallet, send transactions, install skills, run commands or read live chain data: no tool beyond the docs MCP tools is registered. True. The Celo MCP Server page covers balances, blocks, transactions, gas and governance, so the pointer for live data is right.
    • Name and button: public/widget.js panel title "Celo Docs Assistant", launcher "Ask AI", fixed bottom-right. True.
  • Celopedia: npx skills add celo-org/celopedia-skills is the first install line of the celopedia-skills README, which has a ### Claude Code heading saying npx skills add alone is not enough there, so #claude-code resolves; the served Note renders https://github.com/celo-org/celopedia-skills#claude-code.
  • Nothing security-sensitive is published: no rate limit, input cap, message window, denylist reason or gate behaviour; the submit_feedback Note only tells coding assistants with the docs MCP server connected to use the tool, as before.
  • Style: sentence-case headings untouched, callout stays a Note, no promotional or comparative wording, links root-relative without extension, ## Related untouched. Served /build/agents/celopedia shows "Run this in a terminal" and the new Note.
  • PR title and body contain no private-source references; Refs, not Closes, for the weekly-gap issues.

What the PR got right

Two files and a handful of lines aimed at a repeated unanswered question; every limit it states traces to a line of the assistant's code; no internals or numbers published; each review round was answered with a commit or a body edit.

Merge-order hazards

None. The branch sits on current main and follows the #2368 renames. No other open PR I know of edits build/agents/celopedia.mdx or build/agents/use-docs-with-ai.mdx.

@GigaHierz
GigaHierz merged commit 528da8c into main Oct 7, 2026
5 checks passed
@GigaHierz
GigaHierz deleted the palango/assistant-and-celopedia-answers branch October 7, 2026 07:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants