Repository navigation
feat(bird): sistema de logging de execução para benchmarks BIRD - #22
Open
LuizCorrei4 wants to merge 9 commits into
Open
LuizCorrei4 wants to merge 9 commits into
LuizCorrei4 wants to merge 9 commits into
Conversation
… + Factory pattern Updated the bird and the graph the handle kwargs like temperature
- Implement BirdExecutionLogger in src/bird/logger.py with stdout Tee interception and structured .log output. - Log run header (config, filters, timestamp), per-query block (Question, Evidence, RAG Schema, Node trace, Attempt history with prompt/CoT/SQL/sandbox status, and Gold comparison), and run footer (EX accuracy, Pass@1, tokens, elapsed time). - Integrate logger into scripts/test_bird_eval.py with CLI flags --log (on/off) and --log-file. - Save execution log automatically alongside .csv, .md, and .json in reports/bird/run_TIMESTAMP/. - Add comprehensive test coverage in tests/test_bird_eval.py (22 tests passing).
… we can pass a provider like 'openai' to specify and use models that do not have the gpt keyword
…isable schemacrawler by default - Pass model_kwargs to InsightEngine in test_bird_eval.py - Add --schemacrawler and --exploration-selector flags to test_bird_eval.py - Change global usar_schemacrawler default to False across engine, runtime, and CLI - Log enrich_rag, schemacrawler, exploration_selector, and model_kwargs in ExecutionLogger - Update BENCHMARK.md and README.md documentation - Add test coverage for new flags in test_bird_eval.py
…and disable SchemaCrawler by default
- Implement results_f1_score in src/bird/metrics.py for column-level Precision, Recall, and F1 evaluation
- Expand BirdCSVReporter to 19 columns and include Recall/Precision/F1 in Markdown Executive Summary, Difficulty Breakdown, and Failure
Analysis
- Log column-level metrics per query and in the run summary footer in BirdExecutionLogger
- Add CLI flags for --schemacrawler (disabled by default), --enrich-rag, and --use-exploration-selector in scripts/test_bird_eval.py
- Add unit tests in tests/test_bird_eval.py and update src/bird/BENCHMARK.md documentation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Descrição
Implementa a versão simples inicial de logging para as execuções do benchmark BIRD (
Ata 16 - Item 5: Observabilidade e Logging de Execuções).Esta funcionalidade permite que a equipe (Bernardo, Antonio e demais) execute testes de novos modelos (ex: GPT-6 Luna) mantendo rastreabilidade completa do raciocínio, prompts enviados, schemas RAG, prints de execução de nós e histórico de tentativas de autocorreção.
O que foi feito
src/bird/logger.py:BirdExecutionLoggercom context managercapture_trace()via_TeeStream(intercepta prints de console sem suprimir a saída ao vivo do terminal).=== [START QID: ...] ===e=== [END QID: ...] ===).[PLANEJADOR],[RETRIEVER],[SCHEMA],[DATA_EXPLORATION],[EXECUTOR]).scripts/test_bird_eval.py:--log {on,off}(padrão:on) e--log-file <caminho_opcional>..logé gerado automaticamente na mesma pasta da execução (reports/bird/run_YYYY-MM-DD_HH-MM-SS/) ao lado dos arquivos.csv,.mde.json.src/bird/__init__.py:BirdExecutionLogger.tests/test_bird_eval.py:--log off).--log-file).temp_notes/plano_implementacao_logging_bird.mdcom arquitetura técnica e plano de evolução (subpastas por query com prompts brutos + integração com LangSmith / Arize Phoenix).Como testar