Skip to content

Add dashAI document data structure and database model - #920

Merged
cristian-tamblay merged 9 commits into
developfrom
feat/dashai-document
Oct 14, 2026
Merged

cristian-tamblay merged 9 commits into
developfrom
feat/dashai-document

Conversation

@Irozuku

@Irozuku Irozuku commented Oct 14, 2026

Copy link
Copy Markdown
Collaborator

Summary

First of a stacked series that turns the Datasets module into a Data module holding two kinds of data: datasets and documents. This branch only adds the document data structure and its database record; nothing in the UI changes yet.

  • DashAIData is a new common base for DashAIDataset and the new DashAIDocument.
  • DashAIDocument holds parsed content split by DashAI type (Text, Image) in Arrow tables, keeping the reading order (position) and the source page of every item, so tools that already restrict themselves by DashAI type can work on documents.
  • Documents are saved to and loaded from disk (one Arrow file per content type plus document.json), and every path into a document validates its contents.
  • A data_document table and the DataDocument model store uploaded documents, with a DocumentStatus enum for their parsing job.

Type of Change

Check all that apply like this [x]:

  • Backend change
  • Frontend change
  • CI / Workflow change
  • Build / Packaging change
  • Bug fix
  • Documentation

Changes (by file)

  • DashAI/back/dataloaders/classes/dashai_data.py: new DashAIData base class shared by datasets and documents (data_kind()).
  • DashAI/back/dataloaders/classes/dashai_dataset.py: DashAIDataset now extends DashAIData.
  • DashAI/back/dataloaders/classes/dashai_document.py: DocumentItem, DashAIDocument (from_items, items, types, count, as_dataset, with_content), save_document / load_document with atomic writes and strict validation (types, non null content, image bytes, pages from 1, positions unique across types).
  • DashAI/back/dependencies/database/models.py: DataDocument model (name, file info and hash, dataloader and params, content counts, pages, folder, job id, status).
  • DashAI/back/core/enums/status.py: DocumentStatus.
  • DashAI/alembic/versions/a7c3e91b5d20_add_data_document.py: migration creating data_document.
  • tests/back/dataloaders/test_dashai_data.py, test_dashai_document.py, test_document_persistence.py: document structure, reading order and save/load round trips.
  • tests/back/dependencies/test_data_document_model.py, tests/back/migrations/test_add_data_document.py: model and migration (upgrade, downgrade, ORM parity).

Notes (optional)

  • Base branch: develop. The next PR, feat/document-dataloaders, builds on this one.
  • Adds migration a7c3e91b5d20; starting the app upgrades the local database.
  • The legacy RAG document table is untouched here; it is replaced in feat/rag-data-documents.

@Irozuku Irozuku added the back Backend work label Oct 14, 2026
@Irozuku
Irozuku changed the base branch from production to develop October 14, 2026 13:50
@Irozuku
Irozuku added this pull request to stack #922 October 14, 2026 13:52
@cristian-tamblay
cristian-tamblay merged commit 4cdb5d3 into develop Oct 14, 2026
18 checks passed
@cristian-tamblay
cristian-tamblay deleted the feat/dashai-document branch October 14, 2026 21:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

back Backend work

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants