Skip to content

Add durability to Databricks Lakeflow - #1223

Open
yaron2 wants to merge 1 commit into
dapr:mainfrom
yaron2:databricks-1
Open

yaron2 wants to merge 1 commit into
dapr:mainfrom
yaron2:databricks-1

Conversation

@yaron2

@yaron2 yaron2 commented Sep 18, 2026

Copy link
Copy Markdown
Member

This PR adds an extension that turns Databricks Lakeflow data into durable business actions without building your own retry-safe orchestration layer.

dapr.ext.databricks fixes this with one function call:

  • One line, register_workflow_sink(...), turns any Lakeflow streaming table into a trigger for a durable Dapr Workflow - retries, timers, human approval steps, and multi-day processes, all running independently of your pipeline.
  • Retry- and refresh-safe by default. Every action gets a deterministic, collision-safe identity, so a Lakeflow micro-batch retry - or a full pipeline refresh - is automatically recognized as "already handled," never
    re-triggered.
  • Decouples fast ingestion from slow business logic. A workflow that waits three days for a human doesn't hold up your streaming pipeline for a second - Lakeflow hands off and moves on.

Signed-off-by: yaron2 <schneider.yaron@live.com>
@yaron2
yaron2 requested review from a team as code owners September 18, 2026 22:48
@codecov

codecov Bot commented Sep 18, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.77049% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.68%. Comparing base (74c3c54) to head (5ae1cb8).

Files with missing lines Patch % Lines
dapr/ext/databricks/batch_handler.py 97.14% 2 Missing ⚠️
dapr/ext/databricks/identity.py 97.91% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1223      +/-   ##
==========================================
+ Coverage   83.32%   83.68%   +0.35%     
==========================================
  Files         123      132       +9     
  Lines       10250    10494     +244     
==========================================
+ Hits         8541     8782     +241     
- Misses       1709     1712       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant