**Databricks just made a BIG move in the data world. 🚨**
DLT (Delta Live Tables) was once a Databricks-only feature.
**Not anymore.**
Databricks has open-sourced the technology into Apache Spark as **Spark Declarative Pipelines (SDP)**.
In simple words: 👉 You describe **what data pipeline you want** 👉 Spark figures out **how to execute it** 👉 Dependencies are automatically managed 👉 Independent tables can run in parallel 👉 Data quality checks can be defined directly in the pipeline
And the interesting part? 👇
**Your existing DLT code still works.** There’s no forced migration.
The bigger change is this:
**Declarative pipelines are moving from being a Databricks feature to becoming an open Spark standard.**
If you're working with Databricks or Apache Spark, **SDP is definitely something worth learning now.**
Are you already experimenting with **Spark Declarative Pipelines outside Databricks?** 🚀
Software Development Engineer in Test
**Databricks just made a BIG move in the data world. 🚨**
DLT (Delta Live Tables) was once a Databricks-only feature.
**Not anymore.**
Databricks has open-sourced the technology into Apache Spark as **Spark Declarative Pipelines (SDP)**.
In simple words:
👉 You describe **what data pipeline you want**
👉 Spark figures out **how to execute it**
👉 Dependencies are automatically managed
👉 Independent tables can run in parallel
👉 Data quality checks can be defined directly in the pipeline
And the interesting part? 👇
**Your existing DLT code still works.**
There’s no forced migration.
The bigger change is this:
**Declarative pipelines are moving from being a Databricks feature to becoming an open Spark standard.**
If you're working with Databricks or Apache Spark, **SDP is definitely something worth learning now.**
Are you already experimenting with **Spark Declarative Pipelines outside Databricks?** 🚀
1 week ago (edited) | [YT] | 0
View 0 replies