Python Delta Local Pipeline
This article provides a guide on how to execute a simple Delta Table copy locally without Spark using the RTDIP SDK. This pipeline was tested on an M2 Macbook Pro using VS Code in a Python (3.10) environment.
Prerequisites
This pipeline job requires the packages:
Components
| Name | Description |
|---|---|
| PythonDeltaSource | Reads data from a Delta Table. |
| PythonDeltaDestination | Writes to a Delta table. |
Example
Below is an example of how to read from and write to Delta Tables locally without the need for Spark
from rtdip_sdk.pipelines.sources.python.delta import PythonDeltaSource
from rtdip_sdk.pipelines.destinations.python.delta import PythonDeltaDestination
source = PythonDeltaSource("{/path/to/source/table}").read_batch()
destination = PythonDeltaDestination(source, "{/path/to/destination/table}", mode="append").write_batch()