dagron

High-performance DAG execution engine for Python, powered by Rust.

dagron lets you define directed acyclic graphs of tasks and execute them with maximum parallelism, rich observability, and dozens of execution strategies — from simple thread pools to distributed clusters.

The core graph data structure lives in Rust (via PyO3) for zero-copy speed, while the Python layer provides an ergonomic API for building, executing, analyzing, and visualizing your DAGs.

import dagron

dag = (
    dagron.DAG.builder()
    .add_node("extract")
    .add_node("transform")
    .add_node("load")
    .add_edge("extract", "transform")
    .add_edge("transform", "load")
    .build()
)

result = dagron.DAGExecutor(dag).execute({
    "extract": lambda: fetch_data(),
    "transform": lambda: clean(result),
    "load": lambda: write_to_db(result),
})

Features

DAG Builder

Fluent builder pattern, from_records, and Pipeline/task decorator for defining DAGs.

Guide →API →

Parallel Execution

Thread-pool and async executors with topological scheduling and cost-aware planning.

Guide →API →

Incremental Execution

Early-cutoff recomputation — only re-execute what changed.

Guide →API →

Caching

Content-addressable Merkle-tree caching with pluggable backends.

Guide →API →

Checkpointing

Save progress to disk and resume after failures.

Guide →API →

Conditional Execution

Predicate-gated edges that skip branches at runtime.

Guide →API →

Dynamic DAGs

Expand the graph at runtime based on node results.

Guide →API →

Approval Gates

Human-in-the-loop gates that pause execution until approved.

Guide →API →

Resource Scheduling

GPU, CPU, and memory-aware scheduling with blocking acquire/release.

Guide →API →

Distributed Execution

Pluggable backends: threads, multiprocessing, Ray, Celery.

Guide →API →

Tracing & Profiling

Chrome-compatible execution traces and critical-path analysis.

Guide →API →

Graph Analysis

Explain nodes, what-if analysis, lineage tracking, linting, and query DSL.

Guide →API →

Contracts

Type contracts across edges validated at build time.

Guide →API →

Templates

Parameterized DAG templates with placeholder expansion.

Guide →API →

Versioning

Append-only mutation log with time-travel, diffing, and forking.

Guide →API →

DataFrames

Schema validation at edge boundaries for pandas/polars pipelines.

Guide →API →

Plugins & Hooks

Event-driven plugin system with hook registry and auto-discovery.

Guide →API →

Visualization

ASCII, SVG, Mermaid, and live web dashboard.

Guide →API →

Getting Started

Install dagron from PyPI:

pip install dagron

Then head to the Getting Started guide.