Automate Marketing Initiatives with Salesforce Marketing Cloud Learn More

How to Create and Run DAGs in Apache Airflow

Apache Airflow is an open source distributed workflow management platform built for data orchestration. Maxime Beauchemin first started his Airflow project on his Airbnb. After the project’s success, the Apache Software Foundation quickly adopted his Airflow project. Initially, he was hired as an incubator project in 2016 and later as a top-level project in 2019. Airflow allows users to write programmatically, schedule, and control the data pipelines they monitor. A key feature of Airflow is that it will enable users to create planned data pipelines using a flexible Python framework easily.

Introduction to DAG – Directed Acyclic Graph

DAG is the essential component of Apache Airflow. DAG stands for Directed Acyclic Graph. A Directed Acyclic Graph defined in a Python script representing the DAG structure (tasks and their dependencies) as code. This graph has nodes and edges; the edges should always be direct, so they shouldn’t have loops. In other words, a DAG is a data pipeline. A node in a DAG is a task such as downloading a file from S3, querying a MySQL database, or sending an email, and we’ll cover planning a DAG and other basics, covering these topics step-by-step later. Get your first workflow to develop. A simple operator monitored and scheduled by Airflow.

Defining DAG

A DAG is a collection of tasks organized to reflect relationships and dependencies. One advantage of this DAG model is that it provides a relatively simple technique for executing pipelines. Alice says that another benefit of task-based channels is that they are partitioned cleanly into discrete tasks instead of relying on a single monolithic script for all work.


Acyclic functions are essential because they are simple and prevent tasks from getting caught up in circular dependencies. Airflow takes advantage of the acyclic nature of DAGs to solve and execute these task graphs efficiently.

Airflow DAGs Best Practices

Follow the steps below to implement an Airflow DAG on your system.


  • Writing Clean DAGs
  • Designing Reproducible Tasks
  • Handling Data Efficiently
  • Managing the Resources

1. Writing Clean DAGs

It’s easy to get confused when creating an Airflow DAG. For example, DAG code can quickly become unnecessarily complex or difficult to understand, especially when the team members make their DAG vastly different programming styles.

  • Use style conventions: Adopting a consistent and clean programming style and applying it consistently to all Apache Airflow DAGs is one of the first steps to creating clean and consistent DAGs. It’s one. When writing code, the easiest way to make it more transparent and understandable is to use commonly used styles.
  • Centralized credential management: As the Airflow DAG interacts with various systems, many other credentials generate, such as databases, cloud storage, etc. Fortunately, getting connection data from the Airflow connection store makes it easy to persist credentials for custom code.
  • Group tasks together using task groups: Complex DAG airflow can be challenging to understand due to the number of operations required. The new Airflow 2 workgroup feature helps manage these complex systems. Task groups effectively divide tasks into smaller groups to make the DAG structure more manageable and easier to understand.

2. Designing Reproducible Tasks

Aside from developing good DAG code, one of the most demanding parts of creating a successful DAG is making tasks reproducible. It means the user can rerun the job and get the same results, even if the study ran at a different time.

  • Tasks must always be idempotent: Idempotence is one of the essential properties of a good airflow task. Idempotence ensures consistency and resilience in the face of failures. No matter how often you run an idempotent job, the result is always the same.
  • Task results should be deterministic: Creating reproducible tasks and DAGs should be deterministic. A deterministic job should always return the same output given its input.
  • Designing studies using a functional paradigm: It’s easier to create jobs using a functional programming paradigm. Practical programming is writing computer programs that treat computation primarily as an application of mathematical functions and avoid modifying data or changing states.

3. Handling Data Efficiently

Airflow DAGs that process large amounts of data should be carefully designed to be as efficient as possible.

  • Limit the data processed: Limiting data processing to the minimum amount of data necessary to achieve the intended result is the most effective approach to data management. It includes thoroughly examining data sources and assessing whether they are required.
  • Incremental Processing: The main idea behind incremental processing is to split the data into (time-based) ranges and process each DAG run independently. Users can take advantage of incremental processing by running filter/aggregate processes at the overall process stage and performing extensive analysis of the reduced output.
  • Don’t store data on your local file system: When working with data in Airflow, you may want to write data to your local system. As a result, Airflow runs multiple tasks in parallel, and downstream tasks may need help accessing it. The easiest way around this problem is to use a shared memory that all her Airflow employees can access and perform tasks simultaneously.

4. Managing the Resources

When dealing with large volumes of data, it can overburden the Airflow Cluster. As a result, properly managing resources can aid in the reduction of this burden.

  • Managing Concurrency using Pools: When performing many processes in parallel, numerous tasks may require access to the same resource. Airflow uses resource pools to regulate how many jobs can access a resource. Each collection has a set number of slots that offer access to the associated resource.
  • Detect long-running tasks with SLAs and alerts: Airflow’s SLA (Service Level Agreement) mechanism allows users to track job performance. This mechanism enables a user to assign her SLA timeout to her DAG effectively, and if even one of the DAG tasks takes longer than her SLA timeout specified, he will tell Airflow the user will notify you.

Creating your first DAG

In Airflow, a DAG is a Python script containing a set of tasks and their dependencies. What each task does is determined by the task’s operator. For example, defining a job with a Python Operator means that the investigation consists of running Python code.


To create our first DAG, let’s first start by importing the necessary modules:



# We start by importing the DAG object

from airflow import DAG

# We need to import the operators we will use in our tasks From

Airflow.operators.bash_operator import BashOperator

# Then the days ago

function from Airflow.utils Import from date import days ago


Then you can define a dictionary containing the default arguments you want to pass to the DAG. Then apply these arguments to all operators in the DAG.


# Initialize the default arguments passed to the DAG

default_args = {

‘owner’: ‘airflow,’

‘start_date’: days_ago(5),

’email’: [‘’],

’email_on_failure’: False,

’email_on_retry’: False,

‘retry’: 1,

‘retry_delay’: time lag (minutes=5),



As you can see, Airflow provides several standard arguments that make DAG configuration even more accessible. For example, you can easily define the number of retries and the retry delay for DAG execution.


Then you can use the DAG itself. Like:


my_first_dag = DAG(



description=’First DAG’,




The first parameter, first_dag, represents the ID of the DAG, and the scheduling interval represents the interval between two executions of the DAG.


The next step is to define the DAG’s tasks:


task_1 = BashOperator(

task_id=’ first_task ‘,

bash_command=’echo 1′,



task_2 = BashOperator(


bash_command=’echo 2′,




Finally, we need to specify the dependencies. In this case task_2 must run after task_1:



Running the DAG

DAGs should default in the ~/airflow/dags folder. After first testing various tasks using the ‘airflow test’ command to ensure everything configures correctly, you can run the DAG for a specific date range using the ‘airflow backfill’ command:


airflow backfill my_first_dag -s 2020-03-01 -e 2020-03-05


Finally, start the airflow scheduler with the airflow scheduler command, and Airflow will ensure that the DAG runs at the defined interval. Increase.


As we saw today, Apache Airflow is very simple for implementing essential ETL pipelines. Now that you’ve seen the most common Python operators, you know how to run arbitrary Python functions in DAG tasks.


I also know how to transfer data between tasks using XCOM. It is a concept you need to know in Airflow.

The article covered the basics of DAG and the properties required for running DAG and explained in-depth DAG scheduling. The upcoming articles will present details on Operators.


In this article, we have seen the features of Apache Airflow and its user interface components, and we have created a simple DAG. In the upcoming blogs, we will discuss some more concepts like variables and branching and will create a more complex workflow.


This article taught us that an Apache Airflow workflow is a DAG that clearly defines tasks and their dependencies. Similarly, we learned some best practices while creating Airflow DAGs. Many large organizations today rely on Airflow to coordinate many critical data processes. Integrating data from Airflow and other data sources into your cloud data warehouse or destination for further business analysis is essential.

A place for big ideas.

Reimagine organizational performance while delivering a delightful experience through optimized operations.

Top Stories

Enhancing GraphQL with Roles and Permissions
Enhancing GraphQL with Roles and Permissions
GraphQL has gained popularity due to its flexibility and efficiency in fetching data from the server. However, with great power comes great responsibility, especially when it comes to managing access to sensitive data. In this article, we'll explore how to implement roles and permissions in GraphQL APIs to ensure that
Exploring GraphQL with FastAPI A Practical Guide to begin with
Exploring GraphQL with FastAPI: A Practical Guide to begin with
GraphQL serves as a language for asking questions to APIs and as a tool for getting answers from existing data. It's like a translator that helps your application talk to databases and other systems. When you use GraphQL, you're like a detective asking for specific clues – you only get
Train tensorflow object detection model with custom data
Train Tensorflow Object Detection Model With Custom Data
In this article, we'll show you how to make your own tool that can recognize things in pictures. It's called an object detection model, and we'll use TensorFlow to teach it. We'll explain each step clearly, from gathering pictures, preparing data to telling the model what to look for in
Software Development Team
How to deploy chat completion model over EC2?
The Chat Completion model revolutionizes conversational experiences by proficiently generating responses derived from given contexts and inquiries. This innovative system harnesses the power of the Mistral-7B-Instruct-v0.2 model, renowned for its sophisticated natural language processing capabilities. The model can be accessed via Hugging Face at – on a dedicated GPU server g4dn.2xlarge,
How to deploy multilingual embedding model over EC2
How to deploy multilingual embedding model over EC2?
The multilingual embedding model represents a state-of-the-art solution designed to produce embeddings tailored explicitly for chat responses. By aligning paragraph embeddings, it ensures that the resulting replies are not only contextually relevant but also coherent. This is achieved through leveraging the advanced capabilities of the BAAI/bge-m3 model, widely recognized for
Tracking and Analyzing E commerce Performance with Odoo Analytics
Tracking and Analyzing E-commerce Performance with Odoo Analytics
Odoo is famous for its customizable nature. Businesses from around the world choose Odoo because of its scalability and modality. Regardless of the business size, Odoo can cater to the unique and diverse needs of any company. Odoo has proven its capacity and robust quality in terms of helping businesses


          Keep an eye on your inbox for the PDF, it's on its way!

          If you don't see it in your inbox, don't forget to give your junk folder a quick peek. Just in case.

              You have successfully subscribed to the newsletter

              There was an error while trying to send your request. Please try again.

              Zehntech will use the information you provide on this form to be in touch with you and to provide updates and marketing.