Mindful Machines

  As in the previous article   Airflow for beginners — I   we learn about the airflow and how to install the airflow on local. In this article, we will learn about the DAG and how to use the web airflow for scheduling the work and managing it. Let’s start with the DAG and then we will learn about the web airflow. DAG (Directed Acyclic Graph) DAG is a directed graph without a cycle in it. It has a finite number of edges and nodes. In which we can know which thing(node) will come first and we will know the sequence of the nodes. so just think as each node is a work( that you want to execute ) so we can represent the sequencing of the work by the DAG and that’s why DAG is an essential part of the airflow. WEB AIRFLOW So now as you saw that on localhost:8080 if you open it will look like: now, these are the example DAG or work that is added already so for adding your own script(DAG) just follow simple steps mentioned below: Just go to the airflow directory in terminal (it wil...

  PHASE:- 1 ( Introduction and Installation ) Introduction Apache Airflow is a platform for programmatically author schedule and monitor workflows. It is one of the best workflow management system. Airflow was originally developed by Airbnb to manage its data-based operations with a fast-growing data set. Airflow is undergoing incubation at Apache Software foundation as Airbnb has decided to open source it under Apache certification. Apache airflow makes your workflow a little bit simple and organized by allowing you to divide it into small independent task units. Easy to organize and easy to schedule. Your entire workflow can be converted into a DAG with Airflow. Once your workflows are defined by your code it becomes more maintainable. With the feature-rich user interface your workflow pipelines can be easily visualized, monitored and troubleshot. Airflow also provides a rich set of command-line utilities that can be used to perform complex operations on DAG. In short, airflow is...