Skip to main content

Airflow for Beginners — I

 

Image for post

PHASE:- 1 ( Introduction and Installation )

Introduction

Apache Airflow is a platform for programmatically author schedule and monitor workflows. It is one of the best workflow management system.

Airflow was originally developed by Airbnb to manage its data-based operations with a fast-growing data set. Airflow is undergoing incubation at Apache Software foundation as Airbnb has decided to open source it under Apache certification.

Apache airflow makes your workflow a little bit simple and organized by allowing you to divide it into small independent task units. Easy to organize and easy to schedule. Your entire workflow can be converted into a DAG with Airflow.

Once your workflows are defined by your code it becomes more maintainable. With the feature-rich user interface your workflow pipelines can be easily visualized, monitored and troubleshot.

Airflow also provides a rich set of command-line utilities that can be used to perform complex operations on DAG.

In short, airflow is a platform where you can see and schedule your run for the script.

Installation

you can simply instaled the airflow by PIP.

pip install apache-airflow

so this command will installed your airflow local.

To start the airflow run the following command and you are set up with airflow.

airflow initdb   # initating the database for storing the DAGsairflow scheduler  # initating the scheduler for triggering on scheduled timeairflow webserver  # initating the webserver for UI

after this execution, your airflow is fully setup. You can find it in:

localhost:8080

so now you have fully set up with airflow and about execution and creating DAG, we will see in the next article.

THANKS FOR YOUR SUPPORT

Labels:

Comments

Post a Comment

Popular posts from this blog

NEW TREND OF DATA SCIENCE: REINFORCEMENT LEARNING

Reinforcement Learning (RL) is a machine learning method that empowers a specialist to learn in an intuitive environment by performing trial and error utilizing observations from its very own activities and encounters. In spite of the fact that both direct and reinforcement learning use mapping among input and output, not at all like supervised learning where input gave to the specialist is basically the right set of activities for playing out a task, reinforcement learning utilizes prizes and discipline as signs for positive and negative conduct. When compared with unsupervised learning, reinforcement learning is distinctive as far as objectives are taken into consideration. While the objective in unsupervised learning is to discover synonymities and contrasts between data points, in reinforcement learning the objective is to locate a reasonable activity model that would boost the aggregate total reward of the specialist. Reinforcement learning will be a huge thing in Data science in ...
Post a Comment
Read more