Powerful REST API in Airflow 2.0 — what do you need to know?

The business world changes rather dynamically. Many new ideas, methods, and tools are being created all the time. Recently, the Apache Airflow team has introduced a new version of Airflow(2.0). Airflow is a popular piece of workflow management software for creating programs, scheduling and monitoring workflows. This tool has been significantly modified, and new features have been added, including Airflow API REST — this could improve third-party application integration a lot. 

You can’t assess the strengths of the new Airflow 2.0 without installing it. The old version of the software is still available, and if you have not yet upgraded, we’ll be happy to tell you more about how the workflow management tool has evolved. In this article we focus on Airflow REST API and its advantages for Airflow 2.0 business users. 

Airflow version 2.0 — what is new? 

All software has its flaws. As there is no one perfect tool available on the market, it is important to keep improving products in order to stay competitive. Companies are always trying to make their systems more intuitive and easier for users to work with. They also seek to fix the most annoying problems, boost overall efficiency and add new features in order to make their software more attractive to clients. Has the Apache Airflow team succeeded in this?

In fact, there are some significant changes in the new version of Airflow. Users of this software are now provided with:

• Transparent user interface — a new, clear UI with an auto-refresh feature that enables users to access and review information about workflow progress easily.
• Efficient scheduler — in the previous version of Airflow, running DAGs was very time-consuming. Fortunately, the Airflow team has improved its performance.
• “Smarter” sensors — in the new release, Airflow uses different logic to operate more resource-efficiently.
• DAG serialization and DAG versioning — with this additional support, the user can store many versions of serialized DAG representation in the database, which makes it easier to configure Airflow and deploy DAGs.
• Powerful Airflow REST API —  this new feature simplifies third-parties integration. 

With REST API, Airflow can be a great workflow management tool. Do you want to know how the full REST API has changed this software?

SOAP API vs REST API

Before we move to how REST API has changed the way Airflow works, we want to briefly explain the main differences between SOAP API and REST API. An API (application programming interface) is simply a software intermediary — or, to be precise, a collection of rules — that strictly define how two applications should communicate with each other.

SOAP (Simple Object Access Protocol) was designed first. The goal was to create a simple method for exchanging data for programs built on different platforms or using different programming languages. REST (Representational State Transfer), on the other hand, is an architectural pattern. It was created to make it easier to work with complex components.

Both solutions have their drawbacks and benefits, and it’s important to know in which situations each of them should be used. REST API is becoming very popular these days as it, unlike SOAP, allows you to work with different data formats such as Plain text, HTML, XML or JSON, although the most preferred format for transferring data is still JSON. If you are interested in further details, don’t hesitate to ask us for more information. 

Airflow’s API before and after

REST API is a rather unique and preferred API type and has multiple applications. The new version of Airflow should make all of its users very happy. Till now this software came only with an experimental solution in terms of API, and Airflow’s experimental REST API had only limited functionality. 

Now, Airflow’s version 2.0 offers full REST API that gives you a lot of possibilities. Airflow REST API triggers DAGs easily, allowing you to check all of them and manage task instances smoothly — but this represents only a small percentage of how you can benefit from REST API. This powerful API provides users with information about existing connections, so you can leverage this knowledge while using other systems, and offers you a method to add new connections and list them. Additionally, new REST API allows to check scheduler health which can be useful to monitor Airflow state and react quickly in case of an unexpected shutdown, errors etc. 

In Airflow 2.0 you can read and modify variables stored in Airflow as well as Pools or XCom and find the results of certain tasks while a DAG is still being processed. With the REST API, you gain better control of the DAG execution, and you can view a simple DAG representation.

Setting up Airflow architecture for your business organization

Using Airflow for your workflow management has multiple advantages, and we encourage you to install a new version of Airflow and check out how REST API will improve the operation of your software. Maintaining this system and all its numerous components is not easy though. Unfortunately, it requires full-time access to particular resources to manage the platform and the infrastructure beneath it. Sounds complex? 

Don’t worry. Companies such as ours exist to support and help gain access to the powerful business tools that organization needs in order to operate efficiently and grow. We can help you go through Airflow installation and configuration. We offer professional consultation on any matter related to using Airflow 2.0. If your team requires intensive training, we’ll be happy to prepare and conduct a training in your company. Contact us to learn more about our Apache Airflow-related services. 

Check out our blog for more details on Airflow:

• Airflow custom emails
• Differences between Airflow 1.10.x and 2.0
• How to improve Airlfow 2.0 performance with Smart Sensors?