The active and growing Apache Airflow open-source community provides operators (plugins that simplify connections to services) for Apache Airflow to integrate with AWS services. In both cases, access for your Apache Airflow users is controlled by the access control policy you define in AWS Identity and Access Management (IAM). The Apache Airflow Web server can be accessed either over the Internet by selecting the Public network Apache Airflow access mode, or within your VPC by selecting the Private network Apache Airflow access mode. Each environment has its own Apache Airflow metadatabase managed by AWS that is accessible to the Scheduler and Workers Fargate containers via a privately-secured VPC endpoint.Īmazon CloudWatch, Amazon S3, Amazon SQS, Amazon ECR, and AWS KMS are separate from Amazon MWAA and need to be accessible from the Apache Airflow Scheduler(s) and Workers in the Fargate containers. The Apache Airflow Scheduler and Workers are AWS Fargate (Fargate) containers that connect to the private subnets in the Amazon VPC for your environment.
Operators that invoke tasks on Amazon ECS containers, and Kubernetes operators that create and run pods on a Kubernetes cluster are supported.Īll of the components contained in the outer box (in the image below) appear as a single Amazon MWAA environment in your account. Worker fleets – Amazon MWAA offers support for using containers to scale the worker fleet on demand and reduce scheduler outages using Amazon ECS on AWS Fargate. Amazon MWAA automatically sends environment metrics-and if enabled-Apache Airflow logs to CloudWatch.ĪWS integration – Amazon MWAA supports open-source integrations with Amazon Athena, AWS Batch, Amazon CloudWatch, Amazon DynamoDB, AWS DataSync, Amazon EMR, AWS Fargate, Amazon EKS, Amazon Kinesis Data Firehose, AWS Glue, AWS Lambda, Amazon Redshift, Amazon SQS, Amazon SNS, Amazon SageMaker, and Amazon S3, as well as hundreds of built-in and community-created operators and sensors.
Workflow monitoring – View Apache Airflow logs and Apache Airflow metrics in Amazon CloudWatch to identify Apache Airflow task delays or workflow errors without the need for additional third-party tools.
Airflow operators update#
The Amazon MWAA team will update and patch the images for these versions. Streamlined upgrades and patches – Amazon MWAA provides new versions of Apache Airflow periodically. In both cases, access for your Apache Airflow users is controlled by the access control policy you define in AWS Identity and Access Management (IAM), and AWS SSO. The Private network access mode uses a VPC endpoint for your Apache Airflow Web server that is accessible in your VPC. The Public network access mode uses a VPC endpoint for your Apache Airflow Web server that is accessible over the Internet. Public or private access modes – Access your Apache Airflow Web server using a private, or public access mode. Data is also automatically encrypted using AWS Key Management Service, so your environment is secure by default. The Apache Airflow Workers assume these policies for secure access to AWS services.īuilt-in security – The Apache Airflow Workers and Schedulers run in Amazon MWAA's Amazon VPC. Amazon MWAA monitors the Workers in your environment and uses its autoscaling component to add Workers to meet demand, up to and until it reaches the maximum number of Workers you defined.īuilt-in authentication – Enable role-based authentication and authorization for your Apache Airflow Web server by defining the access control policies in AWS Identity and Access Management (IAM).
Airflow operators code#
Amazon MWAA sets up Apache Airflow for you using the same Apache Airflow user interface and open-source code that you can download on the Internet.Īutomatic scaling – Automatically scale Apache Airflow Workers by setting the minimum and maximum number of Workers that run in your environment. Automatic Airflow setup – Quickly setup Apache Airflow by choosing an Apache Airflow version when you create an Amazon MWAA environment.
0 kommentar(er)
