Get Started with Airflow + Google Cloud Platform + Docker

Image from source

My Motivation:

Guide Introduction:

Source: “The AI Hierarchy of Needs”

Technical Sections:

docker build --rm --build-arg AIRFLOW_DEPS="gcp" -t puckel/docker-airflow .
docker-compose -f docker-compose-LocalExecutor.yml up -d
Instruction from
download your keyfile JSON and put it in /home/airflow/gcs/data/keyfile.json
#any libraries you want#docker-compose file under the scheduler and workervolumes:- ./dags:/usr/local/airflow/dags- ./requirements.txt:/requirements.txt
byte_object = BytesIO()
Airflows inner log
Google Cloud Bucket

Data Engineer at Disney, who previous work at Bytedance.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store