Get Started with Airflow + Google Cloud Platform + Docker

Image from source

My Motivation:

Guide Introduction:

Source: “The AI Hierarchy of Needs”

Technical Sections:

docker build --rm --build-arg AIRFLOW_DEPS="gcp" -t puckel/docker-airflow .
docker-compose -f docker-compose-LocalExecutor.yml up -d
Instruction from https://cloud.google.com/composer/docs/how-to/managing/connections
download your keyfile JSON and put it in /home/airflow/gcs/data/keyfile.json
#requirements.txtpandas
numpy
google-cloud-storage
#any libraries you want#docker-compose file under the scheduler and workervolumes:- ./dags:/usr/local/airflow/dags- ./requirements.txt:/requirements.txt
byte_object = BytesIO()
blob.download_to_file(byte_object)
byte_object.seek(0)
Airflows inner log
Google Cloud Bucket

Data Engineer at Disney, who previous work at Bytedance.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store