ETL Processing on GCP Using Dataflow and BigQuery
GSP290
Overview
In this lab you will build several Data Pipelines that will ingest data from a publicly available dataset into BigQuery, using these GCP services:
- GCS - Google Cloud Storage
- Dataflow - Google Dataflow
- BigQuery - BigQuery tables
You will create your own Data Pipeline, including the design considerations, as well as implementation details, to ensure that your prototype meets the requirements. Be sure to open the python files and read the comments when instructed to.
Join Qwiklabs to read the rest of this lab...and more!
- Get temporary access to the Google Cloud Console.
- Over 200 labs from beginner to advanced levels.
- Bite-sized so you can learn at your own pace.
Score
—/100
Create a Cloud Storage Bucket
/ 20
Copy Files to Your Bucket
/ 10
Create the BigQuery Dataset (name: lake)
/ 20
Build a Data Ingestion Dataflow Pipeline
/ 10
Build a Data Transformation Dataflow Pipeline
/ 10
Build a Data Enrichment Dataflow Pipeline
/ 10
Build a Data lake to Mart Dataflow Pipeline
/ 10
Build a Data lake to Mart CoGroupByKey Dataflow Pipeline
/ 10