—/100
Checkpoints
Create a Cloud Storage Bucket
Check my progress
/ 20
Copy Files to Your Bucket
Check my progress
/ 10
Create the BigQuery Dataset (name: lake)
Check my progress
/ 20
Build a Data Ingestion Dataflow Pipeline
Check my progress
/ 10
Build a Data Transformation Dataflow Pipeline
Check my progress
/ 10
Build a Data Enrichment Dataflow Pipeline
Check my progress
/ 10
Build a Data lake to Mart Dataflow Pipeline
Check my progress
/ 20
ETL Processing on Google Cloud Using Dataflow and BigQuery
1 hour
7 Credits
GSP290
Overview
In this lab you build several Data Pipelines that ingest data from a publicly available dataset into BigQuery, using these Google Cloud services:
- Cloud Storage
- Dataflow
- BigQuery
You will create your own Data Pipeline, including the design considerations, as well as implementation details, to ensure that your prototype meets the requirements. Be sure to open the python files and read the comments when instructed to.
Join Qwiklabs to read the rest of this lab...and more!
- Get temporary access to the Google Cloud Console.
- Over 200 labs from beginner to advanced levels.
- Bite-sized so you can learn at your own pace.
GSP290OverviewSetupEnsure that the Dataflow API is successfully enabledDownload the starter codeCreate Cloud Storage BucketCopy Files to Your BucketCreate the BigQuery DatasetBuild a Dataflow PipelineData IngestionReview pipeline python codeRun the Apache Beam PipelineData TransformationRun the Apache Beam PipelineData EnrichmentReview pipeline python codeRun the Apache Beam PipelineData lake to MartReview pipeline python codeRun the Apache Beam PipelineTest your UnderstandingCongratulations!