Introduction to Data Engineering on Google Cloud

Module 1: Data Engineering Tasks and Components

• Explain the role of a data engineer.

• Understand the differences between a data source and a data sink.

• Explain the different types of data formats.

• Explain the storage solution options on Google Cloud.

• Learn about the metadata management options on Google Cloud.

• Understand how to share datasets with ease using Analytics Hub.

• Understand how to load data into BigQuery using the Google Cloud console or the gcloud CLI.

Module 2: Data Replication and Migration

• Explain the baseline Google Cloud data replication and migration architecture.

• Understand the options and use cases for the gcloud command-line tool.

• Explain the functionality and use cases for Storage Transfer Service.

• Explain the functionality and use cases for Transfer Appliance.

• Understand the features and deployment of Datastream.

Module 3: The Extract and Load Data Pipeline Patern

• Explain the baseline extract and load architecture diagram.

• Understand the options of the bq command-line tool.

• Explain the functionality and use cases for BigQuery Data Transfer Service.

• Explain the functionality and use cases for BigLake as a non-extract-load patern.

Module 4: The Extract, Load, and Transform Data Pipeline Patern

• Explain the baseline extract, load, and transform architecture diagram.

• Understand a common ELT pipeline on Google Cloud.

• Learn about BigQuery’s SQL scripting and scheduling capabilities.

• Explain the functionality and use cases for Dataform.

Module 5: The Extract, Transform, and Load Data Pipeline Patern

• Explain the baseline extract, transform, and load architecture diagram.

• Learn about the GUI tools on Google Cloud used for ETL data pipelines.

• Explain batch data processing using Dataproc.

• Learn how to use Dataproc Serverless for Spark for ETL.

• Explain streaming data processing options.

• Explain the role Bigtable plays in data pipelines.

Module 6: Automation Techniques

• Explain the automation paterns and options available for pipelines.

• Learn about Cloud Scheduler and Workflows.

• Learn about Cloud Composer.

• Learn about Cloud Run functions.

• Explain the functionality and automation use cases for Eventarc.

• Data engineers

• Database administrators

• System administrators

Prerequisite:

• Prior Google Cloud experience at the fundamental level using Cloud Shell and accessing products from the Google Cloud console.

• Basic profciency with a common query language such as SQL.

• Experience with data modeling and ETL (extract, transform, load) activities.

• Experience developing applications using a common programming language such as Python.