Modern Data Engineering Course | DevelopIntelligence

Course Summary

The Modern Data Engineering training course presents data engineering concepts for teams who are moving to cloud-based services from legacy systems and want to take advantage of the increased security and availability that the cloud offers. This course will introduce students to the principles of working in the cloud, data warehousing best practices, and cloud-based tools for data engineering.

The course begins with a discussion of the fundamentals of data engineering in the cloud, including strategies for data collection, data storage, and structuring your data. The course then focuses on how to move and transform data in the cloud. The course concludes with data integration concepts and how to visually present data in the cloud.

While this course will focus primarily on the use of AWS to create and maintain data engineering and transformation infrastructure, other cloud providers can be substituted to allow for customization to your organization's specific needs.

Purpose

Learn the basic concepts of data engineering and how to make the shift to the cloud.

Audience

Software Engineers, Data Scientists and Data Engineers who want to learn about data engineering and building data pipelines in the cloud.

Role

Data Engineer - Data Scientist - Software Developer

Skill Level

Introduction

Style

Workshops

Duration

3 Days

Related Technologies

SQL

Cloud Computing Training

Machine Learning Training

Data Science

Download

Productivity Objectives:

Explain different cloud-based services used to build and operate data pipelines that can enable both ETL and ELT workflows.
Define strategies for collecting, storing, transforming, and visualizing data in the cloud.
Utilize AWS services to create and maintain data pipelines.
Understand the role of business intelligence tools.

Learn About Virtual Training

Request Information

Get your team upskilled or reskilled today. Chat with one of our experts to create a custom training proposal. Fully customized at no additional cost.

If you are not completely satisfied with your training class, we'll give you your money back.

about our training

Real-World Content

Project-focused demos and labs using your tool stack and environment, not some canned "training room" lab.
Expert Practitioners

Industry experts with 15+ years of industry experience that bring their battle scars into the classroom.
Experiential Learning

More coding than lecture, coupled with architectural and design discussions.
Fully Customized

One-size-fits-all doesn't apply to training teams. That's where we come in!

What You'll Learn

In the Modern Data Engineering training course, you'll learn:

Collecting Data From Multiple Sources
- Overview of what ETL and ELT are and why they are important
- Using APIs to get data
- Using S/FTP to get data
- Manually uploading data sources
- Extracting data from SQL databases
Storing Data in the Cloud
- Moving data to cloud storage services (S3/Buckets)
- Using Amazon Redshift to store relational data in a data warehouse
- NoSQL storage via DynamoDB
- Elasticache/caching storage
- File storage types (CSV, TSV, JSON, Avro, Parquet, Excel, etc)
Collecting and Structuring Data
- File system recommendations
- Data lake vs Data Warehouse
- Distributed vs non-distributed
- Data structures and filesystems
Extraction of Data
- Overview of data extractions and their importance in overall data architecture
- Moving from the data lake to structure in various tools (Dynamo, Spark, SQL, etc.)
- Provisioning SQL databases around data patterns
- Queueing services (AWS SQS, AWS Kinesis)
- Creating listeners, pollers, and triggers in an extraction architecture
- Lambdas, Terraform, and ethereal infrastructure
- Creating a data extraction integration within an AWS VPC
Data Wrangling
- Overview of data transformations and their importance in overall data architecture
- Data transformations use-cases:
  - Transforming data types from XML to JSON
  - Executing queries to transform data for BI analysis
  - Ensuring robust copies of all data inflow
- In-stream transformations in Dynamo
- Uses of lambda for transforming/restructuring data
- Kinesis in-stream transformations
- Creating an integration with an endpoint
Data Analysis
- Using AWS Quicksight to understand your data in the cloud
- A brief introduction to Tableau, Looker, Metabase and the ecosystem of modern BI tools
- What is data science and where does it fit with data engineering?