Need help finding the right learning solutions? Email Us: [email protected]
- Onboard For Tech Teams
- Reduce initial time to productivity.
- Increase employee tenure.
- Plug-and-play into HR onboarding and career pathing programs.
- Customize for ad-hoc and cohort-based hiring approaches.
- Upskill For Tech Teams
- Upgrade and round out developer skills.
- Tailor to tech stack and specific project.
- Help teams, business units, centers of excellence and corporate tech universities.
- Reskill For Tech Teams
- Offer bootcamps to give employees a running start.
- Create immersive and cadenced learning journeys with guaranteed results.
- Supplement limited in-house L&D resources with all-inclusive programs to meet specific business goals.
- Design For Tech Teams
- Uplevel your existing tech learning framework.
- Extend HR efforts to provide growth opportunities within the organization.
- Prepare your team for an upcoming tech transformation.
Get your team started on a custom learning journey today!
Our Boulder, CO-based learning experts are ready to help!
Course Summary
Home > Courses >Big Data Training >Apache Airflow >Creating & Monitoring Big Data Pipelines with Apache Airflow
The Creating & Monitoring Big Data Pipelines with Apache Airflow training course is designed to demonstrate how to create, schedule and monitor data pipelines using Apache Airflow by programmatically authoring, scheduling and creating workflows.
The course begins with the core functionalities of Apache Airflow and then moves on to building data pipelines. Next, it explores advanced topics, such as start_date and schedule_time, dealing with time zones, and much more. The course concludes by analyzing how to handle monitoring and security with Apache Airflow, as well as managing and deploying workflows in the cloud.
Prerequisites: A basic knowledge of Python and basic understanding of big data tools (Spark, Hive) are expected
Purpose
Promote an in-depth understanding of how to use Apache Airflow to create, schedule and monitor data pipelines.
Audience
Data Engineers familiar with Python and big data tools such as Hive and Spark.
Role
Skill Level
Style
Duration
3 Days
Related Technologies
- Productivity Objectives:
- Utilize code production-grade data pipelines with Airflow
- Schedule & monitor data pipelines using Apache Airflow
- Understand and apply core/advanced concepts of Apache Airflow.
- Create data pipelines using AWS MWAA (Managed Workflow for Apache Airflow)
Request Information
Get your team upskilled or reskilled today. Chat with one of our experts to create a custom training proposal. Fully customized at no additional cost.
If you are not completely satisfied with your training class, we'll give you your money back.
about our training
-
Real-World Content
Project-focused demos and labs using your tool stack and environment, not some canned "training room" lab.
-
Expert Practitioners
Industry experts with 15+ years of industry experience that bring their battle scars into the classroom.
-
Experiential Learning
More coding than lecture, coupled with architectural and design discussions.
-
Fully Customized
One-size-fits-all doesn't apply to training teams. That's where we come in!
What You'll Learn
In the Creating & Monitoring Big Data Pipelines with Apache Airflow training course, you'll learn:
- Understand Core functionalities of Apache Airflow
-
- What is Apache Airflow
- How does Apache Airflow work?
- Installation & Setup
- Understand Airflow Architecture
- Understand core concepts – DAGS/ Tasks/ Operators
- Understand interface – Airflow UI Tour
- Use CLI
- Build Data Pipeline
- Sqoop operator – Ingest Data from RDBMS
- Http Sensor – checking API availability
- File Sensor – Checking File
- Python Operator – Download Data
- Bash Operator – Move data to HDFS
- Hive Operator – Create Hive tables
- Spark Submit Operator – Run Spark Job
- Email Operator – Send email notifications
- Data pipeline in action
- Mastering Apache Airflow
- Understand start_date & schedule_time
- Backfill and Catchup
- Deal with time zones
- Sharing data – XComs in actions
- Retry/Alerts on task failures
- Pools & priority weights
- Understand Different Executors – Local/Celery/Sequential/Kubernetes
- Create customs plugins
- Monitor Apache Airflow
- Understand logging system
- Set up custom logging
- Store logs in S3
- Security in Apache Airflow
- Encrypt sensitive data with Fernet Keys
- Rotate Fernet Keys
- Hide Variables
- Enable Password authentication
- Airflow in cloud
- Utilize Amazon Managed Workflows for Apache Airflow
- Deploy Airflow on Kubernetes cluster on AWS (EKS)
Real-world content
Project-focused demos and labs using your tool stack and environment, not some canned "training room" lab.
Expert Practitioners
Industry experts that bring their battle scars into the classroom.
Experiential Learning
More coding than lecture, coupled with architectural and design discussions.
Fully Customized
One-size-fits-all doesn't apply to training teams. That's where we come in!
Elite Instructor Program
We recently launched our internal Elite Instructor Program. The community driven instructor program is designed to support instructors in transforming students’ lives by consistently showing a world-class level of engagement, ability, and teaching prowess. Reach out today to learn more about our instructors.
Customized Technical Learning Solutions to Help Attract and Retain Talented Developers
Talk to one of our Learning Solution Architects today
Let DI help you design solutions to onboard, upskill or reskill your software development organization. Fully customized. 100% guaranteed.
DevelopIntelligence leads technical and software development learning programs for Fortune 500 companies. We provide learning solutions for hundreds of thousands of engineers for over 250 global brands.
“I appreciated the instructor’s technique of writing live code examples rather than using fixed slide decks to present the material.”
VMwareAbout Us
LET’S DISCUSS
DevelopIntelligence has been in the technical/software development learning and training industry for nearly 20 years. We’ve provided learning solutions to more than 48,000 engineers, across 220 organizations worldwide.
Resources
Thank you for everyone who joined us this past year to hear about our proven methods of attracting and retaining tech talent.
- Boulder, Colorado Headquarters: 980 W. Dillon Road, Louisville, CO 80027
© 2013 - 2022 DevelopIntelligence LLC - Privacy Policy