Apache Spark on Kubernetes is an intensive, two-day, hands-on course focused on running and managing Spark workloads on Kubernetes. Day one opens with a Spark review that refreshes attendees on its data processing model. A Kubernetes introduction follows, covering pods, controllers, and the cluster management model that orchestrates workloads. Each module includes hands-on labs where attendees install and experiment with Spark and Kubernetes side by side. A running-Spark-on-Kubernetes module has students submit their first jobs directly against a cluster. The day closes with scheduling and scaling, where attendees configure Spark to request resources dynamically and scale executor pods up and down with demand.
Day two shifts focus to Kubernetes in the cloud, examining how a managed cluster changes the operational picture for Spark. Attendees learn what to account for when nodes are provisioned and reclaimed automatically. An Apache Airflow overview follows, introducing directed acyclic graphs and task scheduling concepts. A dedicated module then has students deploy and run Airflow on Kubernetes to orchestrate Spark jobs end to end. The course closes with a storage and cost management module, where attendees connect Spark to cloud datastores. They apply practices for controlling compute and storage spend in a dynamically orchestrated environment. Upon completion, attendees have the practical, hands-on skills needed to run Spark effectively in a Kubernetes cluster environment.
Who Should Attend
Application developers, analysts and data scientists
What Attendees Will Learn
Upon completing Apache Spark on Kubernetes, attendees will be able to:
- Review Apache Spark’s data processing model
- Explain Kubernetes cluster management fundamentals
- Run Spark applications on Kubernetes
- Schedule and scale Spark clusters on Kubernetes
- Deploy Kubernetes clusters in the cloud
- Explain Apache Airflow architecture and run Airflow on Kubernetes
- Manage Spark storage and cost in cloud environments
Prerequisites
Students should have taken “Docker Foundation” and “Kubernetes Foundation” courses or have equivalent knowledge.