Skip to content

Data engineering & AI · Corporate training

Sample curriculum

Apache Spark & PySpark

Process larger datasets with distributed transformations, Spark SQL, and streaming foundations.

  • Intermediate
  • Live online and in person
Explore the curriculum Discuss your team’s training

At a glance

Who it’s for

Engineers with the listed foundations who want to deepen this skill or apply it in a different stack.

Starting point: Intermediate

Bring these foundations

  • Python, SQL, and basic data processing.
Foundation sessions can be included when needed.

Technologies & tools

  • Apache Spark
  • PySpark
  • Spark SQL
  • Jupyter / Databricks
Setup and versions agreed for your environment.

A practical destination

Skills to practise.

The sample path works towards these capabilities. Final goals reflect your team’s starting point and agreed scope.

  1. 01

    Reason about distributed data processing and partitions.

  2. 02

    Build DataFrame and SQL transformations with checks.

  3. 03

    Investigate pipeline performance and streaming reliability.

Sample curriculum

Follow the learning route.

Each module connects the concepts to practical work. Open a module to explore both.

01Module 1

Distributed processing

Concepts to understand

  • Spark architecture and execution
  • Sessions, dataframes, and schemas
  • Reading and writing common data formats

Practical work

Read a dataset into Spark and inspect partitions and execution behaviour. Compare the distributed workflow with a local-data approach.

Show the result. Explain one decision. Identify what you would improve.
02Module 2

Transformations & SQL

Concepts to understand

  • Filters, expressions, and aggregations
  • Joins and window functions
  • Spark SQL and reusable transformations

Practical work

Express a transformation using DataFrame operations and SQL. Check the result with representative input and expected records.

Show the result. Explain one decision. Identify what you would improve.
03Module 3

Performance & reliability

Concepts to understand

  • Lazy evaluation and execution plans
  • Partitions, shuffles, and caching
  • Data quality and failure investigation

Practical work

Investigate a join or aggregation using an execution plan. Explain a partitioning or caching choice and its trade-offs.

Show the result. Explain one decision. Identify what you would improve.
04Module 4

Streaming & pipeline design

Concepts to understand

  • Structured Streaming concepts
  • Checkpoints and incremental processing
  • Project review and performance comparison

Practical work

Build a small streaming or incremental pipeline with checkpoints and validation. Demonstrate how failures and reruns are handled.

Show the result. Explain one decision. Identify what you would improve.

A sample project

Bring the skills together.

Build a batch processing pipeline and inspect its execution plan, partitions, and output quality.

Shorter engagements can focus on selected exercises. A larger, learner-owned capstone can be scoped for a longer programme.

Review the work & reasoning

Four useful questions.

  1. 01

    Does the work meet the agreed requirements?

  2. 02

    Can the learner explain the key decisions and trade-offs?

  3. 03

    What happens with invalid input, failure or an unexpected condition?

  4. 04

    What would they change or explore next?

Designed & delivered by Ragav Kumar V.

Training since 2016 · 100+ batches · 3,000+ professionals trained

Live demonstrations, guided exercises, pair work and project reviews connect the concepts to implementation. We agree the goals, starting knowledge, cohort size, delivery and schedule before shaping the final curriculum.

Shape this for your team

Connect the next skill

All data engineering & ai curricula
Foundation

Python programming

Intermediate

Python with AI capabilities

Intermediate

Data analysis with Python