Best practices for parallelizing data pipelines

Recording

With the last decade's developments, we handle much more data and have more computing power available. This increase isn't only visible in data centres but can also be seen on personal computers. In all circumstances, we can leverage the increase only by utilising several compute cores simultaneously, as the number of cores has significantly increased. In contrast, the increase in the performance of a single core has converged.

In this talk, we want to show best practices for adapting your data pipeline to maximise the usage of the available computing power. We will show general patterns that you can apply independently of used technology. To better illustrate these techniques, we will show practical examples in Python.

Uwe Korn is a CTO at the data science company QuantCo. His expertise is in building scalable architectures for machine learning services and the teams & culture around them. Nowadays, he focuses on the data engineering infrastructure needed to provide the building blocks to bring machine learning models into production. As part of his work to provide an efficient data interchange, he became a core committer to the Apache Parquet, Apache Arrow and conda-forge projects.

https://uwekorn.com/