Energyworx.

Processing pipelines were not set up in the optimal Beam pipeline. So, every time the team ran into a case that the Python version of the framework didn’t support, they worked around this limitation by writing code outside of the Beam programming model. This code was a drain on the performance gains Dataflow could provide.

Dataflow was auto-scaling aggressively when a burst of data entered the system, but it didn’t scale down quickly enough after the data had been processed. This led to large unnecessary costs when large batches of data enter the system.

With so many dependencies bound to the version of Apache Beam, it meant updating all dependencies at the same time. This is problematic when upgrading the framework, and the updated version came with an upgrade for a dependent library that had breaking changes. These updates are complicated, and a burden on developers.

Data processing in the Energyworx platform occurs in several stages, and Dataflow has built-in features for deduplicating the data in the platform.

The Nordcloud team had to address data duplications – one of the inherent problems of distributed systems – when migrating away from Dataflow. It meant producing various new implementation methods to counteract this issue.

ubernetes enables granular control over auto-scaling. But this also means more to manage. For best performance across multiple environments with different data, we used a combination of metrics.

We also handled scale-down. The data didn’t enter systematically, rather bursts throughout the day. It needed to guarantee processing within a defined time limit, despite the variables. We used a sufficiently large stabilisation window to solve these issues.

Issues like retries when making API calls, or monitoring the application status – previously covered by Dataflow, now needed to be handled.

This meant more failure handling. When running 1000s of containers in Kubernetes, they often get shut down. Happening halfway through processing or through sending output data causes performance issues and data duplication. We designed new services to handle these situations.

A large part of the codebase was not tightly coupled to the Beam programming model. Therefore, the team was able to remove the Beam framework entirely and dockerise the pipelines, making it possible to run them on Kubernetes without a Beam runner.

While the auto-scaling feature in principle was a big plus, not being able to fine-tune this on a granular level is a significant downside. Kubernetes allows complete control over auto-scaling using Horizontal Pod Auto-scalers.

Upgrading a version of the dependent libraries was often not possible without upgrading the framework version and all the dependent libraries. This is normal when working with a large framework, but also more manageable once we removed the Beam framework, and this allows more freedom to choose dependency libraries.