In this series of posts I will discuss the evolution of machine learning algorithms with regards to scaling and performance. We will start with a naive implementation and progress to more advanced solutions finally reaching state of the art implementations, similar to what companies like Google, Netflix and others use for their data pipelines, recommendation systems or machine learning. A variety of topics will be discussed, from basics of ML, different programming models, impact of distributed environment, specifics of machine learning algorithms as compared to common business applications and much more. For those not particularly interested in machine learning the concepts discussed are chosen carefully to apply to a wide range of applications and ML itself is chosen as a good example.
In my previous blog post we looked into neural networks, their training and investigated a trivial single threaded object oriented implementation. The result was a working example that was, however, not useful in many real world scenarios for its poor performance. With large amounts of data such approach is extremely wasteful and we can achieve vastly better performance through parallelization.