What I chose among BigDL, Horovod and others for Deep Learning on Spark.

As I said, this post was only written in order to help anybody get the hands-on this interesting package. Of course, this is not an exhaustive search. For example, the following tests may be performed:

  • Run this example as a spark-submit instead of doing it interactively from the PySpark Shell;
  • Use more complex networks, such as convnets on the same MNIST or try importing pre-trained BERT models and fine-tune them using BigDL (I don’t even know if there’s the possibility)

When I have some time, I will definitely explore the two bullets above.
Bye and thanks for reading!

--

--

Andrea Guidi

Andrea Guidi

I love data, machine learning and Autoencoders. Control systems/electronic engineer.