A couple of years ago, those of us who follow developments in big data technologies began to hear a new word: Spark.
This was at the stage when many organisations had started to take notice of the Hadoop platform and were trying it out for pilot projects or small production tasks. Many found that while Hadoop was a great way of distributing data across cheap hardware, obtaining the promised analytical insight using some of the applications in the Hadoop ecosystem was not simple. For some people it was all a bit too much effort, and the speed at which results could be produced a little underwhelming.
Author: John Leonard