Nearly every sector of our society is being transformed by moving towards a data-driven decision making paradigm. The so called “big data” revolution is credited with starting this transformative process. The last decade has seen an unprecedented acceleration in the creation of technologies, platforms, and methods to serve the needs of this ongoing big data revolution. However, there is a confusing set of choices to make when undertaking new transformative big data projects. Some examples of choices are: deciding if one should deploy a NoSQL database or a SQL database for the underlying data platform, picking a packaged cloud product v/s rolling your own, and picking the “right” development stack. In this talk, I will describe some common patterns of mistakes in approaching these issues for big data initiatives, and end with some pointers on how to wade through the technology mess to improve the chances of a successful new big data initiative.
This talk should be of interest to both business leaders who want to figure out how to approach and manage a new big data initiative, and to developers who have to actually develop components for a big data project.