What is Hadoop?

I think the best answer is: Hadoop is the clumsy little elephant over there. And in fact, it is. Those who are used to attending my classes and lectures must be thinking, “Uncle is already messing around.” No, I’m not. I guarantee it. This little elephant over there is from Doug Cutting’s son, the creator of “Hadoop” (actually, it’s his son who created it). Doug decided to use this name because it’s easy to pronounce, fun, and above all, unique (I don’t think anyone besides his son knew what a Hadoop was). Being unique has an astronomical advantage in the days of Google! It’s easily searchable on search tools.

Now, we can also say that Hadoop is a new way of storing and manipulating large databases, whether they are normalized or not. In fact, within the Hadoop universe, we don’t even need to organize databases into tables, as we would in a traditional RDBMS.

Hadoop is not for everyone! Hadoop finds its place in large databases. And let’s agree, nothing less than 1TB should be considered large in these Big Data days.

Hadoop does not replace, nor will it ever replace, traditional databases like RDBMS/SGBDRs we see today. Hadoop is not meant for OLTP (transactions, concurrent read and write). Hadoop finds its best use in OLAP operations (large amounts of queries, meaning, reading). One can say that Hadoop is more suited for operations that WRITE once, and READ many times.

Hadoop is a set of tools, an ecosystem that works in harmony to solve the following issues:

Handling big data (large masses of data). No data mass is too big, 1TB, 100TB, 1PB or bigger, much bigger.
Complex data analysis on a large scale.
Processing large logs.
Data warehousing.
Video & image analysis.
Advanced computing: artificial intelligence, machine learning, decision-making.

Hadoop has been used, in the real world, to solve problems like the ones listed below:

Analysis of Terabytes of data provided by thousands of weather sensors scattered around the world, in order to save human lives, through the prediction of disasters and meteorological catastrophes.
Analysis of thousands of Terabytes of disorganized and denormalized financial information, coming from blogs, newspapers, magazines, stock exchanges, to help stock brokers and banks in the efficient and profitable buying and selling of stocks.
Risk determination in financial transactions like credit card operations.
Artificial intelligence projects where massive data needs to be processed in a few seconds.
Profiling consumers in seconds, and offering products with a high chance of sale.

I’ve seen the most varied applications, some even crazy. But what I have seen common in all applications is: Large Data Volumes (>1TB) versus very low response times.

I want to emphasize: Hadoop will not replace MySQL, Oracle, DB2, etc. Hadoop has a different purpose and application. RDBMS and Hadoop will coexist and complement each other over time.

Schedule a meeting here