Running Commentary

Azure is a great cloud platform but its services have mostly terrible names and confusing, often heavily overlapping features between them. And so another one in this series of confusing services is Azure Data Lake. I have tried to make sense of this service and what it does! Azure Data Lake   Azure Data Lake has two components, which are very tightly integrated: a.       Azure Data Lake Analytics (ADLA) b.       Azure Data Lake Storage (ADLS)   ADLA gives us the ability to run analytical jobs (query, extract, aggregate, transform output etc.) on data, which is stored in ADLS, in form of files.   Below ‘temp’ is the name of ADLA account. As we can see ‘tempdl’ is the name of associated ADLS account, which acts as the data source for this ADLA account.   As we can see below, the ‘tempdl’ has got following data stored in form of files. ADLA lets us query, transform and do other operations on this data using USQL (prope...

If you have gone through part 1 of this series, you have used Cloudera's Hadoop Quickstart VM to setup a working instance of Hadoop and have running instance of various Hadoop services. Now it is a good time to go back to a little theory and see how different pieces fit with each other. HortonWorks, another Hadoop distributor, has got an excellent tutorial for Hadoop and each of its accompanying services, which you see in below image (taken from the above URL). You can go to this page and read about Hadoop in detail. However I am going to summarize and simplify some of the content and definitions which make it easy for a beginner to quickly understand and proceed. OK, so this is the definition of Hadoop on HortonWorks site:  Apache Hadoop® is an open source framework for distributed storage and processing of large sets of data on commodity hardware. You will agree that biggest challenges in any computing are very basic: storage and processing. Processing could be any o...

So you have read white papers, blogs and even some books on what is Big Data and how it is transforming world by giving us insights into the data usage through advanced analytical strategies. You might also have read about Hadoop and Map/Reduce. But now what? How do you begin? Theory is not going to cut it, right? You want to get your hands dirty and write some code and setup some clusters, right? Right. So let's start. Admittedly, Hadoop is intimidating. Apart from having a plethora of software (first of all, you need a Linux box!) you also need to have a 'cluster' of machines because Hadoop running on one machine is not really what a real life Hadoop installation looks like. As a beginner you would like to quickly write a 'Hello World' of Hadoop rather than setting up environment. Easiest way to start instantly is using Cloudera's Quickstart VM for Hadoop.(Cloudera is one of the three biggest Hadoop distributors). First of all we need to install a virtu...