MENU

HDFS

       Hadoop Introduction

       Getting Started with HDFS

       Features of HDFS

       How exactly Data is Stored on HDFS Cluster?

       Hadoop Architecture

       Replication 

       SafeMode

       Configuration Files

       Namenode, Secondary Namenode and Datanodes

       Heart Beat Mechanism

       Hadoop CLI Commands 

 

MapReduce

       MapReduce Introduction

       Phases of MapReduce

 

Apache PIG

       PIG Introduction

       Execution Modes of PIG 

       Differences between MapReduce and PIG

HIVE

        Hive Introduction

        Hive Tables

        Hive Metastore

Tutorials

Hi, here are our other tutorial sites. We provide tutorials on Java, Seo and Internet. The following are the links to our tutorials. Please click on the tutorials you want.

• Java

• Seo

• Internet & PC tips

Hive Metastore

Metastore is the internal database for Hive, it is the central repository of Hive Metadata. In this Metastore all the tables information, tables schema, column information, partitioning information will get stored. By default Hive will come up with Derby database as its Metastore. As the default Metastore is Derby, we need not to configure the same, if we want to change the database then we must configure explicitly. 

If we want to configure the Hive Metastore explicitly(other than Derby), we can do so in "hive-site.xml". As part of Metastore configuration there are two parts: 

• Setting of Connection Url
• Setting of Driver name for the same 

The following figure shows you the configuration details. 
 

Configuration of HIVE Metastore

           

Hive Tables

As of now we have seen Hive Introduction. We know that HIVE is used to pose queries on the processed data. Generally in RDBMS we pose queries on Tables. Hive also stores the data in the form of Tables, these tables are stored on HDFS. The following are the two types of tables in HIVE.

• Managed Tables
• External Tables

Hive also stores the metadata of these tables in Hive Metastore. In this post let us see in detail about how to create a table, and how to import data into them.

Managed Tables: 

As the name implies, these tables are managed by Hive Warehouse system. All the Managed tables will be stored in the sub-directories of Hive Warehouse directory. The following is the command tothe Managed Tables.

Managed Table Creation

The following is the example of Managed tables:

Managed table example

External Tables:

As the name implies these are not managed by Warehouse. All the external tabular data is stored in the location specified in the at the time of creation. The following is the syntax to create the External tables. 

External Table Syntax

If we don't specify the HDFS location or external keyword in the command, it will create a Managed table instead of External table. The following is the example of external table. 

External Table Example

  
       


 

Hive Introduction

HIVE is one of the component of Hadoop. It was built by Jeff Hammerbacher and his team at facebook. Hive is a framework for data warehousing on top of HADOOP. This was initially built to run queries on the huge data of facebook. The main use of Hive framework is , using this we can place the processed data into tables, and then we can pose queries on those tables. 

Hive Logo

To work with Hive framework one should have a good idea on SQL, and basic java programming skills. We have HiveQL(Hive Query Language), a dialect of SQL to work with Hive. Hive stores the data in Tabular format.

Apache PIG

       PIG Introduction

       Execution Modes of PIG 

       Differences between MapReduce and PIG

MapReduce

       MapReduce Introduction

       Phases of MapReduce

Hive

        Hive Introduction

        Hive Tables

        Hive Metastore

HDFS

       Hadoop Introduction

       Getting Started with HDFS

       Features of HDFS

       How exactly Data is Stored on HDFS Cluster?

       Hadoop Architecture

       Replication 

       SafeMode

       Configuration Files

       Namenode, Secondary Namenode and Datanodes

       Heart Beat Mechanism

       Hadoop CLI Commands

Java Blog

This is a Tutorial blog for Java Guys. This is especially designed for beginners. This blog contains all the topics of Java technology with a clear explanation. Have a look at this blog by clicking the image below. Follow the blog regularly for updates.

CLICK HERE TO REDIRECT TO JAVA BLOG

www.javagraduates.blogspot.com

Difference between MapReduce and PIG

Difference between MapReduce and PIG: 

Both PIG and MapReduce does the same work. Both are used to process the data. When the PIG program is executed internally it converts into a MapReduce job and process the data. The following are the some of the differences between MR and PIG.

MapReduce	Apache PIG
MapReduce program expects the programming language skills for writing the business logic.	In Apache PIG there is no need of much programming skills. The entire program is based on PIG transformations.
Amount of code is very large; we must write huge programming code.	Amount of code is very less when compared to MapReduce program. 200 lines of MapReduce program is equivalent to 10 lines of Pig script.
MapReduce program is compiled and executed directly.	Pig script internally converts into MapReduce program and gets executed.
Writing and executing MapReduce programming is a bit complex task.	Writing and Executing PIG script is a simple task when compared with MapReduce.

Different modes of Pig Execution

Different modes of Pig Execution:

Pig has two execution modes or types. They are:

1. Local Mode
2. MapReduce Mode

Now let us see each execution mode in detail.

Local Mode: 

In Local Mode of Pig execution, all the input data will be taken from local file system. After execution it provides output on top of local file system.  In local mode, Pig runs in a single JVM and accesses the local filesystem. This mode of suitable only for small datasets and when trying out Pig. To start the local mode of execution, the following command is used.   

• # pig -x local

The above command starts Grunt. Grunt is the Pig interactive shell. 

MapReduce Mode/HDFS Mode/ Clustered Mode: 

In this mode Apache Pig will take the input form HDFS paths only, and after processing data it will put output files on top of HDFS. In MapReduce mode of execution, Pig translates queries into MapReduce jobs and runs them on a Hadoop Cluster.                                        

Pig Introduction

Apache PIG Introduction:

Apache PIG is a transformative language. Initially PIG was developed at Yahoo laboratories, later in 2006 it was officially opted by Apache Software Foundation (ASF)  Pig is high productive when compared to MapReduce. Pig raises the level of abstraction for processing Bigdata. 

Apache Pig is one of the component of Hadoop. Pig is the high level language on top of MapReduce. It uses multiple transformations to process the data. The data flow in Pig is based on these transformations. So, we call Pig as Transformative Language/ DataFlow Language. 

  

Initially Pig is called as Pig Latin. When compared to MapReduce, Pig reduces the size of code, 15 lines of Pig code is equal to nearly 200 lines of MapReduce code. When we run the Pig script, it will internally convert into MapReduce jobs. 

As part of this tutorial you can have a look at the following topics:

• Modes of Pig Execution
• Pig Programming example
• Pig UDF's

                              

Phases of MapReduce

A MapReduce Program consists of three different phases. They are:

• Mapper
• Sort and Shuffle
• Reducer

Among all the three phases, Mapper and Reducer are the direct implementation with respect to coding, where as the Sort and Shuffle phase acts as a glue between Mapper and Reducer.  As a developer we are responsible to write the code for Mapper and Reducer phases. The following figure shows how MapReduce processes data. 

 

Data processing in MapReduce

The above figure shows that, Mapper phase takes the input in the form of key value pairs (K,V), and generate the output in the form of (K,V). 

Sort and Shuffle phase takes the input as (K,V) and generates the output int the form of Key and List of Value pairs (K, List(v)). 

Reducer phase takes the input as (K, List(v)), and generates the output as (K,V). Reducer phase output is the Final Output.

As of now you may not understand the diagram clearly, but just have an idea, you can get the clear picture in the future posts. 

                                    

MapReduce Introduction

Hadoop Framework mainly based on two ecosystems, They are HDFS and MapReduce. HDFS is meant for storage and MapRduce is meant for processing. As of now we have seen the storage part i.e, HDFS, now let us have a look at the processing part i.e, MapReduce. 

In the Hadoop world, MapReduce is considered as one of the major component. MapReduce is responsible for the processing of huge amount of data which get stored on top of HDFS. It is also responsible for parallel processing. MapReduce achieves parallel processing by the means of splits, i.e, all the data is divided into multiple chunks and the processing will be done on each in a parallel fashion. 

MapReduce is a programming model for data processing. Even if it is a programming, it is very simple. Hadoop accepts MR programs written in different languages. Mostly people use java to write MR programs. 

Heart Beat Mechanism

Heart Beat Mechanism

As of now, we know that once if the input file is loaded on to the Hadoop Cluster, the file is sliced into blocks, and these blocks are distributed among the cluster. 

Now Job Tracker and Task Tracker comes into picture. To process the data, Job Tracker assigns certain tasks to the Task Tracker. Let us think that, while the processing is going on one DataNode in the cluster is down. Now, NameNode should know that the certain DataNode is down , otherwise it cannot continue processing by using replicas. To make NameNode aware of the status(active / inactive) of DataNodes, each DataNode sends a "Heart Beat Signal" for every 10 minutes(Default). This mechanism is called as HEART BEAT MECHANISM.

Based on this Heart Beat Signal Job Tracker assigns tasks to the Tasks Trackers which are active. If any task tracker is not able to send the signal in the span of 10 mins, Job Tracker treats it as inactive, and checks for the ideal one to assign the task. If there are no ideal Task Trackers, Job Tracker should wait until any Task Tracker becomes ideal. 

We can change the default value of Heart Beat Signal (10 minutes), by configuring in "mapred-site.xml".                            

put, copyFromLocal and moveFormLocal Command

put, copyFromLocal and moveFormLocal Command:

All these commands are used to copy the files from Local File System to the Hadoop File System. put and copyFormLocal Commands copy the file by keeping the original, but moveFromLocal deletes the original and moves the file to Hadoop File System. The following are the syntax of all the three commands.

• -put <localsrc>  <dst>
• -copyFromLocal <localsrc>  <dst>
• -moveFromLocal <localsrc>  <dst>

The following figures shows you the examples of all the three commands.

Hadoop -put Command

Hadoop -copyFromLocal Command

Hadoop -moveFromLocal Command

        

Remove and Recursive Remove Command in Hadoop

-rm Command: 

This command deletes/removes the file from the directory. The following is the syntax of rm command.

• -rm <src>

The -skipTrash option bypasses trash, if enabled, and immediately deletes <src>. The following figure shows you the example. 
 

Hadoop -rm Command

-rmr Command:

This command deletes/removes entire directory from the File System. The following is the syntax of rmr command.

• -rmr <src>

The -skipTrash option bypasses trash, if enabled, and immediately deletes <src>. The following figure shows you the example. 

Hadoop -rmr Command

            

move and copy commands of Hadoop

-mv Command:

This command is used to move the contents from source to destination. This command is even used for renaming. When moving multiple files, the destination must be a Directory.The following is the syntax of move command.

• -mv <src> <dst>

here, src represent Source, and dst represent Destination.

Hadoop -mv command

In the above figure, /user/root/emp1/part-m-00000 is the source file, where as /Mydir2 is the destination.

-cp Command:

This command is used to copy the file from source to destination. When copying multiple files, the destination must be a Directory. The following is the syntax of copy command.

• -cp <src> <dst>

Hadoop cp command

Hadoop df, du, dus Commands

df Command: 

This command shows the capacity, free and used space of the filesystem. If the filesystem has multiple partitions, and no path to a particular partition is specified, then the status of the root partitions will be shown. The following figure shows the example for df command:

Hadoop df command

du Command: 

This command shows the amount of space, in bytes, used by the files that match the specified file pattern. This command is equivalent to the unix command "du -sb <path>/*" in case of a directory, and to "du -b <path>" in case of a file. The output is in the form name(full path) size (in bytes). The following figure shows the example for du command:

Hadoop du command

dus Command:

This command show the amount of space, in bytes, used by the files that match the specified file pattern. This command is equivalent to the unix command "du -sb"  The output is in the form name(full path) size (in bytes). The following figure shows the example for dus command:

Hadoop dus command