Hadoop is a framework used for storing and processing huge amounts of data in a distributed environment.
Its core components work together to handle big data efficiently.

The four core components of Hadoop are:

1. HDFS (Hadoop Distributed File System) – Storage Layer
2. MapReduce – Processing Layer
3. YARN (Yet Another Resource Negotiator) – Resource Management Layer
4. Hadoop Common – Shared Utilities and Libraries

1. HDFS (Hadoop Distributed File System)

 What is HDFS?

HDFS is a distributed file system designed to store very large files across multiple machines.
It provides:

• High storage capacity
• Fault tolerance
• Scalability

It is built to run on commodity hardware.

 Key Features of HDFS

• Fault Tolerance: Data is replicated across multiple nodes.
• Scalability: More nodes can be added easily.
• High Throughput: Optimized for large-scale data processing.
• Flexibility: Stores structured, semi-structured, and unstructured data.

 HDFS Architecture

1. NameNode (Master Node)

The NameNode manages the file system metadata such as:

• File names
• Directories
• Permissions
• Block locations

It controls all DataNodes.

2. DataNode (Slave Node)

DataNodes store the actual data blocks.

Responsibilities:

• Store data
• Handle read/write operations
• Send heartbeat signals to the NameNode

 How HDFS Works

• Large files are divided into blocks.
• Default block size = 128 MB
• Each block is replicated (usually 3 copies).

This ensures data safety even if a node fails.

 Example of HDFS

Suppose you have a 1 TB video file.

HDFS will:

• Split it into 128 MB blocks
• Create approximately 8000 blocks
• Store each block on 3 different DataNodes

So if one machine crashes, data can still be recovered from another copy.

 Real-Life Example of HDFS

Imagine keeping 3 photocopies of an important document in different rooms.
If one room is damaged, the document is still safe in the other rooms.

2. MapReduce

 What is MapReduce?

MapReduce is a programming model used to process large datasets in parallel across a Hadoop cluster.

It works in three phases:

1. Map Phase
2. Shuffle and Sort Phase
3. Reduce Phase

 Phases of MapReduce

1. Map Phase

The mapper processes input data and converts it into key-value pairs.

Example

Input sentence:

Hadoop is fast Hadoop is scalable

Mapper Output:

(Hadoop,1)
(is,1)
(fast,1)
(Hadoop,1)
(is,1)
(scalable,1)

2. Shuffle and Sort Phase

The system groups all similar keys together.

(Hadoop,[1,1])
(is,[1,1])
(fast,[1])
(scalable,[1])

3. Reduce Phase

The reducer combines values and produces the final result.

Final Output:

(Hadoop,2)
(is,2)
(fast,1)
(scalable,1)

 Advantages of MapReduce

• Parallel processing
• Faster execution
• Fault tolerance
• Handles petabytes of data

 Real-Life Example of MapReduce

Imagine exam papers being checked by multiple teachers:

• Map: Teachers check papers separately
• Shuffle: Papers are grouped subject-wise
• Reduce: Final marks are calculated

3. YARN (Yet Another Resource Negotiator)

 What is YARN?

YARN is the resource management framework in Hadoop.

It manages:

• CPU usage
• Memory allocation
• Task scheduling

YARN allows multiple applications like MapReduce, Spark, and Hive to run together.

 Components of YARN

1. ResourceManager (Master)

Responsibilities:

• Allocates cluster resources
• Schedules applications
• Monitors resource usage

2. NodeManager (Slave)

Responsibilities:

• Manages resources on each node
• Executes tasks
• Reports status to ResourceManager

 Example of YARN

Suppose:

• One user runs a Spark job
• Another user runs a MapReduce job

YARN allocates CPU and memory resources efficiently to both applications.

 Real-Life Example of YARN

Think of a school principal:

• Assigns classrooms to teachers
• Ensures resources are properly used

4. Hadoop Common

 What is Hadoop Common?

Hadoop Common is a collection of shared libraries and utilities required by all Hadoop modules.

It provides:

• Java libraries
• Configuration files
• Scripts for starting Hadoop services
• APIs for Hadoop operations

 Features of Hadoop Common

• Supports communication between Hadoop modules
• Provides operating system utilities
• Helps integrate tools like Hive, Pig, HBase, and Sqoop

 Example of Hadoop Common

Just like common system files support all applications in Windows, Hadoop Common supports all Hadoop components.

Summary Table

Simple Real-Life Analogy