Apache Hadoop

Hadoop Distributed File System (HDFS)

The Hadoop Distributed File System (HDFS) is a key component of the Apache Hadoop ecosystem, specifically designed for storing and managing large datasets across a distributed network of computers. HDFS provides high-throughput access to application data, making it suitable for big data applications such as data warehousing, machine learning, and analytics.

Architecture of HDFS

HDFS employs a master/slave architecture, consisting of two main types of nodes:

1. NameNode (Master Node):

   • The NameNode is the master server that manages the filesystem namespace and regulates access to files by clients. It maintains metadata about the files stored in HDFS, including information about file names, permissions, and the locations of data blocks on DataNodes.

   • It does not store the actual data but keeps track of where each block of data is located across the cluster. The metadata is stored in memory for fast access and persisted on disk for reliability.

   • Operations such as opening, closing, renaming files, and directories are executed by the NameNode.

2. DataNodes (Slave Nodes):

   • DataNodes are responsible for storing the actual data blocks. Each DataNode manages storage attached to the nodes it runs on and serves read and write requests from clients.

   • When a file is stored in HDFS, it is split into blocks (default size typically 128 MB or 256 MB), which are distributed across multiple DataNodes for storage.

   • DataNodes communicate with the NameNode by sending periodic heartbeat signals to indicate their status and report on the blocks they store.

Data Storage and Replication

• Block Structure: HDFS divides files into large blocks. Each block is stored independently across different DataNodes to ensure fault tolerance and high availability.

• Replication: To protect against data loss due to hardware failures, HDFS replicates each block across multiple DataNodes. The default replication factor is three, meaning each block is stored on three different nodes: two on the same rack and one on a different rack. This strategy enhances fault tolerance while optimizing network bandwidth during data writes.

Read and Write Operations

• Read Operation:

  1. A client initiates a read request by contacting the NameNode to retrieve metadata about the file's blocks and their locations.

  2. The client then communicates directly with the relevant DataNodes to read the blocks in parallel, which improves performance.

• Write Operation:

  1. Similar to reading, a write request begins with the client contacting the NameNode for metadata about where to store the new blocks.

  2. The client sends data to one DataNode, which then replicates it to other specified DataNodes based on the replication factor35.

Advantages of HDFS

• Scalability: HDFS can easily scale horizontally by adding more DataNodes to accommodate growing datasets without significant reconfiguration.

• Fault Tolerance: The replication mechanism ensures that even if some nodes fail, data remains accessible from other nodes.

• High Throughput: Designed for high throughput rather than low latency, HDFS efficiently handles large volumes of data.

• Cost-Effectiveness: It can run on commodity hardware, making it an economical choice for organizations dealing with big data45.

In summary, HDFS is a robust storage solution tailored for big data applications, providing essential features such as scalability, fault tolerance, and high throughput through its distributed architecture.

YARN, which stands for Yet Another Resource Negotiator, is a key component of the Apache Hadoop ecosystem introduced in Hadoop 2.x. It serves as a resource management layer that allows multiple data processing engines to run concurrently, improving the efficiency and scalability of big data applications. Here’s an overview of its architecture and components.

YARN Architecture

YARN's architecture separates resource management from job scheduling and monitoring, which enhances its flexibility and scalability. The main components of YARN are:

1. Resource Manager (RM)

• Role: The Resource Manager is the master daemon responsible for managing resources across the cluster. It allocates resources to various applications based on their requirements.

• Components:

  • Scheduler: This component is responsible for resource allocation among running applications without monitoring or tracking their status. It operates based on predefined policies, such as Capacity Scheduler or Fair Scheduler, to ensure efficient resource distribution.

  • Application Manager: This manages the lifecycle of application masters, handling job submissions and negotiating the first container for execution.

2. Node Manager (NM)

• Role: Each Node Manager runs on individual nodes within the cluster and manages the execution of containers on that node. It monitors resource usage (CPU, memory, disk) and reports this information back to the Resource Manager.

• Responsibilities:

  • Launching and managing containers as directed by the Application Master.

  • Monitoring the health of the node and reporting any issues to the Resource Manager.

3. Application Master (AM)

• Role: The Application Master is a framework-specific process that negotiates resources from the Resource Manager and coordinates with Node Managers to execute tasks.

• Responsibilities:

  • Managing the application’s lifecycle, including resource requests, task execution, and fault tolerance.

  • Reporting progress and status back to the Resource Manager.

Workflow in YARN

1. Job Submission: A client submits a job to the Resource Manager.

2. Resource Allocation: The Resource Manager allocates a container for the Application Master.

3. Application Master Execution: The Application Master requests additional containers from the Resource Manager as needed.

4. Task Execution: Node Managers launch containers based on instructions from the Application Master, executing tasks in parallel.

5. Monitoring and Completion: The Application Master monitors task execution and reports back to the Resource Manager upon completion.

Advantages of YARN

• Scalability: YARN can efficiently manage thousands of nodes in a cluster, allowing for extensive data processing capabilities.

• Multi-tenancy: It supports running multiple processing frameworks (e.g., MapReduce, Spark, Flink) simultaneously on a single cluster.

• Dynamic Resource Management: YARN dynamically allocates resources based on application needs, optimizing cluster utilization.

In summary, YARN enhances Hadoop's capabilities by providing a robust architecture for managing resources and scheduling jobs across distributed systems, making it an essential component for modern big data processing environments.