Building and Deploying the Hadoop WordCount Application

Home > hadoop > hadoop/mapreduce-word-count-application-in-hadoop" class="text-[#6b38d4] font-semibold hover:underline">MapReduce Word Count Application in Hadoop</a> > hadoop/building-and-deploying-the-hadoop-wordcount-application" class="text-[#6b38d4] font-semibold hover:underline">Building and Deploying the Hadoop WordCount Application</a>

Detailed Explanation

The Hadoop WordCount application exemplifies the classic MapReduce programming model, where a user‑defined Java class (WordCount.java) implements a mapper that emits a (word, 1) pair for each token and a reducer that aggregates these pairs to produce the final frequency count. Building and deploying this application begins with compiling the Java source file using the Hadoop‑provided compiler, which resolves dependencies on the Hadoop core libraries and produces a class file executable on the cluster. The compiled output is then packaged into a JAR (Java ARchive) file, a portable container that bundles the bytecode and any required resources, allowing the Hadoop runtime to distribute the program across the distributed environment.

Once the JAR is ready, the Hadoop ecosystem must be activated. This involves starting all daemons that constitute the cluster: the NameNode, which maintains the metadata of the Hadoop Distributed File System (HDFS); the DataNode, which stores the actual block replicas; the YARN ResourceManager, which orchestrates resource allocation across the cluster; and the YARN NodeManager, which runs on each worker node to manage container lifecycles. After these services are up, administrators can verify the health of HDFS and the YARN cluster through web interfaces typically hosted at http://localhost:50070/hdfs for the file system and http://localhost:8080/cluster for YARN. These consoles display live metrics such as storage utilization, node status, and running applications, providing essential feedback before job submission.

The next phase concerns data preparation. An input directory is created within HDFS, and a sample text file (ex1.txt) is copied into this directory, making the data accessible to the MapReduce framework. The WordCount JAR is then launched with a command that specifies the HDFS path of the input file and an output directory (ex1out) where the result will be stored. Hadoop automatically splits the input file into logical blocks, distributes them to mapper tasks, and subsequently shuffles the intermediate key‑value pairs to reducer tasks. Upon successful completion, the output directory contains one or more part files (e.g., part‑00000) that hold the final word‑count listings.

Finally, the results are retrieved by invoking the HDFS client to concatenate and display the content of the part file. The command bin/hdfs dfs -cat /user/admin/ex1out/part0000* streams the aggregated counts to the console, allowing the user to verify the correctness of the computation.

Examples

Consider a simple input file ex1.txt containing the sentence “big data drives big insights”. After the WordCount job finishes, the part file will list each distinct token followed by its occurrence count: the word “big” appears twice, while “data”, “drives”, and “insights” each appear once. This concrete result demonstrates how the mapper emits a (word, 1) pair for every token, and the reducer sums these values, producing a concise frequency table. The same workflow can be scaled to terabytes of log data simply by placing larger files into HDFS; Hadoop will handle the parallel execution transparently, preserving the same logical steps of compilation, packaging, daemon startup, data staging, job submission, and output retrieval.