Working of PIG LATIN using HADOOP

What is PIG?

Pig is a high-level platform for creating MapReduce programs used with Hadoop. The language for this platform is called Pig Latin. Pig Latin abstracts the programming from the Java MapReduce idiom into a notation which makes MapReduce programming high level, similar to that of SQL for RDBMS systems.



Why use PIG at all?

Pig was initially developed at Yahoo! to allow people using Hadoop® to focus more on analyzing large data sets and spend less time having to write mapper and reducer programs. Like actual pigs, who eat almost anything, the Pig programming language is designed to handle any kind of data—hence the name!

Pig is made up of two components: the first is the language itself, which is called PigLatin, and the second is a runtime environment where PigLatin programs are executed.
Running a program in PIG:

The objective of the program is to compute highest run scored by a baseball player for each year. The file we are referring to has all statistics from 1871-2011 and over 90000 rows. Once we have the highest runs we will extend the script to translate a player id field into the first and last names of the players.

Data can be downloaded from the following link.

http://hortonassets.s3.amazonaws.com/pig/lahman591-csv.zip

This link opens a zip folder containing csv files. We will upload batting.csv and masters.csv

Once the hortonworks sandbox is running log into HUE using the address 127.0.0.1:8000 on your web browser

Credentials :

Login : hue

password : 1111

Now its time to upload our data to hue using interactive option in file browser tab.

CODES:

Below are the codes which are needed to execute our objective from the data. Along with the code I will attempt to explain each line to aid you in your understanding.

batting = load 'Batting.csv' using PigStorage(',');

PigStorage function loads the data, comma as the data delimiter.

raw_runs = FILTER batting BY $1>0;

Filtering the first row of data

runs = FOREACH raw_runs GENERATE $0 as playerID, $1 as year, $8 as runs;

FOREACH statement will iterate through the batting data object and GENERATE pulls out selected fields and assings them names.

grp_data = GROUP runs by (year);

Groups the elements in runs by the year field.

max_runs = FOREACH grp_data GENERATE group as grp,MAX(runs.runs) as max_runs;

Using FOREACH command to find maximum runs for each year

join_max_run = JOIN max_runs by ($0, max_runs), runs by (year,runs);  

join_data = FOREACH join_max_run GENERATE $0 as year, $2 as playerID, $1 as runs;  

DUMP join_data;

We join maximum runs with joins this with the runs data so that we can pick up the player id.  The result will be a dataset containing Year, Player ID and Run.  The last line dumps the data to the output.

Your final script will look as below

OUTPUT:

Thank you