Enjoy

What is big data?

Big data is large amount of data to store and process on the server.

They can be the user activities of big applications such as social media (facebook), search engines (google), Internet of things (sensors, machines, transports etc)

Today 1TB hard drive costs around 50 euro so it’s cheap storing large amount of data but processing it requires long time using SQL database and NoSQL databases are better way for storing these informations and also they are other frameworks for processing them such as Hadoop.

Big Data technologies

There are two classes of technologies to handle big data, they are complementary and usually used together:

• Operational big data: technologies for storing and processing the big data: NoSQL databases
• Analytical Big Data: Systems for Massive Parallel Processing such as Hadoop

Google Hadoop

in the usual client-server approach we have a client that gets the information from the database.

Hadoop splits the task in small parts assigning each one to a computer connected to the network and connects all the information afterward.This is the MapReduce algorithm.

This means that Hadoop has the below components:

• his own distributed file system called HDFS: it’s made of masters NameNode which manages many slaves nodes DataNodes. A file is divided into many blocks stored in set of DataNode. NameNode makes the mapping between the file and the block
• his own common libraries
• the framework called YARN
• the map reduce algorithm: made by a Map task which converts the input in a map and a Reduce task which splits the map in a smaller set of tuple. The map reduce Algorithm has a single master called Job Tracker responsible for resource management and scheduling jobs, and a slave TaskTraker which executes the task

Hadoop works as below:

• the user submits a job to Hadoop with a input/output elements, a MapReduce algorithm, the job configuration
• the jobTraker takes the responsibility of the job submitting it to the slaves
• the task traker executes its task

Here is a guide on how to install Hadoop

Here is a list of the hadoop file system commands

I tested a couple of commands: mkdir and ls, they work fine

Finally I’ve written a simple map reduce example:

Here is an example of the MapReduce algorithm

package com.michele.rizzi.hadoop;
import java.io.IOException;
import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCount {

public static class TokenizerMapper
extends Mapper<Object, Text, Text, IntWritable>{

private final static IntWritable one = new IntWritable(1);
private Text word = new Text();

public void map(Object key, Text value, Context context
) throws IOException, InterruptedException {
StringTokenizer itr = new StringTokenizer(value.toString());
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
context.write(word, one);
}
}
}

public static class IntSumReducer
extends Reducer<Text,IntWritable,Text,IntWritable> {
private IntWritable result = new IntWritable();

public void reduce(Text key, Iterable<IntWritable> values,
Context context
) throws IOException, InterruptedException {
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
result.set(sum);
context.write(key, result);
}
}

public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "word count");
job.setJarByClass(WordCount.class);
job.setMapperClass(TokenizerMapper.class);
job.setCombinerClass(IntSumReducer.class);
job.setReducerClass(IntSumReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}

The mapper sets all words in the context with key=word, value=1.

if the text file is like the one below

Michele Michele Rizzi

the context at the end of the mapper method will have

key:Michele, value:1

key:Michele value:1

key=Rizzi value:1

The reducer automatically groups the items with the same key, the input parameters will be

key=”Michele” values={1,1}

key=”Rizzi” values={1}

Difference Flink Hadoop

Map/Reduce is a protocol where one transformation takes most likely two phases: One map phase, one reduce phase. Between these phases, the results are spilled to disk. If you have more than one transformation, it stacks up.

One of the limitations of Hadoop MapReduce is that the framework can only batch process one job at a time.

MapReduce may be a legacy system for many companies who started their Big Data journey when Hadoop first came out. This framework has been the workhorse for large data projects. One of the significant challenges for systems developed with this framework is its high-latency, batch-mode response. Since MapReduce has evolved over the last 10 years some developers would complain that it’s difficult to maintain because of inherent inefficiencies in its design and code.
Flink tries to combine all transformation into a job. If the job is started the transformations are done in this jobs, intermediate results are kept in memory. One of the highlights of Flink is its own memory management to prevent Garbage Collection and overflows.

Flink provides a true data streaming platform that uses high-performance dataflow architecture. It is also a strong tool for batch processing since Flink handles batch as a special case of streaming. Flink processes streaming data in real-time streams by pipelining data elements as soon as the elements arrive.

Here is available the source code of the application

Should it have a logic?

if I were the architect of the system I wouldn’t have put any logic in that book object so it should be just a container and below are my reasons

1. The clean code says that one class should be as short as possible with just one responsibility. the less methods and the less code is inside the better it is.
2. I imagine the Amazon web site made of the Model View Control pattern and SQL database using the DAO pattern. If that’s implemented using also Spring and the Dependency injenction pattern and Hibernate the system should be designed as below
   1. View: normal HTML  Javascript page so when the user wants to see the list of the book it calls the REST endpoint in the Spring book controller called getBook
   2. Controller: Spring Book controller with the initial business logic for verifying the input of the user, calling the model, populating a response object which contains a list of books and returns it to the view
      1. All logic should be in the controller and its method getBooks, the book should contain just informations
   3. Model: Another REST endpoint which like the controller has the logic. The logic is inside the rest book service and the book object should be just a container.
      1. That rest service can collect the information from the SQL database using hibernate and as usual the Hibernate book object is just a mapping of the book table, all logic should be inside the BookDAO using the DAO design pattern

For me the book should be just a bean POJO with private variables, getters, setters and constructors

The more logic you put inside the less clean is the code

Tomcat and IBM WebSphere are both HTTP Java servers

Server: is a machine always on, which serves the client requests: the client asks for a request the server processes it and replies with a response.

HTTP is the Hyper Text transfer protocol, it’s a request- response protocol for exchanging web pages.
A Java HTTP web server is a HTTP server which has a JVM (Java virtual machine) which allows to execute Java web applications.

A java web application can be stored in two archives: War and EAR.

• WAR or web archive is an archive made of the below structure:
  • 
  • this archive is made of the below files
    • web.xml is the application descriptors which contains the configuration information of the application
    • lib: is a folder with all jar files necessary to make working our project
    • classes: all project byte codes
    • JSP: the Java server Pages which contain html, css javascript code and Java code as well. After the user clicks on the button it may execute the java code inside the JSP or can call a class stored in the “classes folder” which can use a jar file contained in the lib folder.
  • this archive can be made in many ways
    • using the eclipse export tool: export as->war
    • using the Java building tools such as Ant, Maven, Gradle
• EAR entreprise Archive is a more complex archive which can contain more war and the EJB JAR. The EJB or Enterprise Java Beans are necessary for implementing the session, the persistence of data and the message queue. EJB are used by Java front end framework such as JSF and Struts. If using Spring it has their own beans and it can make the above functions without needing EJB
  • 

After having clear the function about a Java web application and a java web server we can compare Tomcat and WebSphere

• Tomcat: freeware open source server that is light and fast it doesn’t have a EJB container and it allows only to deploy EAR applications
• Web Sphere: Private IBM server very complex and expensive with the EJB container and the opportunity to install war and ear files. WebSphere allows to use JNDI, Queues and many other technologies.

Both servers work with the java web applications (war) with JSP, Class files but being more widely supported Web Sphere is used for complex projects such as banks and insurances rather then Tomcat used for small projects such as web sites and small applications.

Xml can respect a set of rules defined in two ways: the old DTD (Document Type Definition) and the more recent XML schema.

XML is useful for the below purposes

• Web services SOAP use the XML for the communication between the two involved endpoints and for their main configuration file (WSDL)
• making the communication between different software, it’s much better than the CSV (Comma separated value) because being hierarchically an xml document can be easily converted in an object (that’s called “Marshalling”) and vice-versa (“Un-Marshalling”)
• Storing configuration information: the application descriptor web.xml, the JSF configuration file faces-config.xml, the struts config file, the hibernate config file are just few examples about XML configurations file.

the only problem of the xml is that it’s redundant so every opened tag should be closed and the tag name is replicated two times

<tag_name>
tag value
<tag_name>

if the tag name is huge and it has not value so we transmit on line many characters with no value

A better format is the JSON which can be converted to an object but it replaces tags with parenthesis, so it’s not replicated any information.

How to process an XML file using Java?

We need to parse it using XPath and to add a node everywhere.

XPATH

let’s try to inquire the below xml

<?xml version="1.0" encoding="UTF-8"?>
<Employees>
<Employee id="1">
<age>29</age>
<name>Pankaj</name>
<gender>Male</gender>
<role>Java Developer</role>
</Employee>
<Employee id="2">
<age>35</age>
<name>Lisa</name>
<gender>Female</gender>
<role>CEO</role>
</Employee>
<Employee id="3">
<age>40</age>
<name>Tom</name>
<gender>Male</gender>
<role>Manager</role>
</Employee>
<Employee id="4">
<age>25</age>
<name>Meghna</name>
<gender>Female</gender>
<role>Manager</role>
</Employee>
</Employees>

private static List<String> getFemaleEmployeesName(Document doc, XPath xpath) {
List<String> list = new ArrayList<>();
try {
//create XPathExpression object
XPathExpression expr =
xpath.compile("/Employees/Employee[gender='Female']/name/text()");
//evaluate expression result on XML document
NodeList nodes = (NodeList) expr.evaluate(doc, XPathConstants.NODESET);
for (int i = 0; i < nodes.getLength(); i++)
list.add(nodes.item(i).getNodeValue());
} catch (XPathExpressionException e) {
e.printStackTrace();
}
return list;
}

private static List<String> getEmployeeNameWithAge(Document doc, XPath xpath, int age) {
List<String> list = new ArrayList<>();
try {
XPathExpression expr =
xpath.compile("/Employees/Employee[age>" + age + "]/name/text()");

NodeList nodes = (NodeList) expr.evaluate(doc, XPathConstants.NODESET);
for (int i = 0; i < nodes.getLength(); i++)
list.add(nodes.item(i).getNodeValue());
} catch (XPathExpressionException e) {
e.printStackTrace();
}
return list;
}

private static String getEmployeeNameById(Document doc, XPath xpath, int id) {
String name = null;
try {
XPathExpression expr =
xpath.compile("/Employees/Employee[@id='" + id + "']/name/text()");
name = (String) expr.evaluate(doc, XPathConstants.STRING);
} catch (XPathExpressionException e) {
e.printStackTrace();
}

return name;
}

 

Below is the method to add a node in the name of a female employer

private static List<String> addNodesInsideFemaleEmployeesName(Document doc, XPath xpath) {
List<String> list = new ArrayList<>();
try {
//create XPathExpression object
XPathExpression expr =
xpath.compile("/Employees/Employee[gender='Female']/name/text()");
//evaluate expression result on XML document
NodeList nodes = (NodeList) expr.evaluate(doc, XPathConstants.NODESET);
for (int i = 0; i < nodes.getLength(); i++){

Element node = doc.createElement("New_Node"+i);
Element nodeInside = doc.createElement("newnodeInside");

nodeInside.setTextContent("This is the content"); //adds content
nodeInside.setAttribute("attrib", "attrib_value"); //adds an attribute
node.appendChild(nodeInside);

nodes.item(i).getParentNode().insertBefore(node, nodes.item(i).getNextSibling());
}
} catch (XPathExpressionException e) {
e.printStackTrace();
}
return list;
}

 

below is the result

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<Employees>
<Employee id="1">
<age>29</age>
<name>Pankaj</name>
<gender>Male</gender>
<role>Java Developer</role>
</Employee>
<Employee id="2">
<age>35</age>
<name>Lisa<New_Node0>
<newnodeInside attrib="attrib_value">This is the content</newnodeInside>
</New_Node0>
</name>
<gender>Female</gender>
<role>CEO</role>
</Employee>
<Employee id="3">
<age>40</age>
<name>Tom</name>
<gender>Male</gender>
<role>Manager</role>
</Employee>
<Employee id="4">
<age>25</age>
<name>Meghna<New_Node1>
<newnodeInside attrib="attrib_value">This is the content</newnodeInside>
</New_Node1>
</name>
<gender>Female</gender>
<role>Manager</role>
</Employee>
</Employees>

if needed to insert the node outside the name it’s necessary just to modify the XPATH query that will be

/Employees/Employee[gender='Female']/name

the project is available here