Enjoy

What is big data?

Big data is large amount of data to store and process on the server.

They can be the user activities of big applications such as social media (facebook), search engines (google), Internet of things (sensors, machines, transports etc)

Today 1TB hard drive costs around 50 euro so it’s cheap storing large amount of data but processing it requires long time using SQL database and NoSQL databases are better way for storing these informations and also they are other frameworks for processing them such as Hadoop.

Big Data technologies

There are two classes of technologies to handle big data, they are complementary and usually used together:

• Operational big data: technologies for storing and processing the big data: NoSQL databases
• Analytical Big Data: Systems for Massive Parallel Processing such as Hadoop

Google Hadoop

in the usual client-server approach we have a client that gets the information from the database.

Hadoop splits the task in small parts assigning each one to a computer connected to the network and connects all the information afterward.This is the MapReduce algorithm.

This means that Hadoop has the below components:

• his own distributed file system called HDFS: it’s made of masters NameNode which manages many slaves nodes DataNodes. A file is divided into many blocks stored in set of DataNode. NameNode makes the mapping between the file and the block
• his own common libraries
• the framework called YARN
• the map reduce algorithm: made by a Map task which converts the input in a map and a Reduce task which splits the map in a smaller set of tuple. The map reduce Algorithm has a single master called Job Tracker responsible for resource management and scheduling jobs, and a slave TaskTraker which executes the task

Hadoop works as below:

• the user submits a job to Hadoop with a input/output elements, a MapReduce algorithm, the job configuration
• the jobTraker takes the responsibility of the job submitting it to the slaves
• the task traker executes its task

Here is a guide on how to install Hadoop

Here is a list of the hadoop file system commands

I tested a couple of commands: mkdir and ls, they work fine

Finally I’ve written a simple map reduce example:

Here is an example of the MapReduce algorithm

package com.michele.rizzi.hadoop;
import java.io.IOException;
import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCount {

public static class TokenizerMapper
extends Mapper<Object, Text, Text, IntWritable>{

private final static IntWritable one = new IntWritable(1);
private Text word = new Text();

public void map(Object key, Text value, Context context
) throws IOException, InterruptedException {
StringTokenizer itr = new StringTokenizer(value.toString());
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
context.write(word, one);
}
}
}

public static class IntSumReducer
extends Reducer<Text,IntWritable,Text,IntWritable> {
private IntWritable result = new IntWritable();

public void reduce(Text key, Iterable<IntWritable> values,
Context context
) throws IOException, InterruptedException {
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
result.set(sum);
context.write(key, result);
}
}

public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "word count");
job.setJarByClass(WordCount.class);
job.setMapperClass(TokenizerMapper.class);
job.setCombinerClass(IntSumReducer.class);
job.setReducerClass(IntSumReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}

The mapper sets all words in the context with key=word, value=1.

if the text file is like the one below

Michele Michele Rizzi

the context at the end of the mapper method will have

key:Michele, value:1

key:Michele value:1

key=Rizzi value:1

The reducer automatically groups the items with the same key, the input parameters will be

key=”Michele” values={1,1}

key=”Rizzi” values={1}

Difference Flink Hadoop

Map/Reduce is a protocol where one transformation takes most likely two phases: One map phase, one reduce phase. Between these phases, the results are spilled to disk. If you have more than one transformation, it stacks up.

One of the limitations of Hadoop MapReduce is that the framework can only batch process one job at a time.

MapReduce may be a legacy system for many companies who started their Big Data journey when Hadoop first came out. This framework has been the workhorse for large data projects. One of the significant challenges for systems developed with this framework is its high-latency, batch-mode response. Since MapReduce has evolved over the last 10 years some developers would complain that it’s difficult to maintain because of inherent inefficiencies in its design and code.
Flink tries to combine all transformation into a job. If the job is started the transformations are done in this jobs, intermediate results are kept in memory. One of the highlights of Flink is its own memory management to prevent Garbage Collection and overflows.

Flink provides a true data streaming platform that uses high-performance dataflow architecture. It is also a strong tool for batch processing since Flink handles batch as a special case of streaming. Flink processes streaming data in real-time streams by pipelining data elements as soon as the elements arrive.

Here is available the source code of the application

AngularJS2 doesn’t allow to use the <script> tag and the only way for making a project using that technology is using NPM:

• install all dependencies on NodeJS using NPM including the WebPack
• deploy the application on the NodeJS instance running locally on your computer
• building a HTML/JS file for making it running on the Apache

That’s much more different from the usual way of working which needed just to write a html file importing the js file inside and opening the html file using a normal browser for testing the application

ReactJS allows the user to choose between using the <script> tag or making the application using nodejs and NPM.

My previous applications were made using ReactJS and the <script> tag, today I’ve tested the second way of working.

What I did was to:

• Download a ReactJS seed project which has already set up webpack, npm, grunt, gulp and contains a guide about how to run the project locally and how to build the project in production
• I installed React Scroll spy on that seed project
• I replaced the source code of the ReactJS seed project with the React Scroll spy NPM plugin project
• I installed React Bootstrap
• I integrated the bootstrap modal to that project

Basically I’ve written few lines of code but I’ve done a huge configuration-integration work and now I think I’m able to work with ReactJS and NPM without using the <script> tag which I believe will be mandatory in the future

Here is available the result

Should it have a logic?

if I were the architect of the system I wouldn’t have put any logic in that book object so it should be just a container and below are my reasons

1. The clean code says that one class should be as short as possible with just one responsibility. the less methods and the less code is inside the better it is.
2. I imagine the Amazon web site made of the Model View Control pattern and SQL database using the DAO pattern. If that’s implemented using also Spring and the Dependency injenction pattern and Hibernate the system should be designed as below
   1. View: normal HTML  Javascript page so when the user wants to see the list of the book it calls the REST endpoint in the Spring book controller called getBook
   2. Controller: Spring Book controller with the initial business logic for verifying the input of the user, calling the model, populating a response object which contains a list of books and returns it to the view
      1. All logic should be in the controller and its method getBooks, the book should contain just informations
   3. Model: Another REST endpoint which like the controller has the logic. The logic is inside the rest book service and the book object should be just a container.
      1. That rest service can collect the information from the SQL database using hibernate and as usual the Hibernate book object is just a mapping of the book table, all logic should be inside the BookDAO using the DAO design pattern

For me the book should be just a bean POJO with private variables, getters, setters and constructors

The more logic you put inside the less clean is the code

The HTML is processed by the HTTP client (Browser) for showing the web pages.

An evolution of the HTML document is the XML (eXtended markup language) which allows to convert any object to this type of data and make the communication between different systems or storing configuration.

The HTML pages can have different color, style, font so it was necessary one more language for split the structure of the page from his style and CSS were born. CSS is Cascade Style Sheet because it works in cascade from the root of the HTML document to the leaf.

The HTML pages have a logic inside about how to populate it, how to validate the choice of the user and about its behavior, how to communicate with the back end using SOAP (REST). The way for making this logic is using Javascript

Javascript is a language for making dynamic HTML document adding nodes, making conditions, check if the input of the user is correct.

Initially we had the first three technologies:

• HTML: structure of the page, its skeleton
• CSS: style of the page
• Javascript: make HTML and CSS dynamic

One Next library useful for making Javascript easier to use is JQuery, JQuery allows also to create plugins easy to embed in your web page for instance if you need a table with fixed header you can download the Jquery plugin and embed in your web page

Another step forward was making the front end framework which contains already written component made of HTML,CSS, Javascript in this way the developer shouldn’t make one menu from the scratch but it’s enough using the component.

These frameworks are the Twitter Bootstrap and Foundation

Another improvement is making CSS “Object oriented” with the possibility of making inheritance and reuse of styles and they were born LESS, SASS and all CSS preprocessors

Another idea was to make a Javascript server in this way the Javascript developer can work back end side and it was born NodeJS.

I was able to integrate NodeJS with Mongodb, Rest services and SQL database. it’s a real back end server made of Javascript

There are many Javascript libraries and write a huge HTML file with many imports on its head is bad for this reason NodeJS allows to execute the build like Maven and Gradle for Java using NPM (Node Package Manager) and the evolute building tools like GruntJS and GulpJS.

At this time we had many already built plugins:

• Javascript pure: allows to do everything but it’s complex
• Foundation and Bootstrap with components already built which are mobile first
• Jquery:  with plugin already built

Instead of building a HTML page from the scratch it’s enough to find the component and embed it in your page.

Another challenge was making Javascript more object oriented and the first framework which allows to do that is RequireJS

var messages = require('./messages');
var print = require('print');
print(messages.getHello());

Messages:

define(function () {
return {
getHello: function () {
return 'Hello World';
}
};
});

so now Javascript is object Oriented.

Then Google invented its AngularJS which is very powerful.

Now Javascript is an hybrid between functions for navigating HTML document and Object oriented.

A huge step forward was made by ReactJS and EXTJS because both these languages are fully object oriented: you don’t need to make HTML tag of the div but just to use the “Div” object which will be rendered as DIV.

The developer doesn’t need to write HTML tags but just using an object provided by the technology which will be rendered as the necessary tags.

And now there is the monster AngularJS2 which is fully object oriented like ReactJS and EXTJs and needs to write a perfect Javascript code:

Javascript is a language without rules so you can do something like this:

var a=0; //a is a number

a="hello";//now a is a string

a.color="red";//now a is an object

a=function (){}; //now a is a function

if you embed this code in your html page with the script tag it works.

If you use AngularJS2 it doesn’t work because you should compile the code with the Angular framework and then build it in a HTML file.

So the tag script doesn’t work anymore

Comparison between all technologies

• Using only HTML, CSS, Javascript for making an HTML page is difficult even because the most of the site are responsive etc. Why build one component from the scratch when there is one already built and tested carefully?
• Bootstrap,Foundation, Jquery: made plugins for extending the old version of Javascript not fully object oriented but made of functions for make dynamic a HTML/ CSS document
• AngularJS, ReactJS, AngularJS2, ExtJS, RequireJS: made a new way of using Javascript Object oriented making a conversion from their objects in HTML tags. You don’t need to write HTML tags anymore but just the object using your framework and it will be converted in HTML tag automatically
• SASS LESS and all CSS preprocessors make object oriented CSS styles
• NodeJS with NPM, Grunt and Gulp is an evolute Javascript web server
• AngularJS2: is not only a object oriented framework but also it requires to use NodeJS for validating the web page and for building it. AngularJS2 is a little bit more complex than the other front end frameworks but it’s better structured.

Integration of technologies

If you work with HTML file you can use all technologies together: HTML, CSS, Javascript, JQuery plugin together with a Foundation component and a Bootstrap component. That’s easy to do it’s just enough to import the js file in your HTML document

If you work in an Object Oriented Domain the integration with the technologies is very hard because an ReactJS class is finally converted in HTML tags so you don’t have the fully control of the final HTML document.

Fortunately there are ReactJS classes which produce Bootstrap components etc.

So if you choose to use ReactJS your web application it should be built using only ReactJS, the same for AngularJs2 and ExtJs.

AngulatJS1 is an hybrid so it uses html tag with special attributes such as ng-app, ng-if, ng-switch so it’s easily integrable with Bootstrap Foundation and other frameworks.

 Finally which technology should I choose?

It’s necessary to choose one technology fully tested and with a plenty of components already built

AngularJS is great even if it’s not fully object oriented it can used with JQuery, Bootstrap Foundation and all plugins. I believe it’s the best technology to use.

ReactJS has been built since a long time and if it’s not possible to use with all JQUERY / Bootstrap /Foundation components, there are many libraries and many components already built and it’s fully object oriented

AngularJS2 it’s the best framework but it has been built two months ago and I believe there are few components already built and since it’s object oriented it cannot be integrated with Jquery plugins bootstrap foundation. If needed to do something and it’s not available any component already built you should make it on your own. Is it worthy?

I believe we should wait a little bit more for using Angular2 in real important big projects