What is big data?

Big data is large amount of data to store and process on the server.

They can be the user activities of big applications such as social media (facebook), search engines (google), Internet of things (sensors, machines, transports etc)

Today 1TB hard drive costs around 50 euro so it’s cheap storing large amount of data but processing it requires long time using SQL database and NoSQL databases are better way for storing these informations and also they are other frameworks for processing them such as Hadoop.

Big Data technologies

There are two classes of technologies to handle big data, they are complementary and usually used together:

• Operational big data: technologies for storing and processing the big data: NoSQL databases
• Analytical Big Data: Systems for Massive Parallel Processing such as Hadoop

Google Hadoop

in the usual client-server approach we have a client that gets the information from the database.

Hadoop splits the task in small parts assigning each one to a computer connected to the network and connects all the information afterward.This is the MapReduce algorithm.

This means that Hadoop has the below components:

• his own distributed file system called HDFS: it’s made of masters NameNode which manages many slaves nodes DataNodes. A file is divided into many blocks stored in set of DataNode. NameNode makes the mapping between the file and the block
• his own common libraries
• the framework called YARN
• the map reduce algorithm: made by a Map task which converts the input in a map and a Reduce task which splits the map in a smaller set of tuple. The map reduce Algorithm has a single master called Job Tracker responsible for resource management and scheduling jobs, and a slave TaskTraker which executes the task

Hadoop works as below:

• the user submits a job to Hadoop with a input/output elements, a MapReduce algorithm, the job configuration
• the jobTraker takes the responsibility of the job submitting it to the slaves
• the task traker executes its task

Here is a guide on how to install Hadoop

Here is a list of the hadoop file system commands

I tested a couple of commands: mkdir and ls, they work fine

Finally I’ve written a simple map reduce example:

Here is an example of the MapReduce algorithm

package com.michele.rizzi.hadoop;
import java.io.IOException;
import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCount {

public static class TokenizerMapper
extends Mapper<Object, Text, Text, IntWritable>{

private final static IntWritable one = new IntWritable(1);
private Text word = new Text();

public void map(Object key, Text value, Context context
) throws IOException, InterruptedException {
StringTokenizer itr = new StringTokenizer(value.toString());
while (itr.hasMoreTokens()) {
word.set(itr.nextToken());
context.write(word, one);
}
}
}

public static class IntSumReducer
extends Reducer<Text,IntWritable,Text,IntWritable> {
private IntWritable result = new IntWritable();

public void reduce(Text key, Iterable<IntWritable> values,
Context context
) throws IOException, InterruptedException {
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
result.set(sum);
context.write(key, result);
}
}

public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
Job job = Job.getInstance(conf, "word count");
job.setJarByClass(WordCount.class);
job.setMapperClass(TokenizerMapper.class);
job.setCombinerClass(IntSumReducer.class);
job.setReducerClass(IntSumReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}

The mapper sets all words in the context with key=word, value=1.

if the text file is like the one below

Michele Michele Rizzi

the context at the end of the mapper method will have

key:Michele, value:1

key:Michele value:1

key=Rizzi value:1

The reducer automatically groups the items with the same key, the input parameters will be

key=”Michele” values={1,1}

key=”Rizzi” values={1}

Difference Flink Hadoop

Map/Reduce is a protocol where one transformation takes most likely two phases: One map phase, one reduce phase. Between these phases, the results are spilled to disk. If you have more than one transformation, it stacks up.

One of the limitations of Hadoop MapReduce is that the framework can only batch process one job at a time.

MapReduce may be a legacy system for many companies who started their Big Data journey when Hadoop first came out. This framework has been the workhorse for large data projects. One of the significant challenges for systems developed with this framework is its high-latency, batch-mode response. Since MapReduce has evolved over the last 10 years some developers would complain that it’s difficult to maintain because of inherent inefficiencies in its design and code.
Flink tries to combine all transformation into a job. If the job is started the transformations are done in this jobs, intermediate results are kept in memory. One of the highlights of Flink is its own memory management to prevent Garbage Collection and overflows.

Flink provides a true data streaming platform that uses high-performance dataflow architecture. It is also a strong tool for batch processing since Flink handles batch as a special case of streaming. Flink processes streaming data in real-time streams by pipelining data elements as soon as the elements arrive.

Here is available the source code of the application

Should it have a logic?

if I were the architect of the system I wouldn’t have put any logic in that book object so it should be just a container and below are my reasons

1. The clean code says that one class should be as short as possible with just one responsibility. the less methods and the less code is inside the better it is.
2. I imagine the Amazon web site made of the Model View Control pattern and SQL database using the DAO pattern. If that’s implemented using also Spring and the Dependency injenction pattern and Hibernate the system should be designed as below
   1. View: normal HTML  Javascript page so when the user wants to see the list of the book it calls the REST endpoint in the Spring book controller called getBook
   2. Controller: Spring Book controller with the initial business logic for verifying the input of the user, calling the model, populating a response object which contains a list of books and returns it to the view
      1. All logic should be in the controller and its method getBooks, the book should contain just informations
   3. Model: Another REST endpoint which like the controller has the logic. The logic is inside the rest book service and the book object should be just a container.
      1. That rest service can collect the information from the SQL database using hibernate and as usual the Hibernate book object is just a mapping of the book table, all logic should be inside the BookDAO using the DAO design pattern

For me the book should be just a bean POJO with private variables, getters, setters and constructors

The more logic you put inside the less clean is the code

Tomcat and IBM WebSphere are both HTTP Java servers

Server: is a machine always on, which serves the client requests: the client asks for a request the server processes it and replies with a response.

HTTP is the Hyper Text transfer protocol, it’s a request- response protocol for exchanging web pages.
A Java HTTP web server is a HTTP server which has a JVM (Java virtual machine) which allows to execute Java web applications.

A java web application can be stored in two archives: War and EAR.

• WAR or web archive is an archive made of the below structure:
  • 
  • this archive is made of the below files
    • web.xml is the application descriptors which contains the configuration information of the application
    • lib: is a folder with all jar files necessary to make working our project
    • classes: all project byte codes
    • JSP: the Java server Pages which contain html, css javascript code and Java code as well. After the user clicks on the button it may execute the java code inside the JSP or can call a class stored in the “classes folder” which can use a jar file contained in the lib folder.
  • this archive can be made in many ways
    • using the eclipse export tool: export as->war
    • using the Java building tools such as Ant, Maven, Gradle
• EAR entreprise Archive is a more complex archive which can contain more war and the EJB JAR. The EJB or Enterprise Java Beans are necessary for implementing the session, the persistence of data and the message queue. EJB are used by Java front end framework such as JSF and Struts. If using Spring it has their own beans and it can make the above functions without needing EJB
  • 

After having clear the function about a Java web application and a java web server we can compare Tomcat and WebSphere

• Tomcat: freeware open source server that is light and fast it doesn’t have a EJB container and it allows only to deploy EAR applications
• Web Sphere: Private IBM server very complex and expensive with the EJB container and the opportunity to install war and ear files. WebSphere allows to use JNDI, Queues and many other technologies.

Both servers work with the java web applications (war) with JSP, Class files but being more widely supported Web Sphere is used for complex projects such as banks and insurances rather then Tomcat used for small projects such as web sites and small applications.

I’ve been studied english in a deep way for more than two months at least one hour a day and I would write a feedback of my actual knowledgement what is done good and what I need to improve.

There are many steps that you must follow when you learn a new language:

• you learn the grammar, the voucabulary etc
• you become to explain everything.
•  you become able to explain everithing with a lot of vocabulary, synonyms that are different on the basis of the environment:

For instance it is better to don’t use one word more time in a sentence. You have to use a synonym.

If you want to say an apology you can say “sorry” (informal) or “I would like to tell you my apology” (formal)

• you replace the thought of your original language with the thought of your destination language when you speak with yourself
• you become able to write a complex text
• you become able to understand a film like the following without the subtitles

Now I think to have an high level of english because I’m able to talk about everything as in the present as in the past as in the future by using a lot of synonyms and also I’m able to choose the correct word on the basis of the environment (formal or informal) and I speak with myself in english

My problems are the following

• I some times confuse some words such as few and little:

I know that little is used for uncountable nouns such as “little money” but instinctively I say “few money” because in my mind few=much=uncountable many=little=countable (wrong). I need more extercises but American and British Academy have few extercises, is better Douolingo and when I finish this course I will do Duolingo tests.

• There are some words that I never remember because I rarely use them but they are written on my english memory test
• In my opinion the most difficult skill to have is undrestanding spoken english such as videos even if the interlocutor speaks fastly and with slangs because in this case first of all you must understand all the terms used into the phrase and then you’ll undrestand the sense of this one.

I think that the advanced level of American and British academy is not enough to speak english perfectly for this reason I would like to have the following goals:

1. I would like to finish all the American and British academy courses by the 10th of November 2014 this course is necessary to understand the english rules and the english accent gradually
2. I would like to finish All Duolingo three by the end of january 2015. The duolingo tests are more complex than ABAEnglish tests
3. when I’ve free time I’ll watch videos and films in english language with subtitles in italian and also I’ll read news in english
4. At the end of my duolingo test I’ll correct all articles written here and I would like to discover how much mistakes I’ve done to know how my ability is increase in these days
5. when I’m ready I’ll try to do IELTS exam

I do my best to speak english very well asap I hope that I’m able to reach this goal.

GANTT DIAGRAM

I wrote above small examples about how to plan the common activities with some professional instruments.

At the end of each course I will share the deliverable also called milestone: my English certificate.

This is very optimistic planning for this reason I think that this is a challenge.

I’m not hurry and I think that it’s better to end all my activities later but perfectly than end everything in time but wrongly.

The problem, however, is that this is horribly inefficient and it teaches developers to ignore how the underlying database works. In fact, it’s so inefficient that it just about mapping relational database systems in ORM system for this reasons were born a lot of new databases nosql like Apache Cassandra, MongoDB and others.
I only studied one of these technologies: Mongodb, because in my opinion it is the most used.
The differences between MySQL and mongodb are the following.

Data Representation

MySQL represents data in tables and rows.

MongoDB represents data as collections of JSON documents.

A JSON document is very much like what you would be working with in your application layer. If you are using javascript, it’s exactly what you’re working with. If you’re using PHP, it’s just like an associative array. If you’re using python, its just like a dictionary object.

Querying

The SQL in MySQL stands for Structured Query Language. That’s because you have to put together a string in this query language that is parsed by the database system. This is what makes SQL injection attacks possible.

MongoDB uses object JSON querying.

Relationships

One of the best things about MySQL and relational databases in general is the almighty JOIN operation. This allows you to perform queries across multiple tables.

MongoDB does not support joins, but it does multi-dimensional data types such as arrays and even other documents. Placing one document inside another is referred to as embedding. For example, if you were to create a blog using MySQL, you would have a table for posts and a table for comments. In MongoDB you might have a single collection of posts, and an array of comments within each post.

Transactions

Another great thing about MySQL is its support for atomic transactions. The ability to contain multiple operations within a transaction and roll back the whole thing as if it were a single operation.

MongoDB does not support transactions, but single operations are atomic.

Schema Definition

MySQL requires you to define your tables and columns before you can store anything, and every row in a table must have the same columns.

One of my favourite things about MongoDB is that you don’t define the schema. You just drop in documents, and two documents within a collection don’t even need to have the same fields.

Performance

MySQL often has  poor performance. Well if you are using an ORM, performance will likely increased. If you are using a simple database wrapper and you’ve indexed your data correctly, you’ll get good performance

By sacrificing things like joins and providing excellent tools for performance analysis, MongoDB can perform much better than a relational database. You still need to index your data and the truth is that the vast majority applications out there don’t have enough data to notice the difference.

When should you use MySQL?

If your data structure fits nicely into tables and rows, MySQL will offer you robust and easy interaction with your data. If it’s performance that is your concern, there is a good chance you don’t really need MongoDB. Most likely, you just need to index your data properly. If you require SQL or transactions, you’ll have to stick with MySQL.

When should you use MongoDB?

If your data seems complex to model in a relational database system, or if you find yourself de-normalizing your database schema or coding around performance issues you should consider using MongoDB. If you find yourself trying to store serialized arrays or JSON objects, that’s a good sign that you are better off MongoDB. If you can’t pre-define your schema or you want to store records in the same collection that have different fields, that’s another good reason.

Example of mongodb use

> use mydb

switched to db mydb
> users ={"name": "Michele", "surname":"Rizzi" }
{ "name" : "Michele", "surname" : "Rizzi" }
> db.users.insert(users)
WriteResult({ "nInserted" : 1 })
> departments2={"name": "CS","value": "Computer Science" }
{ "name" : "CS", "value" : "Computer Science" }
> db.department.insert(departments)
WriteResult({ "nInserted" : 1 })
> db.department.insert(departments2)
WriteResult({ "nInserted" : 1 })

> db.users.find()
{ "_id" : ObjectId("53dbe6cc8f9fa523bbaef558"), "name" : "Michele", "surname" : "Rizzi" }
> db.department.find()
{ "_id" : ObjectId("53dbe7128f9fa523bbaef559"), "name" : "TC", "value" : "Telecomunications" }
{ "_id" : ObjectId("53dbe71f8f9fa523bbaef55a"), "name" : "CS", "value" : "Computer Science" }
> db.department.find().sort({name:1})
{ "_id" : ObjectId("53dbe71f8f9fa523bbaef55a"), "name" : "CS", "value" : "Computer Science" }
{ "_id" : ObjectId("53dbe7128f9fa523bbaef559"), "name" : "TC", "value" : "Telecomunications" }
> db.department.find().sort({name:-1})
{ "_id" : ObjectId("53dbe7128f9fa523bbaef559"), "name" : "TC", "value" : "Telecomunications" }
{ "_id" : ObjectId("53dbe71f8f9fa523bbaef55a"), "name" : "CS", "value" : "Computer Science" }
> db.department.update ({name:"TC"},{ $set: {name:"TLC"}})
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
> db.department.find().sort({name:-1})
{ "_id" : ObjectId("53dbe7128f9fa523bbaef559"), "name" : "TLC", "value" : "Telecomunications" }
{ "_id" : ObjectId("53dbe71f8f9fa523bbaef55a"), "name" : "CS", "value" : "Computer Science" }
> db.department.remove({name: "CS"})
WriteResult({ "nRemoved" : 1 })
> db.department.find().sort({name:-1})
{ "_id" : ObjectId("53dbe7128f9fa523bbaef559"), "name" : "TLC", "value" : "Telecomunications" }
> db.department.update ({name:"TC"}, {$set: {namee:"TLC"}})
WriteResult({ "nMatched" : 0, "nUpserted" : 0, "nModified" : 0 })
> db.department.find().sort({name:-1})
{ "_id" : ObjectId("53dbe7128f9fa523bbaef559"), "name" : "TLC", "value" : "Telecomunications", "namee" : "TLC" }
> ObjectId()
ObjectId("53dbeb448f9fa523bbaef55b")
> ObjectId()
ObjectId("53dbeb468f9fa523bbaef55c")
> ObjectId()
ObjectId("53dbeb478f9fa523bbaef55d")
> ObjectId()
ObjectId("53dbeb4a8f9fa523bbaef55e")
> var id=ObjectId()
> id
ObjectId("53dbeb658f9fa523bbaef55f")
> db.department.update ({name:"TLC"}, {$set: {id: id}})
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
> db.department.find().sort({name:-1})
{ "_id" : ObjectId("53dbe7128f9fa523bbaef559"), "name" : "TLC", "value" : "Telecomunications", "namee" : "TLC", "id" : ObjectId("53dbeb658f9fa523bbaef55f") }
> db.users
> db.users.update({},{$set: {depId:id}})
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
> db.users.find()
{ "_id" : ObjectId("53dbe6cc8f9fa523bbaef558"), "name" : "Michele", "surname" : "Rizzi", "depId" : ObjectId("53dbeb658f9fa523bbaef55f") }

> db.users.find({name: $in: ["Michele","bla"]}})
2014-08-01T21:47:34.652+0200 SyntaxError: Unexpected token :
> db.users.find({$and:[{name: {$in: ["Michele","bla"]}}]})
{ "_id" : ObjectId("53dbe6cc8f9fa523bbaef558"), "name" : "Michele", "surname" : "Rizzi", "depId" : ObjectId("53dbeb658f9fa523bbaef55f") }

> db.users.find({$and:[{name: {$in: ["Michele","bla"]}}]},{name:1})
{ "_id" : ObjectId("53dbe6cc8f9fa523bbaef558"), "name" : "Michele" }
> db.users.find({$and:[{name: {$in: ["Michele","bla"]}}]})
{ "_id" : ObjectId("53dbe6cc8f9fa523bbaef558"), "name" : "Michele", "surname" : "Rizzi", "depId" : ObjectId("53dbeb658f9fa523bbaef55f") }
> db.users.find({$and:[{name: {$in: ["Michele","bla"]}}]},{id:1})
{ "_id" : ObjectId("53dbe6cc8f9fa523bbaef558") }

 

> var names=db.users.find({$and:[{name: {$in: ["Michele","bla"]}}]},{name:1});
> var names_id=names.hasNext()? names.next():null;
> db.users.find({name: names_id.name});
{ "_id" : ObjectId("53dbe6cc8f9fa523bbaef558"), "name" : "Michele", "surname" : "Rizzi", "depId" : ObjectId("53dbeb658f9fa523bbaef55f") }

> var s={ _id: ObjectId("50a8240b927d5d8b5891743c"), cust_id: "abc123", ord_date: new Date("Oct 04, 2012"), status: 'A', price: 25, items: [ { sku: "mmm", qty: 5, price: 2.5 }, { sku: "nnn", qty: 5, price: 2.5 } ]}

> s
{
"_id" : ObjectId("50a8240b927d5d8b5891743c"),
"cust_id" : "abc123",
"ord_date" : ISODate("2012-10-03T22:00:00Z"),
"status" : "A",
"price" : 25,
"items" : [
{
"sku" : "mmm",
"qty" : 5,
"price" : 2.5
},
{
"sku" : "nnn",
"qty" : 5,
"price" : 2.5
}
]
}
> var mapFunction1 = function() { emit(this.cust_id, this.price);};
> var reduceFunction1 = function(keyCustId, valuesPrices) { return Array.sum(valuesPrices);};
> db.users.insert(s)
WriteResult({ "nInserted" : 1 })
> db.users.mapReduce( mapFunction1, reduceFunction1, { out: "map_reduce_example" })
{
"result" : "map_reduce_example",
"timeMillis" : 343,
"counts" : {
"input" : 2,
"emit" : 2,
"reduce" : 0,
"output" : 2
},
"ok" : 1,
}
> var res1=db.users.mapReduce( mapFunction1, reduceFunction1, { out: "map_reduce_example" })
> db [res1.result].find()
{ "_id" : null, "value" : null }
{ "_id" : "abc123", "value" : 25 }

Example of integration with java

//connection

Mongo mongo = new Mongo("localhost", 27017);
DB db = mongo.getDB("database");

DBCollection collection = db.getCollection("collection");

//INSERT

BasicDBObject document = new BasicDBObject();
document.put("user", "Michele");

collection.insert(document);

//UPDATE WITH SET: REPLACE THE VALUE OF ONE FIELD

BasicDBObject updateDocument = new BasicDBObject();
updateDocument.append("$set", new BasicDBObject().append("user", "Ing Michele"));

collection.update(document, updateDocument);

//READ

BasicDBObject documentSearch = new BasicDBObject();
documentSearch.put("user", "Ing Michele");

DBCursor cursorDocJSON = collection.find(documentSearch);

while (cursorDocJSON.hasNext()) {
System.out.println(cursorDocJSON.next());
}

//DELETE

collection.remove(documentSearch);

The problem is that the information online is not structured like a database:

Web page is complex document with pictures, videos, images and meta information of mark up to specify the interaction with human and graphic effects.

A way to index a text is to use a system of information retrieval like lucene but it is not enough because Lucene works only parsing text on your web page.

Lucene makes a structure called index composed by a pointer of the document and many keywords that this document contains. When you are looking for some information Lucene searches them in these keyword and assigns a special outcome called ranking.

The disadvantage of Lucene is that it performs the simple research in string that is composed by keywords of the document. Lucene doesn’t know the semantic of the text but knows only the keyword contained in this document.

The best way to perform a research on written text document is to add them a specific meta-information by introducing another tags to the html document.

These tags are not visible to the human but are necessary to improve the index of this page for a external information retrieval system.

This is called Semantic WEB

Semantic web works using a special XML file with all rules of the domain, all the constraints of it and the objects written here must respect these rules.

This XML file is called ontology and it is the knowledge base (KB) of the system.

In this way it’s possible to convert the static not structured text into a xml structured document with a lot of constraints and rules (like database) and it’s possible to make queries to this system. The language that executes these queries is OWL (Ontology Web Language)

There are a lot of XML schemas and a lot of rules to make these semantic web a lot of ontology web languages but the most widespread used is RDF

RDF

RDF is the acronym of Resource Description framework and it is the standard model for data interchange on the Web

This is an example of RDF written by me using Apache Jena:

<xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
<xmlns:vcard="http://www.w3.org/2001/vcard-rdf/3.0#" >
<rdf:Description rdf:about="http://www.rizzimichele.it">
<vcard:N rdf:nodeID="A0"/>
<vcard:FN>Michele Rizzi</vcard:FN>
</rdf:Description>
<rdf:Description rdf:nodeID="A0">
<vcard:Family>Rizzi</vcard:Family>
<vcard:Given>Michele</vcard:Given>
</rdf:Description>
</rdf:RDF>

In the precedent example I written that the address www.rizzimichele.it is related at vcard descripted below that has Family Rizzi and Name: Michele.

This is a simple RDF example but I can make a complex XML like a WSDL and XSD that contains the identity card of web service.

Here is linked a complex RDF example

For example, the following are tags to define the domain.

<rdf:type rdf:resource="http://www.w3.org/2002/07/owl#InverseFunctionalProperty"/>
<rdfs:comment xml:lang="en">
Note that hasTopping is inverse functional because isToppingOf is functional
</rdfs:comment>
<rdfs:domain rdf:resource="#Pizza"/>
<rdfs:subPropertyOf rdf:resource="#hasIngredient"/>
<rdfs:range rdf:resource="#PizzaTopping"/>
<owl:inverseOf rdf:resource="#isToppingOf"/>
</owl:ObjectProperty>

The property of #hasTopping is subproperty of #hasIngredient: each ingredient is Topping

hasTopping has the inverse of isTopping for passive voice.

hasTopping has a specified domains: pizza names.

This is an object that respects all rules and constraints on the system

<owl:Class rdf:about="#FourCheesesTopping">
<rdfs:label xml:lang="pt">CoberturaQuatroQueijos</rdfs:label>
<rdfs:label xml:lang="it">Quattro Formaggi</rdfs:label>
<rdfs:subClassOf>
<owl:Restriction><owl:onProperty rdf:resource="#hasSpiciness"/>
<owl:someValuesFrom rdf:resource="#Mild"/>
</owl:Restriction>
</rdfs:subClassOf>
<rdfs:subClassOf rdf:resource="#CheeseTopping"/>
</owl:Class>

This pizza is the subclass of cheeseTopping with mid spiciness

Apache JENA

Jena is a java library to write or read RDF language and other owl languages such as TTL, it can also to perform a query to do a reasoning about the xml rdf tree to search the information that you need

this is an example of jena query

select ?pizza where {?pizza a owl:Class ; rdfs:subClassOf ?restriction. ?restriction owl:onProperty pizza:hasSpiciness ; owl:someValuesFrom pizza:Mild }

In the previous query I am looking for a pizza with Spiciness Mild

My project

I tested four examples of applications with Apache Jena and I made one my application to write a simple RDF file.

When I tried to read a customized RDF file to execute a customized queries I get the following error:

Exception in thread "main" com.hp.hpl.jena.shared.PropertyNotFoundException: http://www.w3.org/2001/vcard-rdf/3.0#N at com.hp.hpl.jena.rdf.model.impl.ModelCom.getRequiredProperty(ModelCom.java:1243) at com.hp.hpl.jena.rdf.model.impl.ResourceImpl.getRequiredProperty(ResourceImpl.java:170)
at org.apache.jena.example.helloworld.Tutorial6.main(Tutorial6.java:52)

I am not an expert of Lucene or Apache jena these are not widespread technologies and there are few forums and manuals to learn about them, when it will be necessary to work with this framework I will spend much time to try and I will perform a lot of tests  to fix these problems and to study this framework but in my opinion it is enough for now 

How I can search a lot of information in many documents or web pages?

I could search these information using a information retrieval system.

Information retrieval

The information retrival system executes queries to text unstructured document but this file is not ready to search information inside it because the user searches only few keyword that are written in sparse positions in the document and also the document can have a synonym of this keyword etc for this reason it is necessary to do a preprocessing operation.

The most important preprocessing operations are the following:

• Stemming: Replacing words with their stems. For instance the English stem “bikes” is replaced with “bike”; now query “bike” can find both documents containing “bike” and those containing “bikes”.
• Stop Words Filtering: Common words like “the”, “and” and “a” rarely add any value to a search. Removing them shrinks the index size and increases performance. It may also reduce some “noise” and actually improve search quality.
• Text Normalization: Stripping accents and other character markings can make for better searching.
• Synonym Expansion: Adding in synonyms at the same token position as the current word can mean better matching when users search with words in the synonym set.
• Get most frequency terms

The information retrieval system executes the precedent operation and saves the resulted document and the pointer of the original document in a index file.

When the user searches the keyword, this keyword is find into the index file and the result is shown to the users grouped by ranking (level of relevance) when the user clicks on the link, he can open the document.

Apache Lucene

Lucene is the framework composed by many libraries to execute queries on text documents like an information retrieval system.

The principal libs are the following:

• org.apache.lucene.document.Document: allows to store into the index principal information (for example title or contain or creation date) called field of the document. When executing the query it’s necessary to specify what property of the document is searched
• org.apache.lucene.analysis.standard.StandardAnalyzer: the analyser that processes the documents to extract the information to save into the index, for example removes stopword, executes stemming and all the four preprocessing operation written above
• org.apache.lucene.search.Query: executes the query
• org.apache.lucene.search.ScoreDoc: executes the ranking of the results
• org.apache.poi.extractor.ExtractorFactory: is very useful for indexing office document like word or excel and to execute queries inside these documents

My Project

I tested all the precedent libraries of Lucene for searching information into word and excel documents and they run well: Lucene creates index files, executes the query and shows the results ordered by ranking.

There are a lot of analysers:

• org.apache.lucene.analysis.standard.StandardAnalyzer
• org.apache.lucene.analysis.en.EnglishAnalyzer
• org.apache.lucene.analysis.it.ItalianAnalyzer

each analyser makes different preprocessing because English stopwords are different from the Italian stopwords, the same for synonyms  and stemming.

The pre processing step is not executed automatically but it is necessary to use the method analyzer.tokenStream

when I execute the following code

for (File f : queue) {

FileReader  fr = new FileReader(f);
TokenStream ts=analyzer.tokenStream("contents", ExtractorFactory.createExtractor(f).getText());
doc.add(new TextField("contents",ts));
ts.reset();
ts.close();
writer.addDocument(doc); }

the console shows the following error:

java.lang.IllegalStateException: TokenStream contract violation: reset()/close() call missing, reset() called multiple times, or subclass does not call super.reset(). Please see Javadocs of TokenStream class for more information about the correct consuming workflow.
at org.apache.lucene.analysis.Tokenizer$1.read(Tokenizer.java:111)

This is not the correct way to add the token that contains the text preprocessed at the indexed document.

when I add the field of the document to the indexed document there are no exceptions and the query is executed correctly  but when I use a token the previous exception occurs

In my opinion the another way to make the fourth precedents preprocessing is manually, by making many simple dictionaries:

• Stop word dictionary a simple xml file with all words that it will be removed while are processed documents for making the index and it will be removed from the user queries
• stem synonym dictionary: simple database table with two column: key:base term, another column:term to replace.
  • For example BASE TERM:eat; TERM TO REPLACE:ate,eating,eats, have a meal
  • I use a table that loads in a class file  when the aplication starts by executing the following query “select * from stem_synonym_dictionary”
    
  • when I process a document or a query and I found eating or have a meal, I replace it with eat (simple search into string)

JSF: Java Server Facets

JSF in my opinion is the most used front end framework because it is easy and stable: you can make a very complex and secure web application  in few time using it.

This framework has the principal component called “portlet”

each portlet is independent by the others for this reason this framework is scalable and easy to maintaining.

each portlet has the following files:

• Session bean with all the data necessary to run the page
• Request bean with all the operation and the action that the user can do
• JSP and XHTML file that shows the result
• Property file with the labels or db connections etc

The defect of this framework is that stores everything in session so it stores everything in ram jvm memory of the server.

This is a huge problem because the RAM of the java virtual machine of the server is expensive.

when the RAM memory of the server is full the server shows an “memory out of bound exception” and it is necessary to restart the server.

The second problem of JSF is the conversion of HTML tags:

JSF uses particular tags, each tag has a class that performs the operation, here is a small list of jsf tags

when you insert the tag <h:form></h:form> and you see the page rendered, this tag is converted in <form action=”"></form>

The conversion of tags is performed to simplify the operation of writing code: it is easiest to write JSF tags and they are rendered in html tags but it is a problem when you want to integrate the page with a special javascript or jquery component:

For example it is very difficult to integrate a jquery slider component with JSF because the slider requires a particular div sequence that are difficult to produce using JSF tags

JSF is easy to use, is stable, scalable, easy to maintenance and each page is independent from the others and requires a lot of jvm memory of the server and also it is difficult to integrate with javascript, jquery, bootstrap component

My  web application affittavetrina is developed using JSF for this reason it is expensive and unuseful because when there are connect a lot of people it might occurs the java memory out of bound exception but in the past  days I didn’t knew Spring model and view

Spring MVC

is in my opinion the best framework to make web application, it has the following characteristics:

• Client (view): the client is a simple JSP file written in html language with special tags (JSTL javascript tag language) and other tags for example the property like jsf for the internationalization. It is fully compatible with all component like jquery, javascript, ecc
•  Controller: Spring beans that allow to communicate via rest with htmlServletRequest and htmlServletResponse and you can set the information that you need in  session.
• Model you can choose a web services or another rest services etc.

The client with for example a jquery ajax component can send a rest request with the selection of url, methode (get, post) this request is processed by  server that sends the response to the client

Spring MVC is more complex than JSF because it is necessary to develop everything from the scratch, but requires less JVM memory (you can store in session only the information that you need) and it is fully customizable.

it is less stable and less maintainable because the jsp page written in html is more complex then the page written in jsf tags

Today I made a small project in Spring MVC to store a complex object in session and to read it. Another example is to pass this object in post and in get to the controller and to read it, I tested also the internationalization that runs correctly.

1. Presentation layer
2. business layer
3. persistence layer
4. database (ETL)

Each layer have a lot of frameworks to simplify the way to make it:

1. Presentation tier (JSF, Primefaces, Richfaces, struts) that I used in affittavetrina
2. Business tier:web services, RMI, EJB etc
3. persistence tier: hibernate, jpa
4. database: oracle, mysql, DB2

But there is an universal framework that allows to make the whole J2EE, the tiers from 1 to 3, this framework is SPRING and offers the following tools:

• Spring Model and view: these libraries allow to make the presentation tier in simple html (without any conversion like jsf) and the browser calls a server using REST services: browser of the computer client sends a request to the server via GET or POST HTTP and the server receives this request, writes the response and sends it to the client.
• Spring Beans is the way for making the communication between client-server: server exposes these beans that client calls to get particular service. These beans contain all business logic that calls external web services or performs particular operations. Spring beans are better than EJB because they are easy to configure and they has three important functionality:
  • Dependency injection: simplify to bind two beans one with the other and separates the creation of a client’s dependencies from its own behaviour: when one or more dependencies (or services) are injected, or passed by reference, into a dependent object (or client) they become part of the client’s state.
  • Aspect programming: allows the developer to call a method of a bean after, before or around that an event is generated.
  • Scheduling: it is possible to schedule the execution of a particular method of the bean for example once a week or once a minute or one time every 6000 ms etc. Like the Quartz framework
• JDBC Template is the tool of Spring for the persistence of the data.

My Spring knowledge

I Used Spring model and view to develop only one small project for one company and I tested this technology to develop Affittavetrina but I preferred JSF because last year, when I developed affittaVetrina I didn’t know Spring well and I didn’t know that JSF causes memory out of bound exception.

Today I have made an example using all the two functionalities of Spring bean: Aspect programming and dependency injection and in a few hour I finished both these my small examples.

Today is the first day that I used aspect programming and I think that it is like the technology  used by WordPress  to call modules.

And also today I tested a simple scheduler of the execution of a Spring method. Every minute the method is executed as I expected.

Spring beans are used by mule technology that allows the integration of different systems for making workflows called mule flows, I use this technology in EON energy.