Skip to main content

Posts

ThreeJs : South Korea's Entry statistics from Asia

     By using ThreeJS (HTML5 supported library) I have been modifying existing code and visualizing the amount of people's entry to South Korea from various countries of Asia. Japan and China is on the top.

Building Term Matrix on HBase and Calculating TF-IDF

     Finding term's related terms in real-time would not be easy if you have 2 millions of tweets and 10 millions of words. Although Hadoop can process huge amount of data in parallel it is not real-time. What if we want to see top  related words for the term "축구" in real-time. Top related words can be found by calculating TF(Term frequency) values for each word a certain set of documents. Then we calculate IDF(Inverted Document Frequency) values by looking at all documents. IDF helps to exclude  meaningless words (이, 다, 나, 너, 니, 하고, 보다  and so on) . Then We calculate TFIDF and sort all of them by the same values. Check out TFIDF on wiki:  http://en.wikipedia.org/wiki/Tf%E2%80%93idf The process is quite complicated, but I will try to explain it as simple as possible. Here are the steps that I went through: 1. Extracting tweetId and the term from Hive into a Text file.    By using Hive I split each tweet into words and separated eac...

Solving java.lang.ClassNotFoundException problem while exporting Jar file from Eclipse

While running my compiled jar files on hadoop cluster I faced this error many times: Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration Usually, I don't use maven and attach libraries manually to the project. The libraries that I use are mostly apache's libraries related to hadoop, hive, hbase and etc.. After successful compilation I export the project as a Jar file. I tried to export in many ways but I faced this ClassNotFoundException every time. So I realized that exporting my project with attached jar libraries is not the optimal solution for the problem. Then I found this solution on StackOverflow http://stackoverflow.com/questions/2096283/including-jars-in-classpath-on-commandline-javac-or-apt It describes the way how to add required (dependency) jar files while running the main jar. It is simple! You just put all jars into the same folder where your main jar file is and Run: >java -cp *:. com.kh.demo.RealtimeTFIDF Note ...

ThreeJs - 3D data visualization on web

     I was into 3D since high school years. My first experience was making 3D models with 3Dsmax. Since few years back Google Chrome had been trying to bring this technology to all desktops through web. With the rise of HTML5 it became possible to render 3D on client side by using WebGL via javascript. However, using WebGL via Javascript was not easy so open-source communities developed various wrapper libraries to simplify using webGL.    Among them ThreeJS ( http://threejs.org/ ) is one the best choice to use IMHO. It is more mature comparing to others, and there are many examples to follow. It is better to see rather than reading, Check out this ThreeJS example:  http://www.georgeandjonathan.com/#6 Is not it amazing? Dynamic, interactive and open-source 3D visualization directly on your Chrome Browser. It runs on the latest Android Tablets and Phones too. In few years, I think everyone will be able to browse 3D sites on their devices. So, by...

Korean NLP on Hive

These days I am building a small platform for doing "Topic Modeling" with Apache Mahout. While working on this project I tried to maximize the speed of NLP processing as fast as possible. Previously, I was running NLP on a single node after retrieving the tweet messages from Hive. Then I found out about UDF(User Defined Functions) which makes it possible to run custom libraries on Hadoop nodes during the mapping process of MapReduce. By using UDF, I  attached open-source Korean NLP library Hannanum and made a library to process NLP and Text processing parallel. Here is how the hive query looks like: >select extractNoun(text) from twitter; I put the source code on GitHub Feel free to use and ask questions. Here is my K-Means dumped cluster output (processing k=100 topics with one-day tweet data) Output.txt

Hive: Creating External Table from existing one

   Apache Hive is an excellent tool to retrieve data from HDFS  using SQL queries. I used Json Serde library to map Json files into Hive table. So initially I created table like this: CREATE EXTERNAL TABLE twitter ( coordinates struct , type:string, created_at string, entities struct , text:string, urls:array , url:string, user_mentions:array , name:string, screen_name:string, favorite_count int, favorited boolean, filter_level string, geo struct , type:string, id_str string, in_reply_to_screen_name string, in_reply_to_status_id_str string, in_reply_to_user_id_str string, lang string, retweet_count int, retweeted boolean, retweeted_status struct , text:string, urls:array , url:string;, user_mentions:array , name:string, screen_name:string, favorite_count:int, favorited:boolean, geo:struct , type:string, id_str:string, in_reply_to_screen_name:string, in_reply_to_status_id_str:string, in_reply_to_user_id_str:string, lang:string, retweet_co...

Running Mahout K-Means example

   CDH4 comes with Mahout library by default. You don't need to install mahout unless if you want to upgrade to the latest version. Mahout is a scalable machine learning library which supports many algorithms for clustering, classification, topic modeling, prediction and recommendation systems. It can have terabytes of input data and process the clustering or classification in less than an hour depending on how powerful is your hadoop cluster. In my case I am using 4 powerful PCs with virtual nodes.    I had a task to model topics from twitter data for 2 months. I tried to run LDA(Latent Dirichlet Allocation - topic modeling algorithm) with R on one PC and it took several hours to build a document matrix of one day tweets(around 800,000) . And for improving results the number of topics K should be bigger than 500 thus exponentially increasing the LDA processing time. It is how I turned into Mahout.    Except Mahout there are other parallel machine learn...