Thursday, November 27, 2014

Tashkent House Price Analysis


Recently  I've got an idea to analyze prices of houses in Tashkent city.
"www.zor.uz" is one of the most famous sites where people sell houses, cars, electronics and etc.
I had to crawl it and collect data from it.
Unfortunately, "www.zor.uz" does not have the old data, the latest data that  I found there was just one month old. They seem to clean their DB every month.

I needed data for previous years. So, I used "Way Back Machine" that has captured points of all website around the world. It is really a cool stuff to try : https://archive.org/index.php
But they use "https" so I had download their certificate and register it to my JRE.

I crawled that site and succeeded to get data starting from 2009.08 (that was the earliest capture point of www.zor.uz)
Interestingly "WayBackMachine" captures websites periodically, depending how often website changes. They have their own logic to capture sites for optimizing their storage.

Anyways, The data I crawled was from specific months of the year.
After crawling the data, for simplicity I just exported data to Excel sheet.
Then filtered data (removed duplicates, meaningless entities, spams, and zero values) and sorted by date.

Here is graph that displays price change for houses with 2 rooms:


You can see that average prices of 2-room houses during 2009.07 ~2014.08  raised from 30,000$ to 42,000$

By drawing the moving average we can identify when the prices was raising and falling:


Interestingly, houses prices raise during summer and it drops during the winter.
There can be more results can retrieved, but at the moment I am too busy with other projects, and will upload more stats later on.



Tuesday, November 18, 2014

Why it is better not to use VirtualBox(VMWare etc..)

Cons

1. VirtualBox can not use full 100% resources of PC. Your hadoop will not be as fast as you expect.

2. You have to start virtual nodes whenever your reboot your PC (unless if you made some auto-start scripts)

3. Several virtual nodes in the same PC can cause mismanagement of resources. If RAM usage of VirtualBox nodes exceeds the maximum then your PC just crashes.


Though, VirtualBox is a good starting point to build experimental hadoop environment.

Pros

1. You can create several nodes in the same PC (if you lack PCs)
2. VirtualBox has Export /Import features that helps you to do "Do once - Copy many" 
3. Do experiments by changing hardware features (add CPUs, reduce RAM etc..)

For learning hadoop VirtualBox can be helpful to build hadoop environment.
However, for production purpose it is better to avoid Virtualbox and use PC itself as a single node.