Using RHive to connect from RStudio to Hive Server on Hadoop Cluster

Scenario:
1. We have cluster running hadoop store for example web logs.
2. We have Hive run on top of hadoop cluster for query data from our web logs.
3. You are a data analyst / scientist (like me :D) using RStudio and you might want to use hive from your RStudio: big benefit of this is that you can store result of query into a dataframe in R language for later process(if you issue query from hive cli you might not store the result).
How to do that?
1. Install RHive package here, you also can find docs on that website.
Just one thing to remind: RHive require rJava package and to install rJava package you need to install jdk from oracle.
2. I assume that you already setup successfully cluster running hadoop and hive.
3. From master of your cluster start hive server using:
hive/bin/> ./hive --service hiveserver2
This command will start Hive Server listen on port 10000 by default.
4. From your machine after successfully install RHive type this:
library("RHive")
Sys.setenv(HADOOP_HOME='/home/tin/tools/hadoop')
Sys.setenv(HIVE_HOME='/home/tin/tools/hive')
Sys.setenv(HADOOP_CONF_DIR='/home/tin/tools/hadoop/conf')
rhive.init()
rhive.env()
rhive.connect(host="ip_of_your_master",user="user_running_hadoop_on_master")

Attention:

On your system you need to download hadoop and hive as you can see above I put hadoop and hive in tools directory at home.

You might need to config hadoop and hive point to your jdk
You don't need to run hadoop or hive on your system.
This is used for RStudio can find some libraries to run.

That's it you now can issue query from RStudio using apache hive.

In case you have any problem about this procedure just email me: imthientin(at)gmail.com I'm willing to help.

Speed layer : Real time views in Lambda architecture

Very first overview about lambda architecture specify its speed layer
http://www.slideshare.net/NitNeiht/speed-layer-real-time-views-in-lamdba-architecture

Hive authorization

scenario: in cluster run hadoop mapreduce and hive. one system run hadoop (ref here) and client run hive (ref here) to run jobs on hadoop.
you have many users access to run hadoop and also have many database tables blah blah.
How to control authorization of each hive' user.?
ok we start.

(source : http://venublog.com/2013/07/16/hadoop-summit-2013-hive-authorization)

From picture above we can see two places we can apply security strategy : RDBMS store metadata and HDFS store real Data of table. In this section I will apply security at RDBMS.
In our mysql databases we already created metastore database (ref to previous tut ) now from log in to mysql using root privileges create user hive with select only to metastore database:

mysql>CREATE USER 'hivetest'@'localhost' IDENTIFIED BY 'mypass';

mysql>use metastore;

mysql>grant select on * to user hivetest;

now we use 'hivetest' and 'mypass' in hive-site.xml at client: 

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
 <property>
      <name>javax.jdo.option.ConnectionURL</name>
      <value>jdbc:mysql://localhost:3306/metastore</value>
      <description>the URL of the MySQL database</description>
 </property>

 <property>
      <name>hive.metastore.warehouse.dir</name>
      <value>/user/hive/warehouse</value>
 </property>


 <property>
      <name>javax.jdo.option.ConnectionDriverName</name>
      <value>com.mysql.jdbc.Driver</value>
 </property>

 <property>
      <name>javax.jdo.option.ConnectionUserName</name>
      <value>hivetest</value>
 </property>

 <property>
      <name>javax.jdo.option.ConnectionPassword</name>
      <value>mypass</value>
 </property> 
</configuration>

log in to hive as previous tut create table :

hive> create table tbl_1(a int);

switch to client: 

hive> select * from tbl_1;// can't select error comes.

back to super user:

hive> grant select on table tbl_1 to user hivetest;

switch to client: now you can select tbl_1 as normal. and this user can only do something granted from super user.

how to setup: 

from your system create new user beside your current user(super user ):hivetest (no need to same user name with mysql user above.) we can log in to these users by ctrl + alt + F1 ; ctrl+alt+F2 for each user. 

This is called hive authorization at metastore level. :)

DOne.

see ya.