Articles

LXC Unprivileged Containers (Ubuntu Xenial 16.04)

In Uncategorized on 24/12/2016 by pier0w

There are a couple tutorials about how to setup unprivileged containers in Ubuntu, but unfortunately both of them fail to mention all the steps required to actually get them working.

Reboot

The first thing you must do after installing LXC and enabling unprivileged containers is to reboot linux. If you don’t reboot you will see this error.

lxc-start 20160311171257.911 ERROR lxc_cgfs - 
  cgfs.c:lxc_cgroupfs_create:1007 - Permission denied - Could not create 
  cgroup '/user/1000.user/c2.session/lxc' in '/sys/fs/cgroup/systemd'.
lxc-start 20160311171257.911 ERROR lxc_cgfs - cgfs.c:cgroup_rmdir:209 - 
  Permission denied - cgroup_rmdir: failed to delete 
  /sys/fs/cgroup/systemd/user/1000.user/c2.session

Global Read and Execute Permissions for ~/.local/share

You must give you ~/.local and ~/.local/share directories global read and execute permissions.

chmod a+rx ~/.local ~/.local/share

Otherwise you will see the fowling errors.

lxc-start 20161224123344.174 ERROR lxc_start - 
    start.c:print_top_failing_dir:103 - Permission denied - 
    Could not access /home/spark/.local. Please grant it x access, 
    or add an ACL for the container root. 
lxc-start 20161224123344.174 ERROR lxc_sync - 
    sync.c:__sync_wait:57 - An error occurred in another process 
    (expected sequence number 3) 
lxc-start 20161224123344.175 ERROR lxc_start - 
    start.c:__lxc_start:1338 - Failed to spawn container "template".

Script

Here is a script that will setup LXC  unprivileged containers correctly for Ubuntu Xenial 16.04.

#!/bin/bash

# Install LXC
sudo apt-get install lxc

SUB_USER_ID1=`grep $USER /etc/subuid | awk -F':' '{print $2}'`
SUB_USER_ID2=`grep $USER /etc/subuid | awk -F':' '{print $3}'`
SUB_GROUP_ID1=`grep $USER /etc/subgid | awk -F':' '{print $2}'`
SUB_GROUP_ID2=`grep $USER /etc/subgid | awk -F':' '{print $3}'`

# Setup unprivileged container default settings
mkdir -p ~/.config/lxc
echo "lxc.id_map = u 0 ${SUB_USER_ID1} ${SUB_USER_ID2}" > \
    ~/.config/lxc/default.conf
echo "lxc.id_map = g 0 ${SUB_GROUP_ID1} ${SUB_GROUP_ID2}" >> \
    ~/.config/lxc/default.conf
echo "lxc.network.type = veth" >> ~/.config/lxc/default.conf
echo "lxc.network.link = lxcbr0" >> ~/.config/lxc/default.conf
echo "$USER veth lxcbr0 2" | sudo tee -a /etc/lxc/lxc-usernet

# Set the correct permissions for ~/.local/share
chmod a+rx ~/.local ~/.local/share

echo "Now reboot your machine."

Articles

Install Proprietary Java on Ubuntu

In Java, Linux, Ubuntu on 13/01/2013 by pier0w

To install the proprietary (non-open) version of Java on Ubuntu just follow these steps.

  1. Download Java.
  2. Extract it.
  3. #> cd /opt
    #> sudo tar xzf ~/jre-<version>-linux-i586.tar.gz
    #> sudo ln -s jre1.<version>/ jre
  4. Add the new java binary to the Ubuntu Java alternatives.
  5. #> sudo update-alternatives --install "/usr/bin/java" "java" "/opt/jre/bin/java" 1
  6. Lastly switch Ubuntu to use the newly intalled Java.
  7. #> update-alternatives --config java
    There are 2 choices for the alternative java (providing /usr/bin/java).
    
      Selection    Path                                           Priority   Status
    ------------------------------------------------------------
    * 0            /usr/lib/jvm/java-6-openjdk-i386/jre/bin/java   1061      auto mode
      1            /opt/jre/bin/java                               1         manual mode
      2            /usr/lib/jvm/java-6-openjdk-i386/jre/bin/java   1061      manual mode
    
    Press enter to keep the current choice[*], or type selection number: 1
    #> java -version
    java version "1.<version>"
    Java(TM) SE Runtime Environment (build 1.<version>)
    Java HotSpot(TM) Client VM (build <build>, mixed mode)

It is also a good idea to set up the JAVA_HOME global environemnt variable.

#> sudo vim /etc/profile.d/java.sh
export JAVA_HOME=/opt/jre
~
~
~
:wq

To pick up the environment variable either restart or dot in the /etc/profile file.

#> . /etc/profile

Articles

MySQL Cluster Installation

In Cluster, Distribution, MySQL on 08/01/2013 by pier0w

Installing MySQL Cluster is surprisingly easy, almost as easy as installing HDFS. MySQL Cluster consists of three main parts; a MySQL Server, a Management Node, and a Data Node. All of these parts can have multiple instances running within a MySQL Cluster setup.

MySQL Server
To use MySQL Cluster a MySQL server must be present, this is just like any other MySQL server that you’re used to, the only difference being the it uses the NDBCLUSTER sorage engine instead of InnoDB or MyISAM. Data is stored and retrieved from MySQL cluster through the connected MySQL instances.
The NDBCLUSTER storage engine is of course the back bone of MySQL Cluster. The next two parts of the installation manage this storage engine.

Management Node
The Management Node is similar to an HDFS Name Node, it is this that keeps track of where the data resides on the cluster of Data Nodes.

Data Node
The Data Node, again just like in HDFS, stores the actual data that is persisted in MySQL Cluster. It is the instances of this that will typically be increase the most to increase read and write performance.

The following installation instructions are for installing on Ubuntu Server so some of the steps may be platform specific.
 

Preliminary Setup

There are two steps that have to be carried out on the installation server before starting the following MySQL Cluster Installation instructions.

Step 1: Create a new mysqlcl user.

#> sudo useradd -m -G adm,cdrom,sudo,dip,plugdev,lpadmin,sambashare -s /bin/bash mysqlcl
#> sudo passwd mysqlcl
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully

 
Step 2: Install the libaio libraries so that MySQL Server will run.

#> sudo apt-get install libaio1

 
It’s also worth noting that the MySQL Cluster components user a lot of RAM, a single Data Node will not start up on a server with less than 1Gig of RAM. To run this installation example you will need at least 4Gigs of RAM on the installation server.
 

Installation

Installing the MySQL Cluster files is simply a case of downloading then extracting a tar ball.

#> cd /opt/
#> tar xzf mysql-cluster-gpl-<version>-linux2.6-i686.tar.gz
#> sudo ln -s mysql-cluster-gpl-<version>-linux2.6-i686/ mysql-cluster

 
It would also be worth adding the MySQL Cluster bin directory to your global PATH environment variable.

#> sudo vim sudo vim /etc/profile.d/mysql-cluster.sh
export PATH=$PATH:/opt/mysql-cluster/bin/
~
~
~
:wq
#> . /etc/profile

 
MySQL Cluster is now installed.
 

MySQL Cluster Manager Configuration

The first thing that needs to be configured is the MySQL Cluster Manager, it needs to be told the hostname of the server it is being installed on, the directory it is going to store it’s data in, the data replication factor, the directory the Data Nodes will store their data, and lastly the ID’s that should be allocated to the data nodes and any clients that will connect to it. The clients being MySQL and any monitoring applications.
The following configuration does all the above, allocating two Data Node ID’s and two client ID’s.

#> cd
#> mkdir -p manager/conf/
#> mkdir manager/data/
#> vim manager/conf/config.ini
[ndb_mgmd]
hostname=localhost
datadir=/home/mysqlcl/manager/data
NodeId=1

[ndbd default]
noofreplicas=1
datadir=/home/mysqlcl/node/data

[ndbd]
hostname=localhost
NodeId=3

[ndbd]
hostname=localhost
NodeId=4

[mysqld]
hostname=localhost
NodeId=50

[mysqld]
hostname=localhost
NodeId=51
~
~
~
:qw

 
This configuration file has been setup to have all the MySQL Cluster components running on the one server, it’s not a very practucal solution, but is a simple starting point.

To start and inspect the MySQL Cluster Manager run the following commands.

#> ndb_mgmd -f /home/mysqlcl/manager/conf/config.ini --initial --configdir=/home/mysqlcl/manager/conf/
#> ndb_mgm -e show
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 4 node(s)
id=3 (not connected, accepting connect from mysqlmaster)
id=4 (not connected, accepting connect from mysqlmaster)

[ndb_mgmd(MGM)] 1 node(s)
id=1 @127.0.1.1 (mysql-5.5.29 ndb-7.2.10)

[mysqld(API)] 2 node(s)
id=50 (not connected, accepting connect from mysqlmaster)
id=51 (not connected, accepting connect from mysqlmaster)

 
So the MySQL Cluster Manager has been started correctly and is waiting for Data Nodes ([ndbd(NDB)]) and client connections ([mysqld(API)]).

Now stop the MySQL Cluster Manager with the following command in prepuration for setting up the MySQL Server and Data Nodes.

#> ndb_mgm -e shutdown
Connected to Management Server at: localhost:1186
1 NDB Cluster node(s) have shutdown.
Disconnecting to allow management server to shutdown.

 

MySQL Server Configuration

The MySQL Server configuration is extremely simple, you just need to create a my.conf that enables the NDBCLUSTER storage engine, sets the data directory, sets the MySQL Server base directory, and sets the default client connection port.

#> cd
#> mkdir -p mysql/conf/
#> mkdir mysql/data/
#> vim mysql/conf/mysql/conf/my.cnf
[mysqld]
ndbcluster
datadir=/home/mysqlcl/mysql/data/
basedir=/opt/mysql-cluster
port=5000
~
~
~
:wq

 
The MySQL Base directory has been set to the MySQL Cluster install directory, this is because it seems that a special MySQL Cluster specific MySQL Client and Server must be used.
Now that MySQL Server is configured it just needs to be initialised.

#> /opt/mysql-cluster/scripts/mysql_install_db --no-defaults --basedir=/opt/mysql-cluster/ --datadir=/home/mysqlcl/mysql/data/
Installing MySQL system tables...
OK
Filling help tables...
OK

To start mysqld at boot time you have to copy
support-files/mysql.server to the right place for your system
...

 
That’s MySQL Server setup, but unfortunately we can’t start it until the Data Nodes have been started.
 

Data Node Configuration

To configure the Data Nodes all you have to do is create their data directory that was previously set in the MySQL Cluster Manager configuration file (~/node/data).

#> cd
#> mkdir node/data

 
That’s it, the Data Nodes are ready.
 

Start Up

So it’s now time to start up MySQL Cluster, first start the MySQL Cluster Manager and the Data Nodes.

#> ndb_mgmd -f /home/mysqlcl/manager/conf/config.ini --initial --configdir=/home/mysqlcl/manager/conf/
#> ndbd -c localhost:1186 1>/tmp/node1.log 2>&1&
#> ndbd -c localhost:1186 1>/tmp/node2.log 2>&1&

 
The startup of the nodes can take a very long time, you can monitor their progress through there log files (node/data/ndb_*_out.log). When a node has finished starting up you should see the following in the log.

...
2013-01-08 00:22:46 [ndbd] INFO     -- Start phase 100 completed
2013-01-08 00:22:46 [ndbd] INFO     -- Start phase 101 completed
2013-01-08 00:22:46 [ndbd] INFO     -- Node started
2013-01-08 00:22:46 [ndbd] INFO     -- Started arbitrator node 1 [ticket=056d0001000632fd]

 
Also you should not see the word “starting” in any of the Data Node definitions in the ndb_mgm -e show command output. The output should look like this.

#> ndb_mgm -e show
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)]	4 node(s)
id=3	@127.0.0.1  (mysql-5.5.29 ndb-7.2.10, Nodegroup: 0, Master)
id=4	@127.0.0.1  (mysql-5.5.29 ndb-7.2.10, Nodegroup: 1)

[ndb_mgmd(MGM)]	1 node(s)
id=1	@127.0.1.1  (mysql-5.5.29 ndb-7.2.10)

[mysqld(API)]	2 node(s)
id=50 (not connected, accepting connect from mysqlmaster)
id=51 (not connected, accepting connect from mysqlmaster)

 
If any of the Data Node definitions look like the following then the startup has not completed.

id=4	@127.0.0.1  (mysql-5.5.29 ndb-7.2.10, starting, Nodegroup: 0)

 
So now that the manager and nodes are started it is possible to start up MySQL Server.

#> mysqld --defaults-file=/home/mysqlcl/mysql/conf/my.cnf 1>/tmp/mysql.log 2>&1 &

 
If it has started correctly you should see that one of the MySQL Cluster client ID’s has been allocated.

#> ndb_mgm -e show
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)]	4 node(s)
id=3	@127.0.0.1  (mysql-5.5.29 ndb-7.2.10, Nodegroup: 0, Master)
id=4	@127.0.0.1  (mysql-5.5.29 ndb-7.2.10, Nodegroup: 1)

[ndb_mgmd(MGM)]	1 node(s)
id=1	@127.0.1.1  (mysql-5.5.29 ndb-7.2.10)

[mysqld(API)]	2 node(s)
id=50	@127.0.1.1  (mysql-5.5.29 ndb-7.2.10)
id=51 (not connected, accepting connect from mysqlmaster)

 
You can also double check that the NDBCLUSTER engine is enabled.

#> mysql -uroot -e "SHOW ENGINES;"
+--------------------+---------+----------------------------------------------------------------+--------------+------+------------+
| Engine             | Support | Comment                                                        | Transactions | XA   | Savepoints |
+--------------------+---------+----------------------------------------------------------------+--------------+------+------------+
| ndbcluster         | YES     | Clustered, fault-tolerant tables                               | YES          | NO   | NO         |
| CSV                | YES     | CSV storage engine                                             | NO           | NO   | NO         |
| MyISAM             | YES     | MyISAM storage engine                                          | NO           | NO   | NO         |
| ndbinfo            | YES     | MySQL Cluster system information storage engine                | NO           | NO   | NO         |
| BLACKHOLE          | YES     | /dev/null storage engine (anything you write to it disappears) | NO           | NO   | NO         |
| MRG_MYISAM         | YES     | Collection of identical MyISAM tables                          | NO           | NO   | NO         |
| ARCHIVE            | YES     | Archive storage engine                                         | NO           | NO   | NO         |
| InnoDB             | DEFAULT | Supports transactions, row-level locking, and foreign keys     | YES          | YES  | YES        |
| PERFORMANCE_SCHEMA | YES     | Performance Schema                                             | NO           | NO   | NO         |
| FEDERATED          | NO      | Federated MySQL storage engine                                 | NULL         | NULL | NULL       |
| MEMORY             | YES     | Hash based, stored in memory, useful for temporary tables      | NO           | NO   | NO         |
+--------------------+---------+----------------------------------------------------------------+--------------+------+------------+

 
Right it’s all up and running so the very last thing to do is create a data base and then a table that uses the NDBCLUSTER engine.

#> mysql -uroot
mysql#> CREATE DATABASE clusterdb;
mysql#> CREATE TABLE users (id INT NOT NULL AUTO_INCREMENT, 
                            email VARCHAR(32) NOT NULL, 
                            first_name VARCHAR(32) NOT NULL, 
                            last_name VARCHAR(32) NOT NULL, 
                            PRIMARY KEY (id), UNIQUE(email)) 
        engine=ndbcluster;

 
And that’s it.
 

Usage

MySQL Cluster is used through the connected MySQL Server instances so the CRUD operations should be very familiar. It is also possible to monitor the distribution of the data accross the different data nodes.

#> mysql -uroot -e "INSERT INTO users (email, first_name, last_name) VALUES ('some.one@there.com', 'Some', 'One');"
#> ndb_desc -c localhost:1186 users -d clusterdb -p
...
-- Per partition info -- 
Partition	Row count	Commit count	Frag fixed memory	Frag varsized memory	Extent_space	Free extent_space	
1        	1        	1           	32768            	32768               	0            	0 

NDBT_ProgramExit: 0 - OK
                	
#> mysql -uroot -e "INSERT INTO users (email, first_name, last_name) VALUES ('some.two@there.com', 'Some', 'Two');"
#> ndb_desc -c localhost:1186 users -d clusterdb -p
...
-- Per partition info -- 
Partition	Row count	Commit count	Frag fixed memory	Frag varsized memory	Extent_space	Free extent_space	
1        	2        	2           	32768            	32768               	0            	0

NDBT_ProgramExit: 0 - OK

#> mysql -uroot -e "INSERT INTO users (email, first_name, last_name) VALUES ('some.three@there.com', 'Some', 'Three');"
#> ndb_desc -c localhost:1186 users -d clusterdb -p
...
-- Per partition info -- 
Partition	Row count	Commit count	Frag fixed memory	Frag varsized memory	Extent_space	Free extent_space	
0         	1        	1           	32768            	32768               	0            	0                 	
1        	2        	2           	32768            	32768               	0            	0

NDBT_ProgramExit: 0 - OK
#> mysql -uroot -e "SELECT * FROM clusterdb.users ORDER BY id;"
+----+----------------------+------------+-----------+
| id | email                | first_name | last_name |
+----+----------------------+------------+-----------+
|  1 | some.one@there.com   | Some       | One       |
|  2 | some.two@there.com   | Some       | Two       |
|  3 | some.three@there.com | Some       | Three     |
+----+----------------------+------------+-----------+

 
The ndb_desc command connects to the second [mysqld] ID that we allocated at the beginning in the MySQL Cluster Manager configuration.

One nice thing about using MySQL Cluster to destribute your data is that you don’t have to give up secondary uniqueness.

#> mysql -uroot -e "INSERT INTO clusterdb.users (email, first_name, last_name) VALUES ('some.three@there.com', 'Some', 'Three');"
ERROR 1062 (23000) at line 1: Duplicate entry 'some.three@there.com' for key 'email'

 
Now that MySQL Cluster is running and usable we might as well shut it down.

#> mysqladmin -u root -h localhost -P 5000 shutdown
#> ndb_mgm -e shutdown

Articles

Hadoop – HDFS – Extra

In Hadoop, Tutorial on 25/02/2012 by pier0w

This extra hadoop setup is for those of you who are a little bit more adventurous. It requires some very small edits to the hadoop startup scripts, so if you would rather just run a vanilla hadoop install then don’t bother reading the rest of this extra tutorial.

The goal of this tutorial is to have hadoop running in what some would consider a slightly cleaner way. That is it will run without having to rely on ssh.

To do this first run through the previous hadoop hdfs tutorial, but leave out anything to do with ssh and also don’t edit the conf/hadoop-env.sh. Having the JAVA_HOME variable set in the conf/hadoop-env.sh file is only required when hadoop executes remote commands through ssh.

The only part our of hadoop cluster that requires ssh by default is the Namenode, so we will be editing the files on this server only.

Now since we have not set the JAVA_HOME variable in the conf/hadoop-env.sh file it will need to be set as a global environment variable for the server. On Ubuntu server this can be done by exporting the environment variables in a Java specific scrip in the /etc/profile.d/ directory.


#> sudo vim /etc/profile.d/java.sh
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk
export JDK_HOME=/usr/lib/jvm/java-6-openjdk
~
~

Once that is setup the first hadoop file to edit is bin/hadoop-daemon.sh, this script uses rsync to make sure that the hadoop files are on the server it is about to start. The reason it does this is so that you can have a central server that can start up multiple Namenodes on remote servers that don’t actually have to have hadoop installed on them.

We are going to edit line 119 in this file and remove the text -e ssh which forces rsync to run through ssh.


#> vim /opt/hadoop/bin/hadoop-daemon.sh
...
    if [ "$HADOOP_MASTER" != "" ]; then
      echo rsync from $HADOOP_MASTER
      rsync -a --delete --exclude=.svn --exclude='logs/*' --exclude='contrib/hod/logs/*' $HADOOP_MASTER/ "$HADOOP_HOME"
    fi
...

Next we are going to edit the bin/start-dfs.sh and bin/stop-dfs.sh scripts and make it so that the Secondary Namenode is started up in the same way as the primary Namenode.


#> vim /opt/hadoop/bin/start-dfs.sh
...
# start dfs daemons
# start namenode after datanodes, to minimize time namenode is up w/o data
# note: datanodes will log connection errors until namenode starts
"$bin"/hadoop-daemon.sh --config $HADOOP_CONF_DIR start namenode $nameStartOpt
"$bin"/hadoop-daemons.sh --config $HADOOP_CONF_DIR start datanode $dataStartOpt
"$bin"/hadoop-daemon.sh --config $HADOOP_CONF_DIR start secondarynamenode
~
~


#> vim /opt/hadoop/bin/stop-dfs.sh
...
. "$bin"/hadoop-config.sh

"$bin"/hadoop-daemon.sh --config $HADOOP_CONF_DIR stop namenode
"$bin"/hadoop-daemons.sh --config $HADOOP_CONF_DIR stop datanode
"$bin"/hadoop-daemon.sh --config $HADOOP_CONF_DIR stop secondarynamenode
~
~

Lastly we will remove any domain names from the conf/slaves file, even localhost. This stops hadoop from trying to start a Datanode up on the Namenode server.


#> vim /opt/hadoop/conf/slaves
~
~

And that’s it, you should now be able to start a Namenode on the server without having to have any special ssh setup.

Articles

Hadoop – HDFS

In Hadoop, Tutorial on 23/02/2012 by pier0w

So you want to have a go at playing with hadoop, but are a little apprehensive because it just looks so complicated and hard to setup. Well DON’T BE! It’s super easy and I’ll show this by first demonstrating below the simple setup for a fully working HDFS cluster.

To get started lets first explain the HDFS structure. It’s a very simple master slave architecture. There is a master, the Namenode, that manages all the slaves, the Datanodes. See the super duper diagram bellow:


            Namenode
       ________|_______
      |                |
Datanode One     Datanode Two

The Namenode manages the distribution of the data within HDFS and the Databodes just store the data. That is HDFS put very very simply. As you can see it doesn’t provide the best redundancy because of the single point of failure. Though, there is a Secondary Namenode that can be used in a time of emergency, but the way to do that is WAY out of the scope of this tutorial. HDFS may not have the most resilient architecture, but it is at least easy to understand.

So how does one set one’s self up the HDFS? Well very easily actually…

Step One: Create a Namenode
To do this get your self a server, an Ubuntu Server VM will do nicely. Make sure the VM is running within a host only network so that it can talk to your host machine and the Datanode VM’s.

When using unix it is always best practice to create a user for the task at hand so lets create a hadoop user and use that from now on. Super users are dangeresque.


#> sudo useradd -m -G users -s /bin/bash hadoop
#> sudo passwd hadoop
# Set the password to what every you like.
#> su - hadoop

Now make sure that the server is running an ssh server. A good way to do this is to login to local host.


#> ssh localhost

If that doesn’t succeed then install the ssh server.


#> sudo apt-get install openssh-server

Now hadoop uses ssh to run it’s commands even when they are on the local machine, because of this you have to make sure that you can automatically ssh into localhost without the need for a password. To do this create an ssh key and add it to you .ssh/authorized_keys2 file.


#> ssh-keygen
# Just press ENTER for all the input request.
#> cat .ssh/id_rsa.pub >> .ssh/authorized_keys2

That’s it, you should now be able to ssh to localhost without having to type in your password.

The next thing to do is make sure your server has a static ip address. So set the IP of the host only network interface to be a static IP within the host only network.


#> sudo vim /etc/network/interfaces
...
# Host Only Network
auto eth0
iface eth0 inet static
        address 192.168.69.201
        netmask 255.255.255.0
        network 192.168.69.0
        broadcast 192.168.69.255
~
~

#> sudo /etc/init.d/networking restart
#> ifconfig
eth0      Link encap:Ethernet HWaddr 00:00:00:00:00:00
          inet addr:192.168.69.201 Bcast:192.168.69.255 Mask:255.255.255.0
...

Lastly hadoop is written in Java so this will need to be installed on the server. It seems that it is only possible now to install the OpenJDK on Ubuntu but this is ok because hadoop runs fine on that JVM.


#> sudo apt-get install openjdk-6-jdk

Now that the server is all ready extract the hadoop tarball somewhere onto the filesystem. /opt/ is as good a place as any. Then do all the configuration that is required for a working Namenode. That is, edit two files.

First edit the conf/core-site.xml file and set the fs.default.name property to be the ip address that you gave the server with a port of your choosing. 9000 seems to be a good default. You’d think that you’d be able to just set the fs.default.name property to be localhost, but unfortunately this does not work. I’m guessing it is because the server will only accept requests for the provided URI and nothing but the local machine can carry out a request to localhost.


#> vim /opt/hadoop/conf/core-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
        <property>
                <name>fs.default.name</name>
                <value>hdfs://192.168.69.201:9000</value>
        </property>
</configuration>
~
~

Then lastly edit the conf/hadoop-env.sh file and uncomment and set the JAVA_HOME variable to the directory that contains the JDK files. On Ubuntu Server this is the /usr/lib/jvm/java-6-openjdk/ directory.


#> vim /opt/hadoop/conf/hadoop-env.sh
...
# The only required environment variable is JAVA_HOME. All others are
# optional. When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on
# remote nodes.

# The java implementation to use. Required.
export JAVA_HOME=/usr/lib/jvm/java-6-openjdk

# Extra Java CLASSPATH elements. Optional.
# export HADOOP_CLASSPATH=
...

And that’s it, you can now start a perfectly running Namenode.


#> /opt/hadoop/bin/hadoop namenode -format # The Namenode must be formatted before it can start up.
#> /opt/hadoop/bin/start-dfs.sh

To check that it is running, browse to the Namenodes ip address on port 50070 e.g. http://192.168.69.201:50070
This will take you to the Namenodes monitoring page.

Step Two: Create a Datanode

The preliminary setup of a Datanode VM is very simple, you just have create the hadoop user and switch into it. Leave the Host only network interface as DHCP, the Datanodes do not require static ip addresses, also below you will see why it is actually really good if their ip addresses are allocated dynamically.

Now extract the same hadoop tarball that you used on the name node into the /opt/ directory on the Datanode. Now all you have to do is configure the Datanodes conf/core-site.xml file with the exact same configuration as the Namenode. By exact I mean character for character, to the point where if you wanted to you could just copy the Namenodes conf/core-site.xml file over the top of the newly extracted one on the Datanode. This now means that the Datanode is configured to be aware of the Namenode.

So I know what you’re now thinking “Hold on a minute there. How does the Datanode know to act as a slave if it has the exact same configuration as the Namenode?”. Well it does because we start it in a completely different way to the Namenode. Where the name node is started with the bin/start-dfs.sh script we are going to start the Datanode with the bin/hadoop-daemon.sh. As follows:


#> /opt/hadoop/bin/hadoop-daemon.sh --config /opt/hadoop/conf start datanode

That’s it your new Datanode should now be up and running. You can confirm this by refreshing the Namenodes monitoring page and you should now see that the Live Nodes number has increased from 0 to 1.

Note: The Datanode could take a little while to register to the name node, you can check on it’s progress in the logs/hadoop-hadoop-datanode-*.log file.

So as you can see all and all setting up hadoop is a very easy process. You can now add extra Datanodes to you hearts content by simply cloning and your first Datanode and starting it up. Since you have left the Host only network interface as DHCP each Datanode will automatically have it’s own unique ip address.

You may also notice if you have read the hadoop documentation that I don’t mention anything about the conf/slaves file. This is because I think that a hadoop cluster should be configured in an extendible way from the beginning so I have shown you how to start a Datanode so that it registers itself with an existing hadoop cluster. Instead of listing the Datanodes within the Namenodes conf/slaves file and having the Namenode start the Datanodes for you.

Articles

Gentoo XBMC Post-Setup

In Uncategorized on 18/04/2011 by pier0w

The very last thing to do is get XBMC to start up automatically on boot.

To do this we will make it so that the xbmc user starts XBMC automatically when ever they login locally to the laptop.

So first get the xbmc user to start X11 on a local login by adding the following to the end of the users .bashrc file.

...
# Put your fun stuff here.
LOGIN_TYPE=`tty | awk '{if(match($0, /pts/)){print "SSH"}else{print "LOCAL"}}'`

if [ $LOGIN_TYPE == "LOCAL" ]
then
        exec startx
fi

unset LOGIN_TYPE

This little bit of script will only start X11 if the user logs in locally.

Next create a file called .xinitrc in the root of the user home directory, anything within this file will be executed by X11 right after it starts up. So this is where we will place the start up of XBMC.

.xinitrc:
#!/bin/bash

exec /usr/bin/xbmc

Now this should work fine if you chose to configure X11 to either ignore or turn off the laptop monitor, but if you did neither of those then you tell XBMC to start up twice. This is because on the first start up it will get the resolution conpletely wrong, but once you close it down then start it again the resolution will be right.

.xinitrc:
#!/bin/bash

xbmc
xbmc

Now get Gentoo to login to the xbmc use on boot, this can be done within the /etc/conf.d/local.start file which is a file that will always be executed at the end of Gentoo’s boot process.

/etc/conf.d/local.start:
# /etc/conf.d/local.start

# This is a good place to load any misc programs
# on startup (use &>/dev/null to hide output)

su - xbmc

So with that all done XBMC is all setup and ready to use.

Link for getting shutdown to work.

Articles

Gentoo XBMC Installation

In Gentoo, XBMC on 18/04/2011 by pier0w

With the Gentoo Installation and Gentoo XBMC Pre-Setup complete it should just be able to be emerged with our any hassle.

#> emerge -av xbmc

When that completes XBMC should be able to be started by simply running the command xbmc from an xterm within an X environment.

#> startx

Now from inside one of the xterms.
#> xbmc

XBMC should now be running.

To get XBMC to display on the TV we need to do create a couple X11 configuration files within the /usr/share/X11/xorg.conf.d/.

The first configuration file will tell X11 to always use the intel driver and also provide labels for the TV and laptop monitor.

/usr/share/X11/xorg.conf.d/11-intel.conf:
Section "Device"
    Identifier "Intel Graphics Card"
    Driver "intel"
    Option "Monitor-LVDS1" "LCD screen"
    Option "Monitor-VGA1" "VGA screen"
EndSection
Section "dri"
    Mode 0666
EndSection

The files name starts with 11 so that it will be executed after the original config file which starts with 10. Any additional config files can have their execution order set in this way.

The "Device" section is where the driver and monitor labels are set, the first name in the Option lines is the actual name that X11 gives the physical devices. X11 gives all video outputs a name that starts with “Monitor-” it then appends that device name on to that, to find the device name of a monitor you can use the xrandr -q command.

Screen 0: minimum 320 x 200, current 3200 x 1080, maximum 4096 x 4096
LVDS1 connected 1280x800+1920+0 (normal left inverted right x axis y axis) 331mm x 207mm
   1280x800       61.0*+
   1024x768       60.0  
   800x600        60.3     56.2  
   640x480        59.9  
VGA1 connected 1920x1080+0+0 (normal left inverted right x axis y axis) 886mm x 498mm
   1920x1080      60.0 +
   1600x1200      60.0  
   1680x1050      60.0  
   1280x1024      75.0     60.0  
   1440x900       75.0     59.9  
   1280x960       60.0  
   1360x768       60.0  
   1280x800       59.8* 
   1152x864       75.0  
   1024x768       75.1     70.1     60.0  
   832x624        74.6  
   800x600        72.2     75.0     60.3  
   640x480        72.8     75.0     66.7     60.0  
   720x400        70.1  
TV1 unknown connection (normal left inverted right x axis y axis)
   848x480        30.0 +
   640x480        30.0 +
   1024x768       30.0  
   800x600        30.0

From the output of this you can see that the laptop screen is called LVD1 and the vga plug is called VGA1 so the X11 names will be Monitor-LVD1 and Monitor-VGA1 respectively.

The "dri" turns on the hardware acceleration.

Now that the monitors have been labelled we need to tell X11 what configuration they are in, this I have done in another config file.

/usr/share/X11/xorg.conf.d/12-dual-screen.conf:
Section "Monitor"
    Identifier "LCD screen"
EndSection

Section "Monitor"
    Identifier "VGA screen"
    Option "LeftOf" "LCD screen"
    Option "PreferredMode" "1920x1080"
EndSection

This tells X11 that there are two monitors associated to the two output devices, it also tells X11 that the "VGA screen" (the TV) should be set to the resolution 1920×1080 and that it is to the left of the "LCD screen".

Now this should work fine with XBMC… and it sort of does. If you start up the X11 server then start up XBMC inside of it you will see that the resolution will be set to the 1280×800, the laptops resolution, but if you then close XBMC but NOT the X11 server then start XBMC again it will be at the right resolution. I have no idea why this is but there you go. Because of this I have created a hack in the next section.

But, if you want to have XBMC start in the correct resolution in a more correct way there are two things you can do.

The first is tell X11 to turn off the laptop monitor, this is good because it saves power and won’t burn out the monitor, but it means when X11 shuts down the monitor won’t turn back on so you are stuck with having to use the TV as the only monitor. In reality this isn’t that bad because you’ll surely be using ssh to do anything with the command line on the laptop any way. So to turn of the laptop monitor use this configuration.

/usr/share/X11/xorg.conf.d/12-dual-screen.conf:
Section "Monitor"
    Identifier "LCD screen"
    Option "Disable" "True"
EndSection

Section "Monitor"
    Identifier "VGA screen"
    Option "PreferredMode" "1920x1080"
EndSection

Though, if for some reason you want the laptop monitor to stay on but still want XBMC to start correctly you can set X11 to ignore the laptop monitor. This is good because it stays on, but bad because it ALWAYS stays on even with the lid close so may burn out the monitor.

/usr/share/X11/xorg.conf.d/12-dual-screen.conf:
Section "Monitor"
    Identifier "LCD screen"
    Option "Ignore" "True"
EndSection

Section "Monitor"
    Identifier "VGA screen"
    Option "PreferredMode" "1920x1080"
EndSection

So now that X11 is configured XBMC should start up and run perfectly on your TV.