Namenode editlog (dfs.name.dir) multiple storage directories

Recently there was a question posted on the Apache Hadoop User mailing list regarding the behavior of specifying multiple storage directories fir property "dfs.name.dir". The documentation on the official site is not clear on what happens if one of the specified storage directories become inaccessible. While the documentation for other properties such as "dfs.data.dir" and "mapred.local.dir" clearly states that the system will ignore any directory that is inaccessible in case some of the specified directories are unavailable. See link below for reference

http://hadoop.apache.org/common/docs/current/hdfs-default.html

There were many comments on the post. See below the link for the complete post

http://lucene.472066.n3.nabble.com/what-will-happen-if-a-backup-name-node-folder-becomes-unaccessible-td1253293.html#a1253293

In this post I am basically going to summarize my tests to prove that it works in the cloudera distribution. So the behavior is that it ignores any directories that are inaccessible and the namenode only bails out when it can't access any of the specified directories. The below series of tests are pretty much self explanatory

hadoop@training-vm:~$ hadoop version
Hadoop 0.20.1+152
Subversion  -r c15291d10caa19c2355f437936c7678d537adf94
Compiled by root on Mon Nov  2 05:15:37 UTC 2009

hadoop@training-vm:~$ jps
8923 Jps
8548 JobTracker
8467 SecondaryNameNode
8250 NameNode
8357 DataNode
8642 TaskTracker

hadoop@training-vm:~$ /usr/lib/hadoop/bin/stop-all.sh
stopping jobtracker
localhost: stopping tasktracker
stopping namenode
localhost: stopping datanode
localhost: stopping secondarynamenode

hadoop@training-vm:~$ mkdir edit_log_dir1

hadoop@training-vm:~$ mkdir edit_log_dir2

hadoop@training-vm:~$ ls
edit_log_dir1  edit_log_dir2

hadoop@training-vm:~$ ls -ltr /var/lib/hadoop-0.20/cache/hadoop/dfs/name
total 8
drwxr-xr-x 2 hadoop hadoop 4096 2009-10-15 16:17 image
drwxr-xr-x 2 hadoop hadoop 4096 2010-08-24 15:56 current

hadoop@training-vm:~$ cp -r /var/lib/hadoop-0.20/cache/hadoop/dfs/name edit_log_dir1

hadoop@training-vm:~$ cp -r /var/lib/hadoop-0.20/cache/hadoop/dfs/name edit_log_dir2

------ hdfs-site.xml added new dirs

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
  <property>
     <name>dfs.permissions</name>
     <value>false</value>
  </property>
  <property>
     <!-- specify this so that running 'hadoop namenode -format' formats the right dir -->
     <name>dfs.name.dir</name>    
    <value>/var/lib/hadoop-  0.20/cache/hadoop/dfs/name,/home/hadoop/edit_log_dir1,
/home/hadoop/edit_log_dir2</value>
  </property>
   <property>
     <name>fs.checkpoint.period</name>
     <value>600</value>
  </property>
  <property>
    <name>dfs.namenode.plugins</name>
    <value>org.apache.hadoop.thriftfs.NamenodePlugin</value>
  </property>
  <property>
    <name>dfs.datanode.plugins</name>
    <value>org.apache.hadoop.thriftfs.DatanodePlugin</value>
  </property>
  <property>
    <name>dfs.thrift.address</name>
    <value>0.0.0.0:9090</value>
  </property>
</configuration>

---- start all daemons

hadoop@training-vm:~$ /usr/lib/hadoop/bin/start-all.sh
starting namenode, logging to 
/usr/lib/hadoop/bin/../logs/hadoop-hadoop-namenode-training-vm.out
localhost: starting datanode, logging to 
/usr/lib/hadoop/bin/../logs/hadoop-hadoop-datanode-training-vm.out
localhost: starting secondarynamenode, logging to 
/usr/lib/hadoop/bin/../logs/hadoop-hadoop-secondarynamenode-training-vm.out
starting jobtracker, logging to 
/usr/lib/hadoop/bin/../logs/hadoop-hadoop-jobtracker-training-vm.out
localhost: starting tasktracker, logging to 
/usr/lib/hadoop/bin/../logs/hadoop-hadoop-tasktracker-training-vm.out


-------- namenode log confirms all dirs taken

2010-08-24 16:20:48,718 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = training-vm/127.0.0.1
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 0.20.1+152
STARTUP_MSG:   build =  -r c15291d10caa19c2355f437936c7678d537adf94; 
compiled by 'root' on Mon Nov  2 05:15:37 UTC 2009
************************************************************/
2010-08-24 16:20:48,815 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: 
Initializing RPC Metrics with hostName=NameNode, port=8022
2010-08-24 16:20:48,819 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: 
Namenode up at: localhost/127.0.0.1:8022
2010-08-24 16:20:48,821 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
 Initializing JVM Metrics with processName=NameNode, sessionId=null
2010-08-24 16:20:48,822 INFO 
org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics: 
Initializing NameNodeMeterics using context object:org.apache.hadoop.metrics.spi.NoEmitMetricsContext
2010-08-24 16:20:48,894 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: 
fsOwner=hadoop,hadoop
2010-08-24 16:20:48,894 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: 
supergroup=supergroup
2010-08-24 16:20:48,894 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
 isPermissionEnabled=false
2010-08-24 16:20:48,903 INFO org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics: 
Initializing
 FSNamesystemMetrics using context object:org.apache.hadoop.metrics.spi.NoEmitMetricsContext
2010-08-24 16:20:48,905 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered 
FSNamesystemStatusMBean
2010-08-24 16:20:48,937 INFO org.apache.hadoop.hdfs.server.common.Storage: Storage directory
 /home/hadoop/edit_log_dir1 is not formatted.
2010-08-24 16:20:48,937 INFO org.apache.hadoop.hdfs.server.common.Storage: Formatting ...
2010-08-24 16:20:48,937 INFO org.apache.hadoop.hdfs.server.common.Storage: Storage directory
 /home/hadoop/edit_log_dir2 is not formatted.
2010-08-24 16:20:48,937 INFO org.apache.hadoop.hdfs.server.common.Storage: Formatting ...
2010-08-24 16:20:48,938 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files = 41
2010-08-24 16:20:48,947 INFO org.apache.hadoop.hdfs.server.common.Storage: 
Number of files under construction = 0
2010-08-24 16:20:48,947 INFO org.apache.hadoop.hdfs.server.common.Storage: 
Image file of size 4357 loaded in 0 seconds.

---- directories confirm in use

hadoop@training-vm:~$ ls -ltr edit_log_dir1
total 12
drwxr-xr-x 4 hadoop hadoop 4096 2010-08-24 16:01 name
-rw-r--r-- 1 hadoop hadoop    0 2010-08-24 16:20 in_use.lock
drwxr-xr-x 2 hadoop hadoop 4096 2010-08-24 16:20 image
drwxr-xr-x 2 hadoop hadoop 4096 2010-08-24 16:20 current

hadoop@training-vm:~$ ls -ltr edit_log_dir2
total 12
drwxr-xr-x 4 hadoop hadoop 4096 2010-08-24 16:01 name
-rw-r--r-- 1 hadoop hadoop    0 2010-08-24 16:20 in_use.lock
drwxr-xr-x 2 hadoop hadoop 4096 2010-08-24 16:20 image
drwxr-xr-x 2 hadoop hadoop 4096 2010-08-24 16:20 current

----- secondary name node checkpoint worked fine

...
2010-08-24 16:27:10,756 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Posted URL 
localhost:50070putimage=1&port=50090&machine=127.0.0.1&token=-
18:1431678956:1255648991179:1282692430000:1282692049090
2010-08-24 16:27:11,008 WARN org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode:

Checkpoint done. New Image Size: 4461
....

--- dirctory put works fine

hadoop@training-vm:~$ hadoop fs -ls /user/training
Found 3 items
drwxr-xr-x   - training supergroup          0 2010-06-30 13:18 /user/training/grep_output
drwxr-xr-x   - training supergroup          0 2010-06-30 13:14 /user/training/input
drwxr-xr-x   - training supergroup          0 2010-06-30 15:30 /user/training/output

hadoop@training-vm:~$ hadoop fs -put /etc/hadoop/conf.with-desktop/hdfs-site.xml /user/training

hadoop@training-vm:~$ hadoop fs -ls /user/training
Found 4 items
drwxr-xr-x   - training supergroup          0 2010-06-30 13:18 /user/training/grep_output
-rw-r--r--   1 hadoop   supergroup        987 2010-08-24 16:25 /user/training/hdfs-site.xml
drwxr-xr-x   - training supergroup          0 2010-06-30 13:14 /user/training/input
drwxr-xr-x   - training supergroup          0 2010-06-30 15:30 /user/training/output


------ delete one of the directories
hadoop@training-vm:~$ rm -rf edit_log_dir2

hadoop@training-vm:~$ ls -ltr
total 4
drwxr-xr-x 5 hadoop hadoop 4096 2010-08-24 16:20 edit_log_dir1

-- namenode logs

No errors/warns in logs

-------- namenode still running

hadoop@training-vm:~$ jps
12426 NameNode
12647 SecondaryNameNode
12730 JobTracker
14090 Jps
12535 DataNode
12826 TaskTracker

----  puts and ls work fine

hadoop@training-vm:~$ hadoop fs -ls /user/training
Found 4 items
drwxr-xr-x   - training supergroup          0 2010-06-30 13:18 /user/training/grep_output
-rw-r--r--   1 hadoop   supergroup        987 2010-08-24 16:25 /user/training/hdfs-site.xml
drwxr-xr-x   - training supergroup          0 2010-06-30 13:14 /user/training/input
drwxr-xr-x   - training supergroup          0 2010-06-30 15:30 /user/training/output

hadoop@training-vm:~$ hadoop fs -put /etc/hadoop/conf.with-desktop/core-site.xml /user/training

hadoop@training-vm:~$ hadoop fs -put /etc/hadoop/conf.with-desktop/mapred-site.xml /user/training

hadoop@training-vm:~$ hadoop fs -ls /user/training
Found 6 items
-rw-r--r--   1 hadoop   supergroup        338 2010-08-24 16:28 /user/training/core-site.xml
drwxr-xr-x   - training supergroup          0 2010-06-30 13:18 /user/training/grep_output
-rw-r--r--   1 hadoop   supergroup        987 2010-08-24 16:25 /user/training/hdfs-site.xml
drwxr-xr-x   - training supergroup          0 2010-06-30 13:14 /user/training/input
-rw-r--r--   1 hadoop   supergroup        454 2010-08-24 16:29 /user/training/mapred-site.xml
drwxr-xr-x   - training supergroup          0 2010-06-30 15:30 /user/training/output

------- secondary namenode checkpoint is successdul

2010-08-24 16:37:11,455 WARN org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: 
Checkpoint done. New Image Size: 4671
....
2010-08-24 16:47:11,884 WARN org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode:
 Checkpoint done. New  Image Size: 4671
...
2010-08-24 16:57:12,264 WARN org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: 
Checkpoint done. New Image Size: 4671

------- after 30 mins

hadoop@training-vm:~$ jps
12426 NameNode
12647 SecondaryNameNode
12730 JobTracker
16256 Jps
12535 DataNode
12826 TaskTracker

Automated backups to Amazon S3

Just got around setting an automated backup to Amazon S3. Until now my backup was limited to Personal Computer -> Home Server -> External USB Drive all under one roof. Over the years I have gathered a lot of document clutter and have been lately scanning them and destroying the originals and that means that if the digital copy is gone then I am kaput. Having a copy on the S3 is good both backup wise and access from anywhere wise. Although I have setup my home server to be accessible from outside its still slow when downloading large files. As of now i have limited S3 to important stuff (just documents), I would eventually like to offload all my digital content o S3 (songs, pictures, videos etc) which would mean shelling more moola to Amazon. I wouldn't be accessing these content frequently so the cost is pretty much the storage cost and off-course the weekly rsync seeks. I also choose the 99.99% (0.10c/gb monthly) reliability over the 99.9999999% (0.15c/gb monthly) reliability as I am conformable with that given that I have a copy on my end too. S3 also has a special offer till Nov 1 wherein they are waving all "Data transfer In" fees. Below are some links that I used to setup :

AWS S3 pricing : http://aws.amazon.com/s3/pricing/
S3Sync and S3cmd tool : http://www.s3sync.net/wiki

Installing S3sync script: http://www.linode.com/wiki/index.php/Backups_with_s3sync

Setting up automated backup using S3sync on S3
http://blog.eberly.org/2006/10/09/how-automate-your-backup-to-amazon-s3-using-s3sync/

General article on home server vs S3 backup : http://jeremy.zawodny.com/blog/archives/007624.html
S3 based backup tools: http://jeremy.zawodny.com/blog/archives/007641.html

Hadoop LZO Installation : Errors and Resolution

A record of our errors and resolution with Hadoop LZO Installation. We previously followed the code base and instructions on

http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ

but later switched to

http://github.com/kevinweil/hadoop-lzo

as there were some improvements and bug fixes.

Error:

java.lang.RuntimeException: native-lzo library not available
 at com.hadoop.compression.lzo.LzopCodec.createDecompressor(LzopCodec.java:91)
 at com.hadoop.mapreduce.LzoSplitRecordReader.initialize(LzoSplitRecordReader.java:52)
 at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:418)
 at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:582)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
 at org.apache.hadoop.mapred.Child.main(Child.java:170)

Resolution: Need to ensure that the lzo lib is in the classpath

$~: ps auxw | grep tasktracker

should show the lzo lib in the classpath list. If not then follow the "Building and configuring" instructions on Kevin's site (link above)

Error:

java.lang.ClassCastException: com.hadoop.compression.lzo.LzopCodec$LzopDecompressor
 cannot be cast to com.hadoop.compression.lzo.LzopDecompressor
 at com.hadoop.mapreduce.LzoSplitRecordReader.initialize(LzoSplitRecordReader.java:52)
 at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:418)
 at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:582)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
 at org.apache.hadoop.mapred.Child.main(Child.java:170)
 
java.lang.IllegalAccessError com.hadoop.compression.lzo.LzopDecompressor cannot access
 superclass com.hadoop.compression.lzo.LzoDecompressor

Resolution:

The above two errors were mainly because of a mixup in the two code base installation. We resolved this by starting from scratch and deleting all old lzo libs and that resolved the error

Testing to see lzo native libs works:

- Create a sample lzo file

$~: echo "hello world" > test.log
$~: lzop test.log

The above should create a test.log.lzo file

$~: hadoop fs -copyFromLocal test.log.lzo /tmp

Local installation test:

$~: hadoop jar /usr/lib/hadoop-0.20/lib/hadoop-lzo-0.4.4.jar com.hadoop.compression.lzo.LzoIndexer /tmp/test.log.lzo
10/08/03 16:40:01 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
10/08/03 16:40:01 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library 
10/08/03 16:40:03 INFO lzo.LzoIndexer: [INDEX] LZO Indexing file /tmp/test.log.lzo, size 0.00 GB...
10/08/03 16:40:03 INFO lzo.LzoIndexer: Completed LZO Indexing in 1.40 seconds (0.00 MB/s).  Index size is 0.01 KB.

Distributed Test:

$~: hadoop jar /usr/lib/hadoop-0.20/lib/hadoop-lzo-0.4.4.jar com.hadoop.compression.lzo.DistributedLzoIndexer /tmp/test.log.lzo
10/08/03 16:42:53 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
10/08/03 16:42:53 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library 
10/08/03 16:42:53 INFO lzo.DistributedLzoIndexer: Adding LZO file /tmp/test.log.lzo to indexing list (no index currently exists)
10/08/03 16:42:53 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
10/08/03 16:44:24 INFO input.FileInputFormat: Total input paths to process : 1
10/08/03 16:44:24 INFO mapred.JobClient: Running job: job_201007251750_0072
10/08/03 16:44:25 INFO mapred.JobClient:  map 0% reduce 0%
10/08/03 16:44:38 INFO mapred.JobClient:  map 100% reduce 0%
10/08/03 16:44:40 INFO mapred.JobClient: Job complete: job_201007251750_0072
10/08/03 16:44:40 INFO mapred.JobClient: Counters: 6
10/08/03 16:44:40 INFO mapred.JobClient:   Job Counters 
10/08/03 16:44:40 INFO mapred.JobClient:     Launched map tasks=1
10/08/03 16:44:40 INFO mapred.JobClient:     Data-local map tasks=1
10/08/03 16:44:40 INFO mapred.JobClient:   FileSystemCounters
10/08/03 16:44:40 INFO mapred.JobClient:     HDFS_BYTES_READ=60
10/08/03 16:44:40 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=8
10/08/03 16:44:40 INFO mapred.JobClient:   Map-Reduce Framework
10/08/03 16:44:40 INFO mapred.JobClient:     Map input records=1
10/08/03 16:44:40 INFO mapred.JobClient:     Spilled Records=0