My Tech Blogs!!!

JSPC Maven Plugin Package Name

2011-10-18T16:28:00.002-07:00

Specifying package name when using the JSPC Maven plugin to pre-compile jsps


<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>jspc-maven-plugin</artifactId>
    <configuration>
     <packageName>com.company.package.name</packageName>
    </configuration>
    <executions>
     <execution>
      <id>jspc</id>
      <goals>
       <goal>compile</goal>
      </goals>
     </execution>
    </executions>
   </plugin>

Amazon AWS Service Status

2011-10-04T07:24:00.007-07:00

AWS currently does not provide an API to query service status. They only provide a health dashboard webpage. Although the page has RSS feed capability, I couldn't detect any specific pattern to watch for. This simple solution below works for now. We have created nagios monitoring out of these one liners


status0 = OK
status2 = Performance Issues
status3 = ERROR

The service and region name below is literally as is from the web page. 

wget -qO- "http://status.aws.amazon.com/" | grep -B 1 "Amazon Elastic Compute Cloud (N. California)" | head -n 1 | 
perl -n -e 'm/images\/([\s\S]+?)\.gif/ && {print "$1\n"}'

status0

wget -qO- "http://status.aws.amazon.com/" | grep -B 1 "Amazon Elastic Compute Cloud (N. Virginia)" | head -n 1 | 
perl -n -e 'm/images\/([\s\S]+?)\.gif/ && {print "$1\n"}'

status0

Rooted my Motorola Atrix

2011-08-13T10:24:00.002-07:00

System version: 4.5.91
Android Version: 2.3.4

Instructions found below:

http://www.digitaltweaker.com/android/phones/2011/07/root-motorola-atrix-4-5-91-on-gingerbread-2-3-4/

If you get the "device not found" error when running "adb shell" then make sure you have sync drivers installed. When you connect Atrix to your computer the first time it installs a bunch of drivers and also gives an option to install "synch drivers". Make sure to install the "sync drivers".

Repository for JBoss RestEasy Releases

2011-07-07T10:41:00.002-07:00

We have been using JBoss RestEasy for developing restful services. The framework is new and we have had to change the repo many times with various version releases and often times the repos fail with permission denied error or dependencies resolution error. A more stable repo for RestEasy is being maintained by the Atlassian folks. We use the below repo and its been working consistently and updates with new releases fairly quickly

https://maven.atlassian.com/content/repositories/jboss-releases

Check Logged Person In Campfire Room

2011-06-01T12:25:00.005-07:00

We use campfire as a sole medium for team communication so its essential every team member is logged in so they are accessible. All team members use Mac and we use the Fluid app to run Campfire as a app. The app logs out automatically sometimes when in not use and we needed a way to get alerted when member was not logged into campfire.

Campfire has a pretty nifty API that allows to query many room attributes. Below script can be used to detect if a person is not logged in. Each of our team member has the below script cronned in their machines to run every 10mins or so to get alerted when they have been logged out. below are the attributes you need to substitute

{authcode here} - To get your auth code in Campfire go to "Edit my Campfire account" or "My Info" at the top screen in Campfire and there you will see your auth code. This is a secret code and should not be shared with others so be watchful and ensure the below script only has read/execute perms for the member user on the box.

"#{id}" - To get the room ID simply login to Campfire and click on the room. The browser url should show the room ID.

{member name} - Should be the same as the name specified when creating campfire account



#!/bin/bash
curl -s -u {authcode here}:X "https://{your specific domain}.campfirenow.com/room/#{id}.xml" > /tmp/campfire-room.xml
present=`grep "{member name}" /tmp/campfire-room.xml | wc -l`
if [ $present -eq 0 ]
  then
   `/usr/local/bin/growlnotify --name "Not Logged In Campfire" -s --message "You are not logged into campfire. 
This might reflect in your performance review next year so you better login now :)"`
fi

Transitioning Traditional Data Support Folks To New Breed ETL Systems

2011-05-08T22:41:00.001-07:00

Like many other folks out there who are discovering the power of data compute grids, we recently transitioned a part of our traditional database based ETL system to Hadoop based processing system. Being in the digital advertising field, we get a whole lot of impression data both from our tracking systems and external. Our existing ETL system is mainly comprised of four components: cleansing, standardization, dimensioning, aggregation.

The cleansing and standardization components that involve a whole lot of text parsing and mapping take the major brunt as they deal with the raw volume of incoming data. As part of the transition we moved these two components to the new system. We have a dedicated product support team that handles most of the daily user data queries. Issues like missing/incorrect data, re-running jobs due to changed dimensions or external data outages, configuring and testing new data fields/sources, generating ad-hoc reports, configuring new clients etc. These folks have a thorough domain knowledge and are well versed with this data. They also fully understand the current ETL data flow and the various business rules thats gets applied as part of the data processing. Technology wise they are comfortable with databases, SQL, basic scripting, Excel and are usually enthusiastic about learning new technologies as need be.

To be able to perform the above mentioned issues, they essentially need a way to slice and dice raw/stage level data and with the data residing in HDFS this becomes an issue. We have been brainstorming ways on how to expose the new system to support folks and below are some options.

Apache Hive: Facebook was the first company to encounter this problem wherein they transitioned their analysts folks from a RDBMS based warehouse to a more scalable hadoop system. They developed Hive which is essentially a data warehouse system for Hadoop that facilitates easy data summarization, ad-hoc queries, and the analysis of large datasets stored in Hadoop compatible file systems. Hive provides a mechanism to project structure onto this data and query the data using a SQL-like language called HiveQL. HiveQL is very similar to SQL although it does not support the full SQL-92 specification. As per my reading it more closely resembles MySql's SQL dialect and that makes sense because Facebook is a mysql shop and a similarity would make the transition easy for its folks. This option stands top in our list of possibilities given its similarity to SQL.

Apache Pig: Apache Pig is also a higher level abstraction for map/reduce. Pig uses Pig Latin language to express data flows. Although this is a powerful tool, it would require the support folks to learn a entirely new language.

Commercial tools like Datameer, IBM Big Sheets: It is a well know fact that Microsoft Excel is most versatile analytic tool. Analysts love the ease of use and the tools it provides to slice and dice/graph datasets. Imagine an Excel like tool with the power of Hadoop. Its essentially what these commercial tools are. We recently received a great demo from the Datameer folks and were impressed by its ease of use and especially its pluggable architecture. Easy and familiar spreadsheet-like interface for business users with complete set of data integration, transformation/analytic and visualization tools. It also has a neat scheduler for cron based job scheduling. This option is also a strong contender in our option set given its ease of use and spreadsheet like usability and feel.

We haven't decided on an option yet. The next couple of weeks will involve closely working with support folks to evaluate these options and ensuring a smooth transition.

Maven3 error "Could not find artifact"

2011-03-18T17:13:00.003-07:00

We recently switched our artifactory repo to a new server and things were fine with build running fine. One of the team dev deployed a new artifact version to the repo and that caused the build to fail. We kept getting the below error on our continuous integration server while the build was working fine on all dev boxes


"Failed to execute goal on project... Could not resolve dependencies for project....SNAPSHOT: The following artifacts could not be resolved: {new artifact}. Failure to find ... in  http://localhost/artifactory/repo was cached in the local repository,  resolution will not be reattempted until the update interval of  artifactory has elapsed or updates are forced"

We debugged this issue for a while and made sure the artifact was deployed correctly to the repo and all that but the main issue was that in the error message the artifact location was wrong. It was looking in the localhost. We checked the POM and ensured the repositories link specified were correct. After quite a bit of head banging we leaned towards a issue with local cache of repo and decided to purge the local repository. We deleted the .m2/repository/artifact directory and that resolved the above error but gave the below error


"Failed to execute goal on project... Could not resolve dependencies for project....SNAPSHOT: The following artifacts could not be resolved: {new artifact} Could not find artifact in artifactory (http://localhost/artifactory/repo) "

We were still looking to find as to why it was not pointing to the correct repo location. The fact that it was working fine on all all places made us believe that there was a specific issue with the continuous integration box so we decided to specify a override on the local repo location so it downloads fresh a copy of all dependencies and thats when we saw a default mirror repo location specified in the local m2 settings.xml file. The file had the below properties set


<mirrors>
    <mirror>
        <id>artifactory</id>
        <mirrorOf>*</mirrorOf>
        <name>Artifactory</name>
        <url>http://localhost/artifactory/repo</url>
    </mirror>
</mirrors>

That was causing the issue. We modified this to point to the new repo location and boom the error was gone

Mac OSX Disable gconsync

2011-03-10T23:46:00.002-07:00

I recently enabled synching with google contacts in ITunes while synching my IPhone. Ever since then I have been getting this nagging pop up where a program called "gconsync" keeps asking me password to access keychain. On my iphone I have now setup google account as exchange account and that automatically synchs my contacts with google so I no longer need to enable the google account synching with itunes but even after disabling that I still kept getting the gconsync popup for keychain access. Here's the way to disable this

Open the Address book app on Mac: /Applications/Address Book.app

Go to Preferences -> Account -> uncheck the "synchronize with Google" option

That should get rid of the popup.

JIRA Current Iteration Filter

2011-03-04T20:38:00.002-07:00

The current version of JIRA we use does not provide a option to query by current iteration. Although the JQL querying option allows to do fairly fine grain searching it fails to provide a current iteration option. One way they could add is by looking at the Iteration start and end date and compare that to current date. I set up various filters for current iteration that search for issue without story points, subtasks with missing hours etc. It was a pain to modify the filters every new iteration to specify the iteration number.

We installed a free JQL query extension that allows a versionList function

https://studio.plugins.atlassian.com/wiki/display/IJQL/VersionList+Function

Once installed you can query the current iteration using the below JQL

project = COLL and fixVersion in unreleasedVersions() and fixVersion in VersionList("Iteration ??")

To make the above work, I had to rename all future iterations to some thing line 'Pre Iteration....". That wasn't a big deal as we only wanted to make sure all historic and current iteration are named correctly.

To query all issues that don't have a story point one can use the below query and subscribe to daily email to get alerts

project = COLL and fixVersion in unreleasedVersions() and fixVersion in VersionList("Iteration ??") and type not in (Bug,Sub-task, "Support Ticket") and storyPoint = empty and (resolution in (Fixed,Completed) or status = Open)

Upgrading Cloudera Hadoop VM From Intrepid (8.10) to Jaunty (9.04) and Configuring Shard Folders with Host OS

2010-11-17T21:03:00.003-07:00

Notes gathered from trying to upgrade Ubuntu version on Cloudera Hadoop VM. The VM comes with version Ubuntu 8.10 Intrepid. Most of the source repo for this Ubuntu version is not supported and any apt-get install errors out with below error


Err http://us.archive.ubuntu.com intrepid-updates/main Packages
  404 Not Found [IP: 91.189.88.46 80]
Err http://us.archive.ubuntu.com intrepid-updates/multiverse Packages
  404 Not Found [IP: 91.189.88.46 80]
Err http://security.ubuntu.com intrepid-security/multiverse Packages
  404 Not Found [IP: 91.189.88.37 80]
Err http://security.ubuntu.com intrepid-security/main Sources
  404 Not Found [IP: 91.189.88.37 80]

This also causes an issue when installing the VMware tools as the tools install needs certain build packages installed however the apt-get won't work. Here's the step by step instruction from upgrading OS to installing VMWare tools to configuring shared folders from Host OS to cloudera vm

The upgrade from 8.10 to 9.04 is seamless and did not break anything on my end. I didn't change the VM a lot so if things would have broken I would have either fixed or downloaded the VM fresh form the Cloudera site :)

-------------- Upgrade O/S --------------

- launch terminal


- sudo do-release-upgrade

This will take close to 15-20mins with decent internet connection. Need to baby sit to confirm various choices. Once complete it will ask for a reboot. After reboot bring up terminal and do


- cat /etc/lsb-release

above should show 9.04 version now.

----------- Install VMWare Tools -----------------

Before doing that we need to install certain libraries. Run these commands


- sudo apt-get update

- sudo apt-get install gcc

- sudo apt-get install linux-headers-{sub from uname command}-server (run uname -a to get the server details and replace server number there)

- Follow instruction in below on how to install VMWare tools in Ubuntu

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1014522

------------ Configuring Shared Folders ------------

Here's a video instuction on how to configure the shared folder for VMware Fusion 3

http://www.youtube.com/watch?v=fxNqdE7IanQ

For VMWare Fusion versions 2 or Older its a little complex

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1013633

Download link for last free version for iStat Menu

2010-10-07T21:56:00.002-07:00

iStat Menus used to be a free download. Its now a paid app. See below link to download the last free version.

Download iStat Menus 2.0 (Last free version; Mac OS X 10.5/10.6 compatible)

Namenode editlog (dfs.name.dir) multiple storage directories

2010-08-24T22:55:00.002-07:00

Recently there was a question posted on the Apache Hadoop User mailing list regarding the behavior of specifying multiple storage directories fir property "dfs.name.dir". The documentation on the official site is not clear on what happens if one of the specified storage directories become inaccessible. While the documentation for other properties such as "dfs.data.dir" and "mapred.local.dir" clearly states that the system will ignore any directory that is inaccessible in case some of the specified directories are unavailable. See link below for reference

http://hadoop.apache.org/common/docs/current/hdfs-default.html

There were many comments on the post. See below the link for the complete post

http://lucene.472066.n3.nabble.com/what-will-happen-if-a-backup-name-node-folder-becomes-unaccessible-td1253293.html#a1253293

In this post I am basically going to summarize my tests to prove that it works in the cloudera distribution. So the behavior is that it ignores any directories that are inaccessible and the namenode only bails out when it can't access any of the specified directories. The below series of tests are pretty much self explanatory



hadoop@training-vm:~$ hadoop version
Hadoop 0.20.1+152
Subversion  -r c15291d10caa19c2355f437936c7678d537adf94
Compiled by root on Mon Nov  2 05:15:37 UTC 2009

hadoop@training-vm:~$ jps
8923 Jps
8548 JobTracker
8467 SecondaryNameNode
8250 NameNode
8357 DataNode
8642 TaskTracker

hadoop@training-vm:~$ /usr/lib/hadoop/bin/stop-all.sh
stopping jobtracker
localhost: stopping tasktracker
stopping namenode
localhost: stopping datanode
localhost: stopping secondarynamenode

hadoop@training-vm:~$ mkdir edit_log_dir1

hadoop@training-vm:~$ mkdir edit_log_dir2

hadoop@training-vm:~$ ls
edit_log_dir1  edit_log_dir2

hadoop@training-vm:~$ ls -ltr /var/lib/hadoop-0.20/cache/hadoop/dfs/name
total 8
drwxr-xr-x 2 hadoop hadoop 4096 2009-10-15 16:17 image
drwxr-xr-x 2 hadoop hadoop 4096 2010-08-24 15:56 current

hadoop@training-vm:~$ cp -r /var/lib/hadoop-0.20/cache/hadoop/dfs/name edit_log_dir1

hadoop@training-vm:~$ cp -r /var/lib/hadoop-0.20/cache/hadoop/dfs/name edit_log_dir2

------ hdfs-site.xml added new dirs

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
  <property>
     <name>dfs.permissions</name>
     <value>false</value>
  </property>
  <property>
     <!-- specify this so that running 'hadoop namenode -format' formats the right dir -->
     <name>dfs.name.dir</name>    
    <value>/var/lib/hadoop-  0.20/cache/hadoop/dfs/name,/home/hadoop/edit_log_dir1,
/home/hadoop/edit_log_dir2</value>
  </property>
   <property>
     <name>fs.checkpoint.period</name>
     <value>600</value>
  </property>
  <property>
    <name>dfs.namenode.plugins</name>
    <value>org.apache.hadoop.thriftfs.NamenodePlugin</value>
  </property>
  <property>
    <name>dfs.datanode.plugins</name>
    <value>org.apache.hadoop.thriftfs.DatanodePlugin</value>
  </property>
  <property>
    <name>dfs.thrift.address</name>
    <value>0.0.0.0:9090</value>
  </property>
</configuration>

---- start all daemons

hadoop@training-vm:~$ /usr/lib/hadoop/bin/start-all.sh
starting namenode, logging to 
/usr/lib/hadoop/bin/../logs/hadoop-hadoop-namenode-training-vm.out
localhost: starting datanode, logging to 
/usr/lib/hadoop/bin/../logs/hadoop-hadoop-datanode-training-vm.out
localhost: starting secondarynamenode, logging to 
/usr/lib/hadoop/bin/../logs/hadoop-hadoop-secondarynamenode-training-vm.out
starting jobtracker, logging to 
/usr/lib/hadoop/bin/../logs/hadoop-hadoop-jobtracker-training-vm.out
localhost: starting tasktracker, logging to 
/usr/lib/hadoop/bin/../logs/hadoop-hadoop-tasktracker-training-vm.out


-------- namenode log confirms all dirs taken

2010-08-24 16:20:48,718 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = training-vm/127.0.0.1
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 0.20.1+152
STARTUP_MSG:   build =  -r c15291d10caa19c2355f437936c7678d537adf94; 
compiled by 'root' on Mon Nov  2 05:15:37 UTC 2009
************************************************************/
2010-08-24 16:20:48,815 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: 
Initializing RPC Metrics with hostName=NameNode, port=8022
2010-08-24 16:20:48,819 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: 
Namenode up at: localhost/127.0.0.1:8022
2010-08-24 16:20:48,821 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
 Initializing JVM Metrics with processName=NameNode, sessionId=null
2010-08-24 16:20:48,822 INFO 
org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics: 
Initializing NameNodeMeterics using context object:org.apache.hadoop.metrics.spi.NoEmitMetricsContext
2010-08-24 16:20:48,894 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: 
fsOwner=hadoop,hadoop
2010-08-24 16:20:48,894 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: 
supergroup=supergroup
2010-08-24 16:20:48,894 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
 isPermissionEnabled=false
2010-08-24 16:20:48,903 INFO org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics: 
Initializing
 FSNamesystemMetrics using context object:org.apache.hadoop.metrics.spi.NoEmitMetricsContext
2010-08-24 16:20:48,905 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered 
FSNamesystemStatusMBean
2010-08-24 16:20:48,937 INFO org.apache.hadoop.hdfs.server.common.Storage: Storage directory
 /home/hadoop/edit_log_dir1 is not formatted.
2010-08-24 16:20:48,937 INFO org.apache.hadoop.hdfs.server.common.Storage: Formatting ...
2010-08-24 16:20:48,937 INFO org.apache.hadoop.hdfs.server.common.Storage: Storage directory
 /home/hadoop/edit_log_dir2 is not formatted.
2010-08-24 16:20:48,937 INFO org.apache.hadoop.hdfs.server.common.Storage: Formatting ...
2010-08-24 16:20:48,938 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files = 41
2010-08-24 16:20:48,947 INFO org.apache.hadoop.hdfs.server.common.Storage: 
Number of files under construction = 0
2010-08-24 16:20:48,947 INFO org.apache.hadoop.hdfs.server.common.Storage: 
Image file of size 4357 loaded in 0 seconds.

---- directories confirm in use

hadoop@training-vm:~$ ls -ltr edit_log_dir1
total 12
drwxr-xr-x 4 hadoop hadoop 4096 2010-08-24 16:01 name
-rw-r--r-- 1 hadoop hadoop    0 2010-08-24 16:20 in_use.lock
drwxr-xr-x 2 hadoop hadoop 4096 2010-08-24 16:20 image
drwxr-xr-x 2 hadoop hadoop 4096 2010-08-24 16:20 current

hadoop@training-vm:~$ ls -ltr edit_log_dir2
total 12
drwxr-xr-x 4 hadoop hadoop 4096 2010-08-24 16:01 name
-rw-r--r-- 1 hadoop hadoop    0 2010-08-24 16:20 in_use.lock
drwxr-xr-x 2 hadoop hadoop 4096 2010-08-24 16:20 image
drwxr-xr-x 2 hadoop hadoop 4096 2010-08-24 16:20 current

----- secondary name node checkpoint worked fine

...
2010-08-24 16:27:10,756 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Posted URL 
localhost:50070putimage=1&port=50090&machine=127.0.0.1&token=-
18:1431678956:1255648991179:1282692430000:1282692049090
2010-08-24 16:27:11,008 WARN org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode:

Checkpoint done. New Image Size: 4461
....

--- dirctory put works fine

hadoop@training-vm:~$ hadoop fs -ls /user/training
Found 3 items
drwxr-xr-x   - training supergroup          0 2010-06-30 13:18 /user/training/grep_output
drwxr-xr-x   - training supergroup          0 2010-06-30 13:14 /user/training/input
drwxr-xr-x   - training supergroup          0 2010-06-30 15:30 /user/training/output

hadoop@training-vm:~$ hadoop fs -put /etc/hadoop/conf.with-desktop/hdfs-site.xml /user/training

hadoop@training-vm:~$ hadoop fs -ls /user/training
Found 4 items
drwxr-xr-x   - training supergroup          0 2010-06-30 13:18 /user/training/grep_output
-rw-r--r--   1 hadoop   supergroup        987 2010-08-24 16:25 /user/training/hdfs-site.xml
drwxr-xr-x   - training supergroup          0 2010-06-30 13:14 /user/training/input
drwxr-xr-x   - training supergroup          0 2010-06-30 15:30 /user/training/output


------ delete one of the directories
hadoop@training-vm:~$ rm -rf edit_log_dir2

hadoop@training-vm:~$ ls -ltr
total 4
drwxr-xr-x 5 hadoop hadoop 4096 2010-08-24 16:20 edit_log_dir1

-- namenode logs

No errors/warns in logs

-------- namenode still running

hadoop@training-vm:~$ jps
12426 NameNode
12647 SecondaryNameNode
12730 JobTracker
14090 Jps
12535 DataNode
12826 TaskTracker

----  puts and ls work fine

hadoop@training-vm:~$ hadoop fs -ls /user/training
Found 4 items
drwxr-xr-x   - training supergroup          0 2010-06-30 13:18 /user/training/grep_output
-rw-r--r--   1 hadoop   supergroup        987 2010-08-24 16:25 /user/training/hdfs-site.xml
drwxr-xr-x   - training supergroup          0 2010-06-30 13:14 /user/training/input
drwxr-xr-x   - training supergroup          0 2010-06-30 15:30 /user/training/output

hadoop@training-vm:~$ hadoop fs -put /etc/hadoop/conf.with-desktop/core-site.xml /user/training

hadoop@training-vm:~$ hadoop fs -put /etc/hadoop/conf.with-desktop/mapred-site.xml /user/training

hadoop@training-vm:~$ hadoop fs -ls /user/training
Found 6 items
-rw-r--r--   1 hadoop   supergroup        338 2010-08-24 16:28 /user/training/core-site.xml
drwxr-xr-x   - training supergroup          0 2010-06-30 13:18 /user/training/grep_output
-rw-r--r--   1 hadoop   supergroup        987 2010-08-24 16:25 /user/training/hdfs-site.xml
drwxr-xr-x   - training supergroup          0 2010-06-30 13:14 /user/training/input
-rw-r--r--   1 hadoop   supergroup        454 2010-08-24 16:29 /user/training/mapred-site.xml
drwxr-xr-x   - training supergroup          0 2010-06-30 15:30 /user/training/output

------- secondary namenode checkpoint is successdul

2010-08-24 16:37:11,455 WARN org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: 
Checkpoint done. New Image Size: 4671
....
2010-08-24 16:47:11,884 WARN org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode:
 Checkpoint done. New  Image Size: 4671
...
2010-08-24 16:57:12,264 WARN org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: 
Checkpoint done. New Image Size: 4671

------- after 30 mins

hadoop@training-vm:~$ jps
12426 NameNode
12647 SecondaryNameNode
12730 JobTracker
16256 Jps
12535 DataNode
12826 TaskTracker

Automated backups to Amazon S3

2010-08-21T19:34:00.002-07:00

Just got around setting an automated backup to Amazon S3. Until now my backup was limited to Personal Computer -> Home Server -> External USB Drive all under one roof. Over the years I have gathered a lot of document clutter and have been lately scanning them and destroying the originals and that means that if the digital copy is gone then I am kaput. Having a copy on the S3 is good both backup wise and access from anywhere wise. Although I have setup my home server to be accessible from outside its still slow when downloading large files. As of now i have limited S3 to important stuff (just documents), I would eventually like to offload all my digital content o S3 (songs, pictures, videos etc) which would mean shelling more moola to Amazon. I wouldn't be accessing these content frequently so the cost is pretty much the storage cost and off-course the weekly rsync seeks. I also choose the 99.99% (0.10c/gb monthly) reliability over the 99.9999999% (0.15c/gb monthly) reliability as I am conformable with that given that I have a copy on my end too. S3 also has a special offer till Nov 1 wherein they are waving all "Data transfer In" fees. Below are some links that I used to setup :

AWS S3 pricing : http://aws.amazon.com/s3/pricing/
S3Sync and S3cmd tool : http://www.s3sync.net/wiki

Installing S3sync script: http://www.linode.com/wiki/index.php/Backups_with_s3sync

Setting up automated backup using S3sync on S3
http://blog.eberly.org/2006/10/09/how-automate-your-backup-to-amazon-s3-using-s3sync/

General article on home server vs S3 backup : http://jeremy.zawodny.com/blog/archives/007624.html
S3 based backup tools: http://jeremy.zawodny.com/blog/archives/007641.html

Hadoop LZO Installation : Errors and Resolution

2010-08-03T17:05:00.003-07:00

A record of our errors and resolution with Hadoop LZO Installation. We previously followed the code base and instructions on

http://code.google.com/p/hadoop-gpl-compression/wiki/FAQ

but later switched to

http://github.com/kevinweil/hadoop-lzo

as there were some improvements and bug fixes.

Error:


java.lang.RuntimeException: native-lzo library not available
 at com.hadoop.compression.lzo.LzopCodec.createDecompressor(LzopCodec.java:91)
 at com.hadoop.mapreduce.LzoSplitRecordReader.initialize(LzoSplitRecordReader.java:52)
 at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:418)
 at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:582)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
 at org.apache.hadoop.mapred.Child.main(Child.java:170)

Resolution: Need to ensure that the lzo lib is in the classpath

$~: ps auxw | grep tasktracker

should show the lzo lib in the classpath list. If not then follow the "Building and configuring" instructions on Kevin's site (link above)

Error:


java.lang.ClassCastException: com.hadoop.compression.lzo.LzopCodec$LzopDecompressor
 cannot be cast to com.hadoop.compression.lzo.LzopDecompressor
 at com.hadoop.mapreduce.LzoSplitRecordReader.initialize(LzoSplitRecordReader.java:52)
 at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:418)
 at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:582)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
 at org.apache.hadoop.mapred.Child.main(Child.java:170)
 
java.lang.IllegalAccessError com.hadoop.compression.lzo.LzopDecompressor cannot access
 superclass com.hadoop.compression.lzo.LzoDecompressor

Resolution:

The above two errors were mainly because of a mixup in the two code base installation. We resolved this by starting from scratch and deleting all old lzo libs and that resolved the error

Testing to see lzo native libs works:

- Create a sample lzo file


$~: echo "hello world" > test.log
$~: lzop test.log

The above should create a test.log.lzo file

$~: hadoop fs -copyFromLocal test.log.lzo /tmp

Local installation test:


$~: hadoop jar /usr/lib/hadoop-0.20/lib/hadoop-lzo-0.4.4.jar com.hadoop.compression.lzo.LzoIndexer /tmp/test.log.lzo
10/08/03 16:40:01 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
10/08/03 16:40:01 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library 
10/08/03 16:40:03 INFO lzo.LzoIndexer: [INDEX] LZO Indexing file /tmp/test.log.lzo, size 0.00 GB...
10/08/03 16:40:03 INFO lzo.LzoIndexer: Completed LZO Indexing in 1.40 seconds (0.00 MB/s).  Index size is 0.01 KB.

Distributed Test:


$~: hadoop jar /usr/lib/hadoop-0.20/lib/hadoop-lzo-0.4.4.jar com.hadoop.compression.lzo.DistributedLzoIndexer /tmp/test.log.lzo
10/08/03 16:42:53 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library
10/08/03 16:42:53 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library 
10/08/03 16:42:53 INFO lzo.DistributedLzoIndexer: Adding LZO file /tmp/test.log.lzo to indexing list (no index currently exists)
10/08/03 16:42:53 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
10/08/03 16:44:24 INFO input.FileInputFormat: Total input paths to process : 1
10/08/03 16:44:24 INFO mapred.JobClient: Running job: job_201007251750_0072
10/08/03 16:44:25 INFO mapred.JobClient:  map 0% reduce 0%
10/08/03 16:44:38 INFO mapred.JobClient:  map 100% reduce 0%
10/08/03 16:44:40 INFO mapred.JobClient: Job complete: job_201007251750_0072
10/08/03 16:44:40 INFO mapred.JobClient: Counters: 6
10/08/03 16:44:40 INFO mapred.JobClient:   Job Counters 
10/08/03 16:44:40 INFO mapred.JobClient:     Launched map tasks=1
10/08/03 16:44:40 INFO mapred.JobClient:     Data-local map tasks=1
10/08/03 16:44:40 INFO mapred.JobClient:   FileSystemCounters
10/08/03 16:44:40 INFO mapred.JobClient:     HDFS_BYTES_READ=60
10/08/03 16:44:40 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=8
10/08/03 16:44:40 INFO mapred.JobClient:   Map-Reduce Framework
10/08/03 16:44:40 INFO mapred.JobClient:     Map input records=1
10/08/03 16:44:40 INFO mapred.JobClient:     Spilled Records=0

Installing Hadoop Native Libraries : JAVA_HOME not set correctly error

2010-07-21T17:24:00.007-07:00

Followed the instructions here to install the hadoop native libraries for the Cloudera distribution. I got the JAVA_HOME not set error when running the ant compile-native target. Here's the step by step procedure to install the native libraries and resolve the error.

Follow the instructions here first to ensure you have the native libraries installed in your OS

http://hadoop.apache.org/common/docs/r0.20.2/native_libraries.html

To configure the hadoop native libraries follow the steps below. Since all our cluster nodes were of the same configuration (architecture and OS wise), I built the native libraries on one and copied onto the other nodes. If you nodes are different then you will have to repeat the below steps on each hadoop node.

Step 1: Install and configure Apache Ant on the hadoop cluster node

Step 2: Download the source of you corresponding hadoop version.


- find your hadoop version

$ hadoop version
Hadoop 0.20.1+169.88
Subversion  -r ded54b29979a93e0f3ed773175cafd16b72511ba
Compiled by root on Thu May 20 22:22:39 EDT 2010

- download the corresponding SRPM for the above version. 
We use Cloudera distribution so I got mine from below link 

http://archive.cloudera.com/redhat/cdh/testing/SRPMS/

- Install the SRPM to get source. This basically dumped the source in my 
/usr/src/redhat/SOURCES/hadoop-0.20.1+169.88 
folder. If it dumped as a tar.gz file then go ahead and untar it.

$rpm -i hadoop-0.20-0.20.1+169.88-1.src.rpm

Step 3: Link the 'src' folder in source to the hadoop installation folder


cd /usr/lib/hadoop-0.20 -- this is basically the HADOOP_HOME
sudo ln -s /usr/src/redhat/SOURCES/hadoop-0.20.1+169.88/hadoop-0.20.1+169.88/src src

Step 4: Run the ant task


ant -Dcompile.native=true compile-native

If the above task fails with the 'jni.h' not found error then configure the JAVA_HOME PATH


[exec] checking jni.h usability... /usr/lib/hadoop-0.20/src/native/configure: line
 19091: test: !=: unary operator expected
[exec] no
[exec] checking jni.h presence... no
[exec] checking for jni.h... no
[exec] configure: error: Native java headers not found. Is $JAVA_HOME set correctly?

BUILD FAILED
/usr/lib/hadoop-0.20/build.xml:466: 
The following error occurred while executing this line:
/usr/lib/hadoop-0.20/build.xml:487: exec returned: 1

To resolve the error open the build.xml file and the add env JAVA_HOME in compile-core-native target. After adding the change should look like below


<exec dir="${build.native}" executable="sh" failonerror="true">
     <env key="OS_NAME" value="${os.name}"/>
     <env key="JAVA_HOME" value="/usr/java"/>
     <env key="OS_ARCH" value="${os.arch}"/>
     <env key="JVM_DATA_MODEL" value="${sun.arch.data.model}"/>
     <env key="HADOOP_NATIVE_SRCDIR" value="${native.src.dir}"/>
     <arg line="${native.src.dir}/configure"/>
</exec>

The above will resolve the issue.

A successful installation should show something like this


$ ls -ltr /usr/lib/hadoop-0.20/build/native/Linux-amd64-64/lib
total 212
-rw-r--r-- 1 root root  12953 Jul 21 16:43 Makefile
-rwxr-xr-x 1 root root  73327 Jul 21 16:43 libhadoop.so.1.0.0
-rw-r--r-- 1 root root    850 Jul 21 16:43 libhadoop.la
-rw-r--r-- 1 root root 114008 Jul 21 16:43 libhadoop.a
lrwxrwxrwx 1 root root     18 Jul 21 17:04 libhadoop.so.1 -> libhadoop.so.1.0.0
lrwxrwxrwx 1 root root     18 Jul 21 17:04 libhadoop.so -> libhadoop.so.1.0.0

Run a test job to make sure native libs are being picked properly. Use the test jar based on your version of hadoop. In the below test ensure that the native libs are getting picked. Check for statement "Successfully loaded & initialized native-zlib library".


$ hadoop jar /usr/lib/hadoop-0.20/hadoop-0.20.1+169.88-test.jar testsequencefile -seed
 0 -count 1000 -compressType RECORD xxx -codec org.apache.hadoop.io.compress.GzipCodec -check
 10/07/22 11:46:27 INFO io.TestSequenceFile: count = 1000
 10/07/22 11:46:27 INFO io.TestSequenceFile: megabytes = 1
 10/07/22 11:46:27 INFO io.TestSequenceFile: factor = 10
 10/07/22 11:46:27 INFO io.TestSequenceFile: create = true
 10/07/22 11:46:27 INFO io.TestSequenceFile: seed = 0
 10/07/22 11:46:27 INFO io.TestSequenceFile: rwonly = false
 10/07/22 11:46:27 INFO io.TestSequenceFile: check = true
 10/07/22 11:46:27 INFO io.TestSequenceFile: fast = false
 10/07/22 11:46:27 INFO io.TestSequenceFile: merge = false
 10/07/22 11:46:27 INFO io.TestSequenceFile: compressType = RECORD
 10/07/22 11:46:27 INFO io.TestSequenceFile: compressionCodec = org.apache.hadoop.io.compress.GzipCodec
 10/07/22 11:46:27 INFO io.TestSequenceFile: file = xxx
 10/07/22 11:46:27 INFO io.TestSequenceFile: creating 1000 records with RECORD compression
 10/07/22 11:46:27 INFO util.NativeCodeLoader: Loaded the native-hadoop library
 10/07/22 11:46:27 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
 10/07/22 11:46:27 INFO compress.CodecPool: Got brand-new compressor
 10/07/22 11:46:27 INFO compress.CodecPool: Got brand-new decompressor
 10/07/22 11:46:28 INFO compress.CodecPool: Got brand-new decompressor
 10/07/22 11:46:28 INFO io.TestSequenceFile: done sorting 1000 debug
 10/07/22 11:46:28 INFO io.TestSequenceFile: sorting 1000 records in memory for debug

Looking online some other folks have had issues with above test. They had to manually add the native libs to JAVA_LIBRARY_PATH. This issue was fixed in https://issues.apache.org/jira/browse/HADOOP-4839. If you are having that issue then here is a link on how to manually add it to path

http://www.mail-archive.com/common-user@hadoop.apache.org/msg04761.html

Ning : Use your own domain from HostMonster

2010-07-16T17:00:00.005-07:00

Here are the instructions on how to set the DNS records to point your domain from HostMonster to Ning. Currently my Ning site is set at "example.ning.com" and I wanted to point it to "example.com". So I purchased the "example.com" on HostMonster. Under cPanel in HostMonster there is a link for "Advanced DNS Editor". Here are the instructions on how to locate the DNS editor

http://helpdesk.hostmonster.com/index.php/kb/article/000559

click on the DNS editor link and choose the "example.com" from the "Select a Domain" dropdown list.

Take a screenshot of the current settings (in case you screw up you can always restore it to what it was earlier using the screenshot)

Step 1: Delete any current "A" type record with name "example.com."

Step 2: Set up a new "A" type record like the below screen shot

Step 3: Set up a new "CNAME" type record like the below screen shot

Now access the "www.example.com" site or "example.com" site and it should point to your ning page. Depending on the set TTL, it might take a while for the new mapping to refresh. You can try to flush your DNS cache using the steps from this site

http://www.tech-faq.com/how-to-flush-dns.html

Hadoop HDFS Error: java.io.IOException: Could not complete write to file

2010-07-15T16:57:00.002-07:00

We recently have been seeing this error in our hadoop name node logs quite a bit.


2010-06-14 05:15:30,428 WARN org.apache.hadoop.hdfs.StateChange: DIR*
 NameSystem.completeFile: failed to complete {filename} because dir.getFileBlocks() is null and pendingFile is null
java.io.IOException: Could not complete write to file {filename} by DFSClient_-44010819
java.io.IOException: Could not complete write to file {filename} by DFSClient_-44010819
 at org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:497)
 at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:512)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:966)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:962)
 at java.security.AccessController.doPrivileged(Native Method)

Further researching, we found that a new map/reduce job was deployed recently that merges daily files. The errors pretty much started showing up after this job was deployed and is cron to run every day. This job takes a bunch of small files as input and merges them to create a merged compressed file. In the process of creating the merging compressed file, it creates a temp file with merged content, verifies the file and then compresses it to a final file after which it deletes the temp file. Errors were being thrown when the temp file was getting deleted. Certain times delete was happening (meta data on namenode) before the file was replicated to all nodes and that was throwing this error. Currently the HDFS API does not provide a way to synchronouos method to wait till file is replicated.

To resolve this, we started using the HDFS trash functionality. HDFS best practices suggests "Enable HDFS trash, and avoid programmatic deletes - prefer the trash facility."If trash is enabled (and, it should be noted, by default it is not), files that are deleted using the Hadoop filesystem shell are moved into a special hidden trash directory, rather than being deleted immediately. The trash directory is deleted periodically by the system. Any files that are mistakenly deleted can be recovered manually by moving them out of the trash directory. The trash deletion period is configurable. The trash facility is a user level feature, it is not used when programmatically deleting files. As a best practice it is recommended considering moving data to be deleted to the trash directory (to be deleted by the system after the given time), rather than doing an immediate delete.

We changed the code to move to trash instead of deleting the code and that resolved the errors in the logs. Below is a common utility method we use across our code base when moving files.


import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.fs.Trash;

public static void moveToTrash(Configuration conf,Path path) throws IOException
{
     Trash t=new Trash(conf);
     boolean isMoved=t.moveToTrash(path);
     if(!isMoved)
     {
 logger.error("Trash is not enabled or file is already in the trash.");
     }
}

Hadoop error logs: org.apache.hadoop.hdfs.server.datanode.BlockAlreadyExistsException

2010-07-06T10:22:00.003-07:00


2010-06-06 23:28:04,195 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
 DatanodeRegistration(10.13.53.128:50010, storageID=DS-1219700830-127.0.0.1-50010-1266876153854, 
infoPort=50075, ipcPort=50020):DataXceiver
org.apache.hadoop.hdfs.server.datanode.BlockAlreadyExistsException: Block
 blk_-7647836931304231161_18783 is valid, and cannot be written to.
    at org.apache.hadoop.hdfs.server.datanode.FSDataset.writeToBlock(FSDataset.java:1069)
    at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.(BlockReceiver.java:98)
    at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:259)
    at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:103)
    at java.lang.Thread.run(Thread.java:619)

Most of the research online pointed to an issue with the open file limit on the cluster instances. Did some analysis and found below


-- current open files [sudhirv@hadoop-cluster-2 ~]$ sudo /usr/sbin/lsof | wc -l
1662
 
-- allowed open file limit cutoff
[hadoop@hadoop-cluster-2 ~]$ ulimit -n
1024

As per the Pro Hadoop book Pg 112 "Hadoop Core uses large numbers of file descriptors for MapReduce, and the DFSClient uses a large number of file descriptors for communicating with the HDFS NameNode and DataNode server processes. Looking at the below commands I see the current open file descriptors greater than set file limit. Most sites immediately bump up the number of file descriptors to 64,000, and large sites, or sites that have MapReduce jobs that open many files, might go higher."

See reference links below to find more suggestions

Decided to set the limit to 8192 and monitor the logs. Modified the open file limit on all hadoop cluster boxes to 8192. Monitored the logs for a few days and we don't see the error anymore.


[root@hadoop-cluster-1 ~]# ulimit -Hn
8192
[root@hadoop-cluster-1 ~]# ulimit -Sn
8192

References:

http://webcache.googleusercontent.com/search?q=cache:sFLquGtEMNEJ:osg-test2.unl.edu/documentation/hadoop/hadoop-installation+hadoop+file+limits&hl=en&client=firefox-a&gl=us&strip=1

simple script to monitor hadoop daemon logs for errors and warning

2010-06-04T14:04:00.007-07:00


#!/bin/sh
check_daemons=$1
log_dir=/var/log/hadoop
warn_file=/tmp/warnings.txt
emails="{email to send errors}"

rm -f /tmp/warnings.txt

for log_file in namenode jobtracker secondarynamenode datanode tasktracker
  do
     echo "$check_daemons" | grep -qw "$log_file"
     if [[ $? -eq 0 ]] ; then            
            echo -e "\n --------------------- $log_file errors and warning ------------------ \n" >> $warn_file
            cat $log_dir/hadoop-hadoop-$log_file-`hostname`.log.`date -d "1 day ago" +%Y-%m-%d` 
| egrep -v 'Checkpoint done' | egrep -A 10 'WARN|ERROR' >> $warn_file
     fi
done

error_line_count=`cat /tmp/warnings.txt | egrep -v "datanode errors and warning" |
 egrep -v "tasktracker errors and warning" | egrep -v "jobtracker errors and warning"
 | egrep -v "namenode errors and warning" | egrep -v "secondarynamenode errors and warning" | egrep -v "^$" | wc -l`

if [ $error_line_count != 0 ] ; then
   mail -s "`hostname` node errors and warnings" $emails < $warn_file
fi

The script can be cronned and it checks the logs from yesterday. You can easily modify the script for standalone runs for any specific dates

Usage:

For a node running namenode, jobtracker run script as

sh get_errors.sh namenode,jobtracker

For a node running datanode, tasktracker run script as

sh get_errors.sh datanode,tasktracker

Updating Cloudera Hadoop To Latest Patch On CentOS

2010-05-27T16:20:00.004-07:00

Warning: Below steps should only be used when updating Hadoop within the same major version. For instance updates like 0.20.1+169.56 to 0.20.1+169.88. When upgrading (example going from 0.20 to 0.21) these steps should not be used.

We have a small cluster in place so the below steps works for us. Bigger cluster nodes will need a more automated way.

Here are the step by step instructions. Ensure no map/reduce jobs/HDFS data write jobs are running.

Step 1: Create a screen session on all hadoop nodes

Step 2: Prepare for any disaster that might happen

On the master node where NameNode is current running, take a latest checkpoint in case things go wrong. This will serve as last image to recover from if things go wrong. To take a manual checkpoint, you need to enter safemode. See below for series of commands to do that.


- check current mode
[sudhirv@hadoop-cluster-3 bin]$ hadoop dfsadmin -safemode get
Safe mode is OFF
 
- enter safe mode
[sudhirv@hadoop-cluster-3 bin]$ hadoop dfsadmin -safemode enter
Safe mode is ON
 
- check safe mode is ON
[sudhirv@hadoop-cluster-3 bin]$ hadoop dfsadmin -safemode get
Safe mode is ON
 
- run manual checkpoint
[sudhirv@hadoop-cluster-1 ~]$ hadoop dfsadmin -saveNamespace
 
-- ensure a checkpoint was taken by looking at image timestamp
[sudhirv@hadoop-cluster-1 ~]$ ls -ltr /var/lib/hadoop-0.20/cache/hadoop/dfs/name/image
total 4
-rw-rw-r-- 1 hadoop hadoop 157 May 26 16:49 fsimage
 
- leave safe mode
[sudhirv@hadoop-cluster-1 ~]$ hadoop dfsadmin -safemode leave
Safe mode is OFF
 
- confirm safe mode is off
[sudhirv@hadoop-cluster-1 ~]$ hadoop dfsadmin -safemode get
Safe mode is OFF

Step 3: Log the current hadoop version and configurations versions. The hadoop version will change after update. The configurations version should still stay the same.


sudo su -
 
[root@hadoop-cluster-3 ~]$ hadoop version
Hadoop 0.20.1+169.56
Subversion  -r 8e662cb065be1c4bc61c55e6bff161e09c1d36f3
Compiled by root on Tue Feb  9 13:40:08 EST 2010
 
[root@hadoop-cluster-3 ~]# alternatives --display hadoop-0.20-conf
hadoop-0.20-conf - status is auto.
 link currently points to /etc/hadoop-0.20/conf.cluster
/etc/hadoop-0.20/conf.empty - priority 10
/etc/hadoop-0.20/conf.pseudo - priority 30
/etc/hadoop-0.20/conf.cluster - priority 50
Current `best' version is /etc/hadoop-0.20/conf.cluster.

Step 4: Shutdown hadoop services on the node. Check to ensure services are turned off. It is recommended to start with nodes that host the Name Node and Job Tracker first as mismatch in versions between NN,JT and DN,TT won't work in most cases.

Step 5: Check to ensure the new hadoop update patch shows up in the yum update list. If not then you would need to configure the repo correctly


yum list updates

Step 5: Run the update


sudo su -
[root@hadoop-cluster-3 ~]# yum update hadoop-0.20

Step 6: Check updated version to ensure update was performed and that config alternatives were not changed


[root@hadoop-cluster-3 ~]# hadoop version
Hadoop 0.20.1+169.88
Subversion  -r ded54b29979a93e0f3ed773175cafd16b72511ba
Compiled by root on Thu May 20 22:22:39 EDT 2010
 
[root@hadoop-cluster-3 ~]# alternatives --display hadoop-0.20-conf
hadoop-0.20-conf - status is auto.
 link currently points to /etc/hadoop-0.20/conf.cluster
/etc/hadoop-0.20/conf.empty - priority 10
/etc/hadoop-0.20/conf.pseudo - priority 30
/etc/hadoop-0.20/conf.cluster - priority 50
Current `best' version is /etc/hadoop-0.20/conf.cluster.

Step 7: Start the hadoop services on the node and tail logs to ensure no errors are being reported.

HDFS Relative path error when running map/reduce jobs

2010-05-19T18:04:00.005-07:00

Recently while running a map reduce we encountered this error


java.io.IOException: Can not get the relative path: 
base = hdfs://hadoop-master:8020/5/_temporary/_attempt_201005150057_0006_r_000000_0
 child = hdfs://hadoop-master/5/_temporary/_attempt_201005150057_0006_r_000000_0/part-r-00000
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getFinalPath(FileOutputCommitter.java:200)
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:146)
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.moveTaskOutputs(FileOutputCommitter.java:165)
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitTask(FileOutputCommitter.java:118)
at org.apache.hadoop.mapred.Task.commit(Task.java:779)
at org.apache.hadoop.mapred.Task.done(Task.java:691)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:414)
at org.apache.hadoop.mapred.Child.main(Child.java:170)

The job tracker and task tracker logs continuously kept failing across many retry attempts and continuously were dumping this error in the logs. Here's the code part that throws the error.

Researched this for quite a while and finally found this excerpt from the Pro Hadoop book "An example of an HDFS URI is hdfs://NamenodeHost[:8020]/. The file system protocol is hdfs, the host to contact for services is NamenodeHost, and the port to connect to is 8020, which is the default port for HDFS. If the default 8020 port is used, the URI may be simplified as hdfs://NamenodeHost/. This value may be altered by individual jobs. You can choose an arbitrary port for the hdfs NameNode."

Currently our "fs.default.name" parameter in core-site.xml file is set to "hdfs://hadoop-master.ic:8020". Based on the above text I went ahead and removed the default port from the parameter and restarted the name node. Ran your job again and voila!!! works now.

Fluid app when signin opens default browser

2010-05-02T21:42:00.002-07:00

I am big fan of Fluid and use it for creating apps for most of my email/social sites. I have also one created for Campfire which I use to communicate with my team. Recently my Campfire app stopped working (not sure if this happened after I upgraded to newer Fluid version or whether the Campfire dev started using a single signon servide ) in the sense then when I click on 'Signin' it was opening my default browser and forwarding me to "https://launchpad.37signals.com/signin". On further googling found that this was happening because the campfire fluid app did not have permission to allow other site domains meaning the signin process was forwarding to a different domain than the campfire specific domain. For instance my campfire domain is "https://icrossing.campfirenow.com" and signin would redirect to "https://launchpad.37signals.com/signin" for authentication. To make it work all it needed was to set the app permission to allow browsing to other domains and that made it work.

Go to fluid app -> preferences -> Advanced and enable the below option

Reliance Connect with Huawei EC325 Modem on Mac OSX

2010-04-11T04:54:00.004-07:00

On a recent trip to India I purchased the Reliance Connect data plan and was given a Huawei EC325 USB modem with a Windows only install CD. Took me a few hours to figure out the setup with Mac OSX on a MacBook Pro and hence the blog.

Download the Mac OSX drivers from here. Under category "USB Modem Huawei USB" download the "Huawei EC325 Driver for McIntosh (both Power PC and Intel X86 Processor". Unzip the downloaded file and install the corresponding pkg file.

After the install (You may need to restart your Mac) go to "System Preferences" -> Network and you should now see the "Huawei Mobile" on the left pane. See below screenshot for configuring the setting (note: These settings are only valid for Reliance NetConnect).

Download Thai version of configuring the modem from here

How to filter queries from Mysql Replication

2010-02-19T12:10:00.002-07:00

So this morning we faced a slave issue (duplicate entry) due to bin log corruption (slave machine unexpectedly died and on reboot had a corrupt table that was causing the duplicate entry error on replicated insert statements). After trying everything to resolve the issue, we ended up with deciding to scrap the slave table and copy the table new from Master. In order to do that we still needed to solve this slave issue and get the slave going and some later time do the master to slave copy. We decided to empty the current slave table and let the slave run however the issue was that this Master - Slave setup is actually a Master <-> Master setup and delete statement wud be replicated to master and we didn't want that. Ideally we wanted the delete statement in this case to not replicate. We achieved this by setting the 'sql_log_bin' session variable to 0 before running the delete statement. Since this is a session based variable it only affects the current session (we tested to make sure. see below). So the steps were

- open a new mysql session with a user that has SUPER privileges
- confirm sql_log_bin is set to 1 using comment "select @@sql_log_bin;"
- set the sql_log_bin to 0 using command "set sql_log_bin=0;"
- confirm variable is correctly set using command "select @@sql_log_bin;"
- run the statements that you don't want replicated
- Quit the mysql client session

Testing to ensure its really session based :)

############## Session 1 #############################

mysql> select @@sql_log_bin;
+---------------+
| @@sql_log_bin |
+---------------+
| 1 |
+---------------+
1 row in set (0.00 sec)

mysql> set sql_log_bin=0;
Query OK, 0 rows affected (0.00 sec)

mysql> select @@sql_log_bin;
+---------------+
| @@sql_log_bin |
+---------------+
| 0 |
+---------------+
1 row in set (0.00 sec)

############## Session 2 #############################

mysql> select @@sql_log_bin;
+---------------+
| @@sql_log_bin |
+---------------+
| 1 |
+---------------+
1 row in set (0.00 sec)

Setting up passwordless access on CentOs 5

2010-02-09T13:56:00.002-07:00

This has been detailed in various places however the part about setting the right permissions on the .ssh entries made it work for. Here's a step by step procedure

- Generate the keys
ssh-keygen -t rsa -P ""

- copy the key to authorized keys
cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys

- ensure only the user has read/write/execute permission on the .ssh dir
chmod 700 $HOME/.ssh

- ensure only the user has read/write permission on authorized keys file
chmod 600 $HOME/.ssh/authorized_keys

Thoughtworks webinars on Distributed Agile Teams

2010-01-14T09:52:00.003-07:00

Attended a pretty educational 2 part series from Thoughtworks on distributed agile teams focussing on communication and facilitating agile meetings. Must watch. designed to address the concerns and risks that come with communicating effectively on a distributed project. The goal of this series is to provide immediate support and detailed suggestions to project teams struggling to work in a distributed environment.

Can you hear me now? Good . . .

Part 1: Remove Barriers to Communication : Mark Rickmeier reviews and rates the best-of-breed tools that address the key communication challenges faced with distributed development. We will evaluate a variety of tools and what they can do for your project, including: Instant messenger/chat tools, desktop sharing, VOIP, Web conferencing, Wiki/collaboration, issue/task tracking, and application lifecycle management.

Part 2: Facilitating Distributed Agile Meetings : This series builds on the tool recommendations from Part 1 to focus on process and innovations used by effective project teams to facilitate four typical Agile meetings between remote locations: Distributed release planning sessions, distributed iteration planning meetings, distributed stand ups, and distributed retrospectives.

Reverting a deleted file in Subversion

2009-12-29T15:35:00.005-07:00

Last week I accidently a file from the SVN repo and had to revert the delete. Here's a procedure to

1. Do a svn log on the directory where the file used to exist. That will show the deleted file commit action


bash> svn log --verbose

deleting MSN creative change
------------------------------------------------------------------------
r1438 | sudhirv | 2009-12-28 15:13:46 -0800 (Mon, 28 Dec 2009) | 1 line
Changed paths:
   D /trunk/lib/ETL/Transform/MSNCreativeTransformByAdId.pm

2. So the above file needs to be reverted. Get the previous version of the file to working copy.


bash> svn copy --revision 1437 \
http://somecompany.com/svn/trunk/lib/ETL/Transform/MSNCreativeTransformByAdId.pm MSNCreativeTransformByAdId.pm

3. Doing a SVN status will now show that there is a file waiting to be added to the rep with previous history. Quoted from the book "The plus sign in the status output indicates that the item isn't merely scheduled for addition, but scheduled for addition “with history”. Subversion remembers where it was copied from. In the future, running svn log on this file will traverse back through the file's resurrection and through all the history it had prior to revision 1438. In other words, this new MSNCreativeTransformByAdId.pm isn't really new; it's a direct descendant of the original, deleted file."


bash> $ svn status
A  +   MSNCreativeTransformByAdId.pm

4. Use the svn commit to commit the file.


bash> svn commit -m "reverting delete to revision 1437" MSNCreativeTransformByAdId.pm

Mysql backups using LVM snapshots

2009-12-16T23:26:00.001-07:00

Logical Volume Manager (LVM) provides the ability to take a snapshot of any logical volume for the purpose of obtaining a backup of a partition in a consistent state. As applications may access files or databases on a partition during a backup some files may be backed up in one state, while later files are backed up after an update has been made, leading to an inconsistent backup. Traditionally the solution has been to mount the partition read-only, apply table-level write locks to databases or shut down the database engine etc.; all measures which adversely impact availability (but not as much as data loss without a backup will). With LVM snapshots it is possible to obtain a consistent backup without compromising availability. The LVM snapshot works by logging the changes to the filesystem to the snapshot partition, rather than mirroring the partition. Thus when you create a snapshot partition you do not need to use space equal to the size of the partition that you are taking a snapshot of, but rather the amount of changes that it will undergo during the lifetime of the snapshot. This is a function of both how much data is being written to the partition and also how long you intend keeping the LVM snapshot. The longer you leave it, the more changes there are likely to be on the file system and the more the snapshot partition will fill up with change information. The higher the rate of change on the partition the shorter the lifespan of the snapshot. If the amount of changes on the LVM partition exceed the size of the snapshot then the snapshot is released. Step by step procedure to backup using snapshot

1. Connect to MySQL and run FLUSH TABLES WITH READ LOCK
2. While holding connection open run: lvcreate -L16G -s -n dbbackup /dev/Main/Data – This will create snapshot named dbbackup for Logical Volume Main/Data . You should specify enough of undo space to hold modifications during backup process – I’ve specified 16GB in this case. If your undo size is not large enough snapshot will get invalidated and backup will be aborted.
3. Snapshot created, now you want to let MySQL Server to continue, which is done by running UNLOCK TABLES or simply closing connection.
4. Mount backup Filesystem: mount /dev/Main/dbbackup /mnt/backup
5. Copy the data from the above mount to tape or any other source
6. Unmount filesystem umount /mnt/backup
7. Remove snapshot: lvremove -f /dev/Main/dbbackup

So how does the LVM snapshot work internally. LVM snapshots are an example of a copy-on-write snapshot solution. When you have an LVM volume with no snapshots, writes to the volume happen as you’d expect. A block is changed, and that’s it. So in the above case before the snapshot is created, any db inserts/updates that causes a write IO is normally written to the data directory. As soon as you create a snapshot, LVM creates a pool of blocks. I believe that this pool also contains a full copy of the metadata (only metadata not the full data) of the volume. When writes happen to the main volume (i.e. the directory off of which snapshot is created), the block being overwritten is copied to this new pool and the new block is written to the main volume. This is the ‘copy-on-write’. Because of this, the more data that gets changed between when a snapshot was taken and the current state of the main volume, the more space will get consumed by that snapshot pool. The normal way that LVM snapshots are used is not for long-term storage, but rather to get a consistent “picture” of the filesystem such that a backup can be taken. Once the backup is done, the snapshot is discarded. When you mount the snapshot, the meta-data written when the snapshot was taken allows the directory tree to be presented. Blocks that haven’t been changed since the snapshot was taken are read directly from the main volume. Blocks that have been changed are read from the snapshot pool. You don’t want to leave snapshots laying around because (a) they’ll fill up and become unusable, and (b) the system’s performance is impacted while a snapshot is active– things get slower.

Hibernate flush only specific object types

2009-11-04T15:55:00.003-07:00

So we have this issue where in certain cases we would only like to flush a certain class of objects and discard the rest. This is sort of a common use case if you are following the one session per request paradigm wherein if a job/request fails and the job status logging is also done via the database/hibernate then you would still want to persist the job log status however the application job data changes can be discarded. Found an excellent post here:

https://forum.hibernate.org/viewtopic.php?f=1&t=983747&sid=4988e082644059b66220836ef4c53806&start=15

So as it turns out there is no way to selectively flush objects. You can use the "evict non flushable objects" methodology (basically bu evicting the objects their changes are discarded) but often times you don't know what all objects were involved in the request.

A better design is to probably open two sessions with each request/job. One for meta data and the other for application business data. That allows better handling of request failures.

Enable Mysql general query log in Mac OSX

2009-10-04T15:42:00.002-07:00

To enable this for the current user follow the below steps

-- create or edit the existing cnf file
vim ~/.my.cnf

-- add the below variable to the file
[server]
log=/tmp/mysql_query.log

-- restart mysql
sudo /Library/StartupItems/MySQLCOM/MySQLCOM stop
sudo /Library/StartupItems/MySQLCOM/MySQLCOM start

To enable the general query log for all users add the "variable" to /etc/my.cnf instead.

Adobe Flash freezes on full screen

2009-08-30T14:08:00.005-07:00

Have been having this problem recently wherein doing a full screen on a flash video freezes the screen. My wife watches movies online occasionally on MegaVideo/Veoh and the problem persists in both as both use flash internally.

While the Flash player is in normal mode, right click on the player and choose "Settings" from the context menu. Remove the checkmark in the box next to "Enable Hardware Acceleration".

Reference links
http://support.mozilla.com/si/kb/Cannot+view+full+screen+Flash+videos
http://www.macromedia.com/support/documentation/en/flashplayer/help/help01.html

Adcenter keyword: Setting match type

2009-08-14T17:07:00.001-07:00

Adcenter is different from other networks as it does not have a special attribute called “match type” on a keyword. Instead each keyword has three bid fields “exact bid (EB)”, “phrase bid (PB)” and “broad bid (BB)”. The keyword match types are determined on whether a bid is specified for that match type. For instance if a keyword has EB=1.5,PB=0.75,BB=0.0 then this keyword is used in exact and phrase matches. This is different in other networks where a keyword has a single “bid” attribute and an attribute to specify match type.

Monitoring Gearman over telnet port 4730

2009-08-07T16:37:00.005-07:00

So the only way to monitor Gearman is via doing a telnet to port 4730. The current monitoring supported commands are fairly basic. I could not locate any documentation on the support commands so had to literally look at the code to figure out the supported commands (the below command documentation is copied from the comments in code). There are plans to include more set of commands in the next release.

Command: STATUS

The output format of this function is tab separated columns as follows, followed by a line consisting of a full stop and a newline (".\n") to indicate the end of output. below are the columns shown

- Function name : A string denoting the name of the function of the job
- Number in queue : A positive integer indicating the total number of jobs for this function in the queue. This includes currently running ones as well (next column)
- Number of jobs running : A positive integer showing how many jobs of this function are currently running
- Number of capable workers : A positive integer denoting the maximum possible count of workers that could be doing this job. Though they may not all be working on it due to other tasks holding them busy.

Command : Workers

This command show the details of various clients registered with the gearmand server. For each worker it shows the following info:

- Peer IP: Client remote host
- Client ID: Unique ID assigned to client
- Functions: List of functions this client has registered for.

Any other command text throws a error "ERR unknown_command Unknown+server+command"



sudhirv@sudhirv:~$ telnet localhost 4730
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

STATUS
.
WORKERS
11 127.0.0.1 - :
.
status
.
workers
11 127.0.0.1 - :
.
some crap
ERR unknown_command Unknown+server+command

#### will add more samples in action when I have this thing up and running ####

Installing Gearman on Ubuntu

2009-08-07T16:01:00.002-07:00

Gearman is a open source job forking system. We have been evaluating its use in our project to share jobs across components. More info on the Gearman project can be found here: http://gearman.org/.

Here are the instructions on installing Gearman on Ubuntu



wget http://launchpad.net/gearmand/trunk/0.9/+download/gearmand-0.9.tar.gz
tar -xvzf gearmand-0.9.tar.gz

# two missing libraries that found at configure time
sudo apt-get install libevent-dev
sudo apt-get install uuid-dev

cd gearmand-0.9
./configure
make
sudo make install
sudo ldconfig

# start gearman
gearmand &

# check gearmand running 
ps auxw | grep [g]earmand

# check germand listening for jobs on tcp port 4730
sudo lsof -i tcp:4730


# errors when starting gearman
Error: gearman: error while loading shared libraries: libgearman.so.1

Solution: `make install` may install libraries into `/usr/local/lib`.  It's possible
 that this directory isn't on your library search path, which will result in an error 
like: "gearman: error while loading shared libraries: libgearman.so.1."  To fix this, 
either add `/usr/local/lib` to `LD_LIBRARY_PATH`, or modify `/etc/ld.so.conf` or 
`/etc/ld.so.conf.d` to add `/usr/local/lib`, then run: sudo ldconfig

Extract process attributes

2009-08-07T13:27:00.003-07:00

Handy one liner scripts to extract various process attributes. You can add additional attributes to the 'ps' command to get other attributes. Below are some samples

-- Process start hour
ps -eo pgid,lstart,cmd | grep "[p]name" | awk '{print $5}' | cut -d\. -f1

-- Process start minute
ps -eo pgid,lstart,cmd | grep "[p]name" | awk '{print $5}' | cut -d\. -f2

--process start day
ps -eo pgid,lstart,cmd | grep "[p]name" | awk '{print $4}'

command to kill a process group

2009-08-07T12:48:00.001-07:00

If pid is negative but not -1, the signal is sent to all processes whose process group ID is equal to the absolute value of pid. The negative pid is specified in this way:

kill -s KILL -- -nn

where nn is the process group ID and may have a range of 2 to 7 digits (nn to nnnnnnn).

kill -s KILL -- -9812753

The format must include the – – – before the nn in order to specify the process group ID

Ubuntu 9.04 resets resolution @ restart

2009-06-06T18:10:00.004-07:00

Spent quite some time today trying to make this work. Installed Ubuntu 9.04 this AM and as expected it was not able to correctly detect the monitor resolution. Installed the latest NVidia drivers (180.44) from Synaptic package and was able to set the right resolution using "nvidia-settings" app. Before saving the configuration made sure that preview of "/etc/X11/xorg.conf" was showing as to what I had set. Restarted the system and the resolution was getting set back to "800x600". Applying a resolution using 'nvidia-settings" was still working in the session however @ logout/restart it would revert back to "800x600". Here's what all I tried and the final step that worked for me:

1) The first question I had was whether nvidia drivers were being loaded at restart or not. One of the Ubuntu knowledge base suggested adding "nvidia" to "/etc/modules" such that it gets loaded at boot time but that didn't work.

.... After many failed google searches I shifted my focus to looking at system logs for any clues......

2) checked this log file after system startup "/var/log/gdm/:0.log" and it was complaining about error loading "type2" and "freetype" modules. I wasn't sure if this was causing the "xorg.comf" to fail so I manually disabled the load of these modules in the "xorg.conf" file under the section module like below. Did a restart but still the same situation however the log "/var/log/gdm/:0.log" was not showing the error this time.

Section "Module"
Load "dbe"
Load "extmod"
# Load "type1"
# Load "freetype"
Load "glx"
EndSection

2) Check log file "/var/log/Xorg.0.log". This log contains details on "xorg.conf" loading. Watchout for any errors in this log. In my case this log looked fine except the last line in the log file that said "[nvidia]: setting resolution 800x600". This baffled me. While the earlier lines in the log file correctly stated loading the correct resolution from 'xorg.conf', this last log line almost seemed like some another process was invoking at startup that was turning the resolution back to '800x600'. Googled around but no clue.

... Took a walk at a park nearby as my brains was about to explode.

3) Lot of sites/forums had samples of "xorg.conf" files from users whose system was working fine. Tried various of those but still no luck.

4) One of the forums post suggested this method. "System --> Preferences --> Display". Choosing that option prompts a question "It appears that your graphics driver does not support the necessary extensions to use this tool. Do you want to use your graphics driver vendor's tool instead?". if you click "yes" it takes you to the "nvidia-settings" tool. If you choose the "No" option, it allows choosing a resolution via the "Display Preferences" window. I select "No" and choose the desired resolution of "1280x1024" from the list and hit "Apply". As usual it worked in the session and voila!!!! it also worked after a restart. The resolution stayed @ "1280x1024".

Obviously there were a lot of other steps that I tried in vain but these seemed worth a note as they provide some clue as to what the issue might be. Hope this post helps.

my google searched keywords: Xorg.log nvidia 800x600, Ubuntu 9.04 resets resolution restart, ubuntu 9.04 stuck at 800x600, ubuntu 9.04 low graphics mode, ubuntu 9.04 loose settings on restart, ubuntu 9.04 lost screen resolution after restart

Mac OSX Burn ISO Image

2009-06-06T12:35:00.003-07:00

Here are the steps.

1) Open Finder
2) Browse "Application/Utilities"
3) Open/double click Disk utility
4) Click "Burn" icon on top. This will ask you to choose a ISO image
5) Once selected it will ask to insert a CD and voila.

Off course its easy but when you've been using Linux and Windows utilities for years, these steps can be a little perplexing and hard to remember.

******************* screenshots **********************

Perl Mysql DBI get deleted row count

2009-01-28T10:52:00.007-07:00

Trying to capture the number of deleted rows when executing a delete statement was returning a "0E0". See below for the script details. What I found was that it was DBI's way of saying zero rows were deleted. The thing to know about DBI is that do() needs to return distinct values to distinguish the occurrence of an error from the case that no rows were affected, so it uses “undef” and “0E0”. The value “undef” indicates an error; it evaluates to false in a Boolean context. The string “0E0” indicates that no rows were affected; it evaluates to true in Boolean contexts but is treated as zero in numeric contexts. If do() returned 0 to indicate no rows were affected, that evaluates to false in a Booelan context and would be difficult from an error. You can do the following to display the row count correctly

Error:


my $deleted = $conn->do("delete from employee where year between 1901 and 1910");
print "deleted $deleted rows.\n"; 

The above would print "deleted 0E0 rows"

Solution:

 
my $deleted = $conn->do("delete from employee where year between 1901 and 1910");
 printf "deleted %d rows.\n", $deleted; 

(OR)

my $deleted += $conn->do("delete from employee where year between 1901 and 1910");
print "deleted $deleted rows.\n";

Adcenter Error Code 1514 : The Negative keywords requires partial match bid

2009-01-26T11:33:00.006-07:00

Got this error when dealing with a keyword update through the Adcenter API. Its a shame that Microsoft has no documentation on this error and none the less anywhere on the internet, hence the drive to blog this. Here's a brief background on Adcenter Negative Keywords:

http://msdn.microsoft.com/en-us/library/bb545038.aspx

So what the error basically means is that "Negative keywords are only allowed on keywords that have a broad or phrase match bid specified". If you try to add/update a keyword with "Not Keywords" and only exact bid specified then you would get this error. In my case this was being caused by a different scenario. We use Apache Axis for our webservice calls and Axis treats a blank string as a valid content string when compared to a null string. So even if the "Not Keywords" was set to blank it was generating the "Negative Keyword" soap element. See below for SOAP source. In the soap below the broad and phrase bid is specified as 0's and the "NegativeKeywords" element is constructed making MSN think that there are negative keywords associated with this keyword and hence the error. I resolved this error by setting the string to NULL instead of a blank ("''").

<?xml version="1.0" encoding="UTF-8"?>
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<soapenv:Header><ns1:ApplicationToken soapenv:actor="http://schemas.xmlsoap.org/soap/actor/next" soapenv:mustUnderstand="0" xmlns:ns1="https://adcenter.microsoft.com/api/advertiser/v5"/><ns2:DeveloperToken soapenv:actor="http://schemas.xmlsoap.org/soap/actor/next" soapenv:mustUnderstand="0" xmlns:ns2="https://adcenter.microsoft.com/api/advertiser/v5"><ns2:Value>xxxx</ns2:Value></ns2:DeveloperToken><ns3:UserCredentials soapenv:actor="http://schemas.xmlsoap.org/soap/actor/next" soapenv:mustUnderstand="0" xmlns:ns3="https://adcenter.microsoft.com/api/advertiser/v5"><ns3:Password>xxxxxx</ns3:Password><ns3:Username>xxxxx</ns3:Username></ns3:UserCredentials></soapenv:Header>
<soapenv:Body>
<UpdateKeywordsRequest xmlns="https://adcenter.microsoft.com/api/advertiser/v5"><AdGroupId>xxxxx</AdGroupId><Keywords>
<Keyword><BroadMatchBid>0.0</BroadMatchBid><ExactMatchBid>0.5</ExactMatchBid><Id>xxxxx</Id><NegativeKeywords><ns4:string xmlns:ns4="http://schemas.microsoft.com/2003/10/Serialization/Arrays"></ns4:string></NegativeKeywords><Param1>xxxx</Param1><Param2></Param2><Param3></Param3><PhraseMatchBid>0.0</PhraseMatchBid></Keyword>
<Keyword><BroadMatchBid>0.0</BroadMatchBid><ExactMatchBid>0.5</ExactMatchBid><Id>xxxxxx</Id><NegativeKeywords><ns5:string xmlns:ns5="http://schemas.microsoft.com/2003/10/Serialization/Arrays"></ns5:string></NegativeKeywords><Param1>xxxxxxx</Param1><Param2></Param2><Param3></Param3><PhraseMatchBid>0.0</PhraseMatchBid></Keyword>
</Keywords>
</UpdateKeywordsRequest>
</soapenv:Body>
</soapenv:Envelope>

<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
<s:Body><s:Fault><faultcode>s:Client</faultcode><faultstring xml:lang="en-US">Invalid client data. Check the SOAP fault details for more information</faultstring>
<detail>
<EditorialApiFaultDetail xmlns="https://adcenter.microsoft.com/api/advertiser/v5" xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
<BatchErrors>
<BatchError><Code>1514</Code><Details i:nil="true"/><Index>0</Index><Message>The Negative keywords requires partial match bid.</Message></BatchError>
<BatchError><Code>1514</Code><Details i:nil="true"/><Index>1</Index><Message>The Negative keywords requires partial match bid.</Message></BatchError>
</BatchErrors>
<OperationErrors/>
<TrackingId>xxxxxxx</TrackingId>
<EditorialErrors/></EditorialApiFaultDetail>
</detail>
</s:Fault></s:Body>
</s:Envelope>

Mysql (errno: 13) can't find file frm

2009-01-20T14:44:00.004-07:00

Recently got this error when I was trying to copy a database on to another server. I basically copied the mysql data directory for the corresponding database on to the new server and logged into mysql client.


mysql> select count(*) from placement; 
Can't find file: './main/placement.frm (errno: 13)

Checked the mysql log and this was being reported for the all frm files for all tables in the db. Looks like this is a permission issue on the data directory. Changed to the following permissions and it seemed to work.


server> sudo chmod -R 660 main
server> sudo chmod 775 main

DBD::mysql::db do failed: The used command is not allowed with this MySQL version

2009-01-15T10:22:00.011-07:00

Our nightly etl process loads a whole lot of data from local files. The recent upgrade of mysql from 5.0.45 --> 5.0.67 was throwing this error

"DBD::mysql::db do failed: The used command is not allowed with this MySQL version"

when using the mysql 'LOAD LOCAL FILE' command. We are not entirely sure if this is an issue caused by the mysql upgrade or an outdated perl DBD:Mysql module. MySQL server has the local_infile set to ON via a show variables command (see below) and user has the FILE permission to the local file. A reinstall of the DBD:Mysql module also did not resolve the issue.


mysql> show variables like 'local%';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| local_infile  | ON    | 
+---------------+-------+

Looking online we came through various links that suggested we needed to specify a option "mysql_local_infile" while connecting to mysql via the DBD:Mysql module. This seemed to resolve our issue.


use strict;
use DBI;
my $dsn = "DBI:mysql:mydb;mysql_local_infile=1"; 
my $user = "me";
my $password = "secret";
my $dbh = DBI->connect($dsn,$user,$password);

References:
http://dev.mysql.com/doc/refman/5.0/en/load-data-local.html
http://www.perlmonks.org/?node_id=728472
http://www.bigresource.com/MYSQL-LOAD-DATA-INFILE-command-to-import-a-text-f
ile-rODHiNex.html

DBD:Mysql warning statements "Use of uninitialized value in concatenation (.) or string"

2009-01-15T09:25:00.002-07:00

We recently upgraded to the latest and greatest version of the DBD:Mysql perl module and have been seeing these warning statements in our logs a lot (believe me a whole lot to the point of annoyance for developers skimming through logs for debug help).

Use of uninitialized value in concatenation (.) or string at /usr/local/perl/lib/site_perl/5.8.8/x86_64-linux-thread-multi/DBI.pm line 929.

For now we went ahead and commented the line no. 929 in the /usr/local/perl/lib/site_perl/5.8.8/x86_64-linux-thread-multi/DBI.pm file and have seen no side effects of that. A fairly weak solution but works for now.

setting up a home server

2009-01-07T17:09:00.005-07:00

I am a software guy by profession and have been lately delving into some system ops at work. Been playing around with a home server for the last month or so mostly for learning purposes. Lots of questions and searching on the internet. Came across this book that is quite useful to understand some of the basics around setting up a home network. "Home Wireless Networking in a Snap (Sams Teach Yourself) (Paperback)" (http://www.amazon.com/gp/product/0672327023). I am a linux guy and although this book is based on setting a windows server, still the router related stuff applies everywhere. Came across this book when I was trying to resolve as to why my home server won't respond to ping although ssh and ftp works. DUH!!! the router is the public interface for the home network and the settings on the router was configured to not respond to ping requests. The WAN setup page on my netgear router has a setting "Allow Pinging of Router Internet Interface" that needs to be enabled. There are some other cool stuff that can be set on the WAN setup page. See description and picture below:

Enable/Disable Firewall

Select the appropriate check box or option button to turn on your router's firewall. Some routers (such as my Netgear router) have the firewall enabled by default, and the configuration page does not provide an enable option; it only allows you to disable the firewall by selecting Disable SPI Firewall. Disabling the firewall opens up your network to the possibility of outside attack. There is actually no good reason to disable the firewall, even if access to gaming or other services is a problem, because all connectivity issues can be resolved with port triggering and port forwarding settings

Specify DMZ Server IP Address

If you want to operate a computer or a server (such as a computer that is acting as a web server or a gaming server) outside the firewall, you can have the router place that computer in the DMZ. This means that the network is still protected from attack but that the DMZ computer could potentially be attacked. To place a computer on your network in the DMZ using a Netgear router, select the Default DMZ Server check box and then enter the IP address of the computer that will be placed in the DMZ. The DMZ isn't really a place; it is a virtual location configured by your WiFi router's firewall. The DMZ is a virtual place that resides between your protected internal network and the public Internet. Placing a computer in the DMZ allows it to communicate with the Internet without the router's firewall inspecting the data flowing to and from the computer. It is not uncommon for computers offering certain services to be placed in the DMZ. Even large corporations sometimes place communication servers in the DMZ so that they do not have to open ports on the firewall to allow access to the server.

Allow Pinging of Router Internet Interface

By default, most WiFi routers are configured so that the router's interface or connection to the Internet cannot be pinged. The Internet interface for you router is actually assigned its IP address by your Internet service provider. So the Internet interface on the router is really its public interface. Allowing the public interface to be pinged can open the router up to attack since it can be "pinged to death." A malicious individual on the Internet could send a barrage of ping packets or oversized ping packets that would actually bring down the router's public interface. This kind of attack is called the "Ping of Death." Enable the router's Internet interface for pinging only if your Internet service provider (or you) needs to ping that interface to determine whether there is a connectivity problem. For my Netgear router, I select the Respond to Ping on Internet Port option to turn on this feature. When you have determined that the interface can be reached by a ping (from you or the ISP technician), I suggest that you disable the feature.

Set MTU Size

The Maximum Transmit Unit (MTU) value for Ethernet networks such as your WiFi network is 1500 bytes. Leave the MTU setting at the default unless your Internet service provider requires that a different setting be used. If you're unsure about the MTU value, contact your ISP. To change the MTU on my Netgear router, I click in the MTU text box and type a different value. Each router provides a slightly different configuration screen for setting the MTU. Your Internet service provider determines the optimal MTU for the network it services by trial and error. The only way you might perceive that you don't have the correct MTU setting for your ISP connection would be a slight slowing of the connection to the Internet—and this would only be in situations where your MTU is set higher than the ISP's and your data packets have to be broken into smaller chunks for transmission. So, bottom line, call your ISP and see whether it uses a special MTU setting.

Mysql increase key length

2008-12-29T08:58:00.003-07:00

We have recently been converting most of our databases from latin to UTF-8 and have encountered issues with smaller Mysql default keylength. Mysql by default allows a max key length of 1024 size. While it was quite a bit of work to change the current data to confirm to the key length, the easier way was to recompile the Mysql source with a custom key length. Here I will detail the steps to do that:


wget http://www.percona.com/mysql/5.0.67-b7/source/mysql-5.0.67-percona-b7-src.tar.gz
tar zxf mysql-5.0.67-percona-b7-src.tar.gz

Now you'll want to make the edits to the myisam.h file to use our longer key length instead of the default 1000 byte keylength. Below is a compare of the file before and after change:


--- /tmp/myisam.h       2008-05-28 13:36:22.000000000 -0700
+++ include/myisam.h    2007-03-15 14:15:41.000000000 -0700
@@ -48,12 +49,12 @@
 #define MI_MAX_KEY                  MAX_INDEXES         /* Max allowed keys */
 #endif
 
-#define MI_MAX_POSSIBLE_KEY_BUFF    (1024+6+6)      /* For myisam_chk */
+#define MI_MAX_POSSIBLE_KEY_BUFF    (4096+6+6)      /* For myisam_chk */
 /*
   The following defines can be increased if necessary.
   But beware the dependency of MI_MAX_POSSIBLE_KEY_BUFF and MI_MAX_KEY_LENGTH.
 */
-#define MI_MAX_KEY_LENGTH           1000            /* Max length in bytes */
+#define MI_MAX_KEY_LENGTH           4000            /* Max length in bytes */
 #define MI_MAX_KEY_SEG              16              /* Max segments for key */
 
 #define MI_MAX_KEY_BUFF  (MI_MAX_KEY_LENGTH+MI_MAX_KEY_SEG*6+8+8)

Next you'll need to run configure / make install


./configure --prefix={choose a name here like /tmp/mysql_mod}
sudo make install

mysql timeout issue

2008-12-24T14:30:00.002-07:00

The sys admins had recently changed the timeout on the firewalls to a fairly low value to deal with the excessive connections issue. This was causing an issue with the mysql servers as the nightly etl and scripts were failing with stale connection issue. Googled around to find a way to set the connection timeout in mysql settings but couldn't find any lead. One of our sys admins suggested we change the tcp timeout on the mysql boxes and that seems to have resolved the issue. here's the command:

sudo /sbin/sysctl net.ipv4.tcp_keepalive_time=90

Fairly Beginner Programming Excercises

2008-11-26T14:32:00.003-07:00

I have been trying to teach programming to one of my friend and occasionally I search online for fun projects which I can give it to him as programming exercises. I will try to document the ones I gave him with notes on what I intended him to learn from the exercise:

******* Loops and Constructs ********

1) Project 1: Print a pyramid. Read (in a loop of course) a number, n, and print a pyramid that has that size. The example below shows what would be printed for n=4.

*
* *
* * *
* * * *

2) Project 2: Printing prime numbers: This is more sort of a classic programming problem and involves knowledge of basic programming constructs such as conditions, looping etc.

3) Project 3: Printing the first n fibonacci numbers. This is again a classic programming problem and involves knowledge of basic programming constructs.

******* Arrays ***********

1) Project 1: Finding missing elements in a array. So basically a array is provided that has certain missing elements like numbers between 1 - 99. The task here is to find those missing numbers from the array. This project basically requires knowledge of arrays and array traversal.

2) Project 3: Sorting a number array: This is again a classic programming problem and can be done in various ways. Knowledge gained here is again arrays.

******* Hash *********

1) Project 1: Parse a file and display words that occur more than n times. Not a requirement that you use a hash here but ideally you would.

2) Project 2:

C# Web Request Using Client Certificate

2008-10-06T18:56:00.005-07:00

Recently worked on this project where a service was accessible over http via client certificate. This is no biggie however took me 4-5 hrs to figure it out as the issue was not with code associated with importing client certificate but with another portion of the code that I overlooked. You know how it goes, stupid mistakes here and there and boom there goes your day. Thinking that the error was with the client certificate code I searched through the anals of Google pages to figure it out and various attempts/learnings made a worth to blog out. BTW kudos to the Fiddler (http://www.fiddlertool.com/fiddler/) tool which allowed me to sniff the http outgoing traffic helping me debug the issue.

So the http service I was trying to access is a secure one and my org was issued a client certificate. There are three ways that I tried to attach a client certificate with the request.

1) Providing direct links to cert files: You can do this via this sample code. Pretty self explanatory. The additional cert you see below is the intermediate certificate agency cert that I had to additionally download.



X509Certificate2Collection certCollect = new X509Certificate2Collection();

X509Certificate2 cert = new X509Certificate2(@"C:\Users\...\x509_verisign-certificate.cer");

X509Certificate2 cert1 = new X509Certificate2(@"C:\Users
\..\Certificate_From_SErvice.cer.pfx", "{client certificate password}");

certCollect.Add(cert);

certCollect.Add(cert1);

2) Exported combined cert file (.cer.p7b format): If you have imported the client/intermediate certificate onto your computer then you can export combined certificate (with all intermediariy certificate) + private key into a .cer.p7b file format. You can use this one file instead of the two used above. Here the code to make it work:



X509Certificate2Collection certCollect = new X509Certificate2Collection();

FileInfo file = new FileInfo(@"C:\Users\...\exported_certificate.cer.p7b");

BinaryReader br = new BinaryReader(File.OpenRead(file.FullName));

byte[] raw = br.ReadBytes((int)file.Length);

SignedCms cms = new SignedCms();

cms.Decode(raw);

certCollect.AddRange(cms.Certificates);

3) Referencing a imported certificate on the workstation: You can reference a imported certificate using the below code. There a various ways to uniquely identify a certificate using 'subject name' or 'serial id' etc. I have accessed it via the subject name.



X509Store store = new X509Store("MY", StoreLocation.CurrentUser);

store.Open(OpenFlags.ReadOnly | OpenFlags.OpenExistingOnly);

X509Certificate2Collection collection = (X509Certificate2Collection)store.Certificates;

X509Certificate2Collection fcollection = 
(X509Certificate2Collection)collection.Find(X509FindType.FindBySubjectName, "{certificate name}", false);

X509Certificate2Collection icollection = 
(X509Certificate2Collection)collection.Find(X509FindType.FindBySubjectName, "VeriSign
Class 1 Individual Subscriber CA - G2", false);

X509Certificate2Enumerator certEnum = fcollection.GetEnumerator();

certEnum.MoveNext();

X509Certificate2 primCertFromStore = certEnum.Current;

certEnum = icollection.GetEnumerator();

certEnum.MoveNext();

X509Certificate2 issuerCertFromStore = certEnum.Current;

certCollect.Add(primCertFromStore);

certCollect.Add(issuerCertFromStore);
            
// use the below code to test if the retrieved certificate is the one you intend to.
//X509Certificate2UI.DisplayCertificate(primCertFromStore);
//X509Certificate2UI.DisplayCertificate(issuerCertFromStore);
store.Close();

You can add the certificate to the request using the below code sample:



....
.......
// Handle any certificate errors on the certificate from the server.
string uri = "request uri";

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uri);

request.ProtocolVersion = HttpVersion.Version11;

request.KeepAlive = true;    
                              
request.Method = "POST";
            
request.Accept = "*/*";      
     
request.ClientCertificates.AddRange(certCollect); 
                       
string str = "POST DATA";

byte[] postBytes = Encoding.UTF8.GetBytes(str);

request.ContentType = "TEXT/XML";

request.UserAgent = "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 
2.0.50727; .NET CLR 1.1.4322; InfoPath.1)";
            
request.ContentLength = postBytes.Length;

Stream requestStream = request.GetRequestStream();

requestStream.Write(postBytes, 0, postBytes.Length);

requestStream.Close();

HttpWebResponse response = (HttpWebResponse)request.GetResponse();

MYSQL duplicate entry error with a non-unique index/key

2008-09-25T15:25:00.002-07:00

This was a strange one. A insert was erroring out due to duplicate entry error on a key that is non -unique. The error was: ERROR 1062 (23000): Duplicate entry '445982' for key 2. Did a check table to check if everything was alright and there it was: an error with the table. Repair quick did not work so had to do a full repair. See below log for query statements:

Here's the log:



mysql> show create table tableA;

......
.....
.....
 PRIMARY KEY  ("category_id"),
  KEY "ix_campaign_category_unpacked_root" ("root_campaign_id"),
  KEY "ix_campaign_category_client" ("client_id","site_id"),
  KEY "ix_campaign_category_site" ("site_id","client_id")
) ENGINE=MyISAM DEFAULT CHARSET=utf8 

-- Error from insert

mysql> ERROR 1062 (23000): Duplicate entry '445982' for key 2

-- check table result

mysql> check table tableA;
+----------------------------------+-------+----------+--------------------------------------------------------------+
| Table                            | Op    | Msg_type | Msg_text                                                     |
+----------------------------------+-------+----------+--------------------------------------------------------------+
| tableA | check | warning  | Table is marked as crashed                                   |
| tableA | check | warning  | 2 clients are using or haven't closed the table properly     |
| tableA | check | warning  | Size of datafile is: 34234576       Should be: 34234400      |
| tableA | check | warning  | Found 0 deleted space in delete link chain. Should be 361132 |
| tableA | check | error    | Found 0 deleted rows in delete link chain. Should be 2385    |
| tableA | check | error    | record delete-link-chain corrupted                           |
| tableA | check | error    | Corrupt                                                      |
+----------------------------------+-------+----------+--------------------------------------------------------------+
7 rows in set (0.00 sec)
 
-- Repair table attempts

mysql> repair local table tableA quick;
+----------------------------------+--------+----------+-------------------------------------------------------------------------------+
| Table                            | Op     | Msg_type | Msg_text                                                                      |
+----------------------------------+--------+----------+-------------------------------------------------------------------------------+
| tableA | repair | error    | Couldn't fix table with quick recovery: Found wrong number of deleted records |
| tableA | repair | error    | Run recovery again without -q                                                 |
| tableA | repair | status   | OK                                                                            |
+----------------------------------+--------+----------+-------------------------------------------------------------------------------+
3 rows in set (9.64 sec)

mysql> repair local table tableA;
+----------------------------------+--------+----------+----------+
| Table                            | Op     | Msg_type | Msg_text |
+----------------------------------+--------+----------+----------+
| tableA | repair | status   | OK       |
+----------------------------------+--------+----------+----------+
1 row in set (2.43 sec)

Mysql duplicate entry error with auto-increment column

2008-09-16T11:45:00.003-07:00

I have been looking at this issue with one of our tables where we were getting a duplicate entry error on key 1 which is basically the primary key on the table. The primary key field is set to auto-increment and was puzzling me as to why it was causing the duplicate entry. Ran the checks to ensure the values were right (see checks below):

Someone suggested to alter the auto-increment value to a higher value and that would reset the next value. So I did a alter statement which took a while as the table was huge. The query to check the next ID value came out correctly (see below for query)

However this still did not resolve the issue. Finally looked in the direction of index and turns out that index might have been corrupted. Ran a repair table to check index and that fixed the value. (see below for query)




*********** checks ********************

--   select max(id) from tableA;
returned: 70809185

--  insert into tableA values (xxxxxx);
Error:  Duplicate entry '70809186' for key 1

--  select * from tableA where id=70809186;
Returned 0 rows

************** alter index **************

-- query to change the next increment value
 ALTER TABLE tbl AUTO_INCREMENT = 70809190;

-- find out whats the next increment value
 SELECT Auto_increment FROM information_schema.tables WHERE table_name='tableA';

************* repair index *****************

--  repair table tableA quick;
quick option repairs the indexes

mysql duplicate key error due to sequence issue

2008-09-03T11:19:00.003-07:00

I recently had to deal with this error so I am blogging it here so the rest can use.

Error: On one of our tables we kept getting a duplicate key exception on insets. The puzzling thing was that the value actually did not exist in the table. Here's a low down:



--- Table definition
CREATE TABLE "job_message" (
  "message_id" bigint(20) NOT NULL auto_increment,
  "run_id" int(11) NOT NULL,
  "object_id" bigint(20) default NULL,
  "date" datetime NOT NULL,
  "priority" tinyint(2) NOT NULL,
  "type" varchar(32) default NULL,
  "msg" varchar(255) NOT NULL,
  "stack_trace" varchar(255) default NULL,
  PRIMARY KEY  ("message_id"),
  KEY "ix_job_message" ("run_id","object_id","date"),
  KEY "ix_job_message_object_id" ("object_id"),
  KEY "ix_job_message_dt" ("date")
) 


-- proof that row did not exist
mysql> select max(message_id) from job_message_200836;
+-----------------+
| max(message_id) |
+-----------------+
|       753222920 |
+-----------------+
1 row in set (0.00 sec)


-- error with insert statement
mysql> insert into job_message (run_id, object_id, date, priority, msg) values (1279408,
 22231896, '2008-09-02 15:53:40', 4, 'test');

ERROR 1062 (23000): Duplicate entry '753222922' for key 1

Thats what was puzzling.

Solution:

Did a lot of googling around but got nothing. I did get some hint that the issue might be with the sequences and that the sequence is not updated but got no clue on how to resolve the issue. Out of desperation I tried to see if the table was healthy and voila there it was. The table was corrupted. The solution was to either repair the table or get a copy from the slave. I tried the repair and it worked fine. I used local in the repair command as I did not want the repair to propagate to slave as the slave copy was fine. See commands below to check and repair a table.



-- check table
mysql> check table job_message_200836;
+-------------------------+-------+----------+-----------------------------------------------------------+
| Table                   | Op    | Msg_type | Msg_text                                                  |
+-------------------------+-------+----------+-----------------------------------------------------------+
| main.job_message_200836 | check | warning  | Size of datafile is: 352975060       Should be: 352974784 |
| main.job_message_200836 | check | error    | Found 3189125 keys of 3189124                             |
| main.job_message_200836 | check | error    | Corrupt                                                   |
+-------------------------+-------+----------+-----------------------------------------------------------+
3 rows in set (0.69 sec)


-- repair table command
mysql> repair local table job_message_200836;
+-------------------------+--------+----------+------------------------------------------------+
| Table                   | Op     | Msg_type | Msg_text                                       |
+-------------------------+--------+----------+------------------------------------------------+
| main.job_message_200836 | repair | warning  | Number of rows changed from 3189124 to 3189125 |
| main.job_message_200836 | repair | status   | OK                                             |
+-------------------------+--------+----------+------------------------------------------------+
2 rows in set (29.93 sec)

TwinView option disabled in Nvidia-Settings

2008-07-27T18:26:00.002-07:00

I am trying to configure dual monitors on my Ubuntu hardy however the 'Twinview' option is disabled in my settings. To enable this option you would have to tweak the 'xorg.conf' file. First of make a backup of the current '/etc/X11/xorg.conf' so you can always revert to this copy. Here are the instructions to modify the file to enable 'Twinview'

- sudo gedit /etc/X11/xorg.conf
- Search for line which says "Twinview" "0". Change the "0" to "1"
- sudo nvidia-settings (now the nvidia driver should automatically detect the second connected screen)

In looking online I also see this option however it did not work for me:

- nvidia-xconfig --twinview
and restarting your X server.

There is a nice article on configuring twinview on the NVidia site: http://us.download.nvidia.com/XFree86/Linux-x86/1.0-9755/README/appendix-g.html

Wicd won't automatically connect to wireless network

2008-07-27T18:20:00.002-07:00

I have been using Wicd on my Ubuntu for quite a while and never really bothered as to why it would not automatically connect to a specified network. Wicd provides an option to specify whether to automatically connect to a network. I finally figured out. You need to add Wicd to your system startup services for it to automatically connect. Here are the steps to do that:

To get the tray icon to automatically appear at boot, go to System > Preferences > Sessions. In the "Startup Programs" tab, click the "New" button. Give it a name ("Wicd" works fine). For the command, enter "/opt/wicd/tray.py"

Ubuntu: Installing NVidia video driver

2008-07-17T19:48:00.002-07:00

I have a Dell Latitude D630 laptop with a NVidia Quadro NVS 135M video card. I recently struggled with installing the driver for this baby on Ubuntu 8.04. I will detail the various solutions I tried and finally the one that worked for me. I have always had issues with this driver however the desire to install the Awant window manager propelled me to deal with this issue. Awant requires that advanced desktop effects be enabled which in turn required a 3D compatible video card. I previously manually modified the xorg.conf to deal with the driver issues but it won't work when I enabled the restricted driver for this card.

The first solution is to simply enable the restricted driver and check if it works for you. Basically under System --> Administration --> Hardware Drivers. My issue was that resolution detection just won't work. Even if I manually chose the card type and monitor type the resolution would default to 800x600. This pointed me to reinstall the packages involved

sudo apt-get install --reinstall linux-restricted-modules-$uname -r} nvidia-glx-new

Re-enable the driver after doing the above step and it still won't detect my resolution. One of the sites suggested using the 'nvidia-xconfig' to generate the 'xorg.conf' and tweak it further via the 'nvidia-settings' util

sudo nvidia-xconfig --> this creates a new xorg.conf file
sudo nvidia-settings --> this is used to tweak the settings

The settings won't show my resolution options as the driver was not correctly installed.

So I looked at some other options and came across this article around installing a proprietary driver downloaded from NVidia site. You can download the latest driver from here:

http://www.nvidia.com/object/unix.html

Here are the instructions to install this driver:

For the driver to install the XServer must be stopped.

cntrl + alt + backspace
sudo /etc/init.d/gdm stop
sudo sh {installed driver}
sudo /etc/init.d/gdm start

I did a restart and upon boot up fired the 'nvidia-settings' on terminal which gave me the options to set the resolution. Hope it helps!!!!

Error installing windows security updates on upgrade from Windows XP Home to Professional

2008-07-12T09:30:00.002-07:00

So I recently upgraded my windows XP Home version to Professional in need of IIS web
server. XP Home does now allow IIS installation (there are various hacks mentioned
on websites but I could not make it to work). The issue was that my windows automatic updates which were working totally fine before the upgrade, started erroring out after the upgrade. The updates were getting downloaded however they won't install. I had a queue of over 90 updates to be installed. The error on the installation screen was unhelpful just saying "Updates were not installed". My further analysis pointed me to the "windows update log file".

C:\WINDOWS\WindowsUpdate.log

To access the log file easily do:
Click Start > Run...
Type/paste %windir%\windowsupdate.log in the "Open:" field
Click OK

So in my logs I found this error (see below for stack trace). Re-Register the Windows Update dll files and that did the trick.

Run the following on command prompt:

regsvr32.exe c:\windows\system32\wuweb.dll
regsvr32.exe c:\windows\system32\wups2.dll
regsvr32.exe c:\windows\system32\wups.dll
regsvr32.exe c:\windows\system32\wucltui.dll
regsvr32.exe c:\windows\system32\wuaueng1.dll
regsvr32.exe c:\windows\system32\wuaueng.dll
regsvr32.exe c:\windows\system32\wuapi.dll

The windows update will automatically restart after a while and shows up the queued updates in system tray. Works from there.

Stacktrace
2008-07-09 22:18:22:703 3812 a78 Misc =========== Logging initialized (build: 7.0.6000.381, tz: -0700) ===========
2008-07-09 22:18:22:703 3812 a78 Misc = Process: C:\WINDOWS\system32\wuauclt.exe
2008-07-09 22:18:22:703 3812 a78 AUClnt FATAL: Error: 0x80004002. wuauclt handler: failed to spawn COM server
2008-07-09 22:18:22:718 1248 8c0 Handler FATAL: 0x80004002: ERROR: Remote update handler container process created (PID: 3812), but exited before signaling event
2008-07-09 22:18:25:906 1248 8c0 Agent * WARNING: Exit code = 0x80004002
2008-07-09 22:18:25:906 1248 8c0 Agent *********
2008-07-09 22:18:25:906 1248 8c0 Agent ** END ** Agent: Installing updates [CallerId = AutomaticUpdates]
2008-07-09 22:18:25:906 1248 e5c AU >>## RESUMED ## AU: Installing update [UpdateId = {707E6500-81C1-4165-8C43-631E09891716}]
2008-07-09 22:18:25:906 1248 8c0 Agent *************
2008-07-09 22:18:25:906 1248 e5c AU # WARNING: Install skipped
2008-07-09 22:18:25:906 1248 8c0 Agent WARNING: WU client failed installing updates with error 0x800040022008-07-09 22:18:25:906 1248 e5c AU >>## RESUMED ## AU: Installing update [UpdateId = {DD167F04-69E8-46A0-83DA-D4A5143EDBF1}]
2008-07-09 22:18:25:906 1248 e5c AU # WARNING: Install skipped
2008-07-09 22:18:25:906 1248 e5c AU >>## RESUMED ## AU: Installing update [UpdateId = {7CB0B704-B2FF-4D5C-B202-B5BA340B7007}]
2008-07-09 22:18:25:906 1248 e5c AU # WARNING: Install skipped
2008-07-09 22:18:25:906 1248 e5c AU >>## RESUMED ## AU: Installing update [UpdateId = {73F11106-7351-465D-9BA5-2881436C0D5A}]
2008-07-09 22:18:25:906 1248 e5c AU # WARNING: Install skipped
2008-07-09 22:18:25:906 1248 e5c AU >>## RESUMED ## AU: Installing update [UpdateId = {383B0108-D3E2-4F2A-9E32-7B7B599BB0F7}]

converting inputstream to octetstream

2008-01-27T16:33:00.000-07:00

The other day I was working with soap attachments and stumbled across OctetStream. Had no idea on this stream type which made me lookaround and came up with this info:

org.apache.axis.attachments.OctetStream corresponds to mime type "application/octetstream"

-- Converting Inputstream to OctetStream

ByteArrayOutputStream baos = new ByteArrayOutputStream();
int byte1 = -1;
while ((byte1 = in.read()) != -1)
baos.write(byte1);
return new OctetStream(baos.toByteArray());

HtmlParser: Parsing by tag attribute

2007-11-06T18:04:00.000-07:00

HTML Parser is a Java library used to parse HTML in either a linear or nested fashion. I have tried various open source parsers like WebHarvest etc and found this one to be the most robust when handling bad and nasty html. My primary purpose in using a parser is to extract content from websites. Other people have other needs from the parser which this might not suffice. It has some pretty cool features like filters, visitors, custom tags and easy to use JavaBeans. It is a fast, robust and well tested package. One of the drawbacks is on the documentation front. There is minimal documentation around and most of the stuff I discovered is by playing around.

So my need here was to be able to extract content from a given tag and the way to identify the tag is by using its ID field. For instance I want to extract the text "some text two" from the below page:

<html><body><div id='one'> some text one </div> <div id='two'> some text two </div></body></html>

Here's the code sample to accomplish this:


import org.htmlparser.Parser;
import org.htmlparser.beans.StringBean;
import org.htmlparser.filters.AndFilter;
import org.htmlparser.filters.HasAttributeFilter;
import org.htmlparser.filters.NodeClassFilter;
import org.htmlparser.tags.Div;
import org.htmlparser.util.NodeList;

....

Parser parse = new Parser("[[url here]]");
// if you have html you can alternativey use the parse.setInputHTML method
NodeList lstNodes = parse.extractAllNodesThatMatch (
     new AndFilter (new NodeClassFilter (Div.class), new HasAttributeFilter ("id")));
if (lstNodes != null && lstNodes.size() > 0)
{
   Div tag = null;
   for (int itr=0; itr<lstNodes.size(); itr++)
   {
      tag = (Div)lstNodes.elementAt(itr);
      String idAttribute = tag.getAttribute("id");
      if (idAttribute != null && idAttribute.equals("two"))
      {
          // this will print the div html <div id='two'> some text two </div>

          System.out.println(tag.toPlainTextString());
          // now I need to extract the text from this div tag
          Parser tagParser = new Parser();
          tagParser.setInputHTML(tag.toHtml());
          StringBean sb = new StringBean ();
          tagParser.visitAllNodesWith (sb);
          System.out.println(sb.getStrings ()); // this will print the content "some text two"
      }
  }
}

Resolving hibernate Duplicate Entry Issue

2007-08-17T13:09:00.000-07:00

So the other day I encountered this issue with hibernate duplicate entry. The error is usually of this sort (See exception stack trace below. We use Spring with hibernate in our project and that explains the DataIntegrityViolationException? being thrown here. Spring categorizes the black box SQLException to more granular level sql exceptions).

I encountered this error when running my testcase. The situation looked fairly simple to resolve at first sight. The exception pointed me to the erroring model "save" call in the code and all I had to do was to ensure that the duplicated key being reported did not already exist in the corresponding model db table. However the model table actually did not contain the erroring key. Then I tried to flush my session in desperate aim to put the blame on hibernate. That did not resolve the issue.

In looking at detail I noticed that there were several other model inserts happening before the erroring model "save" call and given the fact that these objects are related to each other, any of the previous/related inserts might be contributing to this problem. The erroring model was further tough to track becuase the testcase initially cleans all the test related tables in the test database so pretty much all tables were starting with a key 1. So it pretty much boiled down to identifying the erroring model.

I enabled the hibernate logs to print the executing SQL statements (so basically in your log4j properties you can add this line (log4j.logger.net.sf.hibernate.SQL=DEBUG) and this will spit out the executing native sql queries to the root appender). By doing this I was able to find the exact insert statement that was causing the issue. In fact the table erroring out was not being cleaned before the test case got executed and that caused issue with existing data in that table.

Exception stack trace from duplicate entry key issue:

12:37:30,969 WARN  [main] JDBCExceptionReporter.logExceptions(38) - SQL Error: 1062, SQLState: 23000
    [junit] 12:37:30,970 ERROR [main] JDBCExceptionReporter.logExceptions(46) - Duplicate entry '1' for key 1
    [junit] 12:37:30,971 WARN  [main] JDBCExceptionReporter.logExceptions(38) - SQL Error: 1062, SQLState: 23000
    [junit] 12:37:30,971 ERROR [main] JDBCExceptionReporter.logExceptions(46) - Duplicate entry '1' for key 1
    [junit] 12:37:30,973 ERROR [main] JDBCException.<init>(38) - Could not execute JDBC batch update
    [junit] java.sql.BatchUpdateException: Duplicate entry '1' for key 1
    [junit] at com.mysql.jdbc.ServerPreparedStatement.executeBatch(ServerPreparedStatement.java:647)
    [junit] at org.apache.commons.dbcp.DelegatingStatement.executeBatch(DelegatingStatement.java:294)
    [junit] at net.sf.hibernate.impl.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:54)
    [junit] at net.sf.hibernate.impl.BatcherImpl.executeBatch(BatcherImpl.java:126)
    [junit] at net.sf.hibernate.impl.SessionImpl.executeAll(SessionImpl.java:2440)
    [junit] at net.sf.hibernate.impl.SessionImpl.executeInserts(SessionImpl.java:2329)
    [junit] at net.sf.hibernate.impl.SessionImpl.doSave(SessionImpl.java:884)
    [junit] at net.sf.hibernate.impl.SessionImpl.doSave(SessionImpl.java:865)
    [junit] at net.sf.hibernate.impl.SessionImpl.saveWithGeneratedIdentifier(SessionImpl.java:783)
    [junit] at net.sf.hibernate.impl.SessionImpl.save(SessionImpl.java:746)
    [junit] at net.sf.hibernate.impl.SessionImpl.saveOrUpdate(SessionImpl.java:1396)
    [junit] at org.springframework.orm.hibernate.HibernateTemplate$12.doInHibernate(HibernateTemplate.java:583)
    [junit] at org.springframework.orm.hibernate.HibernateTemplate.execute(HibernateTemplate.java:357)
    [junit] at org.springframework.orm.hibernate.HibernateTemplate.saveOrUpdate(HibernateTemplate.java:580)
[junit] Hibernate operation: Could not execute JDBC batch update; SQL []; Duplicate entry '2' for key 1; nested exception is java.sql.BatchUpdateException: Duplicate entry '1' for key 1
    [junit] org.springframework.dao.DataIntegrityViolationException: Hibernate operation: Could not execute JDBC batch update; SQL []; Duplicate entry '1' for key 1; nested exception is java.sql.BatchUpdateException: Duplicate entry '1' for key 1
    [junit] java.sql.BatchUpdateException: Duplicate entry '1' for key 1
    [junit] at com.mysql.jdbc.ServerPreparedStatement.executeBatch(ServerPreparedStatement.java:647)
    [junit] at org.apache.commons.dbcp.DelegatingStatement.executeBatch(DelegatingStatement.java:294)
    [junit] at net.sf.hibernate.impl.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:54)
    [junit] at net.sf.hibernate.impl.BatcherImpl.executeBatch(BatcherImpl.java:126)
    [junit] at net.sf.hibernate.impl.SessionImpl.executeAll(SessionImpl.java:2440)
    [junit] at net.sf.hibernate.impl.SessionImpl.executeInserts(SessionImpl.java:2329)
    [junit] at net.sf.hibernate.impl.SessionImpl.doSave(SessionImpl.java:884)
    [junit] at net.sf.hibernate.impl.SessionImpl.doSave(SessionImpl.java:865)
    [junit] at net.sf.hibernate.impl.SessionImpl.saveWithGeneratedIdentifier(SessionImpl.java:783)
    [junit] at net.sf.hibernate.impl.SessionImpl.save(SessionImpl.java:746)
    [junit] at net.sf.hibernate.impl.SessionImpl.saveOrUpdate(SessionImpl.java:1396)
    [junit] at org.springframework.orm.hibernate.HibernateTemplate$12.doInHibernate(HibernateTemplate.java:583)
    [junit] at org.springframework.orm.hibernate.HibernateTemplate.execute(HibernateTemplate.java:357)
    [junit] at org.springframework.orm.hibernate.HibernateTemplate.saveOrUpdate(HibernateTemplate.java:580)

eclipse debugging tomcat using jpda issues

2007-08-11T02:21:00.000-07:00

So the other day I was getting this issue of "Connection refused" when trying to connect my eclipse debugger to the tomcat localhost with jpda enabled. Try these suggestions:

1) some sites online suggest to add this line at the end of the startup.bat file

...
set JPDA_TRANSPORT=dt_socket
set JPDA_ADDRESS=8000
call "%EXECUTABLE%" jpda start %CMD_LINE_ARGS%
...

The above setting may not work because the values are overidden in the catalina.bat file. So the best way would be to change the values in the catalina.bat file as following

.....
if not ""%1"" == ""jpda"" goto noJpda
set JPDA=jpda
if not "%JPDA_TRANSPORT%" == "" goto gotJpdaTransport
set JPDA_TRANSPORT=dt_socket
:gotJpdaTransport
if not "%JPDA_ADDRESS%" == "" goto gotJpdaAddress
set JPDA_ADDRESS=8000
:gotJpdaAddress
....

2) First ensure that the jpda is listening at the specified port. When tomcat starts with joda enabled it spits out a line of this sort "Listening for transport dt_socket at address: 8000". This line ensures that jpda is started. Make sure you are specifying the similar attributes to eclipse. My eclipse version only allows a connection type of socket so dt_socket was the only option for me.

3) On Windows it may indicate that the Windows Firewall is blocking the connection (it may not notify you of the blockage, even if configured to do so). Go to Control Panel / Windows Firewall / Exceptions / Add Port, and set Name = remote
debugging, Port = 8000, TCP.

Agile Management of Database Changes

2007-07-21T23:31:00.000-07:00

The last couple of weeks I have been researching on a better way to manage our application database changes. We have a small size team (8-9 persons) working on the application and we have a hard time managing the database changes. The way we handle the changes currently is not the most elegant approach but is followed in many team environments.

In the current system we store the database changes as sql statements in source control categorized by the changed component such as tables, sequences, indexes etc. So we basically have a "db" folder under source control containg separate folders for various DB components such as tables, sequences, indexes etc. For each new DB component a new sql file is created and is stored in the corresponding db component folder and changes to existing DB components are added to corresponding DB component creation script.

Here are some of the issues with our current approach:

1) Hard to compare db changes between build versions

2) No elegant way to release db changes to production/staging/other team member dev boxes. In order to push the db changes to production system @ build time, a separate changes.sql file is created and run at release time. Sometimes for developers to catch up with current db state, they need to go through a maze of sql files and manually determine the changes that need to be synched.

3) Hard to switch database to a previous release version. Database changes must be correctly segregated to accomplish the above scenario.

Looking around in search of an elegant approach I stumbled across this concept of agile databases. Of many things that constitute the core principals of agile databases, incremental change and easy change management is one of them. So I found this tool called migratDB that can be used to manage db changes.

MigrateDB is a simple xml based solution that applies all the changes defined by you in the XML file. For each sql change, you provide a pre-condition or test condition for the execution of the sql and the SQL is executed when the condition is met. The tool provides command line support and also gives ANT integration.

Here are some of the salient features (taken from migratDB):

1) Allows construction of a database at a particular version
2) Allows migration from an existing database to a later version
3) Human readable format for releases
4) Release ‘action’ available on multiple environments (i.e., various operating systems) allowing development on a different platform than production
5) Provides complete history of changes for each database object
6) The source code can be branched and merged
7) Allows multiple developers to work with/on the same database source code, at the same time
8) Supports an ‘automated build’ / ‘continuous integration’ enviroment

Here's the setup I played around with:

Database: Postgres

Sample changes.xml file:


<project> 



  <!-- test & action to create a new employee table -->
  <change>
    <sqltest exists="false">
       select * from information_schema.tables where table_name = 'employee';
    </sqltest>
    <sqlaction>
      CREATE TABLE employee (
      emp_id int4 NOT NULL DEFAULT nextval('employee_empId_seq'::text),
      first_name VARCHAR(12) NOT NULL,
      middle_initial CHAR(1) NOT NULL,
      last_name VARCHAR(15) NOT NULL,
      dept_id int4,
      phone_number CHAR(4),
      hire_date DATE,
      job VARCHAR(50),
      education_level int2 NOT NULL,
      sex CHAR(1) ,
      birth_date DATE ,
      salary NUMERIC ,
      bonus NUMERIC ,
      commission NUMERIC ,
      CONSTRAINT employee_emp_id_pk PRIMARY KEY (emp_id),
      CONSTRAINT employee_phone_number_ck CHECK (phone_number >= '0000' AND phone_number <= '9999')
      );
    </sqlaction>
    <sqlaction>
      ALTER TABLE employee OWNER TO pgsql;
    </sqlaction>
    <sqlaction>
      GRANT ALL ON TABLE employee TO pgsql;
    </sqlaction>
  </change>



  <!-- test & action to create a new department table -->
  <change>
    <sqltest exists="false">
       select * from information_schema.tables where table_name = 'department';
    </sqltest>
    <sqlaction>
      CREATE TABLE department (
      dept_id int4 NOT NULL DEFAULT nextval('department_dept_id_seq'::text),
      dept_name VARCHAR(36) NOT NULL,
      manager_emp_id int4,
      admin_dept_id int4,
      location VARCHAR(50),
      CONSTRAINT department_dept_id_pk PRIMARY KEY (dept_id)
      );
    </sqlaction>
    <sqlaction>
      ALTER TABLE department OWNER TO pgsql;
    </sqlaction>
    <sqlaction>
      GRANT ALL ON TABLE department TO pgsql;
    </sqlaction>
 </change>



  <!-- test & action to create a new referential integrity -->
  <change>
    <sqltest exists="false">
      SELECT * FROM information_schema.referential_constraints where
      constraint_name = 'department_admin_dept_id_fk';
    </sqltest>
    <sqlaction>
      alter table department add CONSTRAINT department_admin_dept_id_fk FOREIGN KEY
      (admin_dept_id) REFERENCES department(dept_id)ON DELETE SET NULL;
    </sqlaction>
  </change>




  <!-- test & action to create a new sequence -->
  <change>
    <sqltest exists="false">
      SELECT * FROM pg_catalog.pg_statio_user_sequences WHERE relname = 'department_dept_id_seq';
    </sqltest>
    <sqlaction>
      CREATE SEQUENCE department_dept_id_seq
      INCREMENT 1
      MINVALUE 0
      MAXVALUE 9223372036854775807
      START 1
      CACHE 1;
    </sqlaction>
    <sqlaction>
      ALTER TABLE department_dept_id_seq OWNER TO pgsql;
    </sqlaction>
  </change>




  <!-- test & action to create a new index -->
  <change>
    <sqltest exists="false">
      SELECT * FROM pg_catalog.pg_statio_user_indexes WHERE indexrelname = 'department_admin_dept_id_n_idx';
    </sqltest>
    <sqlaction>
      CREATE INDEX department_admin_dept_id_n_idx ON department (admin_dept_id);
    </sqlaction>
    <sqlaction>
      ALTER INDEX department_admin_dept_id_n_idx OWNER TO pgsql;
    </sqlaction>
  </change>



</project>

The above changes.xml file is stored in source control and can be used to detect database changes between versions. A custom XML parser can be built to track changes for a particular db object.

For databases that do not support data dictionary, an alternative approach called Generic Migrations can be used:

Sample ant target:


<project name="sample_migrateDB">
  <description>Ant build file for database synchronization</description>
  <property file="${basedir}/buildMigrateDB.properties" />

  <target name="init">
    <property value="${db.user}" name="db.user" />
    <property value="${db.password}" name="db.password" />
    <property value="${db.drivername}" name="driver.name" />
    <property value="${db.url}" name="db.url" />
    <echo message="DB Name: ${db.url}" />
  </target>

  <!--  Define the build path -->
  <path id="buildMigrateDB.classpath.ref">
    <!-- ensure the following libs are present in the {lib.dir}: migratedb.jar and postgres jdbc jar -->
    
    <fileset dir="${lib.dir}">
      <include name="*.jar" />
      <include name="*.zip" />
    </fileset>
  </path>

  <property name="buildMigrateDB.classpath" refid="buildMigrateDB.classpath.ref" />
  <!-- End defining the build path  -->

  <taskdef name="dbrelease" classname="net.sf.migratedb.ant.MigrateDbTask" classpath="${buildMigrateDB.classpath}" />

  <target name="dbmigrate" depends="init" description="Create Latest Database Version">

    <dbrelease driver="${db.drivername}"
               url="${db.url}"
               userid="${db.user}"
               password="${db.password}"
               apply="true"
               verbose="true"
               <!-- location for above change xml file -->
               file="${root.dir}/database/postgres/db.xml"
    />
  </target>
</project>

Managing Log4j Appenders @ Runtime

2007-07-19T23:17:00.000-07:00

Many a times when debugging production issues we wish the log level would have been @ DEBUG level or some other lower level to get more insight into the issue. The usual way to debug non-obvious production issue is to reproduce it in you local development environment. In many cases it works but in some cases its just hard to reproduce the exact environment which caused the error. For instance trying to reproduce a complex hibernate transaction error and that sort. In our application we use a hell lot of webservice calls to interact with various third party hosted services. Its almost impractical to log all the SOAP interactions and many a times reproducing the environment which generated the faulty SOAP call is not possible.

For this reason we manage our Log4j appenders @ runtime wherein we can add new appenders and change priority levels of existing appenders @ runtime. Once the stacktrace or the required log is gathered, the priority levels can be reverted back to the non-voluminous/required levels.

We have a Log4j administration page in our application which can only be accessed by super ninjas a.k.a developers. This page can be used to change/add appenders at runtime. The page has sections to add new appenders and change priority levels of existing appenders.

To start with we display all the current appenders with their current priority levels. This can be achieved using the code below (for simplicity sake I am :



 // collect all current appenders
 List appenders = new ArrayList(50);
 Enumeration e = LogManager.getCurrentLoggers();

 while ( e.hasMoreElements() )
 {
    appenders.add(e.nextElement());
 }
 request.setAttribute("appenders",appenders);

 // all possible priority levels
 Priority [] prios = Priority.getAllPossiblePriorities();
 request.setAtrribute("possiblePriorities", prios);

In the JSP page you can use a simple JSTL tag to display the above collected information in a tabular format.



<%@ page import="java.util.*,org.apache.log4j.*"

....
....

<table>
<tr>
 <td> Appender </td>
 <c:forEach var="priority" items='${possiblePriorities}'>
      <td><c:out value='${priority}'/></td>
 </c:forEach>
</tr>

<c:forEach var="appender" items='${appenders}'>
   <tr>    
     <td><c:out value='${appender.getName()}'/></td>
     <c:forEach var="priority" items='${possiblePriorities}'>
      <td>
         <input type="radio" name="'${appender.getName()}'" value="'${priority}'"
            <c:if test='${appender.getChainedPriority() == priority)}'> checked </c:if>
         >
      </td>
     </c:forEach>
   </tr>
</c:forEach>

<tr>
  <td rowspan=5 align="center"> <button type="submit" name="submit" value="update">Update</button></td>
</tr>

</table>

<!-- section of page to add new appender-->

<table style="width:auto;">
  <tr>
     <td align="center">
          <input type="text" name="newLogger" size="70">
     <td>
     <td>
       <select name="newLoggerLevel">
          <c:forEach var="priority" items='${possiblePriorities}'>
             <option value="<c:out value='${priority}' />"><c:out value='${priority}' /></option>
         </c:forEach>
       </select>
     <td>
     <td>
       <button type="submit" name="submit" value="Add">Add Logger</button>
     </td>
  </tr>
</table>

The idea is to get a view of this sort:

Two events can generate from the above page:
1) add a new appender
2) update level of existing appender

Both of these events can be handled with the below servlet code:



import org.apache.log4j.*;

....
....

Enumeration e = LogManager.getCurrentLoggers();
while (e.hasMoreElements())
{
   Logger logger = (Logger) e.nextElement();
   String prio = request.getParameter(logger.getName());
   if (prio != null &&amp; prio.length() > 0)
   {
       Level p = Level.toLevel(prio);
       if (p != null && ! p.equals(logger.getEffectiveLevel()))
       {
           logger.setLevel(p);
       }
   }
}   

// add new loggers desired
String newLogger = request.getParameter("newLogger");
String newLoggerLevel = request.getParameter("newLoggerLevel");
if (newLogger != null)
{
    Level p = Level.toLevel(newLoggerLevel);
    Logger logger = Logger.getLogger(newLogger);
    logger.setLevel(p);
}

Hibernate fetching strategies

2007-06-24T16:35:00.000-07:00

A fetching strategy is the strategy Hibernate will use for retrieving associated objects if the application needs to navigate the association. Fetch strategies may be declared in the O/R mapping metadata, or over-ridden by a particular HQL or Criteria query.

Hibernate3 defines the following fetching strategies:

Join fetching - Hibernate retrieves the associated instance or collection in the same SELECT, using an OUTER JOIN.

<set name="permissions" fetch="join">
    <key column="userId"/>
    <one-to-many class="Permission"/>
</set

<many-to-one name="mother" class="Cat" fetch="join"/>

Can be specified at query level:


User user = (User) session.createCriteria(User.class)
                .setFetchMode("permissions", FetchMode.JOIN)
                .add( Restrictions.idEq(userId) )
                .uniqueResult();

Select fetching - a second SELECT is used to retrieve the associated entity or collection. Unless you explicitly disable lazy fetching by specifying lazy="false", this second select will only be executed when you actually access the association. Use fetch="select" or FetchMode?.Select to accomplish this.
Subselect fetching - a second SELECT is used to retrieve the associated collections for all entities retrieved in a previous query or fetch. Unless you explicitly disable lazy fetching by specifying lazy="false", this second select will only be executed when you actually access the association. Use fetch="subselect" or FetchMode?.SubSelect? to accomplish this.

* Batch fetching - an optimization strategy for select fetching - Hibernate retrieves a batch of entity instances or collections in a single SELECT, by specifying a list of primary keys or foreign keys. You may also enable batch fetching of collections. For example, if each Person has a lazy collection of Cats, and 10 persons are currently loaded in the Sesssion, iterating through all persons will generate 10 SELECTs, one for every call to getCats(). If you enable batch fetching for the cats collection in the mapping of Person, Hibernate can pre-fetch collections. With a batch-size of 8, Hibernate will load 3, 3, 3, 1 collections in four SELECTs. Again, the value of the attribute depends on the expected number of uninitialized collections in a particular Session. Batch fetching of collections is particularly useful if you have a nested tree of items, ie. the typical bill-of-materials pattern. (Although a nested set or a materialized path might be a better option for read-mostly trees.)

<class name="Person">
    <set name="cats" batch-size="3">
    ...
    </set>
</class>

Hibernate also distinguishes between:

Immediate fetching - an association, collection or attribute is fetched immediately, when the owner is loaded.
Lazy collection fetching - a collection is fetched when the application invokes an operation upon that collection. (This is the default for collections.)
"Extra-lazy" collection fetching - individual elements of the collection are accessed from the database as needed. Hibernate tries not to fetch the whole collection into memory unless absolutely needed (suitable for very large collections)
Proxy fetching - a single-valued association is fetched when a method other than the identifier getter is invoked upon the associated object.
"No-proxy" fetching - a single-valued association is fetched when the instance variable is accessed. Compared to proxy fetching, this approach is less lazy (the association is fetched even when only the identifier is accessed) but more transparent, since no proxy is visible to the application. This approach requires buildtime bytecode instrumentation and is rarely necessary.
Lazy attribute fetching - an attribute or single valued association is fetched when the instance variable is accessed. This approach requires buildtime bytecode instrumentation and is rarely necessary.

By default, Hibernate3 uses lazy select fetching for collections and lazy proxy fetching for single-valued associations. These defaults make sense for almost all associations in almost all applications.

To enable lazy property loading, set the lazy attribute on your particular property mappings:

<class name="Document">
       <id name="id">
        <generator class="native"/>
    </id>
    <property name="name" not-null="true" length="50"/>
    <property name="summary" not-null="true" length="200" lazy="true"/>
    <property name="text" not-null="true" length="2000" lazy="true"/>
</class>

XWORK external reference resolver

2007-06-09T16:42:00.000-07:00

XWORK external reference resolver

Sometimes xwork action classes needs to be wired with external references, for example spring maintained resources. In such cases the dependencies are defined using a <external-ref> tag. For example: see below a sample xwork configuration wherein an action class needs to be wired with a spring maintained datasource:

<package name="auditIntegration" extends="auditResultRun">
    <action name="executeAllAudits" class="com.xxx.AllSchemasAuditAction">
      <external-ref name="dataSource" required="true">/db/dw/DataSource</external-ref>
      <param name="nestedActionName">executeAudit</param>
    </action>
  </package>

In such cases external resolvers need to be specified which perform the property setting. XWORK provides a specification using com.opensymphony.xwork.config.ExternalReferenceResolver and provides a default implementation com.opensymphony.xwork.spring.SpringExternalReferenceResolver for spring reference resolver. Custom implementation can be provided to resolve other external references like JNDI dependencies into xwork etc. In any case these external resolvers must be explicitly declared to be used by XWORK. See below for a sample declaration (extends the webwork default config):

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE xwork PUBLIC "-//OpenSymphony Group//XWork 1.0//EN" "http://www.opensymphony.com/xwork/xwork-1.0.dtd">
<xwork>
  <include file="webwork-default.xml"/>
  <package name="xwork-common" extends="webwork-default"
           externalReferenceResolver="com.xxx.SpringExternalReferenceResolver">
 </package>
</xwork>

Sample implementation for the external resolver (code adapted from XWORK com.opensymphony.xwork.spring.SpringExternalReferenceResolver implementation):

public class SpringExternalReferenceResolver implements com.opensymphony.xwork.config.ExternalReferenceResolver
{
    private static final Logger log = Logger.getLogger(SpringExternalReferenceResolver.class);

    /**
     * resolve the references for this invocation
     * @param invocation the invocation to resolve
     * @throws ReferenceResolverException if we had issues.
     */
    public void resolveReferences(ActionInvocation invocation) throws ReferenceResolverException
    {
        ApplicationContext ctx = {{get app context handle here}};
        if (ctx == null)
            throw new IllegalStateException("application context has not been set for this "
                                            + "external reference resolver!");

        List externalRefs = invocation.getProxy().getConfig().getExternalRefs();
        for (Iterator iter = externalRefs.iterator(); iter.hasNext(); )
        {
            ExternalReference reference = (ExternalReference) iter.next();
            if (log.isDebugEnabled())
                log.debug("resolving " + reference.getName() + " to " + reference.getExternalRef());
            if (reference.getExternalRef() == null)
            {
                throw new ReferenceResolverException(
                    "reference " + reference.getName() + " has no external ref");
            }

            Object bean = null;
            try
            {
                // no such bean exception
                bean = ctx.getBean(reference.getExternalRef());
                if (log.isDebugEnabled())
                    log.debug("resolved " + reference.getExternalRef() + " to " +
                              (bean != null ? bean.getClass().toString() : "null"));

                // other exceptions
                Map context = Ognl.createDefaultContext(invocation.getAction());
                if (log.isDebugEnabled())
                    log.debug("setting bean into property " + reference.getName() + " of " +
                              invocation.getAction().getClass());

                // unbelievably, this actually throws a RuntimeException!  Unbelievable
                OgnlUtil.setProperty(reference.getName(), bean, invocation.getAction(),
                                     context, true);
                if (log.isDebugEnabled())
                    log.debug("resolved " + reference.getName() + " to " +
                              reference.getExternalRef() + ": " + bean);
            }
            catch (NoSuchBeanDefinitionException e)
            {
                if (reference.isRequired())
                {
                    //if a dependency is required but wasn't found throw an exception
                    throw new ReferenceResolverException(
                        "Failed to find external reference: " + reference.getExternalRef(), e);
                }
                else
                {
                    log.warn("Bean '" + reference.getExternalRef() +
                             "' could not be found in spring");
                    // just keep going
                    continue;
                }
            }
            catch (Exception e)
            {
                throw new ReferenceResolverException(
                    "Failed to set external reference: " + reference.getExternalRef()
                    + " for bean attribute: " + reference.getName() + ": " + e.getMessage() +
                    " bean hashcode: " + (bean != null ? bean.getClass().hashCode() : -1), e);
            }
        }
        if (log.isDebugEnabled())
            log.debug("external reference resolution for " + invocation.getAction() + " complete.");
    }


}

WS-BPEL

2007-06-09T16:33:00.000-07:00

- What is WS-BPEL?

An XML-based grammar for describing the logic to orchestrate the interaction between Web services in a business process. It defines a set of basic control structures like conditions or loops as well as elements to invoke web services and receive messages from services. It relies on WSDL to express web services interfaces. Message structures can be manipulated, assigning parts or the whole of them to variables that can in turn be used to send other messages.

- Why is it needed?

• Web services --> move towards service-oriented computing
• Applications are viewed as “services”
• Loosely coupled, dynamic interactions
• Heterogeneous platforms
• No single party has complete control
• How do you compose services in this domain?
• WSDL defined Web services have a stateless interaction model
• Messages are exchanged using
• Synchronous invocation
• Uncorrelated asynchronous invocations
• Most “real-world” business processes require a more robust interaction model
• Support for Messages exchanged in a two-way, peer-to-peer conversation lasting minutes, hours, days, etc.
• BPEL provides the ability to express stateful, long-running interactions

- Relationship with WSDL?

BPEL is layered on top of and extends the WSDL service model
• WSDL defines the specific operations allowed
• BPEL defines how WSDL operations are orchestrated to satisfy a business process
• BPEL also specifies extensions to WSDL in support of long-running asynchronous business processes
• Expressed entirely in XML
• Uses and extends WSDL 1.1
• Uses XML Schema 1.0 for the data model

- What is Apache ODE?
Apache ODE (Orchestration Director Engine) executes business processes written following the WS-BPEL standard. It talks to web services, sending and receiving messages, handling data manipulation and error recovery as described by your process definition. It supports both long and short living process executions to orchestrate all the services that are part of your application.

- References
• Real use cases scenarios

ACEGI Authentication Provider Examples

2007-05-25T23:15:00.000-07:00

Acegi provides a very flexible way to configure the authentication provider. By default it provides two implementations of the authentication provider

- InMemoryDaoImpl : Retrieves user details from an in-memory list created by the bean context. So basically the list of users and their passwords are specified in the bean configuration file. (http://www.acegisecurity.org/multiproject/acegi-security/apidocs/org/acegisecurity/userdetails/memory/InMemoryDaoImpl.html). See below for sample bean configuration

<bean id="inMemoryDaoImpl" class="org.acegisecurity.userdetails.memory.InMemoryDaoImpl">
    <property name="userMap">
       <value>
           marissa=koala,ROLE_TELLER,ROLE_SUPERVISOR
           dianne=emu,ROLE_TELLER
           scott=wombat,ROLE_TELLER
           peter=opal,disabled,ROLE_TELLER
       </value>
   </property>
</bean>

- JdbcDaoImpl : Retrieves user details (username, password, enabled flag, and authorities) from a JDBC location. A default database structure is assumed, which most users of this class will need to override, if using an existing schema. This may be done by setting the default query strings used. If this does not provide enough flexibility, another strategy would be to subclass this class and override the MappingSqlQuery instances used, via the initMappingSqlQueries() extension point. (http://www.acegisecurity.org/multiproject/acegi-security/apidocs/org/acegisecurity/userdetails/jdbc/JdbcDaoImpl.html). See below for code sample to configure using a jdbc driver or a datasource. Irrespective of the database used and how a DataSource is obtained, a standard schema must exist in the database

-- USING JDBC Driver
<bean id="dataSource" class="org.springframework.jdbc.datasource.DriverManagerDataSource">
   <property name="driverClassName"><value>org.hsqldb.jdbcDriver</value></property>
   <property name="url"><value>jdbc:hsqldb:hsql://localhost:9001</value></property>
   <property name="username"><value>sa</value></property>
   <property name="password"><value></value></property>
</bean>

-- Using Datasource
<bean id="jdbcDaoImpl" class="org.acegisecurity.userdetails.jdbc.JdbcDaoImpl">
   <property name="dataSource"><ref bean="dataSource"/></property>
</bean>

- Custom implementation : The above two implementations basically implement the UserDetailsService interface. If you have complex needs (such as a special schema or would like a certain UserDetails implementation returned), you'd be better off writing your own UserDetailsService(http://www.acegisecurity.org/multiproject/acegi-security/apidocs/index.html?org/acegisecurity/userdetails/UserDetailsService.html).

Code Sample:

<bean id="authenticationManager" class="org.acegisecurity.providers.ProviderManager">
      <property name="providers">
         <list>
            <ref local="daoAuthenticationProvider"/>
         </list>
      </property>
   </bean>

   <bean id="daoAuthenticationProvider" class="org.acegisecurity.providers.dao.DaoAuthenticationProvider">
      <property name="userDetailsService"><ref bean="UserService"/></property>    
   </bean>

   <bean id="UserService" class = "com.icrossing.xxx.CustomAuthenticationProvider"/>

-- Sample implementation

class CustomAuthenticationProvider implements org.acegisecurity.userdetails.UserDetailsService
{
   public UserDetails loadUserByUsername(String userId) throws UsernameNotFoundException, DataAccessException 
   {
        User user = null;
        GrantedAuthority[] grantedAuthorities = null;
        try {
            user = getUserDAO().lookupUser(userId);
            
            if(user==null) {
                throw new UsernameNotFoundException("Invalid User");            
            }
            
            Set roles = user.getRoles();
            int i = 0;
            grantedAuthorities = new GrantedAuthority[roles.size()];
            for (Iterator iter = roles.iterator(); iter.hasNext(); i++) {
                Role role = (Role) iter.next();
                
                GrantedAuthority authority = new GrantedAuthorityImpl(role.getRole());
                grantedAuthorities[i] = authority;
            }
        } catch (DataStoreException e) {
            throw new DataRetrievalFailureException("Cannot loadUserByUsername userId:"+userId+ " Exception:" + e.getMessage(), e);
        }

        UserDetails userDetails = new org.acegisecurity.userdetails.User(
                user.getUserId(), 
                user.getPassword(),
                user.isEnabled(), //enabled
                user.isEnabled(), //accountNonExpired
                user.isEnabled(), //credentialsNonExpired
                user.isEnabled(), //accountNonLocked
                grantedAuthorities
                );
        return userDetails;
    }
}

FTPClient Default Buffer Policy

2007-05-25T16:43:00.000-07:00

Just found this while researching the buffer policy of org.apache.commons.net.ftp.FTPClient (Apache commons net FTP).

Methods storeFile() and retrieveFile() in FTPClient use a default buffer size of 1024 (http://jakarta.apache.org/commons/net/apidocs/org/apache/commons/net/io/Util.html#DEFAULT_COPY_BUFFER_SIZE).

Methods storeFileAsStream() and retrieveFileStream() do not use a default buffer when file type is BINARY however when file type is ASCII they use a default buffer of 1024. Here's the developer's comment as to why

// We buffer ascii transfers because the buffering has to
// be interposed between ToNetASCIIOutputSream and the underlying
// socket output stream. We don't buffer binary transfers
// because we don't want to impose a buffering policy on the
// programmer if possible. Programmers can decide on their
// own if they want to wrap the SocketOutputStream we return
// for file types other than ASCII.

FTPClient timeout values

2007-05-22T17:39:00.000-07:00

In looking at the docs for org.apache.commons.net.ftp.FTPClient there are three timeouts which can be configured:

setDefaultTimeout : Set the default timeout in milliseconds to use when opening a socket. This value is only used previous to a call to connect() and should not be confused with setSoTimeout() which operates on an the currently opened socket.

setSoTimeout : Set the timeout in milliseconds of a currently open connection. Only call this method after a connection has been opened by connect().

setDataTimeout : Sets the timeout in milliseconds to use when reading from the data connection. This timeout will be set immediately after opening the data connection.

This seemed confusing so I went ahead and peeked at the source code for FTPClient and the whole thing made sense. So basically the FTPClient uses the underlying java.net.Socket and the various timeouts apply at the various stages of socket usage.

if the setDefaultTimeout is set then the underlying java.net.Socket.setSoTimeout() is set and is used default for all connections made using this FTPClient instance. It basically saves you the trouble of calling setSoTimeout after every connection establishment.

if setSoTimeout is set then the underlying java.net.Socket.setSoTimeout() is set for the current connection and at disconnect() the value reverts back to the defaultTimeout set using the setDefaultTimeout. If you call it before connecting, you'll get a NullPointerException

if setDataTimeout is set then the underlying java.net.Socket.setSoTimeout() is set before a read is performed and after the read completion the timeout value is restored to the pre-read state so basically should be called before a data connection is established (e.g., a file transfer) because it doesn't affect an active data connection. Usually when a read() method tries to read data from a socket the program will block until the data arrives. However, if you set the timeout property, the read() will only wait the specified number of milliseconds. Then, if no data was received, it will throw an InterruptedIOException. The data timeout specified blocks for each socket read() call and is not cumulative of all read calls.

It seems obvious that defaultTimeout will suffice the purpose however there might be need have read specific dataTimeouts (e.g., you don't want a 2 gb file transfer to die just because there is a 10 minute loss of connectivity)...

On another note what value is optimal to set for timeout given its implications at various stages? Online research recommends: timeout for connect : 5 secs, write/reads : 120 secs

Update: 08/23/2007
So the interesting fact is after all the babbling above I was not able to make the above timeouts to work So here's what I tried: I have a file of size 57mb and I tried setting various combinations of timeouts for upload:

- set defaultTimeout (120 secs) and dataTimeout (1200 secs) before establishing login connection --> Result: upload failed with timeout

- set defaultTimeout (120 secs) before login connection and dataTimeout (1200 secs) after login connection (1200 secs) - Result: upload failed with timeout

- set defaultTimeout (120 secs) before login connection and after login connection (1200 secs) connection --> Result: upload failed with timeout

- set defaultTimeout (1200 secs) before login connection --> Result: upload succeeded

So I tried the above @ various times to make sure I am not dealing with a nework spike or any of that sort and got the same results. I will update once I find more on this stuff.

OutOfMemory issue in JUnit

2007-05-02T11:04:00.000-07:00

Recently one of my colleague encountered this issue when running a JUnit test. Inspite of increasing the -vm settings the error kept coming. On further research we found that if JUnit is set with "fork=true" the task will be executed in forked VM. So the memory settings of default VM will not effective. So, you need to set maxmemory attribute of JUnit to avoid OutOfMemory? exception.

Example:

<target name="test.class.inner" if="test.class">
      <echo message="test.classpath"/>
      <mkdir dir="${test.output.dir}"/>
      <mkdir dir="${build.dir}/tmp"/>
      <junit dir="${build.dir}" haltonfailure="yes" haltonerror="yes" printsummary="on"
            fork="true" filtertrace="true" '''maxmemory="1024m"'''>
      <sysproperty key="merchantize.env" value="test"/>
....

Hibernate mapping legacy data

2006-12-07T16:59:00.000-07:00

I recently encountered this while working with Hibernate. Mapping legacy data where foreign key does not refer to a primary key of the associated table is complex and is not generated correctly using many of the IDE's out there. You basically have to do some tweakings to make it work.

Two cases arise:
1) when foreign key refers to a unique key of the associated table
2) when foreign key refers to a non-primary/non-unique key

You will get an error of similar sort when mapping without the tweakings in the above cases:

[java] org.hibernate.MappingException: Foreign key (FK6771BFAA1845E8B:CardField [])) must have same number of columns as the referenced primary key (Card [project_id])
> [java] at org.hibernate.mapping.ForeignKey.alignColumns(ForeignKey.java:90)
> [java] at org.hibernate.mapping.ForeignKey.alignColumns(ForeignKey.java:73)
> [java] at org.hibernate.cfg.Configuration.secondPassCompileForeignKeys(Configuration.java:1182)
> [java] at org.hibernate.cfg.Configuration.secondPassCompile(Configuration.java:1089)
> [java] at org.hibernate.cfg.AnnotationConfiguration.secondPassCompile(AnnotationConfiguration.java:302)
> [java] at org.hibernate.cfg.Configuration.buildMappings(Configuration.java:1034)

The trick is to utilize the "property-ref" attribute to solve the above mapping issue.

Consider you have two tables:
Employee (emp_ID, emp_SSN, emp_Name)
Manager (mgr_SSN, mgr_Id)

In employee table empID is the primary key and empSSN is unique. Manager table has a many-to-one relation with the employee table with one manager can be assigned to multiple departments.

In Employee.hbm.xml file the empSSN would be defined like this:

<property name="empSSN" type="java.lang.Integer">
     <column name="emp_SSN" not-null="true" unique="true" />
</property>

<set name="managers" inverse="true">
     <key property-ref="empSSN">
          <column name="mgr_SSN" not-null="true" />
     </key>
     <one-to-many class="com.hibernate.mapping.Manager"/>
</set>

In manager.hbm.xml the property will be defined like this

<property name="mgrSSN" type="java.lang.Integer">
     <column name="mgr_SSN" not-null="true" />
</property>

<many-to-one name="employee" class="com.hibernate.mapping.Employee" update="false" insert="false" fetch="select" property-ref="empSSN" column="mgr_SSN" />

This should make it work!!!

Some other resources:

Post on Serverside
Hibernate Docs

SSL setup with MSN Adcenter

2006-08-29T16:20:00.000-07:00

I will be posting a series of blogs on my experiences with working on a project utilizing the Microsoft adcenter as I encounter them. Let me share my experience with the security set up I struggled with initially

Platform: Linux
environment: J2EE

I kept getting the error

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

in the initial stage. The first time I encountered this error was during the WSDL to JAVA source creation. The WSDL's are hosted on a secure site (https://beta6.api.idss.msn.com/v2) and thus needed the certs to be stored in the keystore. I circumvented it by downloading the WSDL's from browser and supplying local file URL as parameter to axis's WSDL2Java utility and that created the source files.

The error came back to haunt me as the webservice is also hosted on a secure site. I downloaded the certificate from the adcenter UI on sandbox (https://beta1.idss.msn.com/) using IE browser

(Double click the SSL lock icon in the bottom pane in the IE browser --> Details tab --> Copy to file --> [[format]] DER X.509)

and stored the cert in my local keystore using the command

keytool -import -file [[saved cert file loc]] -alias sandbox_api_cert -keystore MSN_Keystore
VM Arguments:
-Djavax.net.ssl.trustStore = [[keystore location]]
-Djavax.net.ssl.trustStorePassword = [[passwd]]

However this did not solve the problem.

On further research I found that "if you use a certificate not signed by a pre-installed certificate authority (=root cert), you need to import both your cert and the root cert". I checked the browser to see the root cert

(Double click the SSL lock icon in the bottom pane in the IE browser --> Certification Path tab)

and voila!!!! the cert was internally chained 3 times (GTE CyberTrust Global Root --> Microsoft Internet Authority --> Microsoft Secure Server Authority --> Beta1.idss.msn.com). I had already entered the cert for Beta1.idss.msn.com in my local keystore (as explained above) and so I proceeded to check if the certs for the other chain members are available in the jdk1.5.0_04/jre/lib/security/cacerts file using the command

keytool -v -list -keystore cacerts | grep gte
Enter keystore password: changeit
Alias name: gtecybertrustglobalca
Alias name: gtecybertrustca
Alias name: gtecybertrust5ca

keytool -v -list -keystore cacerts -alias gtecybertrustglobalca
Enter keystore password: changeit
Alias name: gtecybertrustglobalca
Creation date: May 10, 2002
Entry type: trustedCertEntry

Owner: CN=GTE CyberTrust Global Root, OU="GTE CyberTrust Solutions, Inc.", O=GTE Corporation, C=US
Issuer: CN=GTE CyberTrust Global Root, OU="GTE CyberTrust Solutions, Inc.", O=GTE Corporation, C=US
Serial number: 1a5
Valid from: Wed Aug 12 17:29:00 MST 1998 until: Mon Aug 13 16:59:00 MST 2018
Certificate fingerprints:
MD5: CA:3D:D3:68:F1:03:5C:D0:32:FA:B8:2B:59:E8:5A:DB
SHA1: 97:81:79:50:D8:1C:96:70:CC:34:D8:09:CF:79:44:31:36:7E:F4:74

keytool -v -list -keystore cacerts | grep microsoft
Enter keystore password: changeit

So the middle chain members Microsoft Internet Authority --> Microsoft Secure Server Authority were not available in my cacerts file. I went ahead and downloaded these certs from the browser

(Double click the SSL lock icon in the bottom pane in the IE browser --> Certification Path tab --> click on each certicate --> View Certificate)

and follow the procedure above to save and enter the cert into local keystore. Once I did this the previous problem was resolved however I was getting a new error

"java.security.cert.CertPathValidatorException: signature check failed"

I was pretty sure that one of my certs was wrong so I checked the WSDL to see where the webservice was hosted and it was different from where the UI was hosted. It was hosted on https://beta6.api.idss.msn.com. So I went ahead and downloaded the cert and stored it under local keystore and this resolved all the SSL errors.