Weird IBM RPM Catastrophics Error

Our latest IBM RPM customer is opted for a relatively smaller investment in the solution, their system configuration of RPM involved the following components:

DB2 Express-C 9.5
Tomcat 5.5
JDK 5.0
Windows 2003
IBM RPM 7.1.1.2

Based on the instructions of the official guide of IBM RPM 7.1.1.2, there are some additional steps needed for the proper installation of the solution. You can refer to the guide for further information.

The installation took almost 1 day due to our unfamiliarity with such environment because we usually deal with IBM WAS and DB2 ESE. So we did it slowly to avoid unnecessary mistakes.

The installation was a successful one with no error in the logs.

However, a post installation testing procedure revealed some drastic and yet can be frigthening issues, some of them includes:

* When clicked on Investment Map in the Dashboard, it will respond with a EOleException Catastrophics error with some Windows hexadecimal code that translated to a very general error message.

* When try to save text key in the rich text control in the portlet, it will not be saved.

* More EOleException Catastrophic error in other features.

Very interesting indeed because none of the logs (Windows event, tomcat logs and DB2 logs) were unable to reveal any useful hints on the error.

We decided to open a case with IBM Support since the team will be busy doing other functional works.

The communication was established for almost 2 months with no apparent solution to the problem. We did the regular log compilation, scenario description, screen shots blah blah.... At the end, the support even suggested that the DB2 Express-C v9.5 is not supported by the RPM and thus unable to escalate it further. :-S

Since the project is almost near the completion and the risk of this error haunting us is getting more obvious and serious, the team decided to commit more efforts to fix it.

Here are the problem solving sequences we took:

Note: Our VM Test environment doesn't have the EOleException problem. Thus we believe it is environment specific, not a product bug.


1. Disable all irrelevant services in the OS and test
- Unchanged. Rule out the problem of software conflicting
- Enable back everything

2. Run the RPM Client outside the server machine, i.e. client workstation
- Unchanged. Confirmed it is server based problem. RPM Standalone and Plugins behaves the same.

3. Accidentally we had tried this scenario and found out something interesting.
We opened up the RPM Client and triggered the EOleException catastrophics error and then clicked Ok to continue. Then we recycled the Tomcat while the client was opened. The error will not triggered again and everything is fine, including the rich text control saving problem.
- This has revealed an important hint that the problem is caused by something shared between the client program and the Tomcat in the server.
- Since the client program is a Windows executable, possibly this got to do with some DLLs or something that the client program "loaded" before the Tomcat. I can't be sure since I don't have access to the source codes.

4. We checked the java.library.path from the Tomcat logs and found that there are another copy of xercesImpl.jar and xmlApi.jar in the path. To play safe, we removed these files and tested again. No luck.
- Rules out the problem of JAR class loading because logically the client program and Tomcat doesn't really use them together (I think).

5. A tip that we learnt from the Support is that we can stream out the logs generated by the client program using the "RPMStdIn.exe > logs.txt" and this log will contain information such as stored procedure called with parameters.
- By triggering the error, now the logs.txt contains the stored procedure that causes the error.

6. From 5 above, we copied the call information and executed it directly in DB2
- No errors returned from DB2. This confirmed the problem with how Tomcat or something inside Tomcat interpret or handle the SP call.

7. Confirm the JDBC Driver
- No problem as we directly used the drivers from the local DB2 Express-C.

8. Ruling out so many things and we start to suspect something more fundamental causes this, i.e. the Java execution environment. A check on the Tomcat JRE indicated the use of JRE 1.5.0.2. Cross referenced to Tomcat 5.5 and RPM 7.1.1.2 and none of the manual said anything about a particular level of J2SE 5.0 to be used.
- Since we running out of options, we decided to upgrade the JRE to 1.5.0.16 since that version was used in our VM too (LOL, we realized that too late I guess)
- Bingo, the error is no more.
- To reconfirm, we installed JRE1.5.0.2 in our VM and let the Tomcat used it and we managed to resimulate the problem, which is something that we were unable to perform since the reporting of the case to the Support.
- Note: The mode of the Tomcat in the environment is using jvm.dll. I'm not sure whether the same problem will happen with other modes.


All this havoc just because of insufficient information from the official about supported environments for RPM. Aiks.

P/S: IBM RPM guide never mentioned about the DB2 editions supported and this might be a grey area on the supportability of any RPM deployments.

Professionally, I strongly believe that CA Clarity did a great job on explicitly documented down the supported environments to the details on levels, fix packs, versions and so on. It can help to get rid of weird problem(s) like the one we encountered here.

UDF and SP in IBM RPM

IBM Rational Portfolio Manager (assuming version 7.1.1.2) is using Dynamic Linking Library (DLL) as its implementation for some of the database objects, namely stored procedure (SP) and user defined functions (UDF). More precisely the file name of this dll is IBMRPM.dll and it resides in two locations, i.e. ${DB2_INSTALL_ROOT}\FUNCTION\IBMRPM.dll and ${DB2_INSTALL_ROOT}\FUNCTION\unfenced\IBMRPM.dll.

During RPM installations/migrations, the process will define the linkage between SP/UDF to the appropriate methods in the dll and this piece of information is stored in SYSIBM.SYSROUTINES table's IMPLEMENTATION column.

The problem with IBM RPM installation is that it hardcoded the directory information in IMPLEMENTATION column. This shall causes portability issue when you desire to move the system to another environment where the ${DB2_INSTALL_ROOT} is not consistent with each other. For example, your initial RPM machine might have DB2 at drive C then the latest environment put it in drive E.

In such a setting, after you restored the database successfully into the new environment, you shall get alerts and some errors in the application server's logs specifying database errors with SQLSTATE 42724 REASON CODE 4.

You maybe be tempted to update the path information directly in SYSIBM but the column is only updateable through a proper DROP/CREATE PROCEDURE or ALTER PROCEDURE and in this case the number of objects to change is way too many even if you plan to write a batch script for it.

A shortcut to this problem is to copy the IBMRPM.dll and create the necessary dummy folders that reflect the original path information and paste them there. This works but it's not recommended because it might causes complication next time when doing patching or upgrading of RPM.

The best way to resolve this is to reuse some of the scripts from the installer.

Note: Please do this at your own risk. Backup everything first.


Step 1: Setting Up Environment

To do this, open up an instance of DB2CMD.

Change Directory (CD) to ${RPM_INSTALLER}\Database\DB2\Windows\CSP

If you are using DB2 V9, Please set the following environment variables using:

SET DB2TEMPDIR=${DB2_INSTALL_ROOT}\

Note: Please make sure the above path end with a slash (\)

Then

SET RPMDLLNAME=IBMRPM

You will then login to DB2 by using the user name where the name is the schema name of the objects. For example, DB2ADMIN. If you are not sure what name is that, you can always open up the control center and browse the list of stored procedures and check the schema name column. A common procedure is the SP_LOGON.

db2 connect to MYRPM user db2admin using your_password

Step 2: Recreate SP/UDF

Run the drop_sp.bat by using the following command:

drop_sp.bat > drop_sp.log

Check the drop_sp.log to verify the completion.

Run the create_sp.bat by using the following command:

create_sp.bat > create_sp.log

Check the create_sp.log to verify the completion.

Step 3: Bind packages

Because the dropping of previous SP/UDF invalidated package objects used by RPM, you need to bind them again. Open bindall.bat using notepad or other editor. Copy the for loop command and paste it in a new file. Name the file bindall2.bat. Change the value for QUALIFIER to DB2ADMIN. Save it and run the following command using the previous DB2 Command Window session.

bindall2.bat > bindall2.log

Check the bindall2.log to verify the completion.

You might want to recycle your DB2 process to get a fresh start.

Installing IBM RPM in Microsoft Windows Vista

The IBM RPM 7.1.1 documentation stated that you can install and use IBM RPM client in Microsoft Windows Vista. Is that really true? Yes and No.

No in the sense that the installation is not one step process. Some additional steps are needed to ensure all the necessary components are properly configured. Yes because I tried and the program seems working after the installation. Of course, maybe there will be some bugs I haven't discover yet. I will briefly describe the steps you need to perform.

There are 2 components (RPMBrokers.dll, RPMImpex.dll) to be registered for RPM Rich Client to work and 3 components (RPMBrokers.dll, RPMImpex.dll, RPMPlgin.ocx) for RPM Plugin.

Ok, the ONLY REAL installation in Vista is just the ActiveX component registration is troublesome. If you try to register these components manually after the RPM installation, most probably you will hit by a 0x80004005 error.

Note: The below are the steps I conducted to make the thing works, however I'm not sure whether some of it are mandatory.

The first thing you should do is to enable ActiveX Installer in Vista. You can do this by going to [Control Panel] > [Programs and Features] > [Turn Windows Feature On or Off]. Then from the list select [ActiveX Installer]. Click Ok to proceed.

Then you go to [Control Panel] > [Administrative Tool] > [Services] and turn the [ActiveX Installer] service on (Started).

And finally the real meat is here.

You need to remove the "cached" copy of the regsvr32.exe from Vista.

P/S: It might be helpful to restart your PC first.

Using Windows Explorer, go to C:\Windows\Prefetch, remove all the files where the names start with REGSVR32.

Next, you need to use the regsvr32.exe program shipped along with the IBM RPM. This file can be located in %RPM_INSTALLATION_FOLDER%, i.e. C:\Program Files\IBM\Rational Project Manager\regsvr32.exe. To do this, go to [All Programs] > [Accessories], right click on [Command Prompt] and select [Run as administrator]. After that you need to change directory to the RPM installation folder using command like [CD C:\PROGRAM FILES\IBM\RATIONAL PORTFOLIO MANAGER]. Try to execute the regsvr32. If you succeed, you shall get a message box on how to use it.

You can register the RPMBrokers.dll by typing the command [regsvr32 RPMBrokers.dll]. You will get a message indicating the registration was succeeded.

And before you continue to register the next one, you should perform the removal of REGSVR32 files from PREFETCH folder again.

You can register the RPMImpex.dll by typing the command [regsvr32 RPMImpex.dll]. You will get a message indicating the registration was succeeded.

Again, do the C:\Windows\PREFETCH\REGSVR32 clearing steps.

Lastly, You can register the RPMPlgin.ocx by typing the command [regsvr32 RPMPlgin.ocx]. You will get a message indicating the registration was succeeded.

Taa Daa, you should be able to run the IBM RPM client program now.

Good luck.

IBM Rational Portfolio Manager (RPM) - Installation Best Practices

The followings are some of the best practices or tips to avoid commonn pitfalls when you trying to install a new IBM RPM box. Your situation might not be the same as the one I illustrated here but you still might encounter the same errors/issues that I did.

Best Practice 1: Make Sure Everything else is working First

Start with installing the operating system, in my case, Microsoft Windows 2003 Enterprise Edition SP1 32 bit. Make sure that your operating system version and levels matched with those stated in the Installation Guide.

Then install the database server. I installed IBM DB2 UDB version 8.2.0 Workgroup Unlimited Edition. Immediately, patch up the DB2 to Fix Pack 14 (8.2.7 or 8.1.14).

After this, install the application server. I chosen IBM Websphere Application Server Base 6.1.0.0. You will need to download IBM Update Installer 6.1.0.x, where x is the fix pack version you want to deploy. In my case, I downloaded IBM Update Installer 6.1.0.15 because I want to patch up my WAS to 6.1.0.15.

Once the IBM Update Installer is installed, you shall use it to install IBM WAS 6.1 FP15.

I recommend you to configure your WAS Java Heap Size to larger size.

Possible Issues:

• If you are installing ProjectConsole in the same box as WAS server, make sure you configure the server to use different set of ports.

By end of this best practice, you shall have an environment ready for IBM RPM to invade. Backup/Ghost the system if necessary.

Best practice 2: Make sure your installation image is usable

Some common pitfalls are as follows:

• Contains spaces in the installation directories


• The installation image is in network mapped drive


• You should reset file attributes of all the files in the image folder to write-able.

Best practice 3: Install RPM Database Components


The path to IBM RPM 7.1.1.1 as far as I aware of, requires the prerequisite of RPM 7.1.0.0 installation first.

The best way to tackle this is to run the database scripts for RPM 7.1.0.0
Backup the database for RPM 7.1.0.0
and then directly run the scripts RPM 7.1.1.1 to directly migrate/update it.

Don't intend to install/patch database and midleware RPM components in interweaving steps. Remember, one component at a time.

Once you have database for RPM 7.1.1.1 ready, back it up

Possible Issues:

• DB2TEMPDIR is not specified or point to non existent directory or point to wrong directory.


• The specified database alias already exists in the database server. You need to "drop db yourdatabase".


• You specifies the same user name for both db_user and con_user for installation scenario 2.


• Make sure your tablespace container directories are empty.


• Ensure your PACKAGE DIR doesn't end with a slash. The installation process might end prematurely without error.


• Make sure your db_user or con_user password doesn't contain some special symbols such as !


• Make sure it is DB2 version 8.2. version 9.1 and above are not supported in 7.1.1.1


• If you are sharing your DB2 server with other applications, make sure the DBM CFG's NUMDB is set to value other than 1.


• My experience told me that RPM Installation Wizard only doesn't work with DB2 Workgroup Edition. DB2 ESE is fine but I'm not sure about other edition.

Best practice 4: Install RPM middleware components (EAR deployment)


Firstly, set up the necessary resources, authentications and environment variables as stated in the installation guide.

Then, you shall deploy the EAR for version 7.1.1.1 directly. There is no need to install version 7.1.0.0.

Possible Issues:

• Make sure you know your RPM deployment context root.

Best practice 5: Client Installation

Nothing much here. Make sure you never include the "https://" or "http://" when configuring the location in the Client Login because doing so will make the splash screen hung, well.. at least that happened in my case.

LDAP Integration between Microsoft Active Directory and IBM RPM 7.1

I was pissed off by useless guidance documents scattered in the Internet. My RPM team spent near to 3 hours to get it right. You know what, the biggest portion of the time wasted was when we following the step-by-step instructions that supposed to guide you toward successfull integration between this 2 major components: Microsoft AD and IBM RPM.

com.ibm.rpm.auth.jndi.JNDIController.properties file is the key configuration element that you used for such integration, ONLY for IBM RPM prior to 7.1 version. If you happened to install IBM RPM 7.1 and you will bang your big head on the monitor to basically make it bigger for wondering why the configuration is not working at all.

We started to suspect the correctness of configuring such file when we are diagnosing System.out log file in IBM Websphere AS 6.1. I saw interesting JNDI lookup names such as useLdapAuthentication and ldapConfiguration. So I decided to google again and Taa Daa, found some useful materials from IBM RPM forum.

Apparently the new mechanism in IBM RPM7.1 is to use JNDI object for LDAP configuration lookup instead of previous property files.

The following steps outline this mechanism (I wish to have some screen shots, but not really have time for that):

1. Access IBM WAS Administrative Console

2. Access Resource Environment Entries, under Resources

Basically the entries you created here will be referred by the RPM WAR application using web.xml resource-env-ref. It is good to browse through the list of available resource-env-ref mappings in the web.xml.

3. Create appropriate Resource Environment Entries

a. useLdapConfiguration

This is a switch to enable LDAP integration. Valid value to turn on is only "true".

b. ldapConfiguration

This is where you define the LDAP "connection string".

It must be in one line and each key pair is separated by spaces.

java.naming.provider.url="ldap://your_domain_name.com:389" java.naming.factory.initial="com.sun.jndi.ldap.LdapCtxFactory" java.naming.ldap.version="3"
java.naming.security.protocol="simple"
com.ibm.rpm.auth.jndi.ldapsearchcontext="OU=Your AD OU,dc=your_domain_name, dc=com" com.ibm.rpm.auth.jndi.ldapuseridattr="sAMAccountName"
java.naming.security.principal="your_ad_username"
java.naming.security.credentials="your_ad_password"

Things to note when constructing this string:

• Make sure your testing machine can access Microsoft AD at the specified port. Confirm with the network administrator if necessaery.


• If you are using Non-AD LDAP directory, it is possible that you might need to use the vendor supplied LDAP Context factory and different security protocol.


• ldapsearchcontext is really company specific. You will need to make sure your search context is at the right location, as defined by domain administrators.


• ldapuseridattr is configurable to use different AD attribute to map with RPM user names. I guess it should be possible to use your domain email as RPM login.


• com.ibm.rpm.auth.jndi.ldapuseridattr and java.naming.security.principal must be compatible, meaning if you specify to use distinguishedName as the mapping attribute, make sure your principal string is correct.

It is helpful to use SofTerra LDAP tools to troubleshoot. Click here

IBM Rational Portfolio Manager RPM - Enable Resource to Logon

You will need to do at least the following to enable a resource to logon to RPM.

• Set the Resource Status to Active


• Make sure the effective date and discontinued date are properly configured


• Associate a Calendar with the Resource

IBM Rational Portfolio Manager RPM - Default Resource Password

RPM assigns a default password to any newly created Resource.

ibmrpm