srvctl problem after 10.2.0.4 patch

New migration project of the database and system group in our company is from Oracle Database 9.2.0.5 RAC on LINUX AS3 32 bit to Oracle Database RAC 10.2.0.4 on LINUX ES4 Update6 64 bit. After installing clusterware and the database 10.2.0.1 successfully, i decided to upgrade the database and the clusterware to the last level of patchset 10.2.0.4. After installing the patchset i realised that srvctl cannot open the databases with the command srvctl start database -d PQ. But i can open the databases individually with the sqlplus. The wierd thing is srvctl works with "status" and "close" parameters. When i examine the crs logs i founded the following error logs.

#cat /oracle/product/crs/log/pqvrtsrv1/crsd/crsd.log
2008-07-16 17:52:37.840: [  CRSAPP][1535326560]0StartResource error for ora.PQ.db error code = 1
2008-07-16 17:52:37.892: [  CRSRES][1535326560]0Start of `ora.PQ.db` on member `pqvrtsrv1` failed.

This one error message did not satisfy me to start taking action for the problem hence it has too few information to make a conclusion. I decided to look at the databases alert.log, if i can find any clue about the problem. But the last logs written to the alert.log was the databases closing messages. This may be a clue, because this means srvctl cannot reach the database to open it, now i can start from srvctl settings to examine.

$ srvctl config database -d PQ -a
pqvrtsrv1 PQ1 /oracle/product/db10g
pqvrtsrv2 PQ2 /oracle/product/db10g
DB_NAME: PQ
ORACLE_HOME: /oracle/product/db10g
SPFILE: /pqdata1/PQ/PQ/spfilePQ.ora
DOMAIN: null
DB_ROLE: null
START_OPTIONS: null
POLICY:  AUTOMATIC
ENABLE FLAG: DB ENABLED

Now i can see things. SPFILE configuration is wrong for this database. I clearly remember that i have changed the spfile location to /pqdata1/PQ/spfilePQ.ora. But somehow after the patch is installed, setting are confused. I changed the SPFILE configuration with the correct settings and my problem is now solved.

#srvctl modify database -d PQ -p /pqdata1/PQ/spfilePQ.ora
#srvctl start instance -d PQ -i PQ2
#srvctl status database -d PQ
Instance PQ1 is running on node pqvrtsrv1
Instance PQ2 is running on node pqvrtsrv2

Resources:

Server Control Utility Reference

Changing sysman password in OMS

I had to change the sysman password today. There are nice articles on the web but in summary, if you want to change the sysman password of the Oracle Enterprise Manager Grid Control Server;

1- stop all opmn processes
2- stop the oms
3- stop the agent
4- change the password of the sysman user
5- update the $AGENT_HOME/sysman/emd/targets.xml file
6- update the $OMS_HOME/sysman/config/emoms.properties file
7- start the agent
8- start the oms
9- start all opmn processes (actually they are being started by oms)

Here are the handy articles i used;

http://oraclespin.wordpress.com/2008/07/18/how-to-change-sysman-password-in-oem-oms/
http://thetendjee.wordpress.com/2006/07/19/changing-sysman-and-dbsnmp-passwords-it-aint-that-easy/
http://dbataj.blogspot.com/2007/07/change-sysman-password.html