[oracle@EPRHEL6 admin]$ srvctl status database -d orcl Instance ORCL1 is running on node eprhel5 PRKO-2015 : Error in checking condition of instance on node: eprhel6 [oracle@EPRHEL6 admin]$ sqlplus system/password@ORCL2 SQL*Plus: Release 10.2.0.1.0 - Production on Mon Dec 27 00:03:11 2010 Copyright (c) 1982, 2005, Oracle. All rights reserved. ERROR: ORA-12514: TNS:listener does not currently know of service requested in connect descriptor Enter user-name:
srvctl also complains when i was trying to start the instance on the second node. So i decided to start the instance manually by using sqlplus.
[oracle@EPRHEL6 admin]$ sqlplus "/ as sysdba" SQL*Plus: Release 10.2.0.1.0 - Production on Mon Dec 27 00:03:24 2010 Copyright (c) 1982, 2005, Oracle. All rights reserved. Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options SQL> startup; Oracle instance started. Total System Global Area 599785472 bytes Fixed Size 2022600 bytes Variable Size 188744504 bytes Database Buffers 402653184 bytes Redo Buffers 6365184 bytes Database mounted. Database opened. SQL> alter system register; System altered. SQL> exit Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options [oracle@EPRHEL6 admin]$ sqlplus system/password@ORCL2 SQL*Plus: Release 10.2.0.1.0 - Production on Mon Dec 27 00:04:18 2010 Copyright (c) 1982, 2005, Oracle. All rights reserved. Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options SQL> exit Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options
It seems there is not any problem with the instance itself. Sqlplus barely connects to the instance ORCL2. There should be a problem about the way of communication between srvctl and the instance.
[oracle@EPRHEL6 admin]$ crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....L1.inst application ONLINE ONLINE eprhel5 ora....L2.inst application ONLINE UNKNOWN eprhel6 ora.ORCL.db application ONLINE ONLINE eprhel5 ora....SM1.asm application ONLINE ONLINE eprhel5 ora....L5.lsnr application ONLINE ONLINE eprhel5 ora....el5.gsd application ONLINE ONLINE eprhel5 ora....el5.ons application ONLINE ONLINE eprhel5 ora....el5.vip application ONLINE ONLINE eprhel5 ora....SM2.asm application ONLINE ONLINE eprhel6 ora....L5.lsnr application OFFLINE OFFLINE ora....L6.lsnr application ONLINE ONLINE eprhel6 ora....el6.gsd application ONLINE ONLINE eprhel6 ora....el6.ons application ONLINE ONLINE eprhel6 ora....el6.vip application ONLINE ONLINE eprhel6 [oracle@EPRHEL6 admin]$ srvctl start listener -n EPRHEL6 [oracle@EPRHEL6 admin]$ crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....L1.inst application ONLINE ONLINE eprhel5 ora....L2.inst application ONLINE UNKNOWN eprhel6 ora.ORCL.db application ONLINE ONLINE eprhel5 ora....SM1.asm application ONLINE ONLINE eprhel5 ora....L5.lsnr application ONLINE ONLINE eprhel5 ora....el5.gsd application ONLINE ONLINE eprhel5 ora....el5.ons application ONLINE ONLINE eprhel5 ora....el5.vip application ONLINE ONLINE eprhel5 ora....SM2.asm application ONLINE ONLINE eprhel6 ora....L5.lsnr application OFFLINE OFFLINE ora....L6.lsnr application ONLINE ONLINE eprhel6 ora....el6.gsd application ONLINE ONLINE eprhel6 ora....el6.ons application ONLINE ONLINE eprhel6 ora....el6.vip application ONLINE ONLINE eprhel6 [oracle@EPRHEL6 admin]$ sqlplus system/password@ORCL1 SQL*Plus: Release 10.2.0.1.0 - Production on Mon Dec 27 00:04:35 2010 Copyright (c) 1982, 2005, Oracle. All rights reserved. Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options SQL> show parameter listener; NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ local_listener string remote_listener string LISTENERS_ORCL SQL> exit Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - 64bit Production With the Partitioning, Real Application Clusters, OLAP and Data Mining options
I think there is a problem with listener configuration or the listener resource itself. But everything seems fine except that OFFLINE resource. After searching google a little bit, i found a solution indicates listener configuration. I decided to recreate the listeners with netca. I will first delete listener named LISTENER from both ASM and DB homes using netca and then recreate them only using DB home. Maybe this resolves the problem.
My action plan is first stop all asm and db instances. Manually remove that OFFLINE listener which is very confusing. Remove all the listener configuration from the cluster with netca and recreate using db home. Here we go.
[oracle@EPRHEL6 db]$ lsnrctl status LSNRCTL for Linux: Version 10.2.0.1.0 - Production on 27-DEC-2010 00:22:16 Copyright (c) 1991, 2005, Oracle. All rights reserved. Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521)) STATUS of the LISTENER ------------------------ Alias LISTENER_EPRHEL6 Version TNSLSNR for Linux: Version 10.2.0.1.0 - Production Start Date 27-DEC-2010 00:02:31 Uptime 0 days 0 hr. 19 min. 44 sec Trace Level off Security ON: Local OS Authentication SNMP OFF Listener Parameter File /oracle/product/asm/network/admin/listener.ora Listener Log File /oracle/product/asm/network/log/listener_eprhel6.log Listening Endpoints Summary... (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=172.28.4.226)(PORT=1521))) (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=172.28.4.246)(PORT=1521))) (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC))) Services Summary... Service "+ASM" has 1 instance(s). Instance "+ASM2", status BLOCKED, has 1 handler(s) for this service... Service "+ASM_XPT" has 1 instance(s). Instance "+ASM2", status BLOCKED, has 1 handler(s) for this service... Service "ORCL" has 2 instance(s). Instance "ORCL1", status READY, has 1 handler(s) for this service... Instance "ORCL2", status READY, has 2 handler(s) for this service... Service "ORCLXDB" has 2 instance(s). Instance "ORCL1", status READY, has 1 handler(s) for this service... Instance "ORCL2", status READY, has 1 handler(s) for this service... Service "ORCL_XPT" has 2 instance(s). Instance "ORCL1", status READY, has 1 handler(s) for this service... Instance "ORCL2", status READY, has 2 handler(s) for this service... The command completed successfully [oracle@EPRHEL6 db]$ srvctl stop database -d orcl [oracle@EPRHEL6 db]$ srvctl stop asm -n EPRHEL5 [oracle@EPRHEL6 db]$ srvctl stop asm -n EPRHEL6 [oracle@EPRHEL6 db]$ crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....L1.inst application OFFLINE OFFLINE ora....L2.inst application OFFLINE OFFLINE ora.ORCL.db application OFFLINE OFFLINE ora....SM1.asm application OFFLINE OFFLINE ora....el5.gsd application ONLINE ONLINE eprhel5 ora....el5.ons application ONLINE ONLINE eprhel5 ora....el5.vip application ONLINE ONLINE eprhel5 ora....SM2.asm application OFFLINE OFFLINE ora....L5.lsnr application OFFLINE OFFLINE ora....el6.gsd application ONLINE ONLINE eprhel6 ora....el6.ons application ONLINE ONLINE eprhel6 ora....el6.vip application ONLINE ONLINE eprhel6 [oracle@EPRHEL6 db]$ crs_getperm ora.eprhel6.LISTENER_EPRHEL5.lsnr Name: ora.eprhel6.LISTENER_EPRHEL5.lsnr owner:oracle:rwx,pgrp:dba:rwx,other::r--, [oracle@EPRHEL6 db]$ crs_unregister ora.eprhel6.LISTENER_EPRHEL5.lsnr [oracle@EPRHEL6 db]$ crs_profile -delete ora.eprhel6.LISTENER_EPRHEL5.lsnr CRS-0170: The resource 'ora.eprhel6.LISTENER_EPRHEL5.lsnr' doesn't exist. [oracle@EPRHEL6 db]$ crs_stat -t Name Type Target State Host ------------------------------------------------------------ ora....L1.inst application ONLINE ONLINE eprhel5 ora....L2.inst application ONLINE ONLINE eprhel6 ora.ORCL.db application ONLINE ONLINE eprhel5 ora....SM1.asm application ONLINE ONLINE eprhel5 ora....L5.lsnr application ONLINE ONLINE eprhel5 ora....el5.gsd application ONLINE ONLINE eprhel5 ora....el5.ons application ONLINE ONLINE eprhel5 ora....el5.vip application ONLINE ONLINE eprhel5 ora....SM2.asm application ONLINE ONLINE eprhel6 ora....L6.lsnr application ONLINE ONLINE eprhel6 ora....el6.gsd application ONLINE ONLINE eprhel6 ora....el6.ons application ONLINE ONLINE eprhel6 ora....el6.vip application ONLINE ONLINE eprhel6 [oracle@EPRHEL6 db]$ srvctl status database -d orcl Instance ORCL1 is running on node eprhel5 Instance ORCL2 is running on node eprhel6 [oracle@EPRHEL6 db]$
It seems problem is solved.
