11gR2; Troubleshooting OHAS error – autorun file for ohasd is missing



Introduction:
The present document provides a list of steps to solve OHAS error – autorun file for ohasd is missing.
Environment:
The database server version is 11.2.0.3.4 running on Oracle Enterprise Linux 6. The Oracle clusterware version is 11.2.0.3.4 with Patch Set Update 4 applied.
Problem:
The cluster resources cannot start automatically after a server reboot.
crsctl start res -all gives CRS-4000: Command Get failed, or completed with errors
/var/log/messages file shows the followings messages:
view plaincopy to clipboardprint?
  1. Oct 26 18:27:52 srv300db1 logger: autorun file for ohasd is missing  
  2. Oct 26 18:28:02 srv300db1 logger: autorun file for ohasd is missing  
  3. Oct 26 18:28:12 srv300db1 logger: autorun file for ohasd is missing  
  4. Oct 26 18:28:22 srv300db1 logger: autorun file for ohasd is missing  
  5. Oct 26 18:28:32 srv300db1 logger: autorun file for ohasd is missing  
  6. Oct 26 18:28:42 srv300db1 logger: autorun file for ohasd is missing  
  7. Oct 26 18:28:52 srv300db1 logger: autorun file for ohasd is missing  
  8. Oct 26 18:29:02 srv300db1 logger: autorun file for ohasd is missing  
  9. Oct 26 18:29:12 srv300db1 logger: autorun file for ohasd is missing  
  10. Oct 26 18:29:22 srv300db1 logger: autorun file for ohasd is missing  
  11. Oct 26 18:29:32 srv300db1 logger: autorun file for ohasd is missing  
  12. Oct 26 18:29:42 srv300db1 logger: autorun file for ohasd is missing  
  13. Oct 26 18:29:52 srv300db1 logger: autorun file for ohasd is missing  
  14. Oct 26 18:30:02 srv300db1 logger: autorun file for ohasd is missing  
  15. Oct 26 18:30:12 srv300db1 logger: autorun file for ohasd is missing  
  16. Oct 26 18:30:22 srv300db1 logger: autorun file for ohasd is missing  
  17. Oct 26 18:30:32 srv300db1 logger: autorun file for ohasd is missing  
  18. Oct 26 18:30:42 srv300db1 logger: autorun file for ohasd is missing  
Solution
Follow the steps below to fix the issue;
view plaincopy to clipboardprint?
  1. -bash-4.1# id  
  2. uid=0(root) gid=0(root) groups=0(root),1(bin),2(daemon),3(sys),4(adm),6(disk),10(wheel)  
  3.   
  4. --- Deconfigure Oraclem Restart , OHA stack  
  5.   
  6. -bash-4.1# /opt/11.2.0/grid/crs/install/roothas.pl -verbose -deconfig -f  
  7. Using configuration parameter file: /opt/11.2.0/grid/crs/install/crsconfig_params  
  8. CRS-4639: Could not contact Oracle High Availability Services  
  9. CRS-4000: Command Stop failed, or completed with errors.  
  10. CRS-4639: Could not contact Oracle High Availability Services  
  11. CRS-4000: Command Delete failed, or completed with errors.  
  12. CRS-4544: Unable to connect to OHAS  
  13. CRS-4000: Command Stop failed, or completed with errors.  
  14. Successfully deconfigured Oracle Restart stack  
view plaincopy to clipboardprint?
  1. --- Configure the OHA stack  
  2. -bash-4.1# /opt/11.2.0/grid/crs/install/roothas.pl -verbose  
  3. Using configuration parameter file: /opt/11.2.0/grid/crs/install/crsconfig_params  
  4. LOCAL ADD MODE  
  5. Creating OCR keys for user 'grid', privgrp 'oinstall'..  
  6. Operation successful.  
  7. LOCAL ONLY MODE  
  8. Successfully accumulated necessary OCR keys.  
  9. Creating OCR keys for user 'root', privgrp 'root'..  
  10. Operation successful.  
  11. CRS-4664: Node srv300db1 successfully pinned.  
  12. Adding Clusterware entries to upstart  
  13.   
  14. srv300db1     2012/10/27 03:31:04     /opt/11.2.0/grid/cdata/srv300db1/backup_20121027_033104.olr  
  15. Successfully configured Oracle Grid Infrastructure for a Standalone Server  
  16. -bash-4.1#  
  17.   
  18. -bash-4.1# su - grid  
  19. -bash-4.1$ crsctl check css  
  20. CRS-4530: Communications failure contacting Cluster Synchronization Services daemon  
  21. -bash-4.1$ crsctl status res -t  
  22. --------------------------------------------------------------------------------  
  23. NAME           TARGET  STATE        SERVER                   STATE_DETAILS  
  24. --------------------------------------------------------------------------------  
  25. Local Resources  
  26. --------------------------------------------------------------------------------  
  27. ora.ons  
  28.                OFFLINE OFFLINE      srv300db1  
  29. --------------------------------------------------------------------------------  
  30. Cluster Resources  
  31. --------------------------------------------------------------------------------  
  32. ora.cssd  
  33.       1        OFFLINE OFFLINE  
  34. ora.diskmon  
  35.       1        OFFLINE OFFLINE  
  36. ora.evmd  
  37.       1        ONLINE  ONLINE       srv300db1  
— add ASM resource
view plaincopy to clipboardprint?
  1. -bash-4.1$ srvctl status asm  
  2. PRCR-1001 : Resource ora.asm does not exist  
  3. -bash-4.1$ srvctl add asm  
  4. -bash-4.1$ srvctl status asm  
  5. ASM is not running.  
  6. -bash-4.1$ srvctl start asm  
  7. -bash-4.1$ srvctl status asm  
  8. ASM is running on srv300db1  
  9. -bash-4.1$ crsctl status res -t  
  10. --------------------------------------------------------------------------------  
  11. NAME           TARGET  STATE        SERVER                   STATE_DETAILS  
  12. --------------------------------------------------------------------------------  
  13. Local Resources  
  14. --------------------------------------------------------------------------------  
  15. ora.asm  
  16.                ONLINE  ONLINE       srv300db1                 Started  
  17. ora.ons  
  18.                OFFLINE OFFLINE      srv300db1  
  19. --------------------------------------------------------------------------------  
  20. Cluster Resources  
  21. --------------------------------------------------------------------------------  
  22. ora.cssd  
  23.       1        ONLINE  ONLINE       srv300db1  
  24. ora.diskmon  
  25.       1        OFFLINE OFFLINE  
  26. ora.evmd  
  27.       1        ONLINE  ONLINE       srv300db1  
  28. -bash-4.1$  
MOUNT THE DISKGROUPS DATA and FRA using asmca tool. Click on Mount all button to mount all diskgroups.
view plaincopy to clipboardprint?
  1. -bash-4.1# su - grid  
  2. -bash-4.1$ sqlplus / as sysasm  
  3.   
  4. SQL*Plus: Release 11.2.0.3.0 Production on Fri Oct 26 18:42:09 2012  
  5.   
  6. Copyright (c) 1982, 2011, Oracle.  All rights reserved.  
  7.   
  8.   
  9. Connected to:  
  10. Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production  
  11. With the Automatic Storage Management option  
  12.   
  13. SQL> select NAME from V$ASM_DISKGROUP;  
  14.   
  15. NAME  
  16. ------------------------------  
  17. FRA  
  18. DATA  
  19.   
  20. SQL>  
— Check diskgroup resources added
view plaincopy to clipboardprint?
  1. -bash-4.1$ crsctl status res -t  
  2. --------------------------------------------------------------------------------  
  3. NAME           TARGET  STATE        SERVER                   STATE_DETAILS  
  4. --------------------------------------------------------------------------------  
  5. Local Resources  
  6. --------------------------------------------------------------------------------  
  7. ora.DATA.dg  
  8.                ONLINE  ONLINE       srv300db1  
  9. ora.FRA.dg  
  10.                ONLINE  ONLINE       srv300db1  
  11. ora.asm  
  12.                ONLINE  ONLINE       srv300db1                 Started  
  13. ora.ons  
  14.                OFFLINE OFFLINE      srv300db1  
  15. --------------------------------------------------------------------------------  
  16. Cluster Resources  
  17. --------------------------------------------------------------------------------  
  18. ora.cssd  
  19.       1        ONLINE  ONLINE       srv300db1  
  20. ora.diskmon  
  21.       1        OFFLINE OFFLINE  
  22. ora.evmd  
  23.       1        ONLINE  ONLINE       srv300db1  
  24. -bash-4.1$  
— add Listener resource
view plaincopy to clipboardprint?
  1. -bash-4.1$ srvctl start listener  
  2. PRCN-2044 : No listener exists  
  3. -bash-4.1$  
  4.   
  5. bash-4.1$ srvctl add listener -l LISTENER  
  6. -bash-4.1$ srvctl add listener -l LISTENER_2  
  7. -bash-4.1$ srvctl start listener -l LISTENER  
  8. -bash-4.1$ srvctl start listener -l LISTENER_2  
  9. -bash-4.1$  
  10. -bash-4.1$ crsctl status res -t  
  11. --------------------------------------------------------------------------------  
  12. NAME           TARGET  STATE        SERVER                   STATE_DETAILS  
  13. --------------------------------------------------------------------------------  
  14. Local Resources  
  15. --------------------------------------------------------------------------------  
  16. ora.DATA.dg  
  17.                ONLINE  ONLINE       srv300db1  
  18. ora.FRA.dg  
  19.                ONLINE  ONLINE       srv300db1  
  20. ora.LISTENER.lsnr  
  21.                ONLINE  ONLINE       srv300db1  
  22. ora.LISTENER_2.lsnr  
  23.                ONLINE  ONLINE       srv300db1        
  24. ora.asm  
  25.                ONLINE  ONLINE       srv300db1                 Started  
  26. ora.ons  
  27.                OFFLINE OFFLINE      srv300db1  
  28. --------------------------------------------------------------------------------  
  29. Cluster Resources  
  30. --------------------------------------------------------------------------------  
  31. ora.cssd  
  32.       1        ONLINE  ONLINE       srv300db1  
  33. ora.diskmon  
  34.       1        OFFLINE OFFLINE  
  35. ora.evmd  
  36.       1        ONLINE  ONLINE       srv300db1  
  37. -bash-4.1$  
As Oracle user: spfile.ora file can be obtained by looking for spfile string in the alert log file, we should take the value from the last startup command…
view plaincopy to clipboardprint?
  1. -bash-4.1# su - oracle  
  2. -bash-4.1$ adrci  
  3.   
  4. ADRCI: Release 11.2.0.3.0 - Production on Fri Oct 26 18:47:42 2012  
  5.   
  6. Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.  
  7.   
  8. ADR base = "/opt/app/oracle"  
  9. adrci> show alert  
  10.   
  11. Choose the alert log from the following homes to view:  
  12.   
  13. 1: diag/tnslsnr/srv300db1/listener  
  14. 2: diag/asm/+asm/+ASM  
  15. 3: diag/rdbms/mytstdb/mytstdb  
  16. Q: to quit  
  17.   
  18. Please select option: 3  
  19. Output the results to file: /tmp/alert_7214_14033_mytstdb_1.ado  
The spfile value in alrt log file shows the following ; spfile = “+DATA/mytstdb/spfilemytstdb.ora”
— add Database Instance resource
view plaincopy to clipboardprint?
  1. -bash-4.1# su - oracle  
  2. -bash-4.1$ srvctl add database -d mytstdb -o /opt/app/oracle/product/11.2.0/db_1 -p +DATA/mytstdb/spfilemytstdb.ora -s open  
  3.   
  4. -bash-4.1$ srvctl start database -d mytstdb  
— Check Listener
view plaincopy to clipboardprint?
  1. -bash-4.1$ lsnrctl stat  
  2.   
  3. LSNRCTL for Linux: Version 11.2.0.3.0 - Production on 26-OCT-2012 18:51:52  
  4.   
  5. Copyright (c) 1991, 2011, Oracle.  All rights reserved.  
  6.   
  7. Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))  
  8. STATUS of the LISTENER  
  9. ------------------------  
  10. Alias                     LISTENER  
  11. Version                   TNSLSNR for Linux: Version 11.2.0.3.0 - Production  
  12. Start Date                27-OCT-2012 03:45:56  
  13. Uptime                    0 days 0 hr. 5 min. 56 sec  
  14. Trace Level               off  
  15. Security                  ON: Local OS Authentication  
  16. SNMP                      OFF  
  17. Listener Parameter File   /opt/11.2.0/grid/network/admin/listener.ora  
  18. Listener Log File         /opt/app/oracle/diag/tnslsnr/srv300db1/listener/alert/log.xml  
  19. Listening Endpoints Summary...  
  20.   (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=srv300db1.oracle-class.com)(PORT=1521)))  
  21. Services Summary...  
  22. Service "+ASM" has 1 instance(s).  
  23.   Instance "+ASM", status READY, has 1 handler(s) for this service...  
  24. Service "mytstdb.oracle-class.com" has 1 instance(s).  
  25.   Instance "mytstdb", status READY, has 1 handler(s) for this service...  
  26. Service "mytstdbXDB.oracle-class.com" has 1 instance(s).  
  27.   Instance "mytstdb", status READY, has 1 handler(s) for this service...  
  28. The command completed successfully  
  29. -bash-4.1$  
  30.   
  31. -bash-4.1# su - grid  
  32. -bash-4.1$ crsctl status res -t  
  33. --------------------------------------------------------------------------------  
  34. NAME           TARGET  STATE        SERVER                   STATE_DETAILS  
  35. --------------------------------------------------------------------------------  
  36. Local Resources  
  37. --------------------------------------------------------------------------------  
  38. ora.DATA.dg  
  39.                ONLINE  ONLINE       srv300db1  
  40. ora.FRA.dg  
  41.                ONLINE  ONLINE       srv300db1  
  42. ora.LISTENER.lsnr  
  43.                ONLINE  ONLINE       srv300db1  
  44. ora.LISTENER_2.lsnr  
  45.                ONLINE  ONLINE       srv300db1                   
  46. ora.asm  
  47.                ONLINE  ONLINE       srv300db1                 Started  
  48. ora.ons  
  49.                OFFLINE OFFLINE      srv300db1  
  50. --------------------------------------------------------------------------------  
  51. Cluster Resources  
  52. --------------------------------------------------------------------------------  
  53. ora.cssd  
  54.       1        ONLINE  ONLINE       srv300db1  
  55. ora.diskmon  
  56.       1        OFFLINE OFFLINE  
  57. ora.evmd  
  58.       1        ONLINE  ONLINE       srv300db1  
  59. ora.mytstdb.db  
  60.       1        ONLINE  ONLINE       srv300db1                 Open  
  61. -bash-4.1$  

Comments

  1. Arief PradiptoNovember 29, 2016 at 4:21 AM

    This post helps me a lot! Many thanks

    ReplyDelete

Post a Comment

Popular posts from this blog

SQLNET.ORA Parameters: SQLNET.EXPIRE_TIME & SQLNET.INBOUND_CONNECT_TIMEOUT

Finally after a long long time posting on this blog.  Hi All,  Oracle documentation states SQLNET.EXPIRE_TIME parameter is used to avoid  unused sessions to be kept open in database and  locking resources. For SQLNET.INBOUND_CONNECT_TIMEOUT, it states:  If the client fails to establish a connection and complete authentication in the time specified, then the database server terminates the connection. In addition, the database server logs the IP address of the client and an  ORA-12170: TNS:Connect timeout occurred  error message to tnslsnr diag destination ($ORACLE_HOME/diag/tnslsnr/<listener_name>/trace/<listener.log> . The client receives either an  ORA-12547: TNS:lost contact  or an  ORA-12637: Packet receive failed  error message.and also records in alert log if audit is enabled for IPs.  In our environment both parameters are set for default value which I feel were too low SQLNET.EXPIRE_TIME (0 - dis...
Read more

Administering an Oracle Database Instance Using ORADIM

ORADIM is a command-line tool that is available with Oracle Database. You are required to use ORADIM only if you are manually creating, deleting, or modifying databases. Oracle Database Configuration Assistant is an easier tool to use for this purpose. Starting with Oracle Database 12 c  Release 1 (12.1), ORADIM creates Oracle Database service, Oracle VSS Writer service, and Oracle Scheduler service to run under the Oracle Home User account. If this account is a Windows Local User Account or Windows Domain User Account, then ORADIM prompts for password for that account and accepts the same through  stdin . It is possible to specify both the Oracle Home User and its password using the  -RUNAS osusr[/ospass]  option to  oradim . If the given  osusr  is different from the Oracle Home User, then the Oracle Home User is used instead of  osusr  along with the given  ospass . The following sections describe ORADIM commands and paramet...
Read more

How to resolve "`/root/.gvfs':Permission denied" when running df command from non-root user

When root user is logging in to GNOME, error "`/root/.gvfs':Permission denied" occurs if a normal user runs df. Solution Verified - Updated May 6 2014 at 12:16 AM - English English Environment Red Hat Enterprise Linux (RHEL) 6 GNOME Desktop Environment Issue When root user is logging in to GNOME, the following error occurs if a normal user runs df. $ df -kl Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda2 24670076 3751464 19665444 17% / tmpfs 1030908 100 1030808 1% /dev/shm /dev/sda1 99150 25491 68539 28% /boot df: `/root/.gvfs': Permission denied Resolution To disable gvfs-fuse-daemon , the GVFS_DISABLE_FUSE environment needs to be set. Create the file /etc/X11/xinit/xinitrc.d/00-gvfs-disable-fuse.sh with the follwing contents: GVFS_DISABLE_FUSE=1 export GVFS_DISABLE_FUSE Log out and log in again. If vncserver is running, restart ...
Read more