DB Console HTTPS (SSL) Certificate Expires

Environment

Oracle 10.2.0.5 on Aix 5.3 64-bit (this problem happens on general platforms).

Problem

Failed to start up dbconsole and in $ORACLE_HOME/<hostname>_<ORACLE_SID>/sysman/log/emdctl.trc it appears error

“ERROR ssl: nzos_Handshake failed, ret=29024”

Cause

When the EM DB Console is secured (HTTPS), the SSL certificate that is generated has a lifetime of 6 months. The lifetime of the certificate have been extended by default to 10 years in the 11G and higher versions of the DB Console. So the issue should not be encountered in those versions.

Details

— Today customer reports that db control can’t be started up and it shows the following error:

$ emctl status dbconsole

Agent is already started. Will restart the agent
Stopping agent … stopped.
Starting Oracle Enterprise Manager 10g Database Control …………… failed.

——————————————————————
Logs are generated in directory /gold/GLP/bin/oracle/product/10.2.0/db_1/glpdb_GOLDPROD/sysman/log

— In the log file ORACLE_HOME/<hostname>_<DB_NAME>/sysman/logemdb.nohup it reports this error:

emwd.pl arch is aix-thread-multi—– Mon Jul  4 10:45:16 2011::Console Launched with PID 688370 at time Mon Jul  4 10:45:16 2011 —–
11/07/04 10:45:19 Error starting ORMI-Server.  Unable to bind socket: The socket name is already in use.

— it turns out that the db console has not been stoped cleanly. Try to stop the db control

$ emctl stop dbconsole
Oracle Enterprise Manager 10g Database Control Release 10.2.0.5.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
https://glpdb:1158/em/console/aboutApplication
Stopping Oracle Enterprise Manager 10g Database Control …

click “<Ctrl>-C”

— Failed to shutdown DBConsole Gracefully —
failed.

— Stop dbconsole by killing OS processes ‘oc4j’ and ‘dbconsole’

$ ps -ef |grep oc4j
glporacl  659594       1   0   Jun 12      – 25:36 /gold/GLP/bin/oracle/product/10.2.0/db_1/jdk/bin/java -Xmx256M -DORACLE_HOME=/gold/GLP/bin/oracle/product/10.2.0/db_1 -Doracle.home=/gold/GLP/bin/oracle/product/10.2.0/db_1/oc4j -Doracle.oc4j.localhome=/gold/GLP/bin/oracle/product/10.2.0/db_1/glpdb_GOLDPROD/sysman -DEMSTATE=/gold/GLP/bin/oracle/product/10.2.0/db_1/glpdb_GOLDPROD -Doracle.j2ee.dont.use.memory.archive=true -Djava.protocol.handler.pkgs=HTTPClient -Doracle.security.jazn.config=/gold/GLP/bin/oracle/product/10.2.0/db_1/oc4j/j2ee/OC4J_DBConsole_glpdb_GOLDPROD/config/jazn.xml -Djava.security.policy=/gold/GLP/bin/oracle/product/10.2.0/db_1/oc4j/j2ee/OC4J_DBConsole_glpdb_GOLDPROD/config/java2.policy -Djava.security.properties=/gold/GLP/bin/oracle/product/10.2.0/db_1/oc4j/j2ee/home/config/jazn.security.props -DEMDROOT=/gold/GLP/bin/oracle/product/10.2.0/db_1/glpdb_GOLDPROD -Dsysman.md5password=true -Drepapi.oracle.home=/gold/GLP/bin/oracle/product/10.2.0/db_1 -Ddisable.checkForUpdate=true -Djava.awt.headless=true -jar /gold/GLP/bin/oracle/product/10.2.0/db_1/oc4j/j2ee/home/oc4j.jar -config /gold/GLP/bin/oracle/product/10.2.0/db_1/oc4j/j2ee/OC4J_DBConsole_glpdb_GOLDPROD/config/server.xml

$ ps -ef |grep dbconsole
glporacl  520354 1429626   0 11:03:56  pts/1  0:00 grep dbconsole
glporacl  581636       1   0 10:35:38  pts/1  0:00 /gold/GLP/bin/oracle/product/10.2.0/db_1/perl/bin/perl /gold/GLP/bin/oracle/product/10.2.0/db_1/bin/emwd.pl dbconsole /gold/GLP/bin/oracle/product/10.2.0/db_1/glpdb_GOLDPROD/sysman/log/emdb.nohup

$ kill -9 659594 581636

— Verify if the console server port is still being used by some process.

$ grep ConsoleServerPort  /gold/GLP/bin/oracle/product/10.2.0/db_1/glpdb_GOLDPROD/sysman/config/emoms.properties

oracle.sysman.emSDK.svlt.ConsoleServerPort=1158

— check if the port is still being used by other processes, if so change configuration as demonstrated in note
EMCA or DB Control (DBConsole) Fails with Error starting ORMI-Server [ID 438504.1]

$ netstat -na |grep 1158

<NONE>

— Now try to start dbconsole again and it still failed with error

$ emctl start dbconsole

in emagent.trc it reports:

Thread-1 ERROR ssl: nzos_Handshake failed, ret=29024
Thread-1 ERROR http: 6: Unable to initialize ssl connection with server, aborting connection attempt

Metalink note “PROBLEM : Dbconsole Fails To Start With Error “Nzos_handshake Failed,Ret=29024″ [ID 749243.1]” provides the solution:

1) stop em agent and dbconsole if they rmain hanging

$ ps -ef |grep oc4j
$ ps -ef |grep dbconsole
$ kill -9 <pid_oc4j> <pid_of_dbconsole>

2) Unsecure dbconsole

$ emctl unsecure dbconsole

Oracle Enterprise Manager 10g Database Control Release 10.2.0.5.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
https://glpdb:1158/em/console/aboutApplication
Configuring DBConsole for HTTP…   Done.
DBCONSOLE already stopped…   Done.
Agent successfully stopped…   Done.
Unsecuring dbconsole…   Started.
DBConsole is now unsecured…  Done.
Unsecuring dbconsole…  Sucessful.

3) Secure dbconsole again

$ emctl secure dbconsole
Oracle Enterprise Manager 10g Database Control Release 10.2.0.5.0
Copyright (c) 1996, 2010 Oracle Corporation.  All rights reserved.
http://glpdb:1158/em/console/aboutApplication
Enter Enterprise Manager Root password :
Enter a Hostname for this OMS : glpdb

DBCONSOLE already stopped…   Done.
Agent is already stopped…   Done.
Securing dbconsole…   Started.
Checking Repository…   Done.
Checking Em Key…   Done.
Checking Repository for an existing Enterprise Manager Root Key…   Done.
Fetching Root Certificate from the Repository…   Done.
Updating HTTPS port in emoms.properties file…   Done.
Generating Java Keystore…   Done.
Securing OMS …   Done.
Generating Oracle Wallet Password for Agent….   Done.
Generating wallet for Agent …    Done.
Copying the wallet for agent use…    Done.
Storing agent key in repository…   Done.
Storing agent key for agent …   Done.
Configuring Agent…
Configuring Agent for HTTPS in DBCONSOLE mode…   Done.
EMD_URL set in /gold/GLP/bin/oracle/product/10.2.0/db_1/glpdb_GOLDPROD/sysman/config/emd.properties
Done.
Configuring Key store..   Done.
Securing dbconsole…   Sucessful.

=============================================================================================================================
Note :
When the EM DB Console is secured (HTTPS), the SSL certificate that is generated has a lifetime of 6 months.
The lifetime of the certificate have been extended by default to 10 years in the 11G and higher versions of the DB Console.
So the issue should not be encountered in those versions.
Refer Note 738659.1 DB Console HTTPS (SSL) Certificate Has Expired. Error “ssl: nzos_Handshake failed, ret=29024”
Encountered in Emagent Logs
=============================================================================================================================

4) start db console

$ emctl start dbconsole

Starting Oracle Enterprise Manager 10g Database Control ……………… started.

5) Done.

Reference

PROBLEM : Dbconsole Fails To Start With Error “Nzos_handshake Failed,Ret=29024” [ID 749243.1]

DB Console HTTPS (SSL) Certificate Has Expired. Error “ssl: nzos_Handshake failed, ret=29024” Encountered in Emagent Logs [ID 738659.1]

EMCA or DB Control (DBConsole) Fails with Error starting ORMI-Server [ID 438504.1]

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: