Sunday, July 21, 2013

Oracle Enterprise Manager 12c: Unable to Login using SYSMAN after unsuccessful Upgrade

After an incomplete attempt to upgrade OEM 12.1.0.2.0 to 12.1.0.3.0, a few days later I received a call from my client that the sysman password is not working. Interesting.

We validated the sysman account from the database and it was correct. The issue was on the WebLogic layer.

So, what changed? According to the customer, their Systems Administrators applied OS Security Patches during the weekend.

Other than that, the one apparent thing you need to do as part of the pre-upgrade steps is to copy the emkey to the repository. Did I remove it?

D:\>D:\oracle\app\product\12.1.0\middleware1\oms\bin\emctl status emkey
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
The EMKey  is configured properly, but is not secure. Secure the EMKey by running "emctl config emkey -remove_from_repos".

Apparently not :)

D:\>emctl config emkey -remove_from_repos
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
The EMKey has been removed from the Management Repository.

So, I removed it.

D:\>D:\oracle\app\product\12.1.0\middleware1\oms\bin\emctl status emkey
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
The EMKey is configured properly.

D:\>D:\oracle\app\product\12.1.0\middleware1\oms\bin\emctl stop oms
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
Oracle Management Server is Down

Hmm, no luck. OEM is refusing to let me login as SYSMAN via the console.

D:\>D:\oracle\app\product\12.1.0\middleware1\oms\bin\emctl config oms -change_repos_pwd -use_sys_pwd -sys_pwd mycurrentsecret -new_pwd mynewsecret
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.

Changing passwords in backend ...
Passwords changed in backend successfully.
Updating repository password in Credential Store...
Successfully updated Repository password in Credential Store.
Restart all the OMSs using 'emctl stop oms -all' and 'emctl start oms'.
Successfully changed repository password.

Restart.

D:\>D:\oracle\app\product\12.1.0\middleware1\oms\bin\emctl stop oms -all
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Already Stopped
AdminServer Successfully Stopped
Oracle Management Server is Down

D:\>D:\oracle\app\product\12.1.0\middleware1\oms\bin\emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Windows service OracleManagementServer_EMGC_OMS1_1 successfully started
Oracle Management Server is Up

D:\>D:\oracle\app\product\12.1.0\middleware1\oms\bin\emctl status oms -details
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Console Server Host        : oem.server.mine
HTTP Console Port          : 7789
HTTPS Console Port         : 7800
HTTP Upload Port           : 4890
HTTPS Upload Port          : 4899
EM Instance Home           : D:\oracle\app\product\12.1.0\middleware1\gc_inst\em\EMGC_OMS1
OMS Log Directory Location : D:\oracle\app\product\12.1.0\middleware1\gc_inst\em\EMGC_OMS1/sysman/log
OMS is not configured with SLB or virtual hostname
Agent Upload is locked.
OMS Console is locked.
Active CA ID: 1
Console URL: https:// oem.server.mine:7800/em
Upload URL: https:// oem.server.mine:4899/empbs/upload

WLS Domain Information
Domain Name      : GCDomain
Admin Server Host: oem.server.mine

Managed Server Information
Managed Server Instance Name: EMGC_OMS1
Managed Server Instance Host: oem.server.mine
WebTier is Up
Oracle Management Server is Up

And yet, I still couldn't successfully log into OEM. Think man, think: What else did you change??

Then I remembered this screen pop during the upgrade.



Okay, let’s check the repository.

D:\>%ORACLE_HOME%\bin\sqlplus / as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on Sun Jul 21 15:26:30 2013

Copyright (c) 1982, 2010, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> show parameter job

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
job_queue_processes                  integer     0

Idiot! Go and change it back to its original value!

SQL> alter system set job_queue_processes=1000;

System altered.

SQL> show parameter job

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
job_queue_processes                  integer     1000

Restart OMS.


And magically, I’m now able to log into OEM using the SYSMAN password.

at
Labels: , , , ,

No comments :

Post a Comment

Subscribe to: Post Comments ( Atom )