Sunday, July 21, 2013
Oracle Enterprise Manager 12c: Unable to Login using SYSMAN after unsuccessful Upgrade
After an incomplete attempt to upgrade OEM 12.1.0.2.0 to
12.1.0.3.0, a few days later I received a call from my client that the sysman
password is not working. Interesting.
We validated the sysman account from the database and it was
correct. The issue was on the WebLogic layer.
So, what changed? According to the customer, their Systems
Administrators applied OS Security Patches during the weekend.
Other than that, the one apparent thing you need to do as
part of the pre-upgrade steps is to copy the emkey to the repository. Did I
remove it?
D:\>D:\oracle\app\product\12.1.0\middleware1\oms\bin\emctl
status emkey
Oracle Enterprise Manager Cloud Control 12c
Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
The EMKey
is configured properly, but is not secure. Secure the EMKey by running
"emctl config emkey -remove_from_repos".
|
Apparently not :)
D:\>emctl config emkey -remove_from_repos
Oracle Enterprise Manager Cloud Control 12c
Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
The EMKey has been removed from the Management
Repository.
|
So, I removed it.
D:\>D:\oracle\app\product\12.1.0\middleware1\oms\bin\emctl
status emkey
Oracle Enterprise Manager Cloud Control 12c
Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
The EMKey is configured properly.
D:\>D:\oracle\app\product\12.1.0\middleware1\oms\bin\emctl
stop oms
Oracle Enterprise Manager Cloud Control 12c
Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
Oracle Management Server is Down
|
Hmm, no luck. OEM is refusing to let me login as SYSMAN via
the console.
D:\>D:\oracle\app\product\12.1.0\middleware1\oms\bin\emctl
config oms -change_repos_pwd -use_sys_pwd -sys_pwd mycurrentsecret -new_pwd
mynewsecret
Oracle Enterprise Manager Cloud Control 12c
Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
Changing passwords in backend ...
Passwords changed in backend successfully.
Updating repository password in Credential
Store...
Successfully updated Repository password in
Credential Store.
Restart all the OMSs using 'emctl stop oms -all'
and 'emctl start oms'.
Successfully changed repository password.
|
Restart.
D:\>D:\oracle\app\product\12.1.0\middleware1\oms\bin\emctl
stop oms -all
Oracle Enterprise Manager Cloud Control 12c
Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Already Stopped
AdminServer Successfully Stopped
Oracle Management Server is Down
D:\>D:\oracle\app\product\12.1.0\middleware1\oms\bin\emctl
start oms
Oracle Enterprise Manager Cloud Control 12c
Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
Windows service OracleManagementServer_EMGC_OMS1_1
successfully started
Oracle Management Server is Up
D:\>D:\oracle\app\product\12.1.0\middleware1\oms\bin\emctl
status oms -details
Oracle Enterprise Manager Cloud Control 12c
Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Console Server Host : oem.server.mine
HTTP Console Port : 7789
HTTPS Console Port : 7800
HTTP Upload Port : 4890
HTTPS Upload Port : 4899
EM Instance Home :
D:\oracle\app\product\12.1.0\middleware1\gc_inst\em\EMGC_OMS1
OMS Log Directory Location :
D:\oracle\app\product\12.1.0\middleware1\gc_inst\em\EMGC_OMS1/sysman/log
OMS is not configured with SLB or virtual hostname
Agent Upload is locked.
OMS Console is locked.
Active CA ID: 1
Console URL: https:// oem.server.mine:7800/em
Upload URL: https:// oem.server.mine:4899/empbs/upload
WLS Domain Information
Domain Name
: GCDomain
Admin Server Host: oem.server.mine
Managed Server Information
Managed Server Instance Name: EMGC_OMS1
Managed Server Instance Host: oem.server.mine
WebTier is Up
Oracle Management Server is Up
|
And yet, I still couldn't successfully log into OEM. Think
man, think: What else did you change??
Then I remembered this screen pop during the upgrade.
Okay, let’s check the repository.
D:\>%ORACLE_HOME%\bin\sqlplus / as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Sun Jul
21 15:26:30 2013
Copyright (c) 1982, 2010, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release
11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real
Application Testing options
SQL> show parameter job
NAME TYPE VALUE
------------------------------------ -----------
------------------------------
job_queue_processes integer 0
|
Idiot! Go and change it back to its original value!
SQL> alter system set job_queue_processes=1000;
System altered.
SQL> show parameter job
NAME TYPE VALUE
------------------------------------ -----------
------------------------------
job_queue_processes integer 1000
|
Restart OMS.
And magically, I’m now able to log into OEM using the SYSMAN
password.
Monday, July 8, 2013
Oracle Enterprise Manager 12.1.0.2.0 to 12.1.0.3.0 Upgrade with 12.1.0.1.0 Database
As promised, the upgrade steps for a single instance OMS are
detailed below. My pre-upgrade environment:
Host: OEM 6.3 64bit
OMS: 12.1.0.2.0
Repository: 12.1.0.1.0
How pretentious. Of course I know what I’m doing!!
[oracle@spoon config]$ emctl config emkey
-copy_to_repos_from_file -repos_host spoon.song -repos_port 1521 -repos_sid
monster -repos_user sysman -emkey_file
/u01/app/oracle/middelware/oms/sysman/config/emkey.ora
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
Enter Admin User's Password :
Enter Enterprise Manager Root (SYSMAN) Password :
The EMKey has been copied to the Management Repository. This
operation will cause the EMKey to become unsecure.
After the required operation has been completed, secure the EMKey
by running "emctl config emkey -remove_from_repos".
|
[oracle@spoon config]$ . oraenv
ORACLE_SID = [OEM] ? monster
The Oracle base for
ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1 is /u01/app/oracle
[oracle@spoon config]$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Mon Jul
1 12:40:51 2013
Copyright (c) 1982, 2009, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release
11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real
Application Testing options
SQL> grant execute on dbms_random to dbsnmp;
Grant succeeded.
SQL> ALTER DATABASE ADD LOGFILE SIZE 300M;
SQL> ALTER DATABASE ADD LOGFILE SIZE 300M;
SQL> ALTER DATABASE ADD LOGFILE SIZE 300M;
|
ORACLE_SID = [monster] ? OEM
The Oracle base for
ORACLE_HOME=/u01/app/oracle/middelware/oms is /u01/app/oracle
[oracle@spoon config]$ emctl stop oms
Oracle Enterprise Manager Cloud Control 12c
Release 2
Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
Oracle Management Server is Down
|
A Cup of Tea
And Tomb Raider 2013 later
[root@spoon ~]#
/u01/app/oracle/12.1.0.3/middleware/oms/allroot.sh
Starting to execute allroot.sh .........
Starting to execute
/u01/app/oracle/12.1.0.3/middleware/oms/root.sh ......
Running Oracle 11g root.sh script...
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME=
/u01/app/oracle/12.1.0.3/middleware/oms
Enter the full pathname of the local bin
directory: [/usr/local/bin]:
The file "dbhome" already exists in
/usr/local/bin. Overwrite it? (y/n)
[n]:
The file "oraenv" already exists in
/usr/local/bin. Overwrite it? (y/n)
[n]:
The file "coraenv" already exists in
/usr/local/bin. Overwrite it? (y/n)
[n]:
Entries will be added to the /etc/oratab file as
needed by
Database Configuration Assistant when a database
is created
Finished running generic part of root.sh script.
Now product-specific root actions will be
performed.
/etc exist
/u01/app/oracle/12.1.0.3/middleware/oms
Finished execution of
/u01/app/oracle/12.1.0.3/middleware/oms/root.sh ......
|
After logging into OMS, I can see there are targets in
“Unknown” status.
It seems the agent did not start correctly. That is excepted
because, according to the documentation
“Management Agents, including the central agent installed on
the Oracle Management Service (OMS) host, are not upgraded automatically while
upgrading to Enterprise Manager Cloud Control 12c Release 3
(12.1.0.3). Ensure that you upgrade the central agent installed on the OMS host
immediately after upgrading the old OMS to 12.1.0.3.”
Let’s proceed with the upgrade.
First, ensure that you have the latest Agent Software
available.
Submit it!
Presumably because the “oracle” user doesn’t have sudo
access.
Progress!
I made sure the agent has the latest Oracle Database
(12.1.0.4.0) plugin.
[oracle@spoon bin]$ ./emctl listplugins agent
Oracle Enterprise Manager Cloud Control 12c
Release 3
Copyright (c) 1996, 2013 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
oracle.fmw.gg
12.1.0.1.0
/u01/app/oracle/agent/plugins/oracle.fmw.gg.agent.plugin_12.1.0.1.0
oracle.sysman.beacon 12.1.0.3.0 /u01/app/oracle/agent/plugins/oracle.sysman.beacon.agent.plugin_12.1.0.3.0
oracle.sysman.emas
12.1.0.4.0
/u01/app/oracle/agent/plugins/oracle.sysman.emas.agent.plugin_12.1.0.4.0
oracle.sysman.emrep 12.1.0.3.0
/u01/app/oracle/agent/plugins/oracle.sysman.emrep.agent.plugin_12.1.0.3.0
oracle.sysman.db
12.1.0.4.0
/u01/app/oracle/agent/plugins/oracle.sysman.db.agent.plugin_12.1.0.4.0
oracle.sysman.oh
12.1.0.3.0
/u01/app/oracle/agent/plugins/oracle.sysman.oh.agent.plugin_12.1.0.3.0
oracle.sysman.csa
12.1.0.3.0
/u01/app/oracle/agent/plugins/oracle.sysman.csa.agent.plugin_12.1.0.3.0
|
Before we starting having fun, we need to do house cleaning.
Nice!
Subscribe to:
Posts
(
Atom
)