Sunday, July 21, 2013

Oracle Enterprise Manager 12c: Unable to Login using SYSMAN after unsuccessful Upgrade

After an incomplete attempt to upgrade OEM 12.1.0.2.0 to 12.1.0.3.0, a few days later I received a call from my client that the sysman password is not working. Interesting.

We validated the sysman account from the database and it was correct. The issue was on the WebLogic layer.

So, what changed? According to the customer, their Systems Administrators applied OS Security Patches during the weekend.

Other than that, the one apparent thing you need to do as part of the pre-upgrade steps is to copy the emkey to the repository. Did I remove it?

D:\>D:\oracle\app\product\12.1.0\middleware1\oms\bin\emctl status emkey
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
The EMKey  is configured properly, but is not secure. Secure the EMKey by running "emctl config emkey -remove_from_repos".

Apparently not :)

D:\>emctl config emkey -remove_from_repos
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
The EMKey has been removed from the Management Repository.

So, I removed it.

D:\>D:\oracle\app\product\12.1.0\middleware1\oms\bin\emctl status emkey
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
The EMKey is configured properly.

D:\>D:\oracle\app\product\12.1.0\middleware1\oms\bin\emctl stop oms
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
Oracle Management Server is Down

Hmm, no luck. OEM is refusing to let me login as SYSMAN via the console.

D:\>D:\oracle\app\product\12.1.0\middleware1\oms\bin\emctl config oms -change_repos_pwd -use_sys_pwd -sys_pwd mycurrentsecret -new_pwd mynewsecret
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.

Changing passwords in backend ...
Passwords changed in backend successfully.
Updating repository password in Credential Store...
Successfully updated Repository password in Credential Store.
Restart all the OMSs using 'emctl stop oms -all' and 'emctl start oms'.
Successfully changed repository password.

Restart.

D:\>D:\oracle\app\product\12.1.0\middleware1\oms\bin\emctl stop oms -all
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Already Stopped
AdminServer Successfully Stopped
Oracle Management Server is Down

D:\>D:\oracle\app\product\12.1.0\middleware1\oms\bin\emctl start oms
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Windows service OracleManagementServer_EMGC_OMS1_1 successfully started
Oracle Management Server is Up

D:\>D:\oracle\app\product\12.1.0\middleware1\oms\bin\emctl status oms -details
Oracle Enterprise Manager Cloud Control 12c Release 2
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Console Server Host        : oem.server.mine
HTTP Console Port          : 7789
HTTPS Console Port         : 7800
HTTP Upload Port           : 4890
HTTPS Upload Port          : 4899
EM Instance Home           : D:\oracle\app\product\12.1.0\middleware1\gc_inst\em\EMGC_OMS1
OMS Log Directory Location : D:\oracle\app\product\12.1.0\middleware1\gc_inst\em\EMGC_OMS1/sysman/log
OMS is not configured with SLB or virtual hostname
Agent Upload is locked.
OMS Console is locked.
Active CA ID: 1
Console URL: https:// oem.server.mine:7800/em
Upload URL: https:// oem.server.mine:4899/empbs/upload

WLS Domain Information
Domain Name      : GCDomain
Admin Server Host: oem.server.mine

Managed Server Information
Managed Server Instance Name: EMGC_OMS1
Managed Server Instance Host: oem.server.mine
WebTier is Up
Oracle Management Server is Up

And yet, I still couldn't successfully log into OEM. Think man, think: What else did you change??

Then I remembered this screen pop during the upgrade.



Okay, let’s check the repository.

D:\>%ORACLE_HOME%\bin\sqlplus / as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on Sun Jul 21 15:26:30 2013

Copyright (c) 1982, 2010, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> show parameter job

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
job_queue_processes                  integer     0

Idiot! Go and change it back to its original value!

SQL> alter system set job_queue_processes=1000;

System altered.

SQL> show parameter job

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
job_queue_processes                  integer     1000


Restart OMS.


And magically, I’m now able to log into OEM using the SYSMAN password.
continue reading "Oracle Enterprise Manager 12c: Unable to Login using SYSMAN after unsuccessful Upgrade"

Monday, July 8, 2013

Oracle Enterprise Manager 12.1.0.2.0 to 12.1.0.3.0 Upgrade with 12.1.0.1.0 Database

As promised, the upgrade steps for a single instance OMS are detailed below. My pre-upgrade environment:

Host: OEM 6.3 64bit
OMS: 12.1.0.2.0

Repository: 12.1.0.1.0


How pretentious. Of course I know what I’m doing!!









[oracle@spoon config]$ emctl config emkey -copy_to_repos_from_file -repos_host spoon.song -repos_port 1521 -repos_sid monster -repos_user sysman -emkey_file /u01/app/oracle/middelware/oms/sysman/config/emkey.ora
Oracle Enterprise Manager Cloud Control 12c Release 2 
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Enter Admin User's Password :
Enter Enterprise Manager Root (SYSMAN) Password :
The EMKey has been copied to the Management Repository. This operation will cause the EMKey to become unsecure.
After the required operation has been completed, secure the EMKey by running "emctl config emkey -remove_from_repos".




[oracle@spoon config]$ . oraenv
ORACLE_SID = [OEM] ? monster
The Oracle base for ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1 is /u01/app/oracle
[oracle@spoon config]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on Mon Jul 1 12:40:51 2013

Copyright (c) 1982, 2009, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> grant execute on dbms_random to dbsnmp;

Grant succeeded.

SQL> ALTER DATABASE ADD LOGFILE SIZE 300M;

SQL> ALTER DATABASE ADD LOGFILE SIZE 300M;

SQL> ALTER DATABASE ADD LOGFILE SIZE 300M;





ORACLE_SID = [monster] ? OEM
The Oracle base for ORACLE_HOME=/u01/app/oracle/middelware/oms is /u01/app/oracle
[oracle@spoon config]$ emctl stop oms
Oracle Enterprise Manager Cloud Control 12c Release 2 
Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
Stopping WebTier...
WebTier Successfully Stopped
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
Oracle Management Server is Down





A Cup of Tea



And Tomb Raider 2013 later




[root@spoon ~]# /u01/app/oracle/12.1.0.3/middleware/oms/allroot.sh

Starting to execute allroot.sh .........

Starting to execute /u01/app/oracle/12.1.0.3/middleware/oms/root.sh ......
Running Oracle 11g root.sh script...

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/12.1.0.3/middleware/oms

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
/etc exist
/u01/app/oracle/12.1.0.3/middleware/oms
Finished execution of  /u01/app/oracle/12.1.0.3/middleware/oms/root.sh ......



After logging into OMS, I can see there are targets in “Unknown” status.


It seems the agent did not start correctly. That is excepted because, according to the documentation

Management Agents, including the central agent installed on the Oracle Management Service (OMS) host, are not upgraded automatically while upgrading to Enterprise Manager Cloud Control 12c Release 3 (12.1.0.3). Ensure that you upgrade the central agent installed on the OMS host immediately after upgrading the old OMS to 12.1.0.3.

Let’s proceed with the upgrade.


First, ensure that you have the latest Agent Software available.





Submit it!


Presumably because the “oracle” user doesn’t have sudo access.




Progress!







I made sure the agent has the latest Oracle Database (12.1.0.4.0) plugin.

[oracle@spoon bin]$ ./emctl listplugins agent
Oracle Enterprise Manager Cloud Control 12c Release 3 
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
oracle.fmw.gg       12.1.0.1.0 /u01/app/oracle/agent/plugins/oracle.fmw.gg.agent.plugin_12.1.0.1.0
oracle.sysman.beacon 12.1.0.3.0 /u01/app/oracle/agent/plugins/oracle.sysman.beacon.agent.plugin_12.1.0.3.0
oracle.sysman.emas  12.1.0.4.0 /u01/app/oracle/agent/plugins/oracle.sysman.emas.agent.plugin_12.1.0.4.0
oracle.sysman.emrep 12.1.0.3.0 /u01/app/oracle/agent/plugins/oracle.sysman.emrep.agent.plugin_12.1.0.3.0
oracle.sysman.db    12.1.0.4.0 /u01/app/oracle/agent/plugins/oracle.sysman.db.agent.plugin_12.1.0.4.0
oracle.sysman.oh    12.1.0.3.0 /u01/app/oracle/agent/plugins/oracle.sysman.oh.agent.plugin_12.1.0.3.0
oracle.sysman.csa   12.1.0.3.0 /u01/app/oracle/agent/plugins/oracle.sysman.csa.agent.plugin_12.1.0.3.0

Before we starting having fun, we need to do house cleaning.



Nice!













continue reading "Oracle Enterprise Manager 12.1.0.2.0 to 12.1.0.3.0 Upgrade with 12.1.0.1.0 Database"