"EM daemon is not running" dbconsole status

Related to 11.2.0.2 RDBMS.

I had upgraded a 10g database to 11.2.0.2. I could never get the dbconsole working for this until few days back when someone else had same issue and needed help.

The issue was dbconsole was not starting up.

Note: I have removed actual URL and hostname with "*" symbol to maintain privacy.

$ emctl start dbconsole
Oracle Enterprise Manager 11g Database Control Release 11.2.0.2.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
https://*************:1158/em/console/aboutApplication
Starting Oracle Enterprise Manager 11g Database Control ............................................................................................. failed.
------------------------------------------------------------------
Logs are generated in directory /oracle/product/11.2.0/************_******/sysman/log

$ emctl status dbconsole
Oracle Enterprise Manager 11g Database Control Release 11.2.0.2.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
https://***************:1158/em/console/aboutApplication
EM Daemon is not running.
------------------------------------------------------------------
Logs are generated in directory /oracle/product/11.2.0/*************_******/sysman/log

Even though status is "Em Daemon is not running", I see login screen for https://************:1158/em

So tried logging in using sysman:

As the agent should run on port 3938, which is not running causing below errors when dbconsole service is trying to communicate with the agent:

from emdctl.trc:

2012-01-17 10:59:20,817 Thread-287350144 WARN http: nmehl_connect_internal: connect failed to (************:3938): Connection refused (error = 111)
2012-01-17 10:59:20,818 Thread-287350144 ERROR main: nmectla_agentctl: Error connecting to https://***************:3938/emd/main. Returning status code 1

So we need to find why the agent is not running. Checked agabend.log which shows:

The agentTZRegion value in /oracle/product/11.2.0/*************_*******/sysman/config/emd.properties is not in agreement with what agent thinks it should be.Please verify your environment to make sure that TZ setting has not changed since the last start of the agent.
If you modified the timezone setting in the environment, please stop the agent and exectute 'emctl resetTZ agent' and also execute the script mgmt_target.set_agent_tzrgn(<agent_name>, <new_tz_rgn>) to get the value propagated to repository.

At the same time, emdb.nohup shows:

----- Tue Jan 17 10:57:24 2012::omsRecvDir is set? =  1
---------- Tue Jan 17 10:57:24 2012::osname is linux
---------- Tue Jan 17 10:57:24 2012::Console Launched with PID 25486 at time Tue Jan 17 10:57:24 2012 ---------- Tue Jan 17 10:57:24 2012::heap size used is 384M
---------- Tue Jan 17 10:57:24 2012::perm size used is 400M
---------- Tue Jan 17 10:57:24 2012::joptions used are -server -Xmx384M -XX:MaxPermSize=400M -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=40  
-----sh: line 1: 25491 Segmentation fault      (core dumped) /oracle/product/11.2.0/bin/emdctl validateTZ agent US/Eastern > /oracle/product/11.2.0/************_******/uunUBBW0Yo 2>&1
----- Tue Jan 17 10:57:25 2012::Mismatch detected between timezone in env (US/Eastern) and in /oracle/product/11.2.0/************_******/sysman/config/emd.properties (US/Eastern). Forcing value to latter..
-----sh: line 1: 25524 Segmentation fault      (core dumped) /oracle/product/11.2.0/bin/emdctl validateTZ agent US/Eastern > /oracle/product/11.2.0/************_******/kuoO8Ru0vA 2>&1
----- Tue Jan 17 10:57:26 2012::The agentTZRegion value in /oracle/product/11.2.0/************_******/sysman/config/emd.properties is not in agreement with what agent thinks it should be.Please verify your environment to make sure that TZ setting has not changed since the last start of the agent.If you modified the timezone setting in the environment, please stop the agent and exectute 'emctl resetTZ agent' and also execute the script mgmt_target.set_agent_tzrgn(<agent_name>, <new_tz_rgn>) to get the value propagated to repository
-----

Strange thing is there is NO mismatch, both shows (US/Eastern). Not sure what was happening. I checked /oracle/product/11.2.0/************_******/sysman/config/emd.properties for agentTZRegion value:

$ cat /oracle/product/11.2.0/************_******/sysman/config/emd.properties | grep agentTZ
agentTZRegion=US/Eastern

Found MOS document "Dbconsole 11.2 Does Not Start Logs Shows Timezone Mismatch (Doc ID 1329914.1)".

Oracle instant client was installed on the same Oracle RDBMS Home, removed just that using OUI and now dbconsole is working fine.

If you have similar situation and had created new Database using DBCA. You might have seen below warning message by dbca.

ORA-00001: unique constraint (SYSMAN.PARAMETERS_PRIMARY_KEY) violated

Oracle Database 11.2.0.3 on Redhat linux 5

I wanted to test something withing dbconsole, but my test database dbconsole was not properly configured I guess.

So decided to drop and recreate.
Dropped using "<ORACLE_HOME>/bin/emca -deconfig dbcontrol db -repos drop"

Tried to recreate using "<ORACLE_HOME>/bin/emca -config dbcontrol db -repos create" but it failed with below errors:

WARNING: ORA-00001: unique constraint (SYSMAN.PARAMETERS_PRIMARY_KEY) violated
ORA-06512: at "SYSMAN.EMD_LOADER", line 4986
ORA-06512: at line 1

Jan 9, 2012 5:44:30 PM oracle.sysman.emcp.EMReposConfig invoke
SEVERE: Error creating the repository

Below is the test case:

1. Checked the violating constraint and existing rows before dropping the repository again.

select owner, table_name from user_constraints where constraint_name like '%PARAMETERS_PRIMARY_KEY%'

OWNER    TABLE_NAME
------------------------------------
SYSMAN  MGMT_PARAMETERS


SQL> select count(*) from sysman.mgmt_parameters;

  COUNT(*)
----------
         3

2. Dropped repository to check whether this table gets dropped or not.

emca -deconfig dbcontrol db -repos drop
INFO: Repository successfully dropped

3. Check sysman.mgmt_parameters

select count(*) from sysman.mgmt_parameters
                            *
ERROR at line 1:
ORA-00942: table or view does not exist

4.  Creating dbconsole using "emca -config dbcontrol db -repos create"

STARTED EMCA at Jan 9, 2012 5:41:37 PM
EM Configuration Assistant, Version 11.2.0.3.0 Production
Copyright (c) 2003, 2011, Oracle.  All rights reserved.

Enter the following information:
Database SID: RD11852
Listener port number: 1521
Listener ORACLE_HOME [ /oracle/product/11.2.0.3 ]:
Password for SYS user: 
Password for DBSNMP user: 
Password for SYSMAN user: 
Email address for notifications (optional):
Outgoing Mail (SMTP) server for notifications (optional):
-----------------------------------------------------------------

You have specified the following settings

Database ORACLE_HOME ................ /oracle/product/11.2.0.3

Local hostname ................ ****************
Listener ORACLE_HOME ................ /oracle/product/11.2.0.3
Listener port number ................ 1521
Database SID ................ RD11852
Email address for notifications ...............
Outgoing Mail (SMTP) server for notifications ...............

-----------------------------------------------------------------
Do you wish to continue? [yes(Y)/no(N)]: Y
Jan 9, 2012 5:42:04 PM oracle.sysman.emcp.EMConfig perform
INFO: This operation is being logged at /oracle/cfgtoollogs/emca/RD11852/emca_2012_01_09_17_41_36.log.
Jan 9, 2012 5:42:06 PM oracle.sysman.emcp.EMReposConfig createRepository
INFO: Creating the EM repository (this may take a while) ...

Jan 9, 2012 5:44:30 PM oracle.sysman.emcp.EMReposConfig createRepository
WARNING: ORA-00001: unique constraint (SYSMAN.PARAMETERS_PRIMARY_KEY) violated
ORA-06512: at "SYSMAN.EMD_LOADER", line 4986
ORA-06512: at line 1

Jan 9, 2012 5:44:30 PM oracle.sysman.emcp.EMReposConfig invoke
SEVERE: Error creating the repository
Jan 9, 2012 5:44:30 PM oracle.sysman.emcp.EMReposConfig invoke
INFO: Refer to the log file at /oracle/cfgtoollogs/emca/RD11852/emca_repos_create_<date>.log for more details.
Jan 9, 2012 5:44:30 PM oracle.sysman.emcp.EMConfig perform
SEVERE: Error creating the repository
Refer to the log file at /oracle/cfgtoollogs/emca/RD11852/emca_2012_01_09_17_41_36.log for more details.
Could not complete the configuration. Refer to the log file at /oracle/cfgtoollogs/emca/RD11852/emca_2012_01_09_17_41_36.log for more details.

 I have read MOS notes with similar error but while db upgrade from 10g to 11g and they say we can ignore these errors. But here I am trying to create dbconsole and it does not let me ignore it. I guess I will have to create SR.

I will post the solution once it is available.

BTW below are the rows in sysman.mgmt_parameters table:

SQL> select parameter_name, parameter_value from sysman.mgmt_parameters;

PARAMETER_NAME            PARAMETER_VALUE
------------------------- -------------------------
EST_ANALYZE               OFF
TARGET_POLICY_ENABLED     N
load_interval             300

Does my processor support Hardware Virtualization?

I searched on intel site for my processor "i3-370M".
==>Under Advanced Technologies it showed:
            Intel Virtualization Technology (VT-x) - YES
            Intel Virtualization Technology for Directed I/O (VT-d) - NO

At this moment not sure what does that "NO" mean for my plan of Oracle VM on my laptop. Let's see...

Did Google to find out if there is any other way to find out capability of my PC's processor to support virtualization and found http://www.microsoft.com/windows/virtual-pc/support/configure-bios.aspx

Tool from Microsoft "Hardware-Assisted Virtualization Detection Tool" to check whether the PC has a CPU with support for hardware virtualization and if it does whether I need to change BIOS settings.

Above screenshot is the result from the tool.

SYS_AUTO_SQL_TUNING_TASK exiting with error "1760"

A DBA reported that he is seeing intermittent errors in the alert log related to SYS_AUTO_SQL_TUNING_TASK failing with error 1760.

Begin automatic SQL Tuning Advisor run for special tuning task  "SYS_AUTO_SQL_TUNING_TASK"
SYS_AUTO_SQL_TUNING_TASK exiting with error "1760" for execution "EXEC_11029".  See DBA_ADVISOR_EXECUTIONS for more details.
End automatic SQL Tuning Advisor run for special tuning task  "SYS_AUTO_SQL_TUNING_TASK"
Errors in file **************************************************_j003_****.trc:
ORA-12012: error on auto execute of job 53057
ORA-01760: illegal argument for function
ORA-06512: at "SYS.PRVT_ADVISOR", line 2693
ORA-06512: at "SYS.DBMS_ADVISOR", line 241
ORA-06512: at "SYS.DBMS_SQLTUNE", line 772
ORA-06512: at line 4

My first impression was that may be the objects "SYS.PRVT_ADVISOR", "SYS.DBMS_SQLTUNE" are invalid. But I was wrong, the DBA said, there were no invalid objects under SYS schema.


This seems to be related with Bug 9021724 which is is fixed in 11.2.0.2 but a patch needs to applied for earlier versions if you are seeing these intermittent errors. 




NOTE: This is only related to intermittent ORA-1760 on SQL Tuning Advisor task.

Running wls_basic_audit.py - Weblogic BASIC license compliance

This script will help with verifying that WebLogic Server is being used in accordance with the WebLogic Server "Basic" license.

Reference: MOS note "WebLogic Server Basic License Audit Script [ID 885587.1]"

1. Set the environment from WL_SERVER_HOME/server/bin. Make sure to use below command to avoid "Exception in thread "main" java.lang.NoClassDefFoundError: weblogic/WLST"

Two dots are important in case the "." is not specified in PATH variable.

. ./setWLSEnv.sh

2. java weblogic.WLST wls_basic_audit.py

$ java weblogic.WLST wls_basic_audit.py

Initializing WebLogic Scripting Tool (WLST) ...

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

===============
WLS Basic Audit
===============
12/28/11 4:31 PM
Please enter your username :weblogic
Please enter your password :
Please enter your server URL [t3://localhost:7001] :
Connecting to t3://localhost:7001 with userid weblogic ...
Successfully connected to Admin Server 'AdminServer' that belongs to domain 'INB'.

Warning: An insecure protocol was used to connect to the
server. To ensure on-the-wire security, the SSL port or
Admin port should be used instead.

===========
Domain Info
===========
Checking for administration port enabled: 
Checking for service migration: 
No errors detected
============
Cluster Info
============
Checking cluster type and overload actions: 
No errors detected
===========
Server Info
===========
Checking server mode and overload actions: 
No errors detected
======================
Singleton Service Info
======================
Checking for singleton services: 
No errors detected
======================
Service Migration Info
======================
Checking for service migration: 
No errors detected
========================
Global Work Manager Info
========================
Checking for global Work Managers: 
No errors detected
=============================
Application Work Manager Info
=============================
Checking for application-scoped Work Managers: 
No errors detected
===============
Deployment Info
===============
Checking for application versioning and ordering: 
License compliance failure:  Application reports version 11.1.1.2.0 uses version 11.1.1.2.0
License compliance failure:  Application formsapp version 11.1.1 uses version 11.1.1
Production redeployment strategy involves deploying a new version of an updated application alongside an older version of the same application. WebLogic Server automatically manages client connections so that only new client requests are directed to the new version. Clients already connected to the application during the redeployment continue to use the older version of the application until they complete their work, at which point WebLogic Server automatically retires the older application. This capability is supported by deploying the application in Administration mode, which makes it available only via a configured Administration channel.
In the license for WebLogic Server Basic, deploying any application that uses a version identifier is not permitted.
See the documentation:  http://download.oracle.com/docs/cd/E12839_01/web.1111/e13702/redeploy.htm#DEPGD281
2 error(s) detected
=========================
Application FastSwap Info
=========================
Checking whether FastSwap is enabled in an application: 
No errors detected
========
JMS Info
========
Checking for non-default Unit-of-Work Message Handling Policy: 
Checking for non-default Message Unit-of-Order: 
Checking for Message Store-and-Forward agents: 
No errors detected
=====================
Tuxedo Connector Info
=====================
Checking for WebLogic Tuxedo Connectors: 
No errors detected
=========
SNMP Info
=========
Checking for SNMP agents: 
No errors detected
=====================
Application Mode Info
=====================
Checking for application in Administration mode: 
Location changed to domainRuntime tree. This is a read-only tree with DomainMBean as the root.
For more help, use help(domainRuntime)

Location changed to serverRuntime tree. This is a read-only tree with ServerRuntimeMBean as the root.
For more help, use help(serverRuntime)

No errors detected
======================
Module Deployment Info
======================
Checking for deployment of standalone JMS, JDBC, or WLDF modules: 
No errors detected
===========================
HTTP Publish-Subscribe Info
===========================
Checking for HTTP Publish-Subscribe Server usage: 
No errors detected
==========================
Diagnostics Framework Info
==========================
Checking for WebLogic Diagnostics Framework usage: 
License compliance failure:  Diagnostics framework system resource is being used:  Module-FMWDFW
The WebLogic Diagnostic Framework (WLDF) is a monitoring and diagnostic framework that defines and implements a set of services that run within WebLogic Server processes and participate in the standard server life cycle. Using WLDF, you can create, collect, analyze, archive, and access diagnostic data generated by a running server and the applications deployed within its containers. This data provides insight into the run-time performance of servers and applications and enables you to isolate and diagnose faults when they occur.
In the license for WebLogic Server Basic, the use of WLDF is not permitted.
See the documentation:  http://download.oracle.com/docs/cd/E12839_01/web.1111/e13714/intro.htm#WLDFC108
1 error(s) detected
=======
Summary
=======
3 error(s) detected


Exiting WebLogic Scripting Tool.

I will post more posts on how to take care of these errors reported by compliance script.