Tuesday, January 17, 2012

"EM daemon is not running" dbconsole status

Related to 11.2.0.2 RDBMS.

I had upgraded a 10g database to 11.2.0.2. I could never get the dbconsole working for this until few days back when someone else had same issue and needed help.

The issue was dbconsole was not starting up.

Note: I have removed actual URL and hostname with "*" symbol to maintain privacy.

$ emctl start dbconsole
Oracle Enterprise Manager 11g Database Control Release 11.2.0.2.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
https://*************:1158/em/console/aboutApplication
Starting Oracle Enterprise Manager 11g Database Control ............................................................................................. failed.
------------------------------------------------------------------
Logs are generated in directory /oracle/product/11.2.0/************_******/sysman/log

$ emctl status dbconsole
Oracle Enterprise Manager 11g Database Control Release 11.2.0.2.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
https://***************:1158/em/console/aboutApplication
EM Daemon is not running.
------------------------------------------------------------------
Logs are generated in directory /oracle/product/11.2.0/*************_******/sysman/log


Even though status is "Em Daemon is not running", I see login screen for https://************:1158/em


So tried logging in using sysman:



As the agent should run on port 3938, which is not running causing below errors when dbconsole service is trying to communicate with the agent:

from emdctl.trc:

2012-01-17 10:59:20,817 Thread-287350144 WARN http: nmehl_connect_internal: connect failed to (************:3938): Connection refused (error = 111)
2012-01-17 10:59:20,818 Thread-287350144 ERROR main: nmectla_agentctl: Error connecting to https://***************:3938/emd/main. Returning status code 1

So we need to find why the agent is not running. Checked agabend.log which shows:

The agentTZRegion value in /oracle/product/11.2.0/*************_*******/sysman/config/emd.properties is not in agreement with what agent thinks it should be.Please verify your environment to make sure that TZ setting has not changed since the last start of the agent.
If you modified the timezone setting in the environment, please stop the agent and exectute 'emctl resetTZ agent' and also execute the script mgmt_target.set_agent_tzrgn(<agent_name>, <new_tz_rgn>) to get the value propagated to repository.
At the same time, emdb.nohup shows:
----- Tue Jan 17 10:57:24 2012::omsRecvDir is set? =  1
---------- Tue Jan 17 10:57:24 2012::osname is linux
---------- Tue Jan 17 10:57:24 2012::Console Launched with PID 25486 at time Tue Jan 17 10:57:24 2012 ---------- Tue Jan 17 10:57:24 2012::heap size used is 384M
---------- Tue Jan 17 10:57:24 2012::perm size used is 400M
---------- Tue Jan 17 10:57:24 2012::joptions used are -server -Xmx384M -XX:MaxPermSize=400M -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=40  
-----sh: line 1: 25491 Segmentation fault      (core dumped) /oracle/product/11.2.0/bin/emdctl validateTZ agent US/Eastern > /oracle/product/11.2.0/************_******/uunUBBW0Yo 2>&1
----- Tue Jan 17 10:57:25 2012::Mismatch detected between timezone in env (US/Eastern) and in /oracle/product/11.2.0/************_******/sysman/config/emd.properties (US/Eastern). Forcing value to latter..
-----sh: line 1: 25524 Segmentation fault      (core dumped) /oracle/product/11.2.0/bin/emdctl validateTZ agent US/Eastern > /oracle/product/11.2.0/************_******/kuoO8Ru0vA 2>&1
----- Tue Jan 17 10:57:26 2012::The agentTZRegion value in /oracle/product/11.2.0/************_******/sysman/config/emd.properties is not in agreement with what agent thinks it should be.Please verify your environment to make sure that TZ setting has not changed since the last start of the agent.If you modified the timezone setting in the environment, please stop the agent and exectute 'emctl resetTZ agent' and also execute the script mgmt_target.set_agent_tzrgn(<agent_name>, <new_tz_rgn>) to get the value propagated to repository
-----

Strange thing is there is NO mismatch, both shows (US/Eastern). Not sure what was happening. I checked /oracle/product/11.2.0/************_******/sysman/config/emd.properties for agentTZRegion value:


$ cat /oracle/product/11.2.0/************_******/sysman/config/emd.properties | grep agentTZ
agentTZRegion=US/Eastern

Found MOS document "Dbconsole 11.2 Does Not Start Logs Shows Timezone Mismatch (Doc ID 1329914.1)".

Oracle instant client was installed on the same Oracle RDBMS Home, removed just that using OUI and now dbconsole is working fine.

If you have similar situation and had created new Database using DBCA. You might have seen below warning message by dbca.



6 comments:

  1. Thank you for your website. I've been struggling with this problem for over a month, including opening an SR with Oracle. Oracle never brought up this bug. You are doing a great service to the Oracle DBA community.

    Janet

    ReplyDelete
    Replies
    1. Thank you Janet. Glad that the post helped you.

      Delete
    2. Re: Install Oracle 11gR2 Error?
      Posted: Apr 24, 2013 2:20 PM in response to: Junaid in response to: Junaid
      em_daemon_is_not_running , agent_is_not_running , em_internal_error , error_database_control , time_zone_problem [Edit]

      Click to edit this message... Edit Click to report abuse... Click to reply to this thread Reply
      AOA/Hello,

      Dear All,

      First of all thanks to all of those people who posted solutions in different forums on this issue as this helped me to solve this problem. One more thing, always read documentation and always check log files for error reports as Oracle is not too much interactive.

      The Problem " Error Starting Database Control Please execute the following command(s) " faced at the end of installation of Oracle 10g / 11g is caused due to Time Zone Values in different files, tables and services of Oracle. Interestingly, these values do not change automatically with the change in OS time zone and you have to do it manually.

      You also have to set some environment variables manually like ORACLE_UNQNAME, ORACLE_SID, ORACLE_HOSTNAME etc.

      You also have to set machine name and IP in the hosts file if required so that EM finds the localhost.

      127.0.0.1 localhost
      127.0.0.1 computer name / hostname

      PRAISE ORACLE FOR THIS

      Following is the Solution which worked well for me.

      Step 1: Go to dbhome > hostname_SID > Sysman > Config > open emd.properties in notepad and set the value agentTZRegion=+05:00 to the value of your timezone and click save. You can also do it using export and time zone reset commands.

      Step 2: ALTER DATABASE SET TIME_ZONE='+05:00'; according to your time zone. You can check DB TIME Zone using " select dbtimezone from dual ; ".

      Step 3: In step 2 if you get the error "ORA-30079: cannot alter database timezone when database has TIMESTAMP WITH
      LOCAL TIME ZONE columns" then please FOLLOW following steps

      SQL> select count (*) from dba_tab_columns
      2 where data_type like 'TIMESTAMP%WITH LOCAL TIME ZONE' ;

      COUNT(*)
      1

      In this case there is only one table. The owner and name can be determined by:

      SQL> select owner, table_name from dba_tab_columns
      2 where data_type like 'TIMESTAMP%WITH LOCAL TIME ZONE';

      OWNER TABLE_NAME
      ------------------------------
      OE ORDERS

      SQL> desc oe.orders ;
      Name Null? Type
      -------- ORDER_ID NOT NULL NUMBER(12)
      ORDER_DATE NOT NULL TIMESTAMP(6) WITH LOCAL TIME ZONE
      ORDER_MODE VARCHAR2(8)
      CUSTOMER_ID NOT NULL NUMBER(6)
      ORDER_STATUS NUMBER(2)
      ORDER_TOTAL NUMBER(8,2)
      SALES_REP_ID NUMBER(6)
      PROMOTION_ID NUMBER(6)

      This oe.orders is just a table in the example schema of OE so this colums can be dropped. So please dont install sample schemas if you dont need them.

      SQL> alter table OE.ORDERS drop column ORDER_DATE ;

      Table altered.

      Now the Time Zone can be altered using above command.

      Step 4: Also check the value of " SELECT SESSIONTIMEZONE FROM DUAL; " and if the value differs from os time zone, db time zone , emd.properties / agent time zone then also set it according to your timezone but I believe it always gives the OS time zone.

      Step 5: Stop and Restart all Services and check that Oracle DBConsole service is running or not and if not running then start from command prompt " emctl start dbconsole" .

      NOW THE ENTERPRISE MANAGER WILL WORK WELL BY THE GRACE OF ALMIGHTY ALLAH.

      Regards,

      Tauseef Ahmed
      IT Specialist, TDCP

      Edited by: 1002308 on Apr 24, 2013 2:22 PM

      Delete
    3. Thank you so much bro.... may ALLAH give HIS grace so that u can achieve the ended limits of whatever u want, and will b able to give us solutions of related problems.... Thanks again....
      Tauseef Ahmed g tusi great o..... :)

      Delete
  2. Thank you Thank you Thank you Thank you Thank you Thank you Thank you Thank you Thank you !!!
    This ended my long search for a solution to this problem........

    Really, thanks mate. You're a lifesaver :-)

    ReplyDelete