Although opmnctl status was showing them alive.
Background:
=========
There are two domains configured on this server.
For some reason opmn ports were conflicting for these two domain instances.
And opmnctl startall for one of them was failing with:
opmn for both domain instances were trying to run on:
Starting up the OPMN managed system components ...opmnctl startall: starting opmn and all managed processes...RCV: No such file or directoryCommunication error with the OPMN server local port.Check the OPMN log files/ora11/product/middleware/fmw11g/opmn/bin/opmn: unexpected exit: code512opmnctl startall: opmn failed to start.RCV: No such file or directoryCommunication error with the OPMN server local port.Check the OPMN log filesopmnctl status: opmn is not running.
<port local="6700" remote="6701"/>So we decided to change ports for one of them, by changing ports in opmn.xml ($INSTANCE_HOME/config/OPMN/opmn) to:
<port local="6702" remote="6703"/>
After above change, opmnctl startall worked fine for both domain instances.
Problem with EM:
=============
Now EM showed ohs1 and webcache1 down for the instance whose opmn ports were changed.
Although opmnctl status was showing them alive.
When tried to start ohs1 through EM got below error:
Error flushing write connection to OPMN. The remote OPMN server is probably not running or the OPMN mbean is not properly configured. See the adminserver log for more details.Admin Server Diagnostic log showed:
---------------------------------------
[2012-11-06T10:17:26.987-05:00] [AdminServer] [ERROR] [] [oracle.as.management.mbeans.opmn] [tid: Thread-46] [userId: weblogic] [ecid: 0c198ad5e21e89f5:-411d8c39:13ad5e5ae39:-8000-000000000000013b,1:17426] [APP: NonJ2EEManagement#11.1.1] OPMN mbeanstart failed with the following exception[[
oracle.as.management.opmn.optic.OpticBadConnectException: Error flushing write connection to OPMN
at oracle.as.management.opmn.optic.OpmnPhone.sendRequest(OpmnPhone.java:774)
at oracle.as.management.opmn.optic.OpmnPhone.makePhoneCall(OpmnPhone.java:330)
at oracle.as.management.opmn.optic.OpmnPhone.request(OpmnPhone.java:270)
at oracle.as.management.opmn.optic.OpmnQuery.getBuf(OpmnQuery.java:482)
at oracle.as.management.mbeans.opmn.Opmn.manageProcess(Opmn.java:239)
at oracle.as.management.mbeans.opmn.Opmn.startProcess(Opmn.java:103)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at oracle.as.jmx.framework.standardmbeans.spi.OracleStandardEmitterMBean.doInvoke(OracleStandardEmitterMBean.java:973)
at oracle.adf.mbean.share.AdfMBeanInterceptor.internalInvoke(AdfMBeanInterceptor.java:104)
at oracle.as.jmx.framework.generic.spi.interceptors.AbstractMBeanInterceptor.doInvoke(AbstractMBeanInterceptor.java:252)
at oracle.as.jmx.framework.generic.spi.security.AbstractMBeanSecurityInterceptor.internalInvoke(AbstractMBeanSecurityInterceptor.java:190)
at oracle.as.jmx.framework.generic.spi.interceptors.AbstractMBeanInterceptor.doInvoke(AbstractMBeanInterceptor.java:252)
at oracle.security.jps.ee.jmx.JpsJmxInterceptor$2.run(JpsJmxInterceptor.java:344)
at java.security.AccessController.doPrivileged(Native Method)
at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:313)
at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:413)
at oracle.security.jps.ee.jmx.JpsJmxInterceptor.internalInvoke(JpsJmxInterceptor.java:360)
at oracle.as.jmx.framework.generic.spi.interceptors.AbstractMBeanInterceptor.doInvoke(AbstractMBeanInterceptor.java:252)
at oracle.as.jmx.framework.generic.spi.interceptors.ContextClassLoaderMBeanInterceptor.internalInvoke(ContextClassLoaderMBeanInterceptor.java:103)
at oracle.as.jmx.framework.generic.spi.interceptors.AbstractMBeanInterceptor.doInvoke(AbstractMBeanInterceptor.java:252)
at oracle.as.jmx.framework.generic.spi.interceptors.MBeanRestartInterceptor.internalInvoke(MBeanRestartInterceptor.java:116)
at oracle.as.jmx.framework.generic.spi.interceptors.AbstractMBeanInterceptor.doInvoke(AbstractMBeanInterceptor.java:252)
at oracle.as.jmx.framework.generic.spi.interceptors.LoggingMBeanInterceptor.internalInvoke(LoggingMBeanInterceptor.java:524)
at oracle.as.jmx.framework.generic.spi.interceptors.AbstractMBeanInterceptor.doInvoke(AbstractMBeanInterceptor.java:252)
at oracle.as.jmx.framework.standardmbeans.spi.OracleStandardEmitterMBean.invoke(OracleStandardEmitterMBean.java:887)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
at java.security.AccessController.doPrivileged(Native Method)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
at weblogic.management.mbeanservers.internal.JMXContextInterceptor.invoke(JMXContextInterceptor.java:263)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
at java.security.AccessController.doPrivileged(Native Method)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
at weblogic.management.mbeanservers.internal.SecurityInterceptor.invoke(SecurityInterceptor.java:444)
at weblogic.management.jmx.mbeanserver.WLSMBeanServer.invoke(WLSMBeanServer.java:323)
at weblogic.management.mbeanservers.domainruntime.internal.ManagedMBeanServerConnection.invoke(ManagedMBeanServerConnection.java:330)
at weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.invoke(FederatedMBeanServerInterceptor.java:349)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
at java.security.AccessController.doPrivileged(Native Method)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
at weblogic.management.mbeanservers.internal.JMXContextInterceptor.invoke(JMXContextInterceptor.java:263)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
at java.security.AccessController.doPrivileged(Native Method)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
at weblogic.management.mbeanservers.internal.SecurityMBeanMgmtOpsInterceptor.invoke(SecurityMBeanMgmtOpsInterceptor.java:65)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
at java.security.AccessController.doPrivileged(Native Method)
at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
at weblogic.management.mbeanservers.internal.SecurityInterceptor.invoke(SecurityInterceptor.java:444)
at weblogic.management.jmx.mbeanserver.WLSMBeanServer.invoke(WLSMBeanServer.java:323)
at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder$11$1.run(JMXConnectorSubjectForwarder.java:663)
at java.security.AccessController.doPrivileged(Native Method)
at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder$11.run(JMXConnectorSubjectForwarder.java:661)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder.invoke(JMXConnectorSubjectForwarder.java:654)
at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
at java.security.AccessController.doPrivileged(Native Method)
at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1367)
at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
at javax.management.remote.rmi.RMIConnectionImpl_WLSkel.invoke(Unknown Source)
at weblogic.rmi.internal.ServerRequest.sendReceive(ServerRequest.java:174)
at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:222)
at javax.management.remote.rmi.RMIConnectionImpl_1034_WLStub.invoke(Unknown Source)
at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:993)
at weblogic.management.remote.wlx.ClientProvider$WLXMBeanServerConnectionWrapper.invoke(ClientProvider.java:291)
at weblogic.management.remote.wlx.ClientProvider$WLXMBeanServerConnectionWrapper.invoke(ClientProvider.java:291)
at oracle.sysman.emas.sdk.model.jmx.JMXUtil.invokeOperation(JMXUtil.java:92)
at oracle.sysman.emas.sdk.pcs.PCSHelper.performNonJ2eeComponentOperation(PCSHelper.java:386)
at oracle.sysman.emas.sdk.pcs.PCSProgressTask.handleNonJ2eeComponent(PCSProgressTask.java:801)
at oracle.sysman.emas.sdk.pcs.PCSProgressTask.execute(PCSProgressTask.java:152)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at oracle.sysman.emas.sdk.progress.WorkWrapper.run(WorkWrapper.java:96)
at java.lang.Thread.run(Thread.java:662)
Caused by: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: PKI-07005: Untrusted Cert Chain.
at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1649)
at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:241)
at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:235)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1206)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:136)
at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Handshaker.java:593)
at com.sun.net.ssl.internal.ssl.Handshaker.process_record(Handshaker.java:529)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:893)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1138)
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:632)
at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272)
at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:276)
at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122)
at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212)
at java.io.BufferedWriter.flush(BufferedWriter.java:236)
at oracle.as.management.opmn.optic.OpmnPhone.sendRequest(OpmnPhone.java:772)
... 81 more
Caused by: java.security.cert.CertificateException: PKI-07005: Untrusted Cert Chain.
at oracle.security.pki.ssl.g.a(Unknown Source)
at oracle.security.pki.ssl.g.b(Unknown Source)
at oracle.security.pki.ssl.i.checkServerTrusted(Unknown Source)
at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1198)
... 95 more
]]
Found that topology.xml from $DOMAIN_HOME/opmn still showing port 6701 for instance.
That means Fusion MW control was still trying to communicate with opmn on port 6701. It should be 6703.
Solution:
======
1. Modify topology.xml from $DOMAIN_HOME/opmn to reflect correct port.
2. opmnctl stopall
3. De-register instance
./opmnctl unregisterinstance -instanceName <instName> -adminHost <host> -adminPort <Port#>
4. Re-register instance
./opmnctl registerinstance -adminHost <host> -adminPort <Port#>
5. opmnctl startall
Hope this helps.
thanks for this solution .
ReplyDelete