Tuesday, November 6, 2012

Error flushing write connection to OPMN

Fusion Middleware Control (EM) was showing ohs1 and webcache1 down. 
Although opmnctl status was showing them alive.

Background:
=========
There are two domains configured on this server.

For some reason opmn ports were conflicting for these two domain instances. 
And opmnctl startall for one of them was failing with:


Starting up the OPMN managed system components ...
opmnctl startall: starting opmn and all managed processes...
RCV: No such file or directory
Communication error with the OPMN server local port.
Check the OPMN log files
/ora11/product/middleware/fmw11g/opmn/bin/opmn: unexpected exit: code
512
opmnctl startall: opmn failed to start.
RCV: No such file or directory
Communication error with the OPMN server local port.
Check the OPMN log files
opmnctl status: opmn is not running.
opmn for both domain instances were trying to run on:

<port local="6700" remote="6701"/>
So we decided to change ports for one of them, by changing ports in opmn.xml ($INSTANCE_HOME/config/OPMN/opmn) to:

<port local="6702" remote="6703"/>

After above change, opmnctl startall worked fine for both domain instances.

Problem with EM:
=============
Now EM showed ohs1 and webcache1 down for the instance whose opmn ports were changed.

Although opmnctl status was showing them alive.

When tried to start ohs1 through EM got below error:

Error flushing write connection to OPMN. The remote OPMN server is probably not running or the OPMN mbean is not properly configured. See the adminserver log for more details.
 Admin Server Diagnostic log showed:
---------------------------------------
[2012-11-06T10:17:26.987-05:00] [AdminServer] [ERROR] [] [oracle.as.management.mbeans.opmn] [tid: Thread-46] [userId: weblogic] [ecid: 0c198ad5e21e89f5:-411d8c39:13ad5e5ae39:-8000-000000000000013b,1:17426] [APP: NonJ2EEManagement#11.1.1] OPMN mbeanstart failed with the following exception[[
oracle.as.management.opmn.optic.OpticBadConnectException: Error flushing write connection to OPMN
        at oracle.as.management.opmn.optic.OpmnPhone.sendRequest(OpmnPhone.java:774)
        at oracle.as.management.opmn.optic.OpmnPhone.makePhoneCall(OpmnPhone.java:330)
        at oracle.as.management.opmn.optic.OpmnPhone.request(OpmnPhone.java:270)
        at oracle.as.management.opmn.optic.OpmnQuery.getBuf(OpmnQuery.java:482)
        at oracle.as.management.mbeans.opmn.Opmn.manageProcess(Opmn.java:239)
        at oracle.as.management.mbeans.opmn.Opmn.startProcess(Opmn.java:103)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at oracle.as.jmx.framework.standardmbeans.spi.OracleStandardEmitterMBean.doInvoke(OracleStandardEmitterMBean.java:973)
        at oracle.adf.mbean.share.AdfMBeanInterceptor.internalInvoke(AdfMBeanInterceptor.java:104)
        at oracle.as.jmx.framework.generic.spi.interceptors.AbstractMBeanInterceptor.doInvoke(AbstractMBeanInterceptor.java:252)
        at oracle.as.jmx.framework.generic.spi.security.AbstractMBeanSecurityInterceptor.internalInvoke(AbstractMBeanSecurityInterceptor.java:190)
        at oracle.as.jmx.framework.generic.spi.interceptors.AbstractMBeanInterceptor.doInvoke(AbstractMBeanInterceptor.java:252)
        at oracle.security.jps.ee.jmx.JpsJmxInterceptor$2.run(JpsJmxInterceptor.java:344)
        at java.security.AccessController.doPrivileged(Native Method)
        at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:313)
        at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:413)
        at oracle.security.jps.ee.jmx.JpsJmxInterceptor.internalInvoke(JpsJmxInterceptor.java:360)
        at oracle.as.jmx.framework.generic.spi.interceptors.AbstractMBeanInterceptor.doInvoke(AbstractMBeanInterceptor.java:252)
        at oracle.as.jmx.framework.generic.spi.interceptors.ContextClassLoaderMBeanInterceptor.internalInvoke(ContextClassLoaderMBeanInterceptor.java:103)
        at oracle.as.jmx.framework.generic.spi.interceptors.AbstractMBeanInterceptor.doInvoke(AbstractMBeanInterceptor.java:252)
        at oracle.as.jmx.framework.generic.spi.interceptors.MBeanRestartInterceptor.internalInvoke(MBeanRestartInterceptor.java:116)
        at oracle.as.jmx.framework.generic.spi.interceptors.AbstractMBeanInterceptor.doInvoke(AbstractMBeanInterceptor.java:252)
        at oracle.as.jmx.framework.generic.spi.interceptors.LoggingMBeanInterceptor.internalInvoke(LoggingMBeanInterceptor.java:524)
        at oracle.as.jmx.framework.generic.spi.interceptors.AbstractMBeanInterceptor.doInvoke(AbstractMBeanInterceptor.java:252)
        at oracle.as.jmx.framework.standardmbeans.spi.OracleStandardEmitterMBean.invoke(OracleStandardEmitterMBean.java:887)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
        at java.security.AccessController.doPrivileged(Native Method)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
        at weblogic.management.mbeanservers.internal.JMXContextInterceptor.invoke(JMXContextInterceptor.java:263)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
        at java.security.AccessController.doPrivileged(Native Method)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
        at weblogic.management.mbeanservers.internal.SecurityInterceptor.invoke(SecurityInterceptor.java:444)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServer.invoke(WLSMBeanServer.java:323)
        at weblogic.management.mbeanservers.domainruntime.internal.ManagedMBeanServerConnection.invoke(ManagedMBeanServerConnection.java:330)
        at weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.invoke(FederatedMBeanServerInterceptor.java:349)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
        at java.security.AccessController.doPrivileged(Native Method)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
        at weblogic.management.mbeanservers.internal.JMXContextInterceptor.invoke(JMXContextInterceptor.java:263)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
        at java.security.AccessController.doPrivileged(Native Method)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
        at weblogic.management.mbeanservers.internal.SecurityMBeanMgmtOpsInterceptor.invoke(SecurityMBeanMgmtOpsInterceptor.java:65)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
        at java.security.AccessController.doPrivileged(Native Method)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
        at weblogic.management.mbeanservers.internal.SecurityInterceptor.invoke(SecurityInterceptor.java:444)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServer.invoke(WLSMBeanServer.java:323)
        at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder$11$1.run(JMXConnectorSubjectForwarder.java:663)
        at java.security.AccessController.doPrivileged(Native Method)
        at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder$11.run(JMXConnectorSubjectForwarder.java:661)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
        at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder.invoke(JMXConnectorSubjectForwarder.java:654)
        at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
        at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
        at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1367)
        at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
        at javax.management.remote.rmi.RMIConnectionImpl_WLSkel.invoke(Unknown Source)
        at weblogic.rmi.internal.ServerRequest.sendReceive(ServerRequest.java:174)
        at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:222)
        at javax.management.remote.rmi.RMIConnectionImpl_1034_WLStub.invoke(Unknown Source)
        at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:993)
        at weblogic.management.remote.wlx.ClientProvider$WLXMBeanServerConnectionWrapper.invoke(ClientProvider.java:291)
        at weblogic.management.remote.wlx.ClientProvider$WLXMBeanServerConnectionWrapper.invoke(ClientProvider.java:291)
        at oracle.sysman.emas.sdk.model.jmx.JMXUtil.invokeOperation(JMXUtil.java:92)
        at oracle.sysman.emas.sdk.pcs.PCSHelper.performNonJ2eeComponentOperation(PCSHelper.java:386)
        at oracle.sysman.emas.sdk.pcs.PCSProgressTask.handleNonJ2eeComponent(PCSProgressTask.java:801)
        at oracle.sysman.emas.sdk.pcs.PCSProgressTask.execute(PCSProgressTask.java:152)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at oracle.sysman.emas.sdk.progress.WorkWrapper.run(WorkWrapper.java:96)
        at java.lang.Thread.run(Thread.java:662)
Caused by: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: PKI-07005: Untrusted Cert Chain.
        at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1649)
        at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:241)
        at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:235)
        at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1206)
        at com.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:136)
        at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Handshaker.java:593)
        at com.sun.net.ssl.internal.ssl.Handshaker.process_record(Handshaker.java:529)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:893)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1138)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:632)
        at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
        at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
        at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272)
        at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:276)
        at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122)
        at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212)
        at java.io.BufferedWriter.flush(BufferedWriter.java:236)
        at oracle.as.management.opmn.optic.OpmnPhone.sendRequest(OpmnPhone.java:772)
        ... 81 more
Caused by: java.security.cert.CertificateException: PKI-07005: Untrusted Cert Chain.
        at oracle.security.pki.ssl.g.a(Unknown Source)
        at oracle.security.pki.ssl.g.b(Unknown Source)
        at oracle.security.pki.ssl.i.checkServerTrusted(Unknown Source)
        at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1198)
        ... 95 more

]]

Found that topology.xml from $DOMAIN_HOME/opmn still showing port 6701 for instance. 
That means Fusion MW control was still trying to communicate with opmn on port 6701. It should be 6703.


Solution:
======
1. Modify topology.xml from $DOMAIN_HOME/opmn to reflect correct port.
2. opmnctl stopall
3. De-register instance
./opmnctl unregisterinstance -instanceName <instName> -adminHost <host> -adminPort <Port#>
4. Re-register instance
./opmnctl registerinstance -adminHost <host> -adminPort <Port#>
5. opmnctl startall

Hope this helps.

1 comment: