Error flushing write connection to OPMN

Fusion Middleware Control (EM) was showing ohs1 and webcache1 down. 
Although opmnctl status was showing them alive.

Background:
=========
There are two domains configured on this server.

For some reason opmn ports were conflicting for these two domain instances. 
And opmnctl startall for one of them was failing with:

Starting up the OPMN managed system components ...

opmnctl startall: starting opmn and all managed processes...

RCV: No such file or directory

Communication error with the OPMN server local port.

Check the OPMN log files

/ora11/product/middleware/fmw11g/opmn/bin/opmn: unexpected exit: code

512

opmnctl startall: opmn failed to start.

RCV: No such file or directory

Communication error with the OPMN server local port.

Check the OPMN log files

opmnctl status: opmn is not running.

opmn for both domain instances were trying to run on:

<port local="6700" remote="6701"/>

So we decided to change ports for one of them, by changing ports in opmn.xml ($INSTANCE_HOME/config/OPMN/opmn) to:

<port local="6702" remote="6703"/>

After above change, opmnctl startall worked fine for both domain instances.

Problem with EM:
=============
Now EM showed ohs1 and webcache1 down for the instance whose opmn ports were changed.

Although opmnctl status was showing them alive.

When tried to start ohs1 through EM got below error:

Error flushing write connection to OPMN. The remote OPMN server is probably not running or the OPMN mbean is not properly configured. See the adminserver log for more details.

 Admin Server Diagnostic log showed:
---------------------------------------

[2012-11-06T10:17:26.987-05:00] [AdminServer] [ERROR] [] [oracle.as.management.mbeans.opmn] [tid: Thread-46] [userId: weblogic] [ecid: 0c198ad5e21e89f5:-411d8c39:13ad5e5ae39:-8000-000000000000013b,1:17426] [APP: NonJ2EEManagement#11.1.1] OPMN mbeanstart failed with the following exception[[
oracle.as.management.opmn.optic.OpticBadConnectException: Error flushing write connection to OPMN
        at oracle.as.management.opmn.optic.OpmnPhone.sendRequest(OpmnPhone.java:774)
        at oracle.as.management.opmn.optic.OpmnPhone.makePhoneCall(OpmnPhone.java:330)
        at oracle.as.management.opmn.optic.OpmnPhone.request(OpmnPhone.java:270)
        at oracle.as.management.opmn.optic.OpmnQuery.getBuf(OpmnQuery.java:482)
        at oracle.as.management.mbeans.opmn.Opmn.manageProcess(Opmn.java:239)
        at oracle.as.management.mbeans.opmn.Opmn.startProcess(Opmn.java:103)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at oracle.as.jmx.framework.standardmbeans.spi.OracleStandardEmitterMBean.doInvoke(OracleStandardEmitterMBean.java:973)
        at oracle.adf.mbean.share.AdfMBeanInterceptor.internalInvoke(AdfMBeanInterceptor.java:104)
        at oracle.as.jmx.framework.generic.spi.interceptors.AbstractMBeanInterceptor.doInvoke(AbstractMBeanInterceptor.java:252)
        at oracle.as.jmx.framework.generic.spi.security.AbstractMBeanSecurityInterceptor.internalInvoke(AbstractMBeanSecurityInterceptor.java:190)
        at oracle.as.jmx.framework.generic.spi.interceptors.AbstractMBeanInterceptor.doInvoke(AbstractMBeanInterceptor.java:252)
        at oracle.security.jps.ee.jmx.JpsJmxInterceptor$2.run(JpsJmxInterceptor.java:344)
        at java.security.AccessController.doPrivileged(Native Method)
        at oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:313)
        at oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:413)
        at oracle.security.jps.ee.jmx.JpsJmxInterceptor.internalInvoke(JpsJmxInterceptor.java:360)
        at oracle.as.jmx.framework.generic.spi.interceptors.AbstractMBeanInterceptor.doInvoke(AbstractMBeanInterceptor.java:252)
        at oracle.as.jmx.framework.generic.spi.interceptors.ContextClassLoaderMBeanInterceptor.internalInvoke(ContextClassLoaderMBeanInterceptor.java:103)
        at oracle.as.jmx.framework.generic.spi.interceptors.AbstractMBeanInterceptor.doInvoke(AbstractMBeanInterceptor.java:252)
        at oracle.as.jmx.framework.generic.spi.interceptors.MBeanRestartInterceptor.internalInvoke(MBeanRestartInterceptor.java:116)
        at oracle.as.jmx.framework.generic.spi.interceptors.AbstractMBeanInterceptor.doInvoke(AbstractMBeanInterceptor.java:252)
        at oracle.as.jmx.framework.generic.spi.interceptors.LoggingMBeanInterceptor.internalInvoke(LoggingMBeanInterceptor.java:524)
        at oracle.as.jmx.framework.generic.spi.interceptors.AbstractMBeanInterceptor.doInvoke(AbstractMBeanInterceptor.java:252)
        at oracle.as.jmx.framework.standardmbeans.spi.OracleStandardEmitterMBean.invoke(OracleStandardEmitterMBean.java:887)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
        at java.security.AccessController.doPrivileged(Native Method)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
        at weblogic.management.mbeanservers.internal.JMXContextInterceptor.invoke(JMXContextInterceptor.java:263)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
        at java.security.AccessController.doPrivileged(Native Method)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
        at weblogic.management.mbeanservers.internal.SecurityInterceptor.invoke(SecurityInterceptor.java:444)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServer.invoke(WLSMBeanServer.java:323)
        at weblogic.management.mbeanservers.domainruntime.internal.ManagedMBeanServerConnection.invoke(ManagedMBeanServerConnection.java:330)
        at weblogic.management.mbeanservers.domainruntime.internal.FederatedMBeanServerInterceptor.invoke(FederatedMBeanServerInterceptor.java:349)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
        at java.security.AccessController.doPrivileged(Native Method)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
        at weblogic.management.mbeanservers.internal.JMXContextInterceptor.invoke(JMXContextInterceptor.java:263)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
        at java.security.AccessController.doPrivileged(Native Method)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
        at weblogic.management.mbeanservers.internal.SecurityMBeanMgmtOpsInterceptor.invoke(SecurityMBeanMgmtOpsInterceptor.java:65)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase$16.run(WLSMBeanServerInterceptorBase.java:449)
        at java.security.AccessController.doPrivileged(Native Method)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServerInterceptorBase.invoke(WLSMBeanServerInterceptorBase.java:447)
        at weblogic.management.mbeanservers.internal.SecurityInterceptor.invoke(SecurityInterceptor.java:444)
        at weblogic.management.jmx.mbeanserver.WLSMBeanServer.invoke(WLSMBeanServer.java:323)
        at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder$11$1.run(JMXConnectorSubjectForwarder.java:663)
        at java.security.AccessController.doPrivileged(Native Method)
        at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder$11.run(JMXConnectorSubjectForwarder.java:661)
        at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:363)
        at weblogic.management.mbeanservers.internal.JMXConnectorSubjectForwarder.invoke(JMXConnectorSubjectForwarder.java:654)
        at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427)
        at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72)
        at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1367)
        at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788)
        at javax.management.remote.rmi.RMIConnectionImpl_WLSkel.invoke(Unknown Source)
        at weblogic.rmi.internal.ServerRequest.sendReceive(ServerRequest.java:174)
        at weblogic.rmi.internal.BasicRemoteRef.invoke(BasicRemoteRef.java:222)
        at javax.management.remote.rmi.RMIConnectionImpl_1034_WLStub.invoke(Unknown Source)
        at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:993)
        at weblogic.management.remote.wlx.ClientProvider$WLXMBeanServerConnectionWrapper.invoke(ClientProvider.java:291)
        at weblogic.management.remote.wlx.ClientProvider$WLXMBeanServerConnectionWrapper.invoke(ClientProvider.java:291)
        at oracle.sysman.emas.sdk.model.jmx.JMXUtil.invokeOperation(JMXUtil.java:92)
        at oracle.sysman.emas.sdk.pcs.PCSHelper.performNonJ2eeComponentOperation(PCSHelper.java:386)
        at oracle.sysman.emas.sdk.pcs.PCSProgressTask.handleNonJ2eeComponent(PCSProgressTask.java:801)
        at oracle.sysman.emas.sdk.pcs.PCSProgressTask.execute(PCSProgressTask.java:152)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at oracle.sysman.emas.sdk.progress.WorkWrapper.run(WorkWrapper.java:96)
        at java.lang.Thread.run(Thread.java:662)
Caused by: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: PKI-07005: Untrusted Cert Chain.
        at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Alerts.java:174)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1649)
        at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:241)
        at com.sun.net.ssl.internal.ssl.Handshaker.fatalSE(Handshaker.java:235)
        at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1206)
        at com.sun.net.ssl.internal.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:136)
        at com.sun.net.ssl.internal.ssl.Handshaker.processLoop(Handshaker.java:593)
        at com.sun.net.ssl.internal.ssl.Handshaker.process_record(Handshaker.java:529)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:893)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1138)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:632)
        at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:59)
        at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
        at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272)
        at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:276)
        at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122)
        at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212)
        at java.io.BufferedWriter.flush(BufferedWriter.java:236)
        at oracle.as.management.opmn.optic.OpmnPhone.sendRequest(OpmnPhone.java:772)
        ... 81 more
Caused by: java.security.cert.CertificateException: PKI-07005: Untrusted Cert Chain.
        at oracle.security.pki.ssl.g.a(Unknown Source)
        at oracle.security.pki.ssl.g.b(Unknown Source)
        at oracle.security.pki.ssl.i.checkServerTrusted(Unknown Source)
        at com.sun.net.ssl.internal.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1198)
        ... 95 more

]]

Found that topology.xml from $DOMAIN_HOME/opmn still showing port 6701 for instance. 
That means Fusion MW control was still trying to communicate with opmn on port 6701. It should be 6703.


Solution:
======
1. Modify topology.xml from $DOMAIN_HOME/opmn to reflect correct port.
2. opmnctl stopall
3. De-register instance
./opmnctl unregisterinstance -instanceName <instName> -adminHost <host> -adminPort <Port#>
4. Re-register instance
./opmnctl registerinstance -adminHost <host> -adminPort <Port#>
5. opmnctl startall

Hope this helps.

oradism creation failed for unknown reasons 0 8 11001

On Solaris box, whenever we started database instance, we saw multiple occurrences of below warnings. This started after we setup Automatic Memory Management feature of 11g (memory_target).

WARNING: -------------------------------

WARNING: oradism did not start up correctly.

  Dynamic ISM can not be locked.----------------------------------------

oradism creation failed for unknown reasons 0 8 11001

Tue Oct 30 06:47:00 2012

WARNING: Detected too many memory locking problems.

WARNING: Performance degradation may occur.

 
First thing I checked was file permissions on $ORACLE_HOME/bin/oradism.

It was owned by oracle with 750 permissions.

The oradism utility SHOULD be owned by root with 4755 permissions.

Then I also realized that some other utilities like extjob are also owned by oracle instead of root.

This made me believe that the root.sh was not run. Root.sh script takes care of changing ownership and permissions on some these utilities.

We ran root.sh as root user and that did it. Now we are not seeing above warnings anymore in the alert log of databases.

Step Creating ASInstance failed

We were getting this error during PFRD config.sh execution to create domain and instance.

Checked the install<timestamp>.out file and saw below messages:

java.lang.Exception: oracle.as.provisioning.exception.ASProvisioningException
    at oracle.as.install.classic.ca.standard.InstanceProvisioningTask.doExecute(InstanceProvisioningTask.java:222)
    at oracle.as.install.classic.ca.standard.StandaloneTool.execute(StandaloneTool.java:50)
    at oracle.as.install.classic.ca.standard.StandardProvisionTaskList.execute(StandardProvisionTaskList.java:61)
    at oracle.as.install.classic.ca.ClassicConfigMain.doExecute(ClassicConfigMain.java:124)
    at oracle.as.install.engine.modules.configuration.client.ConfigAction.execute(ConfigAction.java:335)
    at oracle.as.install.engine.modules.configuration.action.TaskPerformer.run(TaskPerformer.java:87)
    at oracle.as.install.engine.modules.configuration.action.TaskPerformer.startConfigAction(TaskPerformer.java:104)
    at oracle.as.install.engine.modules.configuration.action.ActionRequest.perform(ActionRequest.java:15)
    at oracle.as.install.engine.modules.configuration.action.RequestQueue.perform(RequestQueue.java:63)
    at oracle.as.install.engine.modules.configuration.standard.StandardConfigActionManager.start(StandardConfigActionManager.java:158)
    at oracle.as.install.engine.modules.configuration.boot.ConfigurationExtension.kickstart(ConfigurationExtension.java:81)
    at oracle.as.install.engine.modules.configuration.ConfigurationModule.run(ConfigurationModule.java:83)
    at java.lang.Thread.run(Thread.java:619)
Caused by: oracle.as.provisioning.exception.ASProvisioningException
    at oracle.as.provisioning.engine.Config.executeConfigWorkflow_WLS(Config.java:872)
    at oracle.as.install.classic.ca.standard.InstanceProvisioningTask.doExecute(InstanceProvisioningTask.java:218)
    ... 12 more
Caused by: oracle.as.provisioning.engine.CfgWorkflowException
    at oracle.as.provisioning.engine.Engine.processEventResponse(Engine.java:596)
    at oracle.as.provisioning.fmwadmin.ASInstanceProv.createInstance(ASInstanceProv.java:178)
    at oracle.as.provisioning.fmwadmin.ASInstanceProv.createInstanceAndComponents(ASInstanceProv.java:116)
    at oracle.as.provisioning.engine.WorkFlowExecutor._createASInstancesAndComponents(WorkFlowExecutor.java:523)
    at oracle.as.provisioning.engine.WorkFlowExecutor.executeWLSWorkFlow(WorkFlowExecutor.java:439)
    at oracle.as.provisioning.engine.Config.executeConfigWorkflow_WLS(Config.java:866)
    ... 13 more
Caused by: oracle.as.provisioning.util.ConfigException:
Error creating ASInstance inb_inst.
Cause:
An internal operation has failed: Error in starting opmn server
Operation aborted because of a system call failure or internal error
Action:
See logs for more details.
    at oracle.as.provisioning.util.ConfigException.createConfigException(ConfigException.java:123)
    at oracle.as.provisioning.fmwadmin.ASInstanceProv._createInstance(ASInstanceProv.java:317)
    at oracle.as.provisioning.fmwadmin.ASInstanceProv.createInstance(ASInstanceProv.java:166)
    ... 17 more
Caused by: oracle.as.management.opmn.optic.OpticException: Error in starting opmn server
Operation aborted because of a system call failure or internal error
    at oracle.as.management.opmn.optic.OpmnAdmin.executeCommand(OpmnAdmin.java:310)
    at oracle.as.management.opmn.optic.OpmnAdmin.startOpmnServer(OpmnAdmin.java:87)
    at oracle.as.provisioning.fmwadmin.ASInstanceProv._createInstance(ASInstanceProv.java:254)
    ... 18 more
opmnctl start: opmn failed to start. 

 Cause:
An internal operation has failed: Error in starting opmn server
Operation aborted because of a system call failure or internal error

This is mostly related to network issues. So checked /etc/hosts file. It had no issues.

Then tried "nslookup" for the hostname and it timed out for DNS server and then used local DNS(/etc/hosts) to successfully resolve the name.

Later we found that the timeout received from DNS server was the main reason behind this instance creation failure, as the instance creation would timeout and report error if it doesnt complete in certain amount of time which seem to be lower than dns timeout.

We realized that the DNS server setting were incorrect and after correcting that nslookup did not fail. 

Also the instance creation step was successful.

HTH.

TNS-12599: TNS:cryptographic checksum mismatch

Lately noticing these errors a lot in alert.log of our newly upgraded 11g databases.

NI cryptographic checksum mismatch error: 12599.

  VERSION INFORMATION:
        TNS for Linux: Version 11.2.0.2.0 - Production        Oracle Bequeath NT Protocol Adapter for Linux: Version 11.2.0.2.0 - Production
        TCP/IP NT Protocol Adapter for Linux: Version 11.2.0.2.0 - Production
  Time: 13-OCT-2012 14:37:56
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12599

TNS-12599: TNS:cryptographic checksum mismatch
    ns secondary err code: 2526
    nt main err code: 0
    nt secondary err code: 0
    nt OS err code: 0

Our sqlnet.ora does have ASO parameters:

 SQLNET.ENCRYPTION_SERVER = REQUESTED

I see bunch of these errors right after start of the Oracle database. After investigating found that this is caused by dbconsole.

Whenever I tried to login to dbconsole it generated this error message in alert.log. There was no error on dbconsole screen.

I haven't found any solution to this error yet apart from stopping dbconsole. I will update once I have found a solution to this.

ORA-00972 or ORA-00903 while creating an object

I am reading 2 Day DBA for 11gR2 and came across below statement:

A schema object name must abide by certain rules. In addition to being unique within a schema, a schema object name cannot be longer than 30 bytes and must begin with a letter. If you attempt to create an object with a name that violates any of these rules, then the database raises an error.

I was curious to see what error codes would be that if the rules are violated. Below is the answer:


SQL> create table aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa(srno number);
create table aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa(srno number)
             *
ERROR at line 1:
ORA-00972: identifier is too long


SQL> create table aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa(srno number);

Table created.

SQL> create table 1a(srno number);
create table 1a(srno number)
             *
ERROR at line 1:
ORA-00903: invalid table name