IOException: Network adapter could not establish the connection [duplicate]

Question

We started to have this "Network adapter could not establish the connection" problem in our production environment from time to time, I did a lot of google and went through some questions here but still didn't get the solution.

Here are some information about our environment:

1. We are using RAC with 2 oracle instances (version 10.2.0.4).
2. We have several app servers (JBoss AS5) running on top of the RAC.
3. The connection url in oracle-ds.xml is "jdbc:oracle:thin:@(DESCRIPTION =(ADDRESS_LIST =(ADDRESS = (PROTOCOL = TCP)(HOST = db1_vip)(PORT = 1521))(ADDRESS = (PROTOCOL = TCP)(HOST=db2_vip)(PORT = 1521))(LOAD_BALANCE = yes))(CONNECT_DATA=(SERVER = DEDICATED)(SERVICE_NAME = G1db)(FAILOVER_MODE =(TYPE =SELECT)(METHOD = BASIC)(RETRIES = 120)(DELAY = 5))))"
4. We added db1_vip/db2_vip as well the actual ip of 2 oracle instances to the /etc/hosts of both app servers
5. There IS firewall between oracle instances and app servers, but we opened the 1521 port on both oracle instances.

We've run following tests to check if there is any problem on network or oracle TNS listener:

1. run ping from app servers to both oracle instances, everything works fine without any packet loss.
2. telnet the 1521 port from app servers to both oracle instances, nothing wrong.
3. tnsping also works fine.
4. We checked listener log but found nothing valuable.

The most weird thing is that this error happens around 10 times an hour on one app server, but only happens 1 or 2 times a day on another app server.

Can anyone shed some light on this error?

Thanks

[EDIT4]: We found timeout problem when tnsping one oracle instance, so we changed the jdbc url to connect only one instance and found the IOException never happened again, so we think the problem is related with database and the DBA team will continue look into this.

[EDIT3]: We did following tries:

1. disabled the firewall between app servers and database servers
2. Use ip instead of host name
3. use wireshark to check if there is any tcp packet problem.

none of them succeed, someone please help~

[EDIT]: the error stack trace:

java.sql.SQLException: Io Exception: The Network Adapter could not establish the connection
    at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:112)
    at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:146)
    at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:255)
    at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:387)
    at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:414)
    at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:165)
    at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:35)
    at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:801)
    at org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:207)
    ... 5 more

Answer 1

The error

java.sql.SQLException: Io Exception: The Network Adapter could not establish the connection

has nothing to do with Java/JDBC. This is the error you get when there's something wrong with your network setup. My guess, firewall problem. Can you disable it and see if the error is still there?