Apache Ignite nodes fail: Failed to read magic header

Question

Apache Ignite servers deployed on two machines automatically shut down. No concrete clue in logs. I am able to see following warning message in log file :

[00:35:14,047][WARNING][tcp-disco-sock-reader-#86379%null%][TcpDiscoverySpi] Failed to read magic header (too few bytes received) [rmtAddr=/10.96.36.48:47830, locAddr=/10.96.36.48:47500]

How can i debug and solve this issue ?

Configuration:

    <bean id="grid.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
                <property name="peerClassLoadingEnabled" value="true"/>
                <property name="cacheConfiguration">
                                <list>
                                        <bean class="org.apache.ignite.configuration.CacheConfiguration">
                                                <property name="name" value="xyxCache" />
                                                <property name="writeSynchronizationMode" value="FULL_SYNC" />
                                                <property name="cacheMode" value="REPLICATED" />
                                                <property name="rebalanceMode" value="SYNC" />
                                                <property name="readFromBackup" value="true" />
                                                <property name="startSize" value="150000" />
                                                <property name="evictionPolicy">
                                                        <bean class="org.apache.ignite.cache.eviction.lru.LruEvictionPolicy">
                                                                <property name="maxSize" value="1000000" />
                                                        </bean>
                                                </property>
                                        </bean>
                                </list>
                        </property>
<property name="discoverySpi">
            <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
                <property name="ipFinder">
                    <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.multicast.TcpDiscoveryMulticastIpFinder">
                        <property name="addresses">
                            <list>
                                <value>10.96.36.48:47500</value>
                                <value>10.96.36.214:47500</value>
                            </list>
                        </property>
                    </bean>
                </property>
            </bean>
        </property>
    </bean>

Answer 1

Check which process on 10.96.36.48 machine binded to port 47830. For example, netstat -lnp will list the pid and process name next to each listening port. This will work under Linux.