Multi-host Validating Peer Cluster Setup

Question

I am attempting to create a cluster of Hyperledger validating peers, each running on a different host, but it does not appear to be functioning properly.

After starting the root node and 3 peer nodes, this is the output running peer network list on the root node, vp0:

{"peers":[{"ID":{"name":"vp1"},"address":"172.17.0.2:30303","type":1},{"ID":{"name":"vp2"},"address":"172.17.0.2:30303","type":1},{"ID":{"name":"vp3"},"address":"172.17.0.2:30303","type":1}]}

This is the output from the same command on one of the peers, vp3:

{"peers":[{"ID":{"name":"vp0"},"address":"172.17.0.2:30303","type":1},{"ID":{"name":"vp3"},"address":"172.17.0.2:30303","type":1}]}

All of the peers only list themselves and the root, vp0, in their lists.

This is the log output from the root node, vp0: https://gist.github.com/mikezaccardo/f139eaf8004540cdfd24da5a892716cc

This is the log output from one of the peer nodes, vp3: https://gist.github.com/mikezaccardo/7379584ca4f67bce553c288541e3c58e

This is the command I'm running to create the root node:

nohup sudo docker run --name=$HYPERLEDGER_PEER_ID \
            --restart=unless-stopped \
            -i \
            -p 5000:5000 \
            -p 30303:30303 \
            -p 30304:30304 \
            -p 31315:31315 \
            -e CORE_VM_ENDPOINT=http://172.17.0.1:4243 \
            -e CORE_PEER_ID=$HYPERLEDGER_PEER_ID \
            -e CORE_PEER_ADDRESSAUTODETECT=true \
            -e CORE_PEER_NETWORKID=dev \
            -e CORE_PEER_VALIDATOR_CONSENSUS_PLUGIN=pbft \
            -e CORE_PBFT_GENERAL_MODE=classic \
            -e CORE_PBFT_GENERAL_N=$HYPERLEDGER_CLUSTER_SIZE \
            -e CORE_PBFT_GENERAL_TIMEOUT_REQUEST=10s \
            joequant/hyperledger /bin/bash -c "rm config.yaml; cp /usr/share/go-1.6/src/github.com/hyperledger/fabric/consensus/obcpbft/config.yaml .; peer node start" > $HYPERLEDGER_PEER_ID.log 2>&1&

And this is the command I'm running to create each of the other peer nodes:

nohup sudo docker run --name=$HYPERLEDGER_PEER_ID \
            --restart=unless-stopped \
            -i \
            -p 30303:30303 \
            -p 30304:30304 \
            -p 31315:31315 \
            -e CORE_VM_ENDPOINT=http://172.17.0.1:4243 \
            -e CORE_PEER_ID=$HYPERLEDGER_PEER_ID \
            -e CORE_PEER_DISCOVERY_ROOTNODE=$HYPERLEDGER_ROOT_NODE_ADDRESS:30303 \
            -e CORE_PEER_ADDRESSAUTODETECT=true \
            -e CORE_PEER_NETWORKID=dev \
            -e CORE_PEER_VALIDATOR_CONSENSUS_PLUGIN=pbft \
            -e CORE_PBFT_GENERAL_MODE=classic \
            -e CORE_PBFT_GENERAL_N=$HYPERLEDGER_CLUSTER_SIZE \
            -e CORE_PBFT_GENERAL_TIMEOUT_REQUEST=10s \
            joequant/hyperledger /bin/bash -c "rm config.yaml; cp /usr/share/go-1.6/src/github.com/hyperledger/fabric/consensus/obcpbft/config.yaml .; peer node start" > $HYPERLEDGER_PEER_ID.log 2>&1&

HYPERLEDGER_PEER_ID is vp0 for the root node and vp1, vp2, ... for the peer nodes, HYPERLEDGER_ROOT_NODE_ADDRESS is the public IP address of the root node, and HYPERLEDGER_CLUSTER_SIZE is 4.

This is the Docker image that I am using: github.com/joequant/hyperledger

Is there anything obviously wrong with my commands? Should the actual public IP addresses of the peers be showing up as opposed to just 172.17.0.2? Are my logs helpful / is any additional information needed?

Any help or insight would be greatly appreciated, thanks!

Answer 1

I've managed to get a noops cluster working in which all nodes discover each other and chaincodes successfully deploy.

I made a few fixes since my post above:

I now use mikezaccardo/hyperledger-peer image, a fork of yeasy/hyperledger-peer, instead of joequant/hyperledger.

I changed:

-e CORE_PEER_ADDRESSAUTODETECT=true \ 

to:

-e CORE_PEER_ADDRESS=$HOST_ADDRESS:30303 \
-e CORE_PEER_ADDRESSAUTODETECT=false \

so that each peer would advertise its public IP, not private.

And I properly tag my image as the official base image:

 sudo docker tag mikezaccardo/hyperledger:latest hyperledger/fabric-baseimage:latest

Finally, for context, this is all related to my development of a blueprint for Apache Brooklyn which deploys a Hyperledger Fabric cluster. That repository, which contains all of the code mentioned in this post and answer, can be found here: https://github.com/cloudsoft/brooklyn-hyperledger.