How can I set up hyperledger fabric with multiple hosts using Docker?

Question

I work on the Hyperledger Fabric v1.0 and would like to make the Getting Setup work on multiple hosts. For now, 2 would be great.

Here is what I want to do:

1. Host1: start an orderer and 2 peers
2. Host2: start 1 peer
3. Host2: A client creates a channel (using the channel_test.sh updated with the good hosts IP) and join all the 3 peers
4. Host1: Call de deploy.js of the given example to deploy the chaincode

I have a problem on the 3rd step. I think the channel creation works but on my peers log I have the same warnings on the 3 peers:

Remote endpoint claims to be a different peer, expected [host1 IP:8051] but got [172.17.0.4:7051]
Failed obtaining connection for 172.31.9.126:8051, PKIid:[49 55 50 ...] reason: Authentication failure

It looks like they can't communicate with each other. Any idea where the problem is?

I still tried my step 4 but I can't deploy it unless I remove the host2: peer1 from the config.json. And even then, I can only query from the host1: peer0, not the host1: peer2.

Here are the commands I use to set up my network:

Host1: Orderer

docker run --rm -it --name orderer -p 8050:7050 
-e ORDERER_GENERAL_LEDGERTYPE=ram 
-e ORDERER_GENERAL_BATCHTIMEOUT=10s 
-e ORDERER_GENERAL_BATCHSIZE_MAXMESSAGECOUNT=10 
-e ORDERER_GENERAL_MAXWINDOWSIZE=1000 
-e ORDERER_GENERAL_ORDERERTYPE=solo 
-e ORDERER_GENERAL_LOGLEVEL=debug 
-e ORDERER_GENERAL_LISTENADDRESS=0.0.0.0 
-e ORDERER_GENERAL_LISTENPORT=7050 
-e ORDERER_RAMLEDGER_HISTORY_SIZE=100 
sfhackfest22017/fabric-orderer:x86_64-0.7.0-snapshot-c7b3fe0 orderer

Host1: Peer0

docker run --rm -it --name peer0 -p 8051:7051 -p 8053:7053
-v /var/run/:/host/var/run/ -v $BASE_DIR/tmp/peer0:/etc/hyperledger/fabric/msp/sampleconfig 
-e CORE_PEER_ADDRESSAUTODETECT=true 
-e CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock 
-e CORE_LOGGING_LEVEL=DEBUG 
-e CORE_PEER_NETWORKID=peer0 
-e CORE_NEXT=true 
-e CORE_PEER_ENDORSER_ENABLED=true 
-e CORE_PEER_ID=peer0 
-e CORE_PEER_PROFILE_ENABLED=true 
-e CORE_PEER_COMMITTER_LEDGER_ORDERER=$ORDERER_IP:7050 
-e CORE_PEER_GOSSIP_ORGLEADER=true 
-e CORE_PEER_GOSSIP_IGNORESECURITY=true 
sfhackfest22017/fabric-peer:x86_64-0.7.0-snapshot-c7b3fe0 peer node start --peer-defaultchain=false

Host1: Peer2

docker run --rm -it --name peer2 -p 8055:7051 -p 8057:7053 
-v /var/run/:/host/var/run/ -v $BASE_DIR/tmp/peer0:/etc/hyperledger/fabric/msp/sampleconfig
-e CORE_PEER_ID=peer2 
[Other parameters are the same as Peer0]
sfhackfest22017/fabric-peer:x86_64-0.7.0-snapshot-c7b3fe0 peer node start --peer-defaultchain=false

Host2: Peer1

docker run --rm -it --name peer1 -p 8051:7051 
-v /var/run/:/host/var/run/ -v $BASE_DIR/tmp/peer0:/etc/hyperledger/fabric/msp/sampleconfig
-e CORE_PEER_ID=peer1 
[Other parameters are the same as Peer0]
sfhackfest22017/fabric-peer:x86_64-0.7.0-snapshot-c7b3fe0 peer node start --peer-defaultchain=false

Host2: Cli

docker run --rm -it --name cli
    -v /var/run/:/host/var/run/ -v $BASE_DIR/tmp/peer3:/etc/hyperledger/fabric/msp/sampleconfig -v $BASE_DIR/src/github.com/example_cc/example_cc.go:/opt/gopath/src/github.com/hyperledger/fabric/examples/example_cc.go -v $BASE_DIR/channel_test.sh:/opt/gopath/src/github.com/hyperledger/fabric/peer/channel_test.sh
    --workdir /opt/gopath/src/github.com/hyperledger/fabric/peer  
    -e GOPATH=/opt/gopath 
    -e CORE_PEER_ADDRESSAUTODETECT=true
    -e CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock 
    -e CORE_LOGGING_LEVEL=DEBUG 
    -e CORE_NEXT=true 
    -e CORE_PEER_ID=cli 
    -e CORE_PEER_ENDORSER_ENABLED=true 
    -e CORE_PEER_COMMITTER_LEDGER_ORDERER=$ORDERER_IP:8050 
    -e CORE_PEER_ADDRESS=$PEER0_IP:8051 
    sfhackfest22017/fabric-peer:x86_64-0.7.0-snapshot-c7b3fe0 ./channel_test.sh

If you need more information feel free to ask.

Note: I'm not very familiar with docker, any improvement/advice on how I use it is welcome :)

Answer 1

I was able to host hyperledger fabric network on multiple machines using docker swarm mode. Swarm mode provides a network across multiple hosts/machines for the communication of the fabric network components.

This post explains the deployment process.It creates a swarm network and all the other machines join the network. https://medium.com/@wahabjawed/hyperledger-fabric-on-multiple-hosts-a33b08ef24f

It works with Fabric 1.0+

Answer 2

I found a solution that seems to work using docker swarm mode.

1. Initialize a swarm: (docker swarm documentation for mor information)
2. Join the swarm with the other host as a manager

3. Create a network ("hyp-net" in my case)

   docker network create --attachable --driver overlay hyp-net

Changes I had to do:

• Linked the containers with the --link docker parameter
• Added the --network docker parameter (--network=hyp-net)

• Added a new environment varialble todocker run command used:

  -e CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=hyp-net
  

Here are the commands that works for me:

Orderer

docker run --rm -it --network="hyp-net" --name orderer -p 8050:7050 
-e ORDERER_GENERAL_LEDGERTYPE=ram 
-e ORDERER_GENERAL_BATCHTIMEOUT=10s 
-e ORDERER_GENERAL_BATCHSIZE_MAXMESSAGECOUNT=10 
-e ORDERER_GENERAL_MAXWINDOWSIZE=1000 
-e ORDERER_GENERAL_ORDERERTYPE=solo 
-e ORDERER_GENERAL_LOGLEVEL=debug 
-e ORDERER_GENERAL_LISTENADDRESS=0.0.0.0 
-e ORDERER_GENERAL_LISTENPORT=7050 
-e ORDERER_RAMLEDGER_HISTORY_SIZE=100 
-e CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=hyp-net 
sfhackfest22017/fabric-orderer:x86_64-0.7.0-snapshot-c7b3fe0 orderer

Peer0

docker run --rm -it --link orderer:orderer --network="hyp-net" --name peer0 -p 8051:7051 -p 8053:7053 
-v /var/run/:/host/var/run/ -v $BASE_DIR/tmp/peer0:/etc/hyperledger/fabric/msp/sampleconfig  
-e CORE_PEER_ADDRESSAUTODETECT=true 
-e CORE_VM_ENDPOINT=unix:///host/var/run/docker.sock 
-e CORE_LOGGING_LEVEL=DEBUG 
-e CORE_PEER_NETWORKID=peer0 
-e CORE_NEXT=true 
-e CORE_PEER_ENDORSER_ENABLED=true 
-e CORE_PEER_ID=peer0 
-e CORE_PEER_PROFILE_ENABLED=true
-e CORE_PEER_COMMITTER_LEDGER_ORDERER=orderer:7050 
-e CORE_PEER_GOSSIP_ORGLEADER=true 
-e CORE_PEER_GOSSIP_IGNORESECURITY=true 
-e CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=hyp-net 
sfhackfest22017/fabric-peer:x86_64-0.7.0-snapshot-c7b3fe0 peer node start --peer-defaultchain=false

Peer1

docker run --rm -it --network="hyp-net" --link orderer:orderer --link peer0:peer0 [...] -e CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=hyp-net sfhackfest22017/fabric-peer:x86_64-0.7.0-snapshot-c7b3fe0 peer node start --peer-defaultchain=false

Peer2

docker run --rm -it --network="hyp-net" --link orderer:orderer --link peer0:peer0 --link peer1:peer1 [...] -e CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=hyp-net sfhackfest22017/fabric-peer:x86_64-0.7.0-snapshot-c7b3fe0 peer node start --peer-defaultchain=false

Cli

docker run --rm -it --network="hyp-net" --link orderer:orderer --link peer0:peer0 --link peer1:peer1 --link peer2:peer2 [...] -e CORE_VM_DOCKER_HOSTCONFIG_NETWORKMODE=hyp-net sfhackfest22017/fabric-peer:x86_64-0.7.0-snapshot-c7b3fe0 ./channel_test.sh

With this, I am able to deploy, invoke and query my chaincode.