How to add more orderer nodes to a running hyperledger fabric network

Question

I have setup a hyperledger fabric network with 1 orderer node, but don't know how to add more orderer node to a running production hyperledger network.

Any help would be appreciated, thanks.

Answer 1

Firstly, your network ordering service has to be setup as a Kafka one, not solo. You can do this in your configtx.yaml file under OrdererType. You will then also have to create kafka brokers, zookeepers and configure all that. If you are not familiar with this, I found experimenting and studying this repo https://github.com/keenkit/fabric-sample-with-kafka very helpful.

Assuming you have a working network with a Kafka Ordering Service, adding an extra orderer is done via a channel update, which is very similar to adding a new org. There are quite a few steps involved but they are all listed and explained here http://hyperledger-fabric.readthedocs.io/en/release-1.1/channel_update_tutorial.html. I would recommend you understand how adding an org works first, but if you feel comfortable then the only differences for adding an orderer are:

• obviously there is no need to create new org crypto materials but you will need the crypto materials for another orderer
• instead of running the command jq -s '.[0] * {"channel_group":{"groups":{"Application":{"groups": {"Org3MSP":.[1]}}}}}' config.json ./channel-artifacts/org3.json > modified_config.json which adds the new org crypto material to the network, open the json file and look for "OrdererAddresses". There should be an array of orderers there under another tag "addresses". Add your orderer here and just save the file as modified_config.json. You can then just run the same commands moving forward.
• when you sign the envelope using peer channel signconfigtx -f org3_update_in_envelope.pb bootstrap your CLI with an active orderer and use the OrdererMSP as otherwise the orderer will reject your transaction. Organisation MSP's, which are used for adding new orgs, will not work.

To help troubleshoot, I found it easier to initially spin up the 2 Orderer setup that the github repo above creates and then test removing 1 orderer, followed by adding it back in. After that experiment further with adding a 3rd.

Just as a side note you can find all the other things that can be changed with a channel update here: http://hyperledger-fabric.readthedocs.io/en/release-1.1/config_update.html. Click "Click here to see the config" to see an example of the json config (Note: the example is a solo not a Kafka).

Step by step (as requested):

1. In crypto-config under OrdererOrgs: Specs: create a new hostname for your orderer (using the same domain and name as your other).
2. Run the command cryptogen extend --config=./crypto-config.yaml NOTE: the 'extend' part so it generates what you need and not regenerate everything.
3. Spin up a new orderer container that is essentially identical to another orderer except the crypto volumes point to the new crypto generate in step 2, (and perhaps different port depending on your setup). You may notice at this point it is connected to the kafka brokers and has your channel and blocks because it is using the same genesis block. What needs to be done though is the network needs to be made aware of the address of this new orderer.
4. docker exec -it cli bash into your CLI container and bootstrap it with an active orderer information as you will need the OrdererMSP to sign off this change.

Bootstrap e.g. (yours might be different): CORE_PEER_MSPCONFIGPATH=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/users/[email protected]/msp

CORE_PEER_ADDRESS=orderer0.example.com:7050

CORE_PEER_LOCALMSPID=OrdererMSP

CORE_PEER_TLS_ROOTCERT_FILE=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer0.example.com/tls/ca.crt

ORDERER_CA=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer0.example.com/msp/tlscacerts/tlsca.example.com-cert.pem

CHANNEL_NAME=mychannel

5. install jq in the CLI container to convert blocks to json and back apt update && apt install -y jq
6. fetch the latest config block peer channel fetch config config_block.pb -o orderer0.example.com:7050 -c $CHANNEL_NAME --tls --cafile $ORDERER_CA
7. convert to json and trim headers configtxlator proto_decode --input config_block.pb --type common.Block | jq .data.data[0].payload.data.config > config.json
8. open the json file look for "OrdererAddresses" and under that heading there should be another tag "addresses". Add the new IP and PORT for the new orderer in that array. Save the change as modified_config.json
9. covert json form step 7 to block configtxlator proto_encode --input config.json --type common.Config --output config.pb
10. convert json from step 8 to block configtxlator proto_encode --input modified_config.json --type common.Config --output modified_config.pb
11. calculate the delta between block in step 9 and 10 configtxlator compute_update --channel_id $CHANNEL_NAME --original config.pb --updated modified_config.pb --output org3_update.pb
12. change the delta back to json configtxlator proto_decode --input org3_update.pb --type common.ConfigUpdate | jq . > org3_update.json
13. wrap the json in a header echo '{"payload":{"header":{"channel_header":{"channel_id":"mychannel", "type":2}},"data":{"config_update":'$(cat org3_update.json)'}}}' | jq . > org3_update_in_envelope.json
14. convert it back to block configtxlator proto_encode --input org3_update_in_envelope.json --type common.Envelope --output org3_update_in_envelope.pb
15. Since you are bootstrap as an active orderer you can just submit it, as the submitting party gives you a free signature and its the only one you need peer channel update -f org3_update_in_envelope.pb -c $CHANNEL_NAME -o orderer0.example.com:7050 --tls --cafile $ORDERER_CA

Once your peers get this new block, they now know the address of the new orderer and can contact it.