How to add a new orderer in a running hyperledger fabric network using raft?

Question

I want to add a new orderer to my existing running network. Currently my network is as follows :

docker ps -a

CONTAINER ID        IMAGE                               COMMAND                  CREATED              STATUS              PORTS                                        NAMES
998b93eb81c6        hyperledger/fabric-tools:latest     "/bin/bash"              About a minute ago   Up About a minute                                                cli
87bada2d914b        hyperledger/fabric-orderer:latest   "orderer"                About a minute ago   Up About a minute   0.0.0.0:8050->7050/tcp                       orderer2.example.com
5907f35bb5b4        hyperledger/fabric-orderer:latest   "orderer"                About a minute ago   Up About a minute   0.0.0.0:8750->7050/tcp                       orderer6.example.com
7876e35f2fb9        hyperledger/fabric-orderer:latest   "orderer"                About a minute ago   Up About a minute   0.0.0.0:10050->7050/tcp                      orderer4.example.com
fba3185ec9c6        hyperledger/fabric-peer:latest      "peer node start"        About a minute ago   Up About a minute   0.0.0.0:7051->7051/tcp                       peer0.org1.example.com
8b5e4348f04c        hyperledger/fabric-orderer:latest   "orderer"                About a minute ago   Up About a minute   0.0.0.0:9050->7050/tcp                       orderer3.example.com
a5cffb73ceca        hyperledger/fabric-couchdb          "tini -- /docker-ent…"   About a minute ago   Up About a minute   4369/tcp, 9100/tcp, 0.0.0.0:5984->5984/tcp   couchdb0
6be4405ec45b        hyperledger/fabric-orderer:latest   "orderer"                About a minute ago   Up About a minute   0.0.0.0:11050->7050/tcp                      orderer5.example.com
790c9aa84911        hyperledger/fabric-orderer:latest   "orderer"                About a minute ago   Up About a minute   0.0.0.0:8055->7050/tc

I tried adding a orderer in etcdraft file but it isn't working. I am using First-network here.

Answer 1

You can add a new orderer in your RAFT consensus protocol setup by following these steps. It’s a lengthy process so I will also add a script later but for now you can follow these steps. Here, I am using first-network from the fabric samples for starting the network with 5 orderers.

1. First of all in crypto-config under OrdererOrgs: Specs: create a new hostname for your orderer (using the same domain and name as your other).

2. Then, run the command cryptogen extend --config=./crypto-config.yaml NOTE: the 'extend' part so it generates what you need and not regenerate everything.

3. Now first we will add orderer to the system channel and then when it has all the blocks of the system channel then we will move it into application channel so please make sure you are doing it properly.

4. Enter into your cli container by using docker exec -it cli bash and bootstrap it with an active orderer information as you will need the OrdererMSP to sign off this change.

Here are some of the environment variables that you will need to bootstrap cli with orderer:

CORE_PEER_MSPCONFIGPATH=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/users/[email protected]/msp

CORE_PEER_ADDRESS=orderer.example.com:7050

CORE_PEER_LOCALMSPID=OrdererMSP

CORE_PEER_TLS_ROOTCERT_FILE=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer.example.com/tls/ca.crt

ORDERER_CA=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/ordererOrganizations/example.com/orderers/orderer.example.com/msp/tlscacerts/tlsca.example.com-cert.pem

CHANNEL_NAME=[system-channel-name]

4. The next thing is to make sure you have all your binaries working inside your cli container as we will use jq and configtxlator tool here to convert blocks from protobuf to json and back

5. Fetch the latest config block: peer channel fetch config config_block.pb -o orderer.example.com:7050 -c $CHANNEL_NAME --tls --cafile $ORDERER_CA

6. Convert to json and trim headers: configtxlator proto_decode --input config_block.pb --type common.Block | jq .data.data[0].payload.data.config > config.json

7. open the json file look for "ConsensusType" section and under that heading there should be another tag "consenters". And now you have to add new TLS certs in this section of the latest orderer that you have created above. But here the certs are in Base64 encoded form so first you have to lookout for your tlscert then you have to convert in base64 and then insert it in this section.

In my case the tls cert is here:

crypto-config/ordererOrganizations/example.com/orderers/orderer.example.com/tls/server.crt

Now add it into the section given above in the same manner as others are mentioned like this:

   {
               "client_tls_cert": "xxxxxxxxxxxx",
               "host": "new_orderer.example.com",
               "port": 7050,
               "server_tls_cert": "xxxxxxxxxxxx"
   }

Add the base64 encoded cert instead of xxxxxxxx given above and save the change as modified_config.json

8. Convert json form step 6 to block configtxlator proto_encode --input config.json --type common.Config --output config.pb

9. Convert json from step 7 to block configtxlator proto_encode --input modified_config.json --type common.Config --output modified_config.pb

10. Calculate the delta between block in step 8 and 9: configtxlator compute_update --channel_id $CHANNEL_NAME --original config.pb --updated modified_config.pb --output orderer_update.pb

11. change the delta back to json: configtxlator proto_decode --input orderer_update.pb --type common.ConfigUpdate | jq . > orderer_update.json

12. Now, we have a decoded update file – orderer_update.json – that we need to wrap in an envelope message. This step will give us back the header field that we stripped away earlier. We’ll name this file: orderer_update_in_envelope.json

echo '{"payload":{"header":{"channel_header":{"channel_id":"$CHANNEL_NAME", "type":2}},"data":{"config_update":'$(cat orderer_update.json)'}}}' | jq . > orderer_update_in_envelope.json

13. Now we will convert it back into the fully fledged protobuf format that Fabric requires. We’ll name our final update object orderer_update_in_envelope.pb

configtxlator proto_encode --input orderer_update_in_envelope.json --type common.Envelope --output orderer_update_in_envelope.pb

14. Since your cli is already bootstrap as an active orderer you can just submit it, as the submitting party gives you a free signature and its the only one you need:

peer channel update -f orderer_update_in_envelope.pb -c $CHANNEL_NAME -o orderer.example.com:7050 --tls --cafile $ORDERER_CA

Ensure that the node that will be added is part of the system channel by checking that the config block that was fetched includes the certificate of (soon to be) added node.

15. Spin up a new orderer container that is essentially identical to another orderer except the crypto volumes point to the new crypto generate in step 2, (and perhaps different port depending on your setup). And the most important thing is to incorporate this orderer with the latest config block and for that you have to fetch the config block:

peer channel fetch config config_block.pb -o orderer.example.com:7050 -c $CHANNEL_NAME --tls --cafile $ORDERER_CA

then, move this config block to the channel-artifacts folder and add the path to this env variable inside orderer’s docker-compose file:

ORDERER_GENERAL_GENESISFILE=/var/hyperledger/orderer/orderer.genesis.block

After starting your orderer, you may notice at this point it is connected to the raft and the Steprequests are successful and has your channel and blocks because it is using the same genesis block. What needs to be done though is the network needs to be made aware of the address of this new orderer.

16. Wait for the Raft node to replicate the blocks from existing nodes for all channels its certificates have been added to. After this step has been completed, the node begins servicing the channel.

17. Add the endpoint of the newly added Raft node to the channel configuration of the system channel and for that you have to again repeat the whole process of channel update transaction as we have done earlier from (5th to 14th) and the only thing you have to do differently is in the step 7 you have to do this:

Open the json file look for "OrdererAddresses" section and under that heading there should be another tag "addresses". Add the new IP and PORT for the new orderer endpoint in that array. Save the change as modified_config.json and then do the rest as above.

Once your peers get this new block, they now know the address of the new orderer and can contact it.

18. And now you have to repeat all the steps by adding it to the application channel and for that just change this in your docker environment variable:

CHANNEL_NAME=[application-channel-name]

You have to add the same tls certs into the consenters section and then after having replicated blocks of the application channel you can add the orderer’s endpoint in the addresses section and then it will start reflecting all the latest changes that you will make to the application channel.