How to correctly upgrade a runtime on Substrate node?

Question

Followed the create a first substrate chain it all works good.

And then I want to go one step further to customize my on the demo.rs file, here is what I am doing:

1. Totally replace code in demo.rs with code here, there is event involved now .

2. update the lib.rs with

Demo: demo::{Module, Call, Storage, Event<T>},  

and

impl demo::Trait for Runtime {
    type Event = Event;
}

3. run ./scripts/build.rs
4. run ./target/release/node-name --dev

Then I see my updated extrinsic function is NOT listed here on the Polkadot Web App or by upload the substrate_node_template_runtime_wasm.compact.wasm file by following the the step 5 on tutorial

So I have to run the following code to make the update happens:

rm -rf ./target
cargo build --release
./target/release/node-name --dev

By discuss with @shawntabrizi he suggest with the following command

./scripts/build.sh
cargo build --release
./target/release/node-name purge-chain --dev
./target/release/node-name --dev

It seems that without purge-chain both the substrate_node_template_runtime_wasm.compact.wasm and ./target/release/node-name are not updated.

Quoted here

By upgrading the runtime, you're simply switching out the block of code that is going to be receiving extrinsics and reading storage.

But I want know one step deeper, when upgrade a runtime node, what is the difference behind build.sh and cargo build? Is that because that substrate_node_template_runtime_wasm.compact.wasm and/or ./target/release/node-name binary is not updated in the case above?

Answer 1

Let's try to address a few of the different topics you asked:

what is the difference behind build.sh and cargo build

The Substrate runtime is compiled to both a Native binary and a Wasm blob. In Substrate v1.0, these compilation steps are separate. build.sh compiles your runtime to Wasm, while cargo build compiles your entire node (like the CLI, database, etc..) including a native version of your runtime.

It seems that without purge-chain both the substrate_node_template_runtime_wasm.compact.wasm and ./target/release/node-name are not updated.

It is important to understand the specifics happening in the background here. When you run a node, a database is stored locally which has your chain state. So if you start a node with ./target/release/node-name --dev for 50 blocks, stop the node, and then start it again, it will continue right where you left off (at block 51).

Remember that as part of your node's genesis configuration, the Runtime Wasm is stored on chain, and used to determine which version of the runtime you should run (native vs Wasm).

If you recompile your Wasm and your Native binary, and run it without doing anything else, you will see no differences. Even though your node binary is totally new and updated, it is using the same database with the old chain state. This means that in your database, you also have the old Wasm, and when the node does it's check for which version to use, it will fallback to use the Wasm from your database!

If you want your node to pull the latest changes you made, you can do one of two things:

1. Trigger an on-chain upgrade of your Wasm runtime. This will make your database have the latest runtime code, and thus your node would use the latest changes.

2. Purge your chain to restart your genesis. This will delete any old state of your Substrate blockchain, and ultimately populate the chain state with the latest Wasm runtime, which should agree with your node.

My suggestion of:

./scripts/build.sh
cargo build --release
./target/release/node-name purge-chain --dev
./target/release/node-name --dev

Will do the second method, cleaning up your database, and restarting your node from block 0 each time you upgrade your runtime logic. This is usually the easiest thing to do when you are developing a runtime, because there are a number of factors that could cause unexpected behaviors when performing a runtime upgrade.

I changed most of the code, and add Event in it.

Unfortunately, you don't share any code here, which could help debug this issue. Although it is important to note that you cannot use the runtime upgrade feature for every update to your runtime.

You should treat your blockchain like two separate parts which work together:

1. The blockchain storage
2. The blockchain logic

When you do an upgrade, your are basically swapping out the blockchain logic from one thing to another thing. Technically speaking, you could literally swap anything to anything. But practically speaking, it doesn't mean it will work. If your new logic does not understand your current blockchain storage, you will break your chain.

So imagine you introduce a change to a function which assumes you have completely different storage items than before.... well things won't turn out well.

In general, additive changes are just fine for runtime upgrades. Since new features do not affect old features, your storage should always be able to work well with your new runtime. However, if you make a runtime upgrade which assumes something has changed with your blockchain storage, you will need to trigger a migration of those storage items before any runtime logic actually executes. You can do this with a one-off on_initialize call, which will convert one set of storage items to the new format, however implementation details start to kick in when you talk about migrating lots and lots of data...

Anyway, in summary, there are too many factors around upgrading your runtime which could cause issues, maybe like the ones your are seeing. In general, you should not be doing initial development using runtime upgrades. Instead, you should generally purge your chain and start from scratch between iterations of your runtime.