How to understand Watkins's Q(λ) learning algorithm in Sutton&Barto's RL book?

Question

In Sutton&Barto's RL book (link), the Watkins's Q(λ) learning algorithm presented in Figure 7.14: Line 10 "For all s, a:", the "s,a" here is for all the (s,a), while the (s,a) in line 8 and line 9 is for the current (s,a), is this right?

In line 12 and 13, when a'!=a*, execute line 13, so all the e(s,a) will be set to 0, so what's the point of eligibility trace when all the eligibility trace are set to 0, since the situation a'!=a* will happen very often. Even if the situation a'!=a* don't happen very often, but once it happens, the meaning of eligibility trace will totally lose, then the Q will not be updated again, since all the e(s,a)=0, then in every update, the e(s,a) will still be 0 if using the replacing traces.

So, is this an error here?

Answer 1

The idea of eligibility traces is to give credit or blame only to the eligible state-action pairs. The book from Sutton & Barto has a nice illustration of the idea: Backward view of eligibility traces

In Watkin's Q(λ) algorithm you want to give credit/blame to the state-action pairs you actually would have visited, if you would have followed your policy Q in a deterministic way (always choosing the best action).

So the answer to your question is in line 5:

Choose a' from s' using policy derived from Q (e.g. epsilon-greedy)

Because a' is chosen epsilon greedy, there is a little chance (with probability epsilon) that you take an exploratory random step instead of a greedy step. In such a case the whole eligibility trace is set to zero, because it makes no sense to give credit/blame to state-action pairs that have been visited before. The state-action pairs you visited before the random exploratory step deserve no credit/blame for future rewards, hence you delete the whole eligibility trace. In the time steps afterwards you begin to build up a new eligibility trace...

Hope that helped.