gcc accepts and clang rejects this code with nested generic lambdas, why?

Question

According to Godbolt's Compiler Explorer (see demo), the following code compiles with GCC (both 10.2 and trunk) and outputs 3628800, but fails to compile with Clang (both 11.0.1 and trunk). In both cases, -std=c++17 is used. It also compiles with MSVC 19, but not with other compilers. Why is that and whose behavior is correct?

#include <iostream>

int main()
{
   std::cout << [](auto f) { return f(f); }(
        [](auto& h) {
            return [&h](auto n) {
                if (n < 2)
                    return 1;
                else
                    return n * h(h)(n - 1);
                };
            })(10) << std::endl;
}

Further, even GCC and MSVC reject the code if I replace auto n with int n or if I replace if-else with the ternary operator (return n < 2 ? 1 : (n * h(h)(n - 1));).

Answer 1

This program is ill-formed, no diagnostic required, so both implementations (indeed, any implementation) are correct. The rule violated is [temp.res.general]/6.4:

a hypothetical instantiation of a template immediately following its definition would be ill-formed due to a construct that does not depend on a template parameter

The template here is the innermost operator(): it uses the specialization of the immediately containing operator() (with the very closure type of which it is a member as a template argument), which has a deduced return type. "Immediately following" here is (still) within the very return statement from which that type shall be deduced, so the instantiation would be ill-formed due to just h(h), which does not involve a template parameter of that template (i.e., the type of n).

GCC and MSVC don't bother doing the semantic check for the deduced return type until the final instantiation (with int for the 10), by which point the return type is known: it's just the appropriate variant of the innermost lambda. If that innermost lambda is not generic, however, the check takes place during instantiation of the containing operator(), and the same error occurs.

This difference of behavior can be seen in a much simpler example:

auto g();
template<class T> auto f(T x) {return g()-x;}

Clang has already rejected at this point, but GCC accepts, perhaps followed by

auto g() {return 1;}
int main() {return f(1);}