Why can't a class extend a static nested class occurring within it?

Question

This class:

public class OuterChild extends OuterChild.InnerParent {
    public static class InnerParent {
    }
}

Fails to compile:

$ javac OuterChild.java
OuterChild.java:1: error: cyclic inheritance involving OuterChild
public class OuterChild extends OuterChild.InnerParent {
       ^
1 error

because OuterChild would "depend on" itself, because (per §8.1.4 "Superclasses and Subclasses" of The Java Language Specification, Java SE 8 Edition) a class directly depends on any type that "is mentioned in [its] extends or implements clause […] as a qualifier in the fully qualified form of a superclass or superinterface name."

But I don't really understand the motivation here. What is the problematic dependency? Is it just for consistency with the case where InnerParent were non-static (and would therefore end up with a lexically enclosing instance of itself)?

Answer 1

This appears to be a fairly nefarious corner-case, as there are a number of bugs related to cyclic inheritance, often leading to infinite loops, stack overflows, and OOMs in the compiler. Here are some relevant quotes that may offer some insight:

Bug 4326631:

This example is not legal, and this is made clear in the forthcoming 2nd edition of the Java Language Specification. Classes simultaneously related by both inheritance and enclosure are problematical, however the original innerclasses whitepaper did not adequately address the issue, nor did the pre-1.3 compilers implement a consistent policy. In JLS 2nd edition, the rule against cyclic inheritance has been extended to prohibit a class or interface from "depending" on itself, directly or indirectly. A type depends not only on types that it extends or implements, but also on types that serve as qualifiers within the names of those types.

Bug 6695838:

The two class declarations are indeed cyclic; accordingly to JLS 8.1.4 we have that:

Foo depends on Foo$Intf (Foo$Intf appears in the implements clause of Foo)
Foo$Intf depends on Moo$Intf (Moo$Intf appears in the extends clause of Foo$Intf)
Foo$Intf depends on Foo (Foo appears as a qualifier in the extends clause of Foo$Intf)

For transitivity, we have that Foo depends on itself; as such the code should be rejected with a compile-time error.

Bug 8041994:

Stepping back, the directly-depends relationship for classes and interfaces was introduced in JLS2 to clarify JLS1 and to cover superclasses/superinterfaces that are nested classes (e.g. A.B in the Description).

Bug 6660289:

This problem is due to the order in which javac perform attribution of type-variable bounds wrt class attribution.

1) Attribution of class Outer<T extends Outer.Inner>
1a) Attribution of Outer triggers attribution of Outer's type variable
2) Attribution of Outer.T
2a) Attribution of Outer.T triggers attribution of its declared bound
3) Attribution of class Outer.Inner<S extends T>
3a) Attribution of Outer.Inner triggers attribution of Outer.Inner's type variable
4) Attribution of Outer.Inner<S>
4a) Attribution of Outer.Inner.S triggers attribution of its declared bound
5) Attribution of Outer.T - this does nothing but returning the type of T; as you can see, at this stage T's bound has not been set yet on the object representing the type of T.

At a later point, for each attributed type variable, javac performs a check to ensure that the bound of a given type variable does not introduce cyclic inheritance. But we have seen that no bound is set for Outer.T; for this is the reason javac crashes with a NPE when trying to detect a cycle in the inheritance tree induced by the declared bound of Outer.Inner.S.

Bug 6663588:

Type-variable bounds might refer to classes belonging to a cyclic inheritance tree which causes the resolution process to enter a loop when looking up for symbols.

To your specific question of "what is the problematic dependency?" it appears to be a complex compile-time symbol resolution edge case, and the solution introduced in JLS2 was to simply ban cycles introduced by qualifier types as well as actual supertypes.

In other words this could theoretically be made to work with appropriate improvements to the compiler, but until someone comes along and makes that happen it's more practical to just ban this unusual relationship in the language specification.