How can instanceof return true for a class that has a private constructor?

Question

This is a question from this book: https://www.cl.cam.ac.uk/teaching/0506/ConcSys/cs_a-2005.pdf page 28

Can you write an additional Java class which creates an object that, when passed to the test method causes it to print “Here!”? As I say in the code, editing the class A itself, or using library features like reflection, serialization, or native methods are considered cheating! I’ll provide some hints in lectures if nobody can spot it in a week or so. None of the PhD students has got it yet.

public class A {
  // Private constructor tries to prevent A
  // from being instantiated outside this
  // class definition
  //
  // Using reflection is cheating :-)
  private A() {
  }

  //  ’test’ method checks whether the caller has
  //  been able to create an instance of the ’A’
  //  class. Can this be done even though the
  //  constructor is private?
  public static void test(Object o) {
    if (o instanceof A) {
      System.out.println("Here!");
    }
  }
}

I know the question is a lot unclear. I can think of many different 'hack-ish' solutions but not sure if they will be counted as 'cheating' or not :)

I can't find the official answer so asking you for what would be a good answer.

Answer 1

If we consider that nesting class A does not "modify it" (as, technically, all lines of code are intact) then this solution is probably the only valid option:

class B
{
    static

    public class A {
      // Private constructor tries to prevent A
      // from being instantiated outside this
      // class definition
      //
      // Using reflection is cheating :-)
      private A() {
      }

      //  ’test’ method checks whether the caller has
      //  been able to create an instance of the ’A’
      //  class. Can this be done even though the
      //  constructor is private?
      public static void test(Object o) {
        if (o instanceof A) {
          System.out.println("Here!");
        }
      }
    }

    public static void main (String[] args) throws java.lang.Exception
    {
        A.test(new A());
    }
}

What I mean is, technically it follows all the rules:

1. Can you write an additional Java class which creates an object that, when passed to the test method causes it to print “Here!”? - Done
2. As I say in the code, editing the class A itself ... considered cheating! - Technically, the class is unedited. I copy pasted it into my code.
3. ... or using library features like reflection, serialization, or native methods are considered cheating! - Done

If, however, you decide that nesting class A should not be allowed, then I believe there is no proper solution to the problem given the current definition. Also, given the section of the book this task is given in, I bet that the author wanted to make the constructor protected but not private.

Answer 2

Somehow, I don't like this sort of questions. It's from a lecture back in 2005, and according to websearches, it seems that nobody has found "the" solution until now, and no solution has been published.

The constraints are clear, but the question of what is allowed or not is somewhat fuzzy. Every solution could be considered as "cheating", in one or the other way, because a class with a private constructor is not meant to be subclassed. That's a critical security mechanism, and the responsible engineers are working hard to make sure that this security mechanism cannot be trivially circumvented.

So of course, you have to cheat in order to solve this.

Nevertheless, I spent quite a while with this, and here's how I eventually cheated it:

1.) Download the Apache Bytecode Engineering Library, and place the bcel-6.0.jar in one directory.

2.) Create a file CreateB.java in the same directory, with the following contents:

import java.io.FileOutputStream;

import org.apache.bcel.Const;
import org.apache.bcel.generic.*;

public class CreateB
{
    public static void main(String[] args) throws Exception
    {
        ClassGen cg = new ClassGen("B", "A", "B.java",
            Const.ACC_PUBLIC | Const.ACC_SUPER, new String[] {});
        ConstantPoolGen cp = cg.getConstantPool();
        InstructionList il = new InstructionList();
        MethodGen method = new MethodGen(Const.ACC_PUBLIC, Type.VOID,
            Type.NO_ARGS, new String[] {}, "<init>", "B", il, cp);
        il.append(InstructionFactory.createReturn(Type.VOID));
        method.setMaxStack();
        method.setMaxLocals();
        cg.addMethod(method.getMethod());
        il.dispose();
        cg.getJavaClass().dump(new FileOutputStream("B.class"));
    }
}

3.) Compile and execute this class:

javac -cp .;bcel-6.0.jar CreateB.java
java  -cp .;bcel-6.0.jar CreateB

(note: On linux, the ; must be a :). The result will be a file B.class.

4.) Copy the class that was given in the question (verbatim - without any modification) into the same directory and compile it.

5.) Create the following class in the same directory, and compile it:

public class TestA
{
    public static void main(String[] args)
    {
        A.test(new B());
    }
}

6.) The crucial step: Call

java -Xverify:none TestA

The output will be Here!.

---

The key point is that the CreateB class creates a class B that extends A, but does not invoke the super constructor. (Note that an implicit super constructor invocation would normally be added by the compiler. But there's no compiler involved here. The bytecode is created manually). All this would usually fail with a VerifyError when the class is loaded, but this verification can be switched off with -Xverify:none.

So in summary:

• The class A itself is not edited (and also its byte code is not edited, I hope this is clear!)
• No reflection
• No serialization
• No custom native methods