Can a program depend on a library during compilation but not runtime?

Question

A compile-time dependency is generally required at runtime. In maven, a compile scoped dependency will be added to the classpath on runtime (e.g. in wars they will be copied to WEB-INF/lib).

It is not, however, strictly required; for instance, we may compile against a certain API, making it a compile-time dependency, but then at runtime include an implementation that also includes the API.

There may be fringe cases where the project requires a certain dependency to compile but then the corresponding code is not actually needed, but these will be rare.

On the other hand, including runtime dependencies that are not needed at compile-time is very common. For instance, if you're writing a Java EE 6 application, you compile against the Java EE 6 API, but at runtime, any Java EE container can be used; it's this container that provides the implementation.

Compile-time dependencies can be avoided by using reflection. For instance, a JDBC driver can be loaded with a Class.forName and the actual class loaded be configurable through a configuration file.

Each Maven dependency has a scope that defines which classpath that dependency is available on.

When you create a JAR for a project, dependencies are not bundled with the generated artifact; they are used only for compilation. (However, you can still make maven include the dependencies in the built jar, see: Including dependencies in a jar with Maven)

When you use Maven to create a WAR or an EAR file, you can configure Maven to bundle dependencies with the generated artifact, and you can also configure it to exclude certain dependencies from the WAR file using the provided scope.

The most common scope — compile — indicates that the dependency is available to your project on the compile classpath, the unit test compile and execution classpaths, and the eventual runtime classpath when you execute your application. In a Java EE web application, this means the dependency is copied into your deployed application. In a JAR file however, dependencies will not be included when the compile scope is used.

runtime scope indicates that the dependency is available to your project on the unit test execution and runtime execution classpaths, but unlike the compile scope it is not available when you compile your application or its unit tests. A Runtime Dependency is copied into your deployed application, but it is not available during compilation. This is good for making sure you do not mistakenly depend on a specific library. Imagine you have a specific logging implementation being used, but you only want to import a logging facade in your source code. You would include the concrete log library with a runtime scope, so you do not mistakenly rely on it.

Finally, provided scope indicates that the container in which your application executes provides the dependency on your behalf. In a Java EE application, this means the dependency is already on the Servlet container’s or application server’s classpath and is not copied into your deployed application. It also means that you need this dependency for compiling your project.

You need at compile time dependencies which you might need at runtime. However many libraries run without all its possible dependencies. i.e. libraries which can use four different XML libraries, but only needs one to work.

Many libraries, need other libraries in turn. These libraries are not needed at compile time but are needed at runtime. i.e. when the code is actually run.

Generally you are right and probasbly it is the ideal situation if runtime and compile time dependencies are identical.

I will give you 2 example when this rule is incorrect.

If class A depends on class B that depends on class C that depends on class D where A is your class and B, C and D are classes from different third party libraries you need only B and C at compile time and you need also D at runtime. Often programs use dynamic class loading. In this case you do not need classes dynamically loaded by library you are using at compile time. Moreover often the library chooses which implementation to use at runtime. For example SLF4J or Commons Logging can change the target log implementation at runtime. You need only SSL4J itself at compile time.

Opposite example when you need more dependencies at compile time than at runtime. Think that you are developing application that has to work at different environments or operating systems. You need all platform specific libraries at compile time and only libraries needed for current environment at runtime.

I hope my explanations help.

Usually, the static dependencies graph is a sub-graph of the dynamic one, see e.g. this blog entry from the author of NDepend.

That said, there are some exceptions, mainly dependencies that add compiler-support, which becomes invisible at runtime. For instance for code generation as via Lombok or additional checks as via the (pluggable type-)Checker Framework.

I understand the difference between runtime and compile-time and how to differentiate between the two, but I just don't see the need to make a distinction between compile-time and runtime dependencies.

The general compile-time and runtime concepts and the Maven specific compile and runtime scope dependencies are two very different things. You cannot directly compare them as these don't have the same frame : the general compile and runtime concepts are broad while the maven compile and runtime scope concepts is about specifically the dependencies availability/visibility according to the time : compilation or execution.
Don't forget that Maven is above all a javac/java wrapper and that in Java you have a compile time classpath that you specify with javac -cp ... and a runtime classpath that you specify with java -cp ... .
It would be not wrong to consider the Maven compile scope as a way to add a dependency both in the Java compile and runtime classppath (javac and java) while the Maven runtime scope can be seen as a way to add a dependency only in the Java runtime classppath (javac).

What I'm choking on is this: how can a program not depend on something at runtime that it depended on during compilation?

What you describe doesn't have any relationship with runtime and compile scope.
It looks like more to the provided scope that you specify for a dependency to depend to that at compile time but not at runtime.
You use it as you need the dependency to compile but you don't want to include it in the packaged component (JAR, WAR or any others) because the dependency is already provided by the environment : it can be included in the server or any path of the classpath specified as the Java application is started.

If my Java app uses log4j, then it needs the log4j.jar file in order to compile (my code integrating with and invoking member methods from inside log4j) as well as runtime (my code has absolutely no control over what happens once code inside log4j.jar is ran).

In this case yes. But suppose that you need to write a portable code that relies on slf4j as facade in front of log4j to be able to switch to another logging implementation later (log4J 2, logback or any other).
In this case in you pom you need to specify slf4j as a compile dependency (it is the default) but you will specify the log4j dependency as a runtime dependency :

<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-api</artifactId>
    <version>...</version>
</dependency>
<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-log4j12</artifactId>
    <version>...</version>
    <scope>runtime</scope>
</dependency>

In this way, the log4j classes could not be referenced in the compiled code but you will still be able to refer slf4j classes.
If you specified the two dependencies with the compile time, nothing will prevent you from referencing log4j classes in the compiled code and you could so create an undesirable coupling with the logging implementation :

<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-api</artifactId>
    <version>...</version>
</dependency>
<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-log4j12</artifactId>
    <version>...</version>
</dependency>

A common usage of runtime scope is the JDBC dependency declaration. To write portable code, you don't want that the client code may refer classes of the specific DBMS dependency (for example : PostgreSQL JDBC dependency) but you want all the same include it in your application as at runtime the classes are needed to make the JDBC API works with this DBMS.