What are non-lexical lifetimes?

Question

It's easiest to understand what non-lexical lifetimes are by understanding what lexical lifetimes are. In versions of Rust before non-lexical lifetimes are present, this code will fail:

fn main() {
    let mut scores = vec![1, 2, 3];
    let score = &scores[0];
    scores.push(4);
}

The Rust compiler sees that scores is borrowed by the score variable, so it disallows further mutation of scores:

error[E0502]: cannot borrow `scores` as mutable because it is also borrowed as immutable
 --> src/main.rs:4:5
  |
3 |     let score = &scores[0];
  |                  ------ immutable borrow occurs here
4 |     scores.push(4);
  |     ^^^^^^ mutable borrow occurs here
5 | }
  | - immutable borrow ends here

However, a human can trivially see that this example is overly conservative: score is never used! The problem is that the borrow of scores by score is lexical — it lasts until the end of the block in which it is contained:

fn main() {
    let mut scores = vec![1, 2, 3]; //
    let score = &scores[0];         //
    scores.push(4);                 //
                                    // <-- score stops borrowing here
}

Non-lexical lifetimes fix this by enhancing the compiler to understand this level of detail. The compiler can now more accurately tell when a borrow is needed and this code will compile.

A wonderful thing about non-lexical lifetimes is that once enabled, no one will ever think about them. It will simply become "what Rust does" and things will (hopefully) just work.

Why were lexical lifetimes allowed?

Rust is intended to only allow known-safe programs to compile. However, it is impossible to exactly allow only safe programs and reject unsafe ones. To that end, Rust errs on the side of being conservative: some safe programs are rejected. Lexical lifetimes are one example of this.

Lexical lifetimes were much easier to implement in the compiler because knowledge of blocks is "trivial", while knowledge of the data flow is less so. The compiler needed to be rewritten to introduce and make use of a "mid-level intermediate representation" (MIR). Then the borrow checker (a.k.a. "borrowck") had to be rewritten to use MIR instead of the abstract syntax tree (AST). Then the rules of the borrow checker had to be refined to be finer-grained.

Lexical lifetimes don't always get in the way of the programmer, and there are many ways of working around lexical lifetimes when they do, even if they are annoying. In many cases, this involved adding extra curly braces or a boolean value. This allowed Rust 1.0 to ship and be useful for many years before non-lexical lifetimes were implemented.

Interestingly, certain good patterns were developed because of lexical lifetimes. The prime example to me is the entry pattern. This code fails before non-lexical lifetimes and compiles with it:

fn example(mut map: HashMap<i32, i32>, key: i32) {
    match map.get_mut(&key) {
        Some(value) => *value += 1,
        None => {
            map.insert(key, 1);
        }
    }
}

However, this code is inefficient because it calculates the hash of the key twice. The solution that was created because of lexical lifetimes is shorter and more efficient:

fn example(mut map: HashMap<i32, i32>, key: i32) {
    *map.entry(key).or_insert(0) += 1;
}

The name "non-lexical lifetimes" doesn't sound right to me

The lifetime of a value is the time span during which the value stays at a specific memory address (see Why can't I store a value and a reference to that value in the same struct? for a longer explanation). The feature known as non-lexical lifetimes doesn't change the lifetimes of any values, so it cannot make lifetimes non-lexical. It only makes the tracking and checking of borrows of those values more precise.

A more accurate name for the feature might be "non-lexical borrows". Some compiler developers refer to the underlying "MIR-based borrowck".

Non-lexical lifetimes were never intended to be a "user-facing" feature, per se. They've mostly grown large in our minds because of the little papercuts we get from their absence. Their name was mostly intended for internal development purposes and changing it for marketing purposes was never a priority.

Yeah, but how do I use it?

In Rust 1.31 (released on 2018-12-06), you need to opt-in to the Rust 2018 edition in your Cargo.toml:

[package]
name = "foo"
version = "0.0.1"
authors = ["An Devloper <[email protected]>"]
edition = "2018"

As of Rust 1.36, the Rust 2015 edition also enables non-lexical lifetimes.

The current implementation of non-lexical lifetimes is in a "migration mode". If the NLL borrow checker passes, compilation continues. If it doesn't, the previous borrow checker is invoked. If the old borrow checker allows the code, a warning is printed, informing you that your code is likely to break in a future version of Rust and should be updated.

In nightly versions of Rust, you can opt-in to the enforced breakage via a feature flag:

#![feature(nll)]

You can even opt-in to the experimental version of NLL by using the compiler flag -Z polonius.

A sample of real problems solved by non-lexical lifetimes

• Returning a reference from a HashMap or Vec causes a borrow to last beyond the scope it's in?
• Why does HashMap::get_mut() take ownership of the map for the rest of the scope?
• Cannot borrow as immutable because it is also borrowed as mutable in function arguments
• How to update-or-insert on a Vec?
• Is there a way to release a binding before it goes out of scope?
• Cannot obtain a mutable reference when iterating a recursive structure: cannot borrow as mutable more than once at a time
• When returning the outcome of consuming a StdinLock, why was the borrow to stdin retained?
• Collaterally moved error when deconstructing a Box of pairs