Generalizing iteraton method in Rust

Question

Suppose I have some custom collection of Foos:

struct Bar {}
struct Foo {
    bar: Bar
}
struct SubList {
    contents: Vec<Foo>,
}

and suppose I also have a SuperList which is a custom collection of SubLists:

struct SuperList {
    contents: Vec<SubList>,
}

SubList and SuperList each provide a method bars:

impl SubList {
    fn bars(&self) -> impl Iterator<Item = &Bar> + '_ {
        self.contents.iter().map(|x| &x.bar)
    }
}
impl SuperList {
    fn bars(&self) ->  impl Iterator<Item = &Bar> + '_ {
        self.contents.iter().flat_map(|x| x.items())
    }
}

I want to define a trait that provides a method items, and implement that trait on SubList and SuperList so that SubList::items is equivalent to SubList::bars and SuperList::items is equivalent to SuperList::bars, so that I can do this:

fn do_it<T: Buz<Bar>>(buz: &T) {
    for item in buz.items() {
        println!("yay!")
    }
}

fn main() {
    let foos = vec![Foo{ bar: Bar{} }];
    let sublist = SubList{ contents: foos };
    do_it(&sublist);
    let superlist = SuperList{ contents: vec![sublist] };
    do_it(&superlist);
}

I can do what I want with dynamic dispatch:

trait Buz<T> {
    fn items(&self) -> Box<dyn Iterator<Item = &T> + '_>;
}
impl Buz<Bar> for SubList {
    fn items(&self) -> Box<dyn Iterator<Item = &Bar> + '_> {
        SubList::bars(self)
    }
}
impl Buz<Bar> for SuperList {
    fn items(&self) -> Box<dyn Iterator<Item = &Bar> + '_> {
        SuperList::bars(self)
    }
}

However, the following doesn't work:

trait Baz<T> {
    fn items(&self) -> impl Iterator<Item = &T> + '_;
}
impl Baz<Bar> for SubList {
    fn items(&self) -> impl Iterator<Item = &Bar> + '_ {
        SubList::bars(self)
    }
}
impl Baz<Bar> for SuperList {
    fn items(&self) -> impl Iterator<Item = &Bar> + '_ {
        SuperList::bars(self)
    }
}

(error[E0562]: `impl Trait` not allowed outside of function and inherent method return types)

Here's a playground link to what I've tried so far

How can I define a trait Baz which provides an items method to abstract over the bars methods of SubList and SuperList without using dynamic dispatch?

Answer 1

Unfortunately, what you are trying to do is not really possible in Rust right now. Not by design, but simply because some relevant type level features are not implemented or stabilized yet.

Unless you have spare time, an interest in type level stuff and are willing to use nightly: just use boxed iterators. They are simple, they just work, and in most cases it will likely not even hurt performance in a meaningful way.

---

You're still reading? Ok let's talk about it.

As you intuitively tried, impl Trait in return type position would be the obvious solution here. But as you noticed, it doesn't work: error[E0562]: `impl Trait` not allowed outside of function and inherent method return types. Why is that? RFC 1522 says:

Initial limitations:

impl Trait may only be written within the return type of a freestanding or inherent-impl function, not in trait definitions [...] Eventually, we will want to allow the feature to be used within traits [...]

These initial limitations were put in place because the type level machinery to make this work was/is not in place yet:

One important usecase of abstract return types is to use them in trait methods.

However, there is an issue with this, namely that in combinations with generic trait methods, they are effectively equivalent to higher kinded types. Which is an issue because Rust's HKT story is not yet figured out, so any "accidental implementation" might cause unintended fallout.

The following explanation in the RFC is also worth reading.

That said, some uses of impl Trait in traits can be achieved already today: with associated types! Consider this:

trait Foo {
    type Bar: Clone;
    fn bar() -> Self::Bar;
}

struct A;
struct B;

impl Foo for A {
    type Bar = u32;
    fn bar() -> Self::Bar { 0 }
}

impl Foo for B {
    type Bar = String;
    fn bar() -> Self::Bar { "hello".into() }
}

This works and is "basically equivalent" to:

trait Foo {
    fn bar() -> impl Clone;
}

Each impl block can choose a different return type as long as it implements a trait. So why then does impl Trait not simply desugar to an associated type? Well, let's try with your example:

trait Baz<T> {
    type Iter: Iterator<Item = &Bar>;
    fn items(&self) -> Self::Iter;
}

We get a missing lifetime specifier error:

4 |     type Iter: Iterator<Item = &Bar>;
  |                                ^ expected named lifetime parameter

Trying to add a lifetime parameter... we notice that we can't do that. What we need is to use the lifetime of &self here. But we can't get it at that point (in the associated type definition). This limitation is very unfortunate and is encountered in many situations (search term "streaming iterator"). The solution here are GATs: generic associated types. They allow us to write this:

trait Baz<T> {
    //       vvvv  That's what GATs allow
    type Iter<'s>: Iterator<Item = &'s Bar>;
    fn items(&self) -> Self::Iter<'_>;
}

GATs are not fully implemented and certainly not stable yet. See the tracking issue.

But even with GATs, we cannot fully make your example work. That's because you use iterator types that are unnameable, due to using closures. So the impl Baz for ... blocks would not be able to provide the type Iter<'s> definition. Here we can use another feature that is not stable yet: impl Trait in type aliases!

impl Baz<Bar> for SubList {
    type Iter<'s> = impl Iterator<Item = &'s Bar>;
    fn items(&self) -> Self::Iter<'_> {
        SubList::bars(self)
    }
}

This actually works! (Again, on nightly, with these unstable features.) You can see your full, working example in this playground.

This type system work has been going on for a long time and it seems like it's slowly reaching a state of being usable. (Which I am very happy about ^_^). I expect that a few of the foundational features, or at least subsets of them, will be stabilized in the not-too-distant future. And once these are used in practice and we see how they work, more convenience features (like impl Trait in traits) will be reconsidered and stabilized. Or so I think.

Also worth noting that async fns in traits are also blocked by this, since they basically desugar into a method returning impl Future<Output = ...>. And those are also a highly requested feature.

---

In summary: these limitations have been a pain point for quite some time and they resurface in different practical situations (async, streaming iterator, your example, ...). I'm not involved in compiler development or language team discussions, but I kept an eye on this topic for a long time and I think we are getting close to finally resolving a lot of these issues. Certainly not in the next releases, but I see a decent chance we get some of this in the next year or two.