Return lazy iterator that depends on data allocated within the function

Question

I am new to Rust and reading The Rust Programming Language, and in the Error Handling section there is a "case study" describing a program to read data from a CSV file using the csv and rustc-serialize libraries (using getopts for argument parsing).

The author writes a function search that steps through the rows of the csv file using a csv::Reader object and collect those entries whose 'city' field match a specified value into a vector and returns it. I've taken a slightly different approach than the author, but this should not affect my question. My (working) function looks like this:

extern crate csv;
extern crate rustc_serialize;

use std::path::Path;
use std::fs::File;

fn search<P>(data_path: P, city: &str) -> Vec<DataRow>
    where P: AsRef<Path>
{
    let file = File::open(data_path).expect("Opening file failed!");
    let mut reader = csv::Reader::from_reader(file).has_headers(true);

    reader.decode()
          .map(|row| row.expect("Failed decoding row"))
          .filter(|row: &DataRow| row.city == city)
          .collect()
}

where the DataRow type is just a record,

#[derive(Debug, RustcDecodable)]
struct DataRow {
    country: String,
    city: String,
    accent_city: String,
    region: String,
    population: Option<u64>,
    latitude: Option<f64>,
    longitude: Option<f64>
}

Now, the author poses, as the dreaded "exercise to the reader", the problem of modifying this function to return an iterator instead of a vector (eliminating the call to collect). My question is: How can this be done at all, and what are the most concise and idiomatic ways of doing it?

---

A simple attempt that i think gets the type signature right is

fn search_iter<'a,P>(data_path: P, city: &'a str)
    -> Box<Iterator<Item=DataRow> + 'a>
    where P: AsRef<Path>
{
    let file = File::open(data_path).expect("Opening file failed!");
    let mut reader = csv::Reader::from_reader(file).has_headers(true);

    Box::new(reader.decode()
                   .map(|row| row.expect("Failed decoding row"))
                   .filter(|row: &DataRow| row.city == city))
}

I return a trait object of type Box<Iterator<Item=DataRow> + 'a> so as not to have to expose the internal Filter type, and where the lifetime 'a is introduced just to avoid having to make a local clone of city. But this fails to compile because reader does not live long enough; it's allocated on the stack and so is deallocated when the function returns.

I guess this means that reader has to be allocated on the heap (i.e. boxed) from the beginning, or somehow moved off the stack before the function ends. If I were returning a closure, this is exactly the problem that would be solved by making it a move closure. But I don't know how to do something similar when I'm not returning a function. I've tried defining a custom iterator type containing the needed data, but I couldn't get it to work, and it kept getting uglier and more contrived (don't make too much of this code, I'm only including it to show the general direction of my attempts):

fn search_iter<'a,P>(data_path: P, city: &'a str)
    -> Box<Iterator<Item=DataRow> + 'a>
    where P: AsRef<Path>
{
    struct ResultIter<'a> {
        reader: csv::Reader<File>,
        wrapped_iterator: Option<Box<Iterator<Item=DataRow> + 'a>>
    }

    impl<'a> Iterator for ResultIter<'a> {
        type Item = DataRow;

        fn next(&mut self) -> Option<DataRow>
        { self.wrapped_iterator.unwrap().next() }
    }

    let file = File::open(data_path).expect("Opening file failed!");

    // Incrementally initialise
    let mut result_iter = ResultIter {
        reader: csv::Reader::from_reader(file).has_headers(true),
        wrapped_iterator: None // Uninitialised
    };
    result_iter.wrapped_iterator =
        Some(Box::new(result_iter.reader
                                 .decode()
                                 .map(|row| row.expect("Failed decoding row"))
                                 .filter(|&row: &DataRow| row.city == city)));

    Box::new(result_iter)
}

This question seems to concern the same problem, but the author of the answer solves it by making the concerned data static, which I don't think is an alternative for this question.

I am using Rust 1.10.0, the current stable version from the Arch Linux package rust.

Answer 1

CSV 1.0

As I alluded to in the answer for older versions of the crate, the best way of solving this is for the CSV crate to have an owning iterator, which it now does: DeserializeRecordsIntoIter

use csv::ReaderBuilder; // 1.1.1
use serde::Deserialize; // 1.0.104
use std::{fs::File, path::Path};

#[derive(Debug, Deserialize)]
struct DataRow {
    country: String,
    city: String,
    accent_city: String,
    region: String,
    population: Option<u64>,
    latitude: Option<f64>,
    longitude: Option<f64>,
}

fn search_iter(data_path: impl AsRef<Path>, city: &str) -> impl Iterator<Item = DataRow> + '_ {
    let file = File::open(data_path).expect("Opening file failed");

    ReaderBuilder::new()
        .has_headers(true)
        .from_reader(file)
        .into_deserialize::<DataRow>()
        .map(|row| row.expect("Failed decoding row"))
        .filter(move |row| row.city == city)
}

Before version 1.0

The straightest path to convert the original function would be to simply wrap the iterator. However, doing so directly will lead to problems because you cannot return an object that refers to itself and the result of decode refers to the Reader. If you could surmount that, you cannot have an iterator return references to itself.

One solution is to simply re-create the DecodedRecords iterator for each call to your new iterator:

fn search_iter<'a, P>(data_path: P, city: &'a str) -> MyIter<'a>
where
    P: AsRef<Path>,
{
    let file = File::open(data_path).expect("Opening file failed!");

    MyIter {
        reader: csv::Reader::from_reader(file).has_headers(true),
        city: city,
    }
}

struct MyIter<'a> {
    reader: csv::Reader<File>,
    city: &'a str,
}

impl<'a> Iterator for MyIter<'a> {
    type Item = DataRow;

    fn next(&mut self) -> Option<Self::Item> {
        let city = self.city;

        self.reader
            .decode()
            .map(|row| row.expect("Failed decoding row"))
            .filter(|row: &DataRow| row.city == city)
            .next()
    }
}

This could have overhead associated with it, depending on the implementation of decode. Additionally, this might "rewind" back to the beginning of the input — if you substituted a Vec instead of a csv::Reader, you would see this. However, it happens to work in this case.

Beyond that, I'd normally open the file and create the csv::Reader outside of the function and pass in the DecodedRecords iterator and transform it, returning a newtype / box / type alias around the underlying iterator. I prefer this because the structure of your code mirrors the lifetimes of the objects.

I'm a little surprised that there isn't an implementation of IntoIterator for csv::Reader, which would also solve the problem because there would not be any references.

See also:

• How can I store a Chars iterator in the same struct as the String it is iterating on?
• Is there an owned version of String::chars?
• What is the correct way to return an Iterator (or any other trait)?