Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to idiomatically / efficiently pipe data from Read+Seek to Write?

Tags:

io

pipe

rust

I want to take data from random locations in input file, and output them sequentially to output file. Preferably, with no unnecessary allocations.

This is one kind of solution I have figured out:

use std::io::{ self, SeekFrom, Cursor, Read, Write, Seek };

#[test]
fn read_write() {
    // let's say this is input file
    let mut input_file = Cursor::new(b"worldhello");
    // and this is output file
    let mut output_file = Vec::<u8>::new();

    assemble(&mut input_file, &mut output_file).unwrap();

    assert_eq!(b"helloworld", &output_file[..]);
}

// I want to take data from random locations in input file
// and output them sequentially to output file
pub fn assemble<I, O>(input: &mut I, output: &mut O) -> Result<(), io::Error> 
    where I: Read + Seek, O: Write 
{
    // first seek and output "hello"
    try!(input.seek(SeekFrom::Start(5)));
    let mut hello_buf = [0u8; 5];
    try!(input.take(5).read(&mut hello_buf));
    try!(output.write(&hello_buf));

    // then output "world"
    try!(input.seek(SeekFrom::Start(0)));
    let mut world_buf = [0u8; 5];
    try!(input.take(5).read(&mut world_buf));
    try!(output.write(&world_buf));

    Ok(())
}

Let's not worry about I/O latency here.

Questions:

  1. Does the stable Rust have some helper to take x bytes from one stream and push them to another stream? Or do I have to roll my own?
  2. If I have to roll my own, maybe there is a better way?
like image 605
Nercury Avatar asked Nov 01 '15 11:11

Nercury


People also ask

What is the read and write function of a pipe?

Fortunately, the read function returns the number of read bytes, and we can implement a loop that exhausts the data in the pipe, as shown in the next code sample. Note that pipes have fixed capacity, and if the maximum is reached, the write operations will block until the reader retrieves some data.

How does pipe improve code readability in Python?

How Pipe improves code readability in Python —Code snippet by Author. Both codes produce the same results. Yet the second one is more intuitive than the first one. Obviously, it has fewer lines of code as well. This is how Pipe helps us simplify our codes. We can chain operations on a collection without writing separate lines of code.

What is the use of pipe system call?

It provides a unidirectional communication channel, namely a byte stream between two processes, and the data is moved sequentially in one direction. pipe system call is used to create a pipe and acquire file descriptors for its read and write ends.

Why does fork close the read end of a pipe?

Both child and parent close the one end of the pipe because the child process inherits the parent’s file descriptors on’ fork’. Finally, after the sent bytes are read, the read end of the pipe is closed by the child, terminating with an exit call.


1 Answers

You are looking for io::copy:

use std::io::{self, prelude::*, SeekFrom};

pub fn assemble<I, O>(mut input: I, mut output: O) -> Result<(), io::Error>
where
    I: Read + Seek,
    O: Write,
{
    // first seek and output "hello"
    input.seek(SeekFrom::Start(5))?;
    io::copy(&mut input.by_ref().take(5), &mut output)?;

    // then output "world"
    input.seek(SeekFrom::Start(0))?;
    io::copy(&mut input.take(5), &mut output)?;

    Ok(())
}

If you look at the implementation of io::copy, you can see that it's similar to your code. However, it takes care to handle more error cases:

  1. write does not always write everything you ask it to!
  2. An "interrupted" write isn't usually fatal.

It also uses a larger buffer size but still stack-allocates it.

like image 129
Shepmaster Avatar answered Nov 23 '22 14:11

Shepmaster