Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

redis bulk import using --pipe

Tags:

redis

I'm trying to import one million lines of redis commands, using the --pipe feature.

redis_version:2.8.1

cat file.txt | redis-cli --pipe

This results in the following error:

Error reading from the server: Connection reset by peer

Does anyone know what I'm doing wrong?

file.txt contains, for example,

lpush name joe
lpush name bob

edit: I now see there's probably a special format(?) for using pipe mode - http://redis.io/topics/protocol

like image 968
d-_-b Avatar asked Jan 01 '14 00:01

d-_-b


People also ask

How does Redis pipelining work?

Redis pipelining is a technique for improving performance by issuing multiple commands at once without waiting for the response to each individual command. Pipelining is supported by most Redis clients. This document describes the problem that pipelining is designed to solve and how pipelining works in Redis.

How are Redis pipelining and transaction different?

Unlike pipelines, pipelines use special commands to mark the beginning and the end of the transaction, and the server also can queue the commands from a transaction (so the client can send one at a time). So transactions are stateful on the server, it actually keeps track of an ongoing transaction.


1 Answers

To use the pipe mode (a.k.a mass insertion) you must indeed provide your commands directly in Redis protocol format.

The corresponding Redis protocol for LPUSH name joe is:

*3
$5
LPUSH
$4
name
$3
joe

Or as a quoted string: "*3\r\n$5\r\nLPUSH\r\n$4\r\nname\r\n$3\r\njoe\r\n".

This is what your input file must contain.

The Redis documentation includes a Ruby sample to help you generate the protocol: see gen_redis_proto.

like image 141
deltheil Avatar answered Oct 25 '22 05:10

deltheil