Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

More than 4 billion key value pairs in Redis?

Tags:

redis

I am trying to store ip numbers in redis along with associated ISP information. I have Maxmind data and the csv files contain start and end numbers for each ISP.

When querying in SQL I can check for an IP(after converting it to a number) to be available in a range and get the associated ISP.

I was thinking of converting all the ranges to individual numbers and submit all the key values pairs in Redis for faster lookup. This approximately will result in 4 billion key value pairs in the Redis store. I have done this for a few hundred million key value pairs but I am looking for advice/suggestions when moving to 4 billion pairs in Redis. Any performance issues I must be aware of or are there ways I can do this better ?

Thank you for all the suggestions.

UPDATE: Thanks to the suggestions below I could get this working. Thought I'd share the Python code (quick and dirty) for this here :

import redis
import pymysql

conn = pymysql.connect(host='localhost',user='user',passwd='password',db='foo')
cur = conn.cursor()
cur.execute('select startipnum,endipnum,isp from wiki.ipisp order by endipnum;')
result = cur.fetchall()

r = redis.StrictRedis(host='localhost', port=6379, db=0)
ispctr = 1
for row in result:
    tempDict = {'ispname':row[2],'fromval':row[0],'toval':row[1]}
    namefield = ispctr
    r.hmset(namefield,tempDict)
    r.zadd('ispmaxindex',row[1],namefield)
    ispctr = ispctr+1
conn.close()

ipstotest = ['23.23.23.23','24.96.185.10','203.59.91.235','188.66.105.50','99.98.163.93']
for ip in ipstotest:
    ipvalsList = [int(ipoct) for ipoct in ip.split('.')]
    ipnum = (16777216*ipvalsList[0]) + (65536*ipvalsList[1]) + (256*ipvalsList[2]) + ipvalsList[3]
    ipnum = long(ipnum)
    tempVal1 = r.zrangebyscore('ispmaxindex',ipnum,float('Inf'),0,1)
    tempval2 = r.hgetall(tempval1[0])
    print tempval2['ispname']
like image 391
harshsinghal Avatar asked Oct 16 '25 17:10

harshsinghal


1 Answers

You can store 4B items in Redis with no specific degradation of performance, but you need the memory for this (i.e. everything must fit in memory).

The optimal ways to implement this kind of queries with Redis has been described here:

store ip ranges in Redis

and here:

Redis or Mongo for determining if a number falls within ranges?

So the complexity of the optimal solution depends on the fact you consider the ranges of IPs can overlap or not.

like image 112
Didier Spezia Avatar answered Oct 18 '25 21:10

Didier Spezia