Map with TTL option in Go

Question

I need to build a data-structure like this:

map[string]SomeType 

But it must store values for about 10 minutes and then clear it from memory. Second condition is records amount - it must be huge. This data-structure must add at least 2-5K records per second.

So, what is the most correct way in Go to make it?

I'm trying to make goroutine with timeout for each new elemnt. And one(or more) garbage-collector goroutine with channel to receive timeouts and clear elements. But I'm not sure it's the most clear way. Is it Ok to have millions of waiting goroutines with timeouts?

Thanks.

Answer 1

You will have to create a struct to hold your map and provide custom get/put/delete funcs to access it.

Note that 2-5k accesses per second is not really that much at all, so you don't have to worry about that.

Here's a simple implementation:

type item struct {
    value      string
    lastAccess int64
}

type TTLMap struct {
    m map[string]*item
    l sync.Mutex
}

func New(ln int, maxTTL int) (m *TTLMap) {
    m = &TTLMap{m: make(map[string]*item, ln)}
    go func() {
        for now := range time.Tick(time.Second) {
            m.l.Lock()
            for k, v := range m.m {
                if now.Unix() - v.lastAccess > int64(maxTTL) {
                    delete(m.m, k)
                }
            }
            m.l.Unlock()
        }
    }()
    return
}

func (m *TTLMap) Len() int {
    return len(m.m)
}

func (m *TTLMap) Put(k, v string) {
    m.l.Lock()
    it, ok := m.m[k]
    if !ok {
        it = &item{value: v}
        m.m[k] = it
    }
    it.lastAccess = time.Now().Unix()
    m.l.Unlock()
}

func (m *TTLMap) Get(k string) (v string) {
    m.l.Lock()
    if it, ok := m.m[k]; ok {
        v = it.value
        it.lastAccess = time.Now().Unix()
    }
    m.l.Unlock()
    return

}

playground

note(2020-09-23): for some reason the time resolution on the current version of the playground is way off, this works fine, however to try on the playground you have to change the sleep to 3-5 seconds.

Answer 2

Take a look at buntdb.

tinykv is no longer being maintained.

Just for the record, I had the same problem and wrote tinykv package which uses a map internally.

• It uses a heap of time.Time for timeouts, so it does not ranges over the whole map.
• A max interval can be set when creating an instance. But actual intervals for checking the timeout can be any value of time.Duration greater than zero and less than max, based on the last item that timed out.
• It provides CAS and Take functionality.
• A callback (optional) can be set which notifies which key and value got timed out.
• Timeouts can be explicit or sliding.