KDB/Q memory consumption

Question

I have a KDB/Q databse which has around ~2M records per day consuming about ~2G of memory. At end of day it runs some reporting stuff doing joins between the tables and outputting result into files on disk. During the computation the memory usage grows to ~15G. My problem is that once this operation finishes the memory is never released back and until the DB is restarted it consumes all the 15G of memory.

I would like to tell KDB to unload some tables from memory (not drop them though) but I don't want to restart the DB since some other apps are still connecting to it.

Is there a way to tell KDB to unload something from memory?

EDIT:

If anyone finds it interesting I suggest to have a look on .Q.gc[] for KDB 2.5+, looks promising.

Answer 1

Here is the sum up of my research:

• KDB prior to ver. 2.5 allocates 64MB memory chunks on need and never release them. It is able to reuse them though.
• recent KDB versions allow .Q.gc[] call which is on-request call to garbage collector (KDB uses ref. counting btw.)
• this is especially useful when you invoke some memory intensive computation which allocates lot of memory (in my case it was ~20gB) and you want to release the memory after the computation finishes.
• you can always consider putting the memory intensive script into a separate Q process so the memory will be released once the script finishes

Answer 2

For anyone trying this in future the easiest way would have been to:

1. Start a new KDB process.
2. From that process query to select out the smallest limited subsets of data needed.
3. Perform any joins / calculations / writing to file from that process. (allowing the original to carry on processing requests)
4. Close the process, freeing up all memory.

As mentioned by the above posters newer versions of KDB free up memory better but not perfect.

There's a good article on our company website that details KDB+ Memory Management: http://timestored.com/kdbGuides/memoryManagement