I find myself again and again faced with a similar problem: there's some piece of code that processes data as it arrives from the user/network/produces of some sort. For efficiency reasons, I don't want to call flush()
or commit()
on every piece of data that I receive, but only occasionally.
I usually come up with code like this:
class Processor {
private final static MAX_SAVE_PERIOD = 60000;
private final static MIN_SAVE_PERIOD = 20000;
private final static int MAX_BUFFER = 10000;
Arraylist<Data> dataBuffer = new Arraylist<Data>();
private long lastSave = 0;
public Saver() {
new Timer().schedule(new TimerTask() {
periodicSave();
}, MAX_SAVE_PERIOD, MAX_SAVE_PERIOD);
Runtime.getRuntime().addShutdownHook(new Thread(new Runnable() {
public void run() {
periodicSave();
}
}));
}
public synchronized void processData(Data data) {
dataBuffer.add(data);
if(dataBuffer.size() >= MAX_BUFFER) {
saveData();
}
}
private synchronzied void periodicSave() {
if(!dataBuffer.isEmpty()) {
saveData();
}
}
private void saveData() {
if (System.currentTimeMillis() - lastSave < MIN_SAVE_PERIOD) return;
...
lastSave = System.currentTimeMillis();
}
}
I get the distinct feeling that I'm reinventing the wheel every time I write this, and what's more, I keep changing stuff every time I write this kind of code, depending if the various parts make sense in the specific context.
It seems to me that this is a very common pattern, but I don't remember seeing it named or implemented as a library utility. As long as I have to implement this myself, I keep facing analysis paralysis whenever I reimplement it. Please help me!
UPDATE: After I wrote this, I realized that I haven't accounted for flushing the buffer before JVM shutdown, so I added a shutdown hook in the constructor. Now I've realized that this code will still not work properly if the shutdown happens less than MIN_SAVE_PERIOD
milliseconds after the last save, so I should probably refactor saveData
. It's happening again.
Your code already says it: it's called buffering.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With