map chunking strategy, rechunk lag issue

Question

I'm having a horrible time coming up with a good question Title... sorry/please edit if your brain is less shot than mine.

I am having some issues handling my game's maps client side. My game is tile based using 32x32 pixel tiles. My first game map was 1750 x 1750 tiles. I had a bunch of layers client side, but managed to cut it down to 2 (ground and buildings). I was previously loading the entire map's layers into memory(short arrays). When I jumped to 2200 x 2200 tiles I noticed an older pc having some issues with out of memory (1GB+). I wish there was a data type between byte and short(I am aiming for ~1000 different tiles). My game supports multiple resolutions so the players visible space may show 23,17 tiles for a 800x600 resolution all the way up to 45,29 tiles for 1440x1024 (plus) resolutions. I use Tiled to draw my maps and output the 2 layers into separate text files using a format similar to the following (0, 0, 2, 0, 3, 6, 0, 74, 2...) all on one line.

With the help of many SO questions and some research I came up with a map chunking strategy. Using the player's current coordinates, as the center point, I load enough tiles for 5 times the size of the visual map(largest would be 45*5,29*5 = 225,145 tiles). The player is always drawn in the center and the ground moves beneath him/her(when you walk east the ground moves west). The minimap is drawn showing one screen away in all directions, to be three times the size of the visible map. Please see the below(very scaled down) visual representation to explain better than I likely explained it.

My issue is this: When the player moves "1/5th the chunk size tiles away" from the original center point (chunkX/Y) coordinates, I call for the game to re scan the file. The new scan will use the current coordinates of the player as it's center point. Currently this issue I'm having is the rechunking takes like .5s on my pc(which is pretty high spec). The map does not update for like 1-2 tile moves.

To combat the issue above I tried to run the file scanning in a new thread (before hitting the 1/5th point) into a temporary arraybuffer. Then once it was done scanning, I would copy the buffer into the real array, and call repaint(). Randomly I saw some skipping issues with this which was no big deal. Even worse I saw it drawing a random part of the map for 1-2 frames. Code sample below:

private void checkIfWithinAndPossiblyReloadChunkMap(){
    if (Math.abs(MyClient.characterX - MyClient.chunkX) + 10 > (MyClient.chunkWidth / 5)){ //arbitrary number away (10)
        Runnable myRunnable = new Runnable(){
            public void run(){
                logger.info("FillMapChunkBuffer started.");

                short chunkXBuffer = MyClient.characterX;
                short chunkYBuffer = MyClient.characterY;

                int topLeftChunkIndex = MyClient.characterX - (MyClient.chunkWidth / 2) + ((MyClient.characterY - (MyClient.chunkHeight / 2)) * MyClient.mapWidth); //get top left coordinate of chunk
                int topRightChunkIndex = topLeftChunkIndex + MyClient.chunkWidth - 1; //top right coordinate of chunk

                int[] leftChunkSides = new int[MyClient.chunkHeight];
                int[] rightChunkSides = new int[MyClient.chunkHeight];

                for (int i = 0; i < MyClient.chunkHeight; i++){ //figure out the left and right index points for the chunk
                    leftChunkSides[i] = topLeftChunkIndex + (MyClient.mapWidth * i);
                    rightChunkSides[i] = topRightChunkIndex + (MyClient.mapWidth * i);
                }

                MyClient.groundLayerBuffer = MyClient.FillGroundBuffer(leftChunkSides, rightChunkSides);
                MyClient.buildingLayerBuffer = MyClient.FillBuildingBuffer(leftChunkSides, rightChunkSides);

                MyClient.groundLayer = MyClient.groundLayerBuffer;
                MyClient.buildingLayer = MyClient.buildingLayerBuffer;
                MyClient.chunkX = chunkXBuffer;
                MyClient.chunkY = chunkYBuffer;

                MyClient.gamePanel.repaint();

                logger.info("FillMapChunkBuffer done.");
            }
        };
        Thread thread = new Thread(myRunnable);
        thread.start();
    } else if (Math.abs(MyClient.characterY - MyClient.chunkY) + 10 > (MyClient.chunkHeight / 5)){ //arbitrary number away (10)
        //same code as above for Y
    }
}

public static short[] FillGroundBuffer(int[] leftChunkSides, int[] rightChunkSides){
    try {
        return scanMapFile("res/images/tiles/MyFirstMap-ground-p.json", leftChunkSides, rightChunkSides);
    } catch (FileNotFoundException e) {
        logger.fatal("ReadMapFile(ground)", e);
        JOptionPane.showMessageDialog(theDesktop, getStringChecked("message_file_locks") + "\n\n" + e.getMessage(), getStringChecked("message_error"), JOptionPane.ERROR_MESSAGE);
        System.exit(1);
    }
    return null;
}

private static short[] scanMapFile(String path, int[] leftChunkSides, int[] rightChunkSides) throws FileNotFoundException {
    Scanner scanner = new Scanner(new File(path));
    scanner.useDelimiter(", ");

    int topLeftChunkIndex = leftChunkSides[0];
    int bottomRightChunkIndex = rightChunkSides[rightChunkSides.length - 1];

    short[] tmpMap = new short[chunkWidth * chunkHeight];
    int count = 0;
    int arrayIndex = 0;

    while(scanner.hasNext()){
        if (count >= topLeftChunkIndex && count <= bottomRightChunkIndex){ //within or outside (east and west) of map chunk
            if (count == bottomRightChunkIndex){ //last entry
                tmpMap[arrayIndex] = scanner.nextShort();
                break;
            } else { //not last entry
                if (isInsideMapChunk(count, leftChunkSides, rightChunkSides)){
                    tmpMap[arrayIndex] = scanner.nextShort();
                    arrayIndex++;
                } else {
                    scanner.nextShort();
                }
            }
        } else {
            scanner.nextShort();
        }

        count++;
    }

    scanner.close();
    return tmpMap;
}

I really am at my wits end with this. I want to be able to move on past this GUI crap and work on real game mechanics. Any help would be tremendously appreciated. Sorry for the long post, but trust me a lot of thought/sleepless nights has gone into this. I need the SO experts ideas. Thanks so much!!

p.s. I came up with some potential optimization ideas (but not sure these would solve some of the issue):

• split the map files into multiple lines so I can call scanner.nextLine() 1 time, rather than scanner.next() 2200 times

• come up with a formula that given the 4 corners of the map chunk will know if a given coordinate lies within it. this would allow me to call scanner.nextLine() when at the farthest point on chunk for a given line. this would require the multiline map file approach above.

• throw away only 1/5th of the chunk, shift the array, and load the next 1/5th of the chunk

Answer 1

Make sure scanning the file has finished before starting a new scan.

Currently you'll start scanning again (possibly in every frame) while your centre is too far away from the previous scan centre. To fix this remember you are scanning before you even start and enhance your far away condition accordingly.

// MyClient.worker represents the currently running worker thread (if any)
if(far away condition && MyClient.worker == null) {
    Runnable myRunnable = new Runnable() {
        public void run(){
            logger.info("FillMapChunkBuffer started.");

            try {
                short chunkXBuffer = MyClient.nextChunkX;
                short chunkYBuffer = MyClient.nextChunkY;

                int topLeftChunkIndex = MyClient.characterX - (MyClient.chunkWidth / 2) + ((MyClient.characterY - (MyClient.chunkHeight / 2)) * MyClient.mapWidth); //get top left coordinate of chunk
                int topRightChunkIndex = topLeftChunkIndex + MyClient.chunkWidth - 1; //top right coordinate of chunk

                int[] leftChunkSides = new int[MyClient.chunkHeight];
                int[] rightChunkSides = new int[MyClient.chunkHeight];

                for (int i = 0; i < MyClient.chunkHeight; i++){ //figure out the left and right index points for the chunk
                    leftChunkSides[i] = topLeftChunkIndex + (MyClient.mapWidth * i);
                    rightChunkSides[i] = topRightChunkIndex + (MyClient.mapWidth * i);
                }

                // no reason for them to be a member of MyClient
                short[] groundLayerBuffer = MyClient.FillGroundBuffer(leftChunkSides, rightChunkSides);
                short[] buildingLayerBuffer = MyClient.FillBuildingBuffer(leftChunkSides, rightChunkSides);


                MyClient.groundLayer = groundLayerBuffer;
                MyClient.buildingLayer = buildingLayerBuffer;
                MyClient.chunkX = chunkXBuffer;
                MyClient.chunkY = chunkYBuffer;
                MyClient.gamePanel.repaint();
                logger.info("FillMapChunkBuffer done.");
            } finally {
                // in any case clear the worker thread
                MyClient.worker = null;
            }
        }
    };

    // remember that we're currently scanning by remembering the worker directly
    MyClient.worker = new Thread(myRunnable);
    // start worker
    MyClient.worker.start();
}

Preventing a rescan before the previous rescan has completed presents another challenge: what to do if you walk diagonally i.e. you reach the situation where in x you're meeting the far away condition, start scanning and during that scan you'll meet the condition for y to be far away. Since you choose the next scan centre according to your current position, this problem should not arise as long as you have a large enough chunk size.

Remembering the worker directly comes with a bonus: what do you do if you need to teleport the player/camera at some point while you are scanning? You can now simply terminate the worker thread and start scanning at the new location: you'll have to check the termination flag manually in MyClient.FillGroundBuffer and MyClient.FillBuildingBuffer, reject the (partially computed) result in the Runnable and stop the reset of MyClient.worker in case of an abort.

If you need to stream more data from the file system in your game think of implementing a streaming service (extend the idea of the worker to one that's processing arbitrary file parsing jobs). You should also check if your hard drive is able to perform reading from multiple files concurrently faster than reading a single stream from a single file.

Turning to a binary file format is an option, but won't save much in terms of file size. And since Scanner already uses an internal buffer to do its parsing (parsing integers from a buffer is faster than filling a buffer from a file), you should first focus on getting your worker running optimally.