How to properly overwrite content of file using android storage access framework

Question

>> Background

I want to use the SAF (Storage Access Frameword) to save data files of my app to the user's desired location on storage media. I first create the file at app dedicated folder and then copy it to the file that users has selected from file chooser dialog (the codes are comming later).

this procedure works perfectly for new files but for existing files though the file chooser warns for overwriting the file, the final file will not erased before writing.

By counting the bytes written and investigating the files using hex editor, the code writes correct bytes to the output stream but: if the existing file has more bytes than bytes to be written, the final overwritten file is corrupted (Not actually corrupted, please see the next sections for clarify) and if the existing has less bytes than bytes to be written, the final overwritten file is correct.

>> More Detail and codes

I use the codes bellow to show the issue (the jpg is as sample): and i will try to work with two files:

file1.jpg 166,907 bytes
file2.jpg 1,323,647 bytes
file3.jpg The final file with variable size

First i will copy the file1 it to user selected folder with the name file3 (the destination file), then overwrite it with file2 and finaly i will overwrite it again with file1. see what are the codes and what happens:

Code for call the file chooser:

val intent = Intent(Intent.ACTION_CREATE_DOCUMENT).apply {
    addCategory(Intent.CATEGORY_OPENABLE)
    type = "image/jpeg"
}
startActivityForResult(intent, request)

Now at onActivityResult() i process the data as bellow:

contentResolver.openOutputStream(fileUri)?.use {output->
        val input = FileInputStream(File(getExternalFilesDir(null),"Pictures/file1.jpg"))
    // file1.jpg for first run, file2.jpg for 2nd run and file1.jpg again for 3rd run     
        copyStream(input, output)

    }

And the code for copying the stream:

@Throws(IOException::class)
fun copyStream(input: InputStream, output: OutputStream) {
    val buffer = ByteArray(1024)
    var bytesRead = input.read(buffer)
    while (bytesRead > 0) {
        output.write(buffer, 0, bytesRead)
        bytesRead = input.read(buffer)
    }
    input.close()
    //The output will be closes by kotlin standard function "use" at previous code
}

Now at first run the file3.jpg is exactly the same as file1.jpg. also the file3.jpg is same as file2.jpg at the second run. but at third run which overwrites file3.jpg with content of file1.jpg (Which has kess bytes than file3.jpg) the size of file3.jpg remains 1,323,647 bytes and the first 166,907 bytes are same as file1.jpg and the remaining bytes until 1,323,647 are same as file2.jpg that was written at 2nd run.

this is content of the files in hex:

file1.jpg

0000:0000 | FF D8 FF E1  09 49 45 78  69 66 00 00  49 49 2A 00 | ÿØÿá.IExif..II*.
0000:0010 | 08 00 00 00  09 00 0F 01  02 00 06 00  00 00 7A 00 | ..............z.
...
0002:8BE0 | 56 5E 2A EC  C7 36 6D B1  57 1C D5 CD  95 8A BB 2F | V^*ìÇ6m±W.ÕÍ..»/
0002:8BF0*| 36 6C 55 AD  F2 F3 65 60  43 FF D9*                | 6lU.òóe`CÿÙ     

file2.jpg

0000:0000 | FF D8 FF E0  00 10 4A 46  49 46 00 01  01 00 00 01 | ÿØÿà..JFIF......
0000:0010 | 00 01 00 00  FF E1 01 48  45 78 69 66  00 00 49 49 | ....ÿá.HExif..II
...
0002:8BC0 | F2 07 23 D4  57 CA 7E 13  FD A9 23 B5  86 2D 3E 4D | ò.#ÔWÊ~.ý©#µ.->M
0002:8BD0 | 66 7B 58 D1  42 A3 4D 6A  57 80 38 C9  CF EB 5E 93 | f{XÑB£MjW.8ÉÏë^.
0002:8BE0 | E1 3F DA 36  CA EA 10 2E  7C 49 0B C4  E3 21 F6 8C | á?Ú6Êê..|I.Äã!ö.
0002:8BF0*| 9F D6 BB 63  8B A3 86 D5  34 B5 D9*E8  D2 E9 D7 AE | .Ö»c.£.Õ4µÙèÒé×®
0002:8C00 | B7 34 9F B5  85 18 C6 B5  DF 2E FA 6B  AD B6 5D BC | ·4.µ..Ƶß.úk.¶]¼
0002:8C10 | F7 3D 6E F3  C3 50 6B 56  32 D9 CC 14  AB AE 30 C3 | ÷=nóÃPkV2ÙÌ.«®0Ã
...
0014:3260 | E8 8B 0A CE  4E 47 AD 4A  92 B2 E4 E6  8B 3B 7F 34 | è..ÎNG.J.²äæ.;.4
0014:3270 | 1C 55 D8 6C  14 83 BA 88  AB 98 46 4D  33 FF D9    | .UØl..º.«.FM3ÿÙ 

file3.jpg (After the 3rd run)

0000:0000 | FF D8 FF E1  09 49 45 78  69 66 00 00  49 49 2A 00 | ÿØÿá.IExif..II*.
0000:0010 | 08 00 00 00  09 00 0F 01  02 00 06 00  00 00 7A 00 | ..............z.
...
0002:8BD0 | D9 B1 43 BA  E6 39 B7 CD  8A B5 97 9B  36 29 76 5E | Ù±Cºæ9·Í.µ..6)v^
0002:8BE0 | 56 5E 2A EC  C7 36 6D B1  57 1C D5 CD  95 8A BB 2F | V^*ìÇ6m±W.ÕÍ..»/
//content of file1 continues with content of file2 (Next line)
0002:8BF0*| 36 6C 55 AD  F2 F3 65 60  43 FF D9*E8  D2 E9 D7 AE | 6lU.òóe`CÿÙèÒé×®
0002:8C00 | B7 34 9F B5  85 18 C6 B5  DF 2E FA 6B  AD B6 5D BC | ·4.µ..Ƶß.úk.¶]¼
0002:8C10 | F7 3D 6E F3  C3 50 6B 56  32 D9 CC 14  AB AE 30 C3 | ÷=nóÃPkV2ÙÌ.«®0Ã
0002:8C20 | 8C F3 83 5E  55 3D 86 A1  F0 EB C5 72  E9 C6 62 E2 | .ó.^U=.¡ðëÅréÆbâ
...
0014:3260 | E8 8B 0A CE  4E 47 AD 4A  92 B2 E4 E6  8B 3B 7F 34 | è..ÎNG.J.²äæ.;.4
0014:3270 | 1C 55 D8 6C  14 83 BA 88  AB 98 46 4D  33 FF D9    | .UØl..º.«.FM3ÿÙ 

As you can see the file3 starts with content of file1 and after the final bytes of file1 (FF D9) at third group at line 0002:8BF0 it continues with content of file2 (E8 D2) (Star points)

I tested the process of copying same files directly inside app's dedicated folder but the results were correct with correct file3 for all three runs. the problem was just for SAF.

Answer 1

I Found the answer after three days of searching and a day after asking here. I didn't remove the question as others may encounter the same problem. The nature of the problem was taking me to the wrong way. It does not just arise from copying streams but also when writing e.g. 4 bytes (bbbb) to overwrite file with 8 bytes (aaaaaaaa). it creates a file with first 4 new bytes and then 4 old bytes! (bbbbaaaa).

So the answer is in FileOutputStream(). have the size of bytes that is written to the file (input.channel.size()) or (output.cannel.position()) and truncate remaining bytes (output.channel.truncate(size)).

as the code in question, i changed to:

contentResolver.openOutputStream(fileUri)?.use {output->
    output as FileOutputStream
    FileInputStream(File(getExternalFilesDir(null),"Pictures/file1.jpg")).use{input->
        copyStream(input, output)
        // this new line removes bytes beyond the input file size
        output.channel.truncate(input.channel.size())
        // or
        // output.channel.truncate(output.channel.position())
    }
}

UPDATE 2019/09/15

Thanks to the first comment from @mjanssen you can also get the same result by putting the output.channel.truncate(0) just before copying the file, copyStream(input, output):

contentResolver.openOutputStream(fileUri)?.use {
    output-> output as FileOutputStream
    FileInputStream(File(getExternalFilesDir(null),"Pictures/file1.jpg")).use{input->
    output.channel.truncate(0)            
    copyStream(input, output)
    }
}

That's it

Answer 2

Having wasted several days trying to understand why I was getting corrupt data at the end of the file when I was overwriting it, I thankfully found this post. My testing showed the same results as the OP. When writing new contents that are shorter than the existing file contents, residual data is left at the end of the file - leading to a 'corrupt' file. My, incorrect, assumption was that writing to the file would completely remove/overwrite the existing contents.

Using the Android Developers example on 'Edit A Document' ...(https://developer.android.com/training/data-storage/shared/documents-files#edit) highlighted the problem, since using their code example only overwrites the first record. IE a 100 line text file (for example), is still a 100 line text file if you use their code example, only the first record is changed. Useful if you want to just modify specific records 'in situ', but not if you want to completely replace the file contents !

Here is a modified, java version, of the sample code that ensures that the output is 'emptied' first before being overwritten (employing the approach described by the OP above)

    private void overwriteDocument(Uri uri) {
        try {
            ParcelFileDescriptor pfd = getActivity().getContentResolver().
                    openFileDescriptor(uri, "w");
            FileOutputStream fileOutputStream =
                    new FileOutputStream(pfd.getFileDescriptor());


            // Use this code to ensure that the file is 'emptied' 
            FileChannel fChan=fileOutputStream.getChannel();
            fChan.truncate(0);


            fileOutputStream.write(("Overwritten at " + System.currentTimeMillis() +
                    "\n").getBytes());
            // Let the document provider know you're done by closing the stream.
            fileOutputStream.close();
            pfd.close();
        } catch (FileNotFoundException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }