GitHub API returns wrong file content

Question

I am trying to create file using GitHub API. File should contain a single byte 0xDD. I am using following command:

http PUT https://api.github.com/repos/koluch/test/contents/data.txt "message"="New file" "content"="3Q==" Authorization:"token <OAUTH_TOKEN>"

3Q== is a Base64 representation of 0xDD byte. This request returns 200; however, when I am trying to retrieve file using GET request...

http GET https://api.github.com/repos/koluch/test/contents/data.txt 

... it returns following JSON:

{
    "_links": {
        "git": "https://api.github.com/repos/koluch/test/git/blobs/4e3bc519eef7d5a35fff67687eaee65616832e45",
        "html": "https://github.com/koluch/test/blob/master/data.txt",
        "self": "https://api.github.com/repos/koluch/test/contents/data.txt?ref=master"
    },
    "content": "77+9\n",
    "download_url": "https://raw.githubusercontent.com/koluch/test/master/data.txt",
    "encoding": "base64",
    "git_url": "https://api.github.com/repos/koluch/test/git/blobs/4e3bc519eef7d5a35fff67687eaee65616832e45",
    "html_url": "https://github.com/koluch/test/blob/master/data.txt",
    "name": "data.txt",
    "path": "data.txt",
    "sha": "4e3bc519eef7d5a35fff67687eaee65616832e45",
    "size": 1,
    "type": "file",
    "url": "https://api.github.com/repos/koluch/test/contents/data.txt?ref=master"
}

Field content contains 77+9\n value, which is not my 0xDD byte. When I am using download url, everything is fine.

Does anybody know what's going on?

Answer 1

I'm still in the loop with github support. so far no progress. but I think I know the problem. it looks like github tries to guess the filetype of a file and if it thinks it's a text file, uses the detected encoding to create the buffer before encoding it as base64.

this is at least what I see form the response headers of the raw request. My workaround is to check the filesize, and if it doesn't match, fetch the blob again using the raw api:

      const res = await octokit.repos.getContents(params);
      let data = Buffer.from(res.data.content, 'base64');
      if (data.length !== res.data.size) {
        log.debug(`base64 decoded content length doesn't match: ${data.length} != ${res.data.size}`);
        log.debug(`fetching content from ${res.data.download_url}`);
        const result = await rp.get({
          url: res.data.download_url,
          resolveWithFullResponse: true,
          encoding: null,
        });
        data = result.body;
      }