Updating a file in Amazon S3 bucket

Question

I am trying to append a string to the end of a text file stored in S3. Currently I just read the contents of the file into a String, append my new text and resave the file back to S3. Is there a better way to do this. I am thinkinig when the file is >>> 10MB then reading the entire file would not be a good idea so how should I do this correctly?

Current code [code]

private void saveNoteToFile( String p_note ) throws IOException, ServletException    
{
    String str_infoFileName =  "myfile.json"; 

    String existingNotes = s3Helper.getfileContentFromS3( str_infoFileName  ); 
    existingNotes += p_note;
    writeStringToS3( str_infoFileName , existingNotes );        
}

public void writeStringToS3(String p_fileName, String p_data) throws IOException 
{
  ByteArrayInputStream byteArrayInputStream = new ByteArrayInputStream( p_data.getBytes());

  try {
      streamFileToS3bucket(  p_fileName, byteArrayInputStream, p_data.getBytes().length);
  } 
  catch (AmazonServiceException e)
  {
      e.printStackTrace();
  } catch (AmazonClientException e)
  {
      e.printStackTrace();
  }
}

public void streamFileToS3bucket( String p_fileName,  InputStream input, long size)
{
    //Create sub folders if there is any in the file name.
    p_fileName = p_fileName.replace("\\", "/");
    if( p_fileName.charAt(0) == '/')
    {
        p_fileName = p_fileName.substring(1, p_fileName.length());
    }
    String folder = getFolderName( p_fileName );
    if( folder.length() > 0)
    {
        if( !doesFolderExist(folder))
        {
            createFolder( folder );
        }
    }
    ObjectMetadata metadata =  new ObjectMetadata();
    metadata.setContentLength(size);
    AccessControlList acl = new AccessControlList();
    acl.grantPermission(GroupGrantee.AllUsers, Permission.Read);

    s3Client.putObject(new PutObjectRequest(bucket, p_fileName , input,metadata).withAccessControlList(acl));
}

[/code]

Answer 1

It's not possible to append to an existing file on AWS S3. When you upload an object it creates a new version if it already exists:

If you upload an object with a key name that already exists in the bucket, Amazon S3 creates another version of the object instead of replacing the existing object

Source: http://docs.aws.amazon.com/AmazonS3/latest/UG/ObjectOperations.html

The objects are immutable.

It's also mentioned in these AWS Forum threads:

https://forums.aws.amazon.com/message.jspa?messageID=179375 https://forums.aws.amazon.com/message.jspa?messageID=540395

Answer 2

It's not possible to append to an existing file on AWS S3.

You can delete existing file and upload new file with same name.

Configuration

    private string bucketName = "my-bucket-name-123";
    private static string awsAccessKey = "AKI............";
    private static string awsSecretKey = "+8Bo..................................";

    IAmazonS3 client = new AmazonS3Client(awsAccessKey, awsSecretKey, 
                                           RegionEndpoint.APSoutheast2);

    string awsFile = "my-folder/sub-folder/textFile.txt";

    string localFilePath = "my-folder/sub-folder/textFile.txt";

To Delete

    public void DeleteRefreshTokenFile()
    {
        try
        {
            var deleteFileRequest = new DeleteObjectRequest
            {
                BucketName = bucketName,
                Key = awsFile
            };
            DeleteObjectResponse fileDeleteResponse = client.DeleteObject(deleteFileRequest);
        }
        catch (Exception ex)
        {
            throw new Exception(ex.Message);
        }
    }

To Upload

        public void UploadRefreshTokenFile()
        {
            FileInfo file = new FileInfo(localFilePath);
            try
            {
                PutObjectRequest request = new PutObjectRequest()
                {
                    InputStream = file.OpenRead(),
                    BucketName = bucketName,
                    Key = awsFile
                };
                PutObjectResponse response = client.PutObject(request);
            }
            catch (Exception ex)
            {
                throw new Exception(ex.Message);
            }
        }