Pyspark Save dataframe to S3

Question

I want to save dataframe to s3 but when I save the file to s3 , it creates empty file with ${folder_name}, in which I want to save the file.

Syntax to save the dataframe :-

f.write.parquet("s3n://bucket-name/shri/test")

It saves the file in test folder but it creates $test under shri .

Is there a way I can save it without creating that extra folder?

Answer 1

I was able to do it by using below code.

df.write.parquet("s3a://bucket-name/shri/test.parquet",mode="overwrite")