S3 Select retrieve headers in the CSV

Question

I am trying to fetch a subset of records from a CSV stored in an S# bucket using the following code:

s3 = boto3.client('s3')
bucket = bucket
file_name = file

sql_stmt = """SELECT S.* FROM s3object S LIMIT 10"""


req = s3.select_object_content(
    Bucket=bucket,
    Key=file,
    ExpressionType='SQL',
    Expression=sql_stmt,
    InputSerialization = {'CSV': {'FileHeaderInfo': 'USE'}},
    OutputSerialization = {'CSV': {}},
)

records = []
for event in req['Payload']:
    if 'Records' in event:
        records.append(event['Records']['Payload'])
    elif 'Stats' in event:
        stats = event['Stats']['Details']


file_str = ''.join(r.decode('utf-8') for r in records)

select_df = pd.read_csv(StringIO(file_str))
df = pd.DataFrame(select_df)
print(df)

This successfully yields the records but misses out on headers.

I read here S3 Select CSV Headers that S3 Select does not yield headers at all. So, is it possible to retrieve the headers of a CSV file in S3 in any other way?

Answer 1

Change InputSerialization={'CSV': {"FileHeaderInfo": "Use"}},

TO InputSerialization={'CSV': {"FileHeaderInfo": "NONE"}},

Then, it will print full content, including the header.

Explanation:

FileHeaderInfo accepts one of "NONE" OR "USE" OR "IGNORE".

Use NONE option rather then USE, it will then print header as well, as NONE tells that you need header as well for processing.

Here is reference. https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.select_object_content

I hope it helps.