Parse cloudwatch logs using filter patterns

Question

I have this line of lambda function log in cloudwatch that I receive by mail :

 /aws/lambda/sns-function | 2017/01/10/[$LATEST]425d9138c8d54ab57l0766ba74fdfd4p | 2017-01-10T00:04:30.734Z | 2017-01-10 00:04:30,734 :: ERROR :: error creating /tmp/tmpkRWp3S_20170110/file20170115.tar.gz: Command `['/bin/tar', '--create', '-z', '--file', u'/tmp/tmpkRWp3S_20170110/file20170115.tar.gz', '--', './']' returned non-zero exit status 1

As explained in this doc I want to put filter patterns to get only the important data. For me, I want to get the date only once because in the line above, I have twice this information: 2017-01-10T00:04:30.734Z I tried to use a pattern like this :

[...,timestamp,level,message=*ERROR*,...]

but I got this error:

2017-01-17 10:45:58,091 :: ERROR :: logGroup: '/aws/lambda/sns-function' - logStream: 'None'
2017-01-17 10:45:58,091 :: ERROR :: An error occurred (InvalidParameterException) when calling the FilterLogEvents operation: Duplicate field '...'

How can I parse the log to get the date once ?

Answer 1

Metric filters help you search for and match terms, phrases, or values in your log events. They do not remove values from the log event (the timestamp in your case). You could modify your script to exclude the timestamp from the output (since it is already included).

Also, you're using a metric filter for space-delimited log events. Your delimiter seems to be ::, which wouldn't work in this case. The metric filter will interpret this as a single field. If you want to use this metric filter, you can enclose each field in square brackets [] or two double quotes "".

For example, you can use this pattern [timestamp, result=ERROR, message, exit_status=*1*] for the following log event:

[2017-01-10 00:04:30,734] [ERROR] "error creating /tmp/tmpkRWp3S_20170110/file20170115.tar.gz: Command `['/bin/tar', '--create', '-z', '--file', u'/tmp/tmpkRWp3S_20170110/file20170115.tar.gz', '--', './']' returned non-zero" "exit status 1"

NOTE: The reason for the error is that ellipsis … should occur only once in the pattern.