Menu
I am trying to use Logs Insights with data containing JSON in one of the fields, and to parse the JSON fields
My data looks like the following when I put it in insights with the starter code
fields @timestamp, @message | sort @timestamp desc | limit 25 How can I easily extract the path variable in my nested JSON to perform aggregations on it ? By looking at some documentation, I thought @message.path would work but it does not seem so. Has anyone successfully interpreted JSON logs in Insights
EDIT : Sample of what my data looks like
# @timestamp @message 1 2018-12-19 23:42:52.000 I, [2018-12-19T23:42:52.629855 #23447] INFO -- : [2ce588f1-c27d-4a55-ac05-62a75b39e762] {"method":"GET","path":"/api/v1/professionals/ID","format":"json","controller":"API::V1::Public::ProfessionalsController","action":"show","status":200,"duration":285.27,"view":222.36,"time":"2018-12-19T23:42:52.344+00:00","params":{"include":"user,tags,promotions,company_sector,similar_professionals.tags,similar_professionals.user","format":"json","compress":false,"id":"ID"},"@timestamp":"2018-12-19T23:42:52.629Z","@version":"1","message":"[200] GET /api/v1/professionals/ID (API::V1::Public::ProfessionalsController#show)"} @logStream i-05d1d61ab853517a0 @message I, [2018-12-19T23:42:52.629855 #23447] INFO -- : [2ce588f1-c27d-4a55-ac05-62a75b39e762] {"method":"GET","path":"/api/v1/professionals/ID","format":"json","controller":"API::V1::Public::ProfessionalsController","action":"show","status":200,"duration":285.27,"view":222.36,"time":"2018-12-19T23:42:52.344+00:00","params":{"include":"xxx","format":"json","compress":false,"id":"ID"},"@timestamp":"2018-12-19T23:42:52.629Z","@version":"1","message":"[200] GET /api/v1/professionals/ID (API::V1::Public::ProfessionalsController#show)"} @timestamp 1545262972000 2 2018-12-19 23:42:16.000 I, [2018-12-19T23:42:16.723472 #851] INFO -- : [ea712503-eb86-4a6e-ab38-ddbcd6c2b4d0] {"method":"GET","path":"/api/v1/heartbeats/new","format":"json","controller":"API::V1::Public::HeartbeatsController","action":"new","status":201,"duration":9.97,"view":3.2,"time":"2018-12-19T23:42:16.712+00:00","params":{"format":"json","compress":false},"@timestamp":"2018-12-19T23:42:16.722Z","@version":"1","message":"[201] GET /api/v1/heartbeats/new (API::V1::Public::HeartbeatsController#new)"} 
679
We are happy to announce support for monitoring JSON-formatted logs with CloudWatch Logs. This capability enables you to create graphs and receive notifications when your JSON-formatted log events contain terms or match conditions that you choose.
Use the sort command to display log events in ascending ( asc ) or descending ( desc ) order. Use the limit command to specify the number of log events that you want your query to return. Use the parse command to extract data from a log field and create an ephemeral field that you can process in your query.
CloudWatch Logs Insights enables you to interactively search and analyze your log data in Amazon CloudWatch Logs. You can perform queries to help you more efficiently and effectively respond to operational issues.
CloudWatch Logs Insights automatically discovers fields in logs from AWS services such as Amazon Route 53, AWS Lambda, AWS CloudTrail, and Amazon VPC, and any application or custom log that emits log events as JSON. You can use CloudWatch Logs Insights to search log data that was sent to CloudWatch Logs on November 5, 2018 or later.
By sending in JSON-formatted logs we are making AWS do all the heavy lifting. Insights will automatically parse the content and create fields based on the JSON message. Insights can extract a maximum of 1000 log event fields from a JSON log.
To take advantage of the discoverability of JSON fields that CloudWatch Log Insights provides, you must change the default configuration of your Apache HTTP Server to format the logs in JSON format. The log agent then publishes the JSON format log file to CloudWatch Logs.
Insights can extract a maximum of 1000 log event fields from a JSON log. It will also allow access to nested JSON fields using the dot notation and flatten arrays into a list of field names and values. Below you can see 20 fields have automatically been extracted by CloudWatch allowing us to search, filter and produce metrics based on that data.
CloudWatch Insights Logs automatically discovers fields for the following log types:
Lambda logs
CloudWatch Logs Insights automatically discovers log fields in Lambda logs, but only for the first embedded JSON fragment in each log event (Note: emphasis mine). If a Lambda log event contains multiple JSON fragments, you can parse and extract the log fields by using the parse command. For more information, see Fields in JSON Logs.
CloudTrail logs
See fields in JSON logs.
Source: Supported Logs and Discovered Fields
If @message is I, [2018-12-11T13:20:27] INFO -- : {"method":"GET"}
Then you can select and filter the fields like so:
fields @timestamp, @message, method | filter method = "GET" | sort @timestamp desc It works with nested fields too, i.e. params.format = "json" or results.0.firstName = "Paul".
79
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
