I am trying to get logs from ClouldTrail into ElasticSearch so that we can see what is going on in our AWS account better.
I have set up both Logstash and ElasticSearch on my machine (Ubuntu 14.04), and can push text from stdin
to ElasticSearch. However when I try to use the S3 input nothing is added to ElasticSearch.
Here is the conf file Im using, I have removed my amazon keys
input {
s3 {
bucket => 'ko-cloudtrail-log-bucket'
secret_access_key => ''
access_key_id => ''
delete => false
interval => '60'
region => 'eu-west-1'
type => 'CloudTrail'
codec => cloudtrail {}
}
}
output {
stdout {}
elasticsearch {
host => '127.0.0.1'
}
}
I have install the logstash-codec-cloudtrail codec but the documentation is pretty sparse.
I get no errors in my terminal even when running Logstash with -v and nothing is printed to stdout
. Is there something I am missing?
Here's my cloudtrail input. It works great with one minor issue- it duplicates records. As the prefix
indicates, I put the cloudtrail logs at s3://bucketname/cloudtrail, not the root.
The mutations are optional. The eventSource mutation is to make the logs a little more readable, and the ruby ingest_time gives me a date the record showed up in ELK- otherwise, it only has the time of the event. Finally, I drop a very common record that just adds noise to my system.
input {
s3 {
bucket => "bucketname"
delete => false
interval => 60 # seconds
prefix => "cloudtrail/"
type => "cloudtrail"
codec => "cloudtrail"
credentials => "/etc/logstash/s3_credentials.ini"
sincedb_path => "/opt/logstash_cloudtrail/sincedb"
}
}
filter {
if [type] == "cloudtrail" {
mutate {
gsub => [ "eventSource", "\.amazonaws\.com$", "" ]
add_field => {
"document_id" => "%{eventID}"
}
}
if ! [ingest_time] {
ruby {
code => "event['ingest_time'] = Time.now.utc.strftime '%FT%TZ'"
}
}
if [eventSource] == "elasticloadbalancing" and [eventName] == "describeInstanceHealth" and [userIdentity.userName] == "secret_username" {
drop {}
}
}
}
The credentials.ini format is explained on the s3 input page; it's just this:
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With