Is there a way to generate the AWS Console URLs for CloudWatch Log Group filters?

Question

I would like to send my users directly to a specific log group and filter but I need to be able to generate the proper URL format. For example, this URL

https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/
%252Fmy%252Flog%252Fgroup%252Fgoes%252Fhere/log-events/$3FfilterPattern$3D$255Bincoming_ip$252C$2Buser_name$252C$2Buser_ip$2B$252C$2Btimestamp$252C$2Brequest$2B$2521$253D$2B$2522GET$2B$252Fhealth_checks$252Fall$2B*$2522$252C$2Bstatus_code$2B$253D$2B5*$2B$257C$257C$2Bstatus_code$2B$253D$2B429$252C$2Bbytes$252C$2Burl$252C$2Buser_agent$255D$26start$3D-172800000

will take you to a log group named /my/log/group/goes/here and filter messages with this pattern for the past 2 days:

[incoming_ip, user_name, user_ip , timestamp, request != "GET /health_checks/all *", status_code = 5* || status_code = 429, bytes, url, user_agent]

I can decode part of the URL but I don't know what some of the other characters should be (see below), but this doesn't really look like any standard HTML encoding to me. Does anyone know a encoder/decoder for this URL format?

%252F == /
$252C == ,
$255B == [
$255D == ]
$253D == =
$2521 == !
$2522 == "
$252F == _
$257C == |

$2B == +
$26 == &
$3D == =
$3F == ?

Answer 1

I had to do a similar thing to generate a back link to the logs for a lambda and did the following hackish thing to create the link:

const link = `https://${process.env.AWS_REGION}.console.aws.amazon.com/cloudwatch/home?region=${process.env.AWS_REGION}#logsV2:log-groups/log-group/${process.env.AWS_LAMBDA_LOG_GROUP_NAME.replace(/\//g, '$252F')}/log-events/${process.env.AWS_LAMBDA_LOG_STREAM_NAME.replace('$', '$2524').replace('[', '$255B').replace(']', '$255D').replace(/\//g, '$252F')}`

Answer 2

A colleague of mine figured out that the encoding is nothing special. It is the standard URI percent encoding but applied twice (2x). In javascript you can use the encodeURIComponent function to test this out:

let inp = 'https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/'

console.log(encodeURIComponent(inp))
console.log(encodeURIComponent(encodeURIComponent(inp)))

This piece of javascript produces the expected output on the second encoding stage:

https%3A%2F%2Fconsole.aws.amazon.com%2Fcloudwatch%2Fhome%3Fregion%3Dus-east-1%23logsV2%3Alog-groups%2Flog-group%2F
https%253A%252F%252Fconsole.aws.amazon.com%252Fcloudwatch%252Fhome%253Fregion%253Dus-east-1%2523logsV2%253Alog-groups%252Flog-group%252F

Caution

At least some bits use the double encoding, not the whole link though. Otherwise all special characters would occupy 4 characters after double encoding, but some still occupy only 2 characters. Hope this helps anyway ;)

Answer 3

My complete Javascript solution based on @isaias-b answer, which also adds a timestamp filter on the logs:

const logBaseUrl = 'https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group';
const encode = text => encodeURIComponent(text).replace(/%/g, '$');
const awsEncode = text => encodeURIComponent(encodeURIComponent(text)).replace(/%/g, '$');
const encodeTimestamp = timestamp => encode('?start=') + awsEncode(new Date(timestamp).toJSON());
const awsLambdaLogBaseUrl = `${logBaseUrl}/${awsEncode('/aws/lambda/')}`;
const logStreamUrl = (logGroup, logStream, timestamp) =>
  `${awsLambdaLogBaseUrl}${logGroup}/log-events/${awsEncode(logStream)}${timestamp ? encodeTimestamp(timestamp) : ''}`;

Answer 4

First of all I'd like to thank other guys for the clues. Further goes the complete explanation how Log Insights links are constructed.

Overall it's just weirdly encoded conjunction of an object structure that works like that:

• Part after ?queryDetail= is object representation and {} are represented by ~()

• Object is walked down to primitive values and the latter are transformed as following:

  • encodeURIComponent(value) so that all special characters are transformed to %xx
  • replace(/%/g, "*") so that this encoding is not affected by top level ones
  • if value type is string - it is prefixed with unmatched single quote

  To illustrate:

  "Hello world" -> "Hello%20world" -> "Hello*20world" -> "'Hello*20world"
  

• Arrays of transformed primitives are joined using ~ and as well put inside ~() construct

Then, after primitives transformation is done - object is joined using "~".

After that string is escape()d (note that not encodeURIComponent() is called as it doesn't transform ~ in JS).

After that ?queryDetail= is added.

And finally this string us encodeURIComponent()ed and as a cherry on top - % is replaced with $.

Let's see how it works in practice. Say these are our query parameters:

const expression = `fields @timestamp, @message
    | filter @message not like 'example'
    | sort @timestamp asc
    | limit 100`;

const logGroups = ["/application/sample1", "/application/sample2"];

const queryParameters = {
  end: 0,
  start: -3600,
  timeType: "RELATIVE",
  unit: "seconds",
  editorString: expression,
  isLiveTrail: false,
  source: logGroups,
};

Firstly primitives are transformed:

const expression = "'fields*20*40timestamp*2C*20*40message*0A*20*20*20*20*7C*20filter*20*40message*20not*20like*20'example'*0A*20*20*20*20*7C*20sort*20*40timestamp*20asc*0A*20*20*20*20*7C*20limit*20100";

const logGroups = ["'*2Fapplication*2Fsample1", "'*2Fapplication*2Fsample2"];

const queryParameters = {
  end: 0,
  start: -3600,
  timeType: "'RELATIVE",
  unit: "'seconds",
  editorString: expression,
  isLiveTrail: false,
  source: logGroups,
};

Then, object is joined using ~ so we have object representation string:

const objectString = "~(end~0~start~-3600~timeType~'RELATIVE~unit~'seconds~editorString~'fields*20*40timestamp*2C*20*40message*0A*20*20*20*20*7C*20filter*20*40message*20not*20like*20'example'*0A*20*20*20*20*7C*20sort*20*40timestamp*20asc*0A*20*20*20*20*7C*20limit*20100~isLiveTrail~false~source~(~'*2Fapplication*2Fsample1~'*2Fapplication*2Fsample2))"

Now we escape() it:

const escapedObject = "%7E%28end%7E0%7Estart%7E-3600%7EtimeType%7E%27RELATIVE%7Eunit%7E%27seconds%7EeditorString%7E%27fields*20*40timestamp*2C*20*40message*0A*20*20*20*20*7C*20filter*20*40message*20not*20like*20%27example%27*0A*20*20*20*20*7C*20sort*20*40timestamp*20asc*0A*20*20*20*20*7C*20limit*20100%7EisLiveTrail%7Efalse%7Esource%7E%28%7E%27*2Fapplication*2Fsample1%7E%27*2Fapplication*2Fsample2%29%29"

Now we append ?queryDetail= prefix:

const withQueryDetail = "?queryDetail=%7E%28end%7E0%7Estart%7E-3600%7EtimeType%7E%27RELATIVE%7Eunit%7E%27seconds%7EeditorString%7E%27fields*20*40timestamp*2C*20*40message*0A*20*20*20*20*7C*20filter*20*40message*20not*20like*20%27example%27*0A*20*20*20*20*7C*20sort*20*40timestamp*20asc*0A*20*20*20*20*7C*20limit*20100%7EisLiveTrail%7Efalse%7Esource%7E%28%7E%27*2Fapplication*2Fsample1%7E%27*2Fapplication*2Fsample2%29%29"

Finally we URLencode it and replace % with $ and vois la:

const result = "$3FqueryDetail$3D$257E$2528end$257E0$257Estart$257E-3600$257EtimeType$257E$2527RELATIVE$257Eunit$257E$2527seconds$257EeditorString$257E$2527fields*20*40timestamp*2C*20*40message*0A*20*20*20*20*7C*20filter*20*40message*20not*20like*20$2527example$2527*0A*20*20*20*20*7C*20sort*20*40timestamp*20asc*0A*20*20*20*20*7C*20limit*20100$257EisLiveTrail$257Efalse$257Esource$257E$2528$257E$2527*2Fapplication*2Fsample1$257E$2527*2Fapplication*2Fsample2$2529$2529"

Of course reverse operation can be performed as well.

That's all folks. Have fun, take care and try to avoid doing such a weird stuff yourselves. :)

Answer 5

I have created a bit of Ruby code that seems to satisfy the CloudWatch URL parser. I'm not sure why you have to double escape some things and then replace % with $ in others. I'm guessing there is some reason behind it but I couldn't figure out a nice way to do it, so I'm just brute forcing it. If you have something better, or know why they do this, please add a comment.

NOTE: The filter I tested with is kinda basic and I'm not sure what might need to change if you get really fancy with it.

# Basic URL that is the same across all requests
url = 'https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/'

# CloudWatch log group
log_group = '/aws/my/log/group'

# Either specify the instance you want to search or leave it out to search all instances
instance = '/log-events/i-xxxxxxxxxxxx'
 OR
instance = '/log-events'

# The filter to apply.
filter = '[incoming_ip, user_name, user_ip , timestamp, request, status_code = 5*, bytes, url, user_agent]'

# Start time.  There might be an End time as well but my queries haven't used 
# that yet so I'm not sure how it's formatted.  It should be pretty similar
# though.
hours = 48
start = "&start=-#{hours*60*60*1000}"

# This will get you the final URL
final = url + CGI.escape(CGI.escape(log_group)) + instance + '$3FfilterPattern$3D' + CGI.escape(CGI.escape(filter)).gsub('%','$') + CGI.escape(start).gsub('%','$')

Answer 6

A bit late but here is a python implementation

def get_cloud_watch_search_url(search, log_group, log_stream, region=None,):
    """Return a properly formatted url string for search cloud watch logs

    search = "{$.message: "You are amazing"}
    log_group = Is the group of message you want to search
    log_stream = The stream of logs to search
    """

    url = f'https://{region}.console.aws.amazon.com/cloudwatch/home?region={region}'

    def aws_encode(value):
        """The heart of this is that AWS likes to quote things twice with some substitution"""
        value = urllib.parse.quote_plus(value)
        value = re.sub(r"\+", " ", value)
        return re.sub(r"%", "$", urllib.parse.quote_plus(value))

    bookmark = '#logsV2:log-groups'
    bookmark += '/log-group/' + aws_encode(log_group)
    bookmark += "/log-events/" + log_stream
    bookmark += re.sub(r"%", "$", urllib.parse.quote("?filterPattern="))
    bookmark += aws_encode(search)

    return url + bookmark

This then allows you to quickly verify it.

>>> real = 'https://us-west-2.console.aws.amazon.com/cloudwatch/home?region=us-west-2#logsV2:log-groups/log-group/$252Fapp$252Fdjango/log-events/production$3FfilterPattern$3D$257B$2524.msg$253D$2522$2525s$2525s+messages+to+$2525s+pk$253D$2525d...$2522$257D'
>>> constructed = get_cloud_watch_search_url(None, search='{$.msg="%s%s messages to %s pk=%d..."}', log_group='/app/django', log_stream='production', region='us-west-2')
>>> real == constructed
True