Logging to CloudWatch from EC2 instances

Question

My EC2 servers are currently hosting a website that logs each registered user's activity under their own separate log file on the local EC2 instance, say username.log. I'm trying to figure out a way to push log events for these to CloudWatch using the PHP SDK without slowing the application down, AND while still being able to maintain a separate log file for each registered member of my website.

I can't for the life of me figure this out:

OPTION 1: How can I log to CloudWatch asynchronously using the CloudWatch SDK? My PHP application is behaving VERY sluggishly, since each log line takes roughly 100ms to push directly to CloudWatch. Code sample is below.

OPTION 2: Alternatively, how could I configure an installed CloudWatch Agent on EC2 to simply OBSERVE all of my log files, which would basically upload them asynchronously to CloudWatch for me in a separate process? The CloudWatch EC2 Logging Agent requires a static "configuration file" (AWS documentation) on your server which, to my knowledge, needs to lists out all of your log files ("log streams") in advance, which I won't be able to predict at the time of server startup. Is there any way around this (ie, simply observe ALL log files in a directory)? Config file sample is below.

All ideas are welcome here, but I don't want my solution to simply be "throw all your logs into a single file, so that your log names are always predictable".

Thanks in advance!!!

---

OPTION 1: Logging via SDK (takes ~100ms / logEvent):

// Configuration to use for the CloudWatch client
$sharedConfig = [
    'region'  => 'us-east-1',
    'version' => 'latest',
    'http'    => [
        'verify' => false
    ]
];

// Create a CloudWatch client
$cwClient = new Aws\CloudWatchLogs\CloudWatchLogsClient($sharedConfig);

// DESCRIBE ANY EXISTING LOG STREAMS / FILES
$create_new_stream = true;
$next_sequence_id = "0";
$result = $cwClient->describeLogStreams([
        'Descending' => true,
        'logGroupName' => 'user_logs',
        'LogStreamNamePrefix' => $stream,
]);
// Iterate through the results, looking for a stream that already exists with the intended name
// This is so that we can get the next sequence id ('uploadSequenceToken'), so we can add a line to an existing log file
foreach ($result->get("logStreams") as $stream_temp) {
    if ($stream_temp['logStreamName'] == $stream) {
        $create_new_stream = false;
        if (array_key_exists('uploadSequenceToken', $stream_temp)) {
            $next_sequence_id = $stream_temp['uploadSequenceToken'];
        }
        break;
    }
}   

// CREATE A NEW LOG STREAM / FILE IF NECESSARY
if ($create_new_stream) { 
    $result = $cwClient->createLogStream([
        'logGroupName' => 'user_logs',
        'logStreamName' => $stream,
    ]);
}

// PUSH A LINE TO THE LOG *** This step ALONE takes 70-100ms!!! ***

$result = $cwClient->putLogEvents([
    'logGroupName' => 'user_logs',
    'logStreamName' => $stream,
    'logEvents' => [
        [
            'timestamp' => round(microtime(true) * 1000),
            'message' => $msg,
        ],
    ],
    'sequenceToken' => $next_sequence_id
]);

OPTION 2: Logging via CloudWatch Installed Agent (note that config file below only allows hardcoded, predermined log names as far as I know):

[general]
state_file = /var/awslogs/state/agent-state  

[applog]
file = /var/www/html/logs/applog.log
log_group_name = PP
log_stream_name = applog.log
datetime_format = %Y-%m-%d %H:%M:%S

Answer 1

Looks like we have some good news now... not sure if it's too late!

CloudWatch Log Configuration

So to answer the doubt,

Is there any way around this (ie, simply observe ALL log files in a directory)?

yes, we can mention log files and file paths using wild cards, which can help you in having some flexibility in configuring from where the logs are fetched and pushed to the log streams.