Why did Cloudwatch stop logging Sagemaker?

Question

I have a Sagemaker instance running for a while now. I didn't change anything in between, but now I can't see new logs on Cloudwatch anymore. The old logs are still there, but no new ones since 2 days.

The Sagemaker instance is still running. It's just not logging anymore. And as the code didn't change and I don't have anything time-dependant in there, I'm pretty sure I hit a limit. But I don't know which one:

• The Log group has only one log stream
• The single log stream has a size of 175MB.

I found CloudWatch Logs Limits and CloudWatch Events Limits, but that didn't help me.

What could be the problem? How can I investigate it?

According to AWS docs this should not happen. The general AWS support did not help.

Answer 1

First, it doesn't sound like you're doing anything wrong. Logs should just show up in CloudWatch without you having to do anything, without size or time limits. If they start at all, then we know permissions were set up properly -- unless you modified IAM in the middle of the run. If the logs stop mid job, then either the actual job stopped outputting to stdout/stderr for some reason or this is an operational glitch with the service's log processing. Contacting AWS support (here, in the AWS forums, or through tech support) is the right way to deal with this -- giving somebody in AWS the account id and job name will enable them to look into exactly what happened.

Also, sorry this has gone unanswered for so long. Judging by the activity here, it seemed like a lot of people might have hit this problem. But I'm also guessing & hoping that the problem was a temporary internal service glitch that has been resolved. If anybody is still seeing this problem (after October 2018), please leave a comment so we know it still needs attention. Or better yet open a new question (not ideal from an SO perspective, but that's more likely to get somebody's attention at AWS).

Thanks for using Amazon SageMaker, and thanks for the feedback!

-An AWS employee