AWS Glue Job logs still going to output and error logs

Question

I have configured a AWS Glue Job in which I have enabled continuous-logging and then defined the continuous-logging in the job-parameters.

Parameters:

1. --enable-continuous-cloudwatch-log - true
2. --enable-continuous-log-filter - false
3. --continuous-log-logStreamPrefix - test
4. --continuous-log-logGroupName or --continuous-log-logGroup - custom_log_group

But still my Glue Job log group points to "/aws-glue/jobs" and the links under Cloudwatch logs

1. 'All Logs' opens default log group - /aws-glue/jobs/logs-v2. Though the l,ink opens wrong location but its actually writing to my custom_log_group. But it only contains glue context logs. Not my java library logs which is being referenced in glue job.
2. 'Output Logs' goes to - /aws-glue/jobs/output
3. 'Error Logs' goes to - /aws-glue/jobs/error - This shows all my java library logs.

I want all these logs to go my custom_log_group. What am I missing here.

Answer 1

I also spend a lot of time on this and realized Glue 5.0 does not support custom log group. When I revert 5.0 to 4.0, I verified logs appeared in custom log group.

https://docs.aws.amazon.com/glue/latest/dg/monitor-continuous-logging-enable.html

Continuous logging is only available in AWS Glue 4.0 and earlier.