Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

AWS Fargate Task Debugging, what does "CannotPullContainerError ... invalid reference format" mean?

I have docker container stored on AWS ECR, using this container I have created Task . When I run Task over Fargate , the service moves from PROVISIONING > PENDING (red color) > STOPPED states .

There is no log either in Service Event , or CloudWatch . Absolutely no way to know what is the real problem .

Any idea which way to move, AWS document did not helped.

like image 560
RajDev Avatar asked Dec 16 '17 07:12

RajDev


People also ask

Can you ssh into fargate task?

This is a lot of work (and against security best practices) to simply exec into a container (running on an EC2 instance). Furthermore, ECS users deploying tasks on Fargate did not even have this option because with Fargate there are no EC2 instances you can ssh into.

What is the difference between ECS and fargate?

' ECS delivers more control over the infrastructure, but the trade-off is the added management that comes with it. Fargate is the better option for ease of use as it takes infrastructure management out of the equation allowing you to focus on just the tasks to be run.


4 Answers

I saw this error also. In my case the problem was that I had an invalid image value in my container definition. The image is generally a repository_url and image tag as per image instructions at https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html

like image 115
Marc Avatar answered Oct 07 '22 01:10

Marc


The AWS Fargate has a simple but scuttle info missing , this is what has lead you to your issue .Let me explain step by step.

First step is to create a Task definition , when you do that you can define

 Entry point ["/bin/ls"]
Command     [">","/list.txt","|","echo","/list.txt"]

This is equivalient to /bin/ls > /list.txt | echo /list.txt .

Please note the spaces in between has to be replaced by comma. What AWS Farget team forgot was that in the TaskDefinition > Add Container section in the wizard , this comma delimiter echo,helloworld is provided in the text box [![enter image description here][1]][1]. But in the Task form there is not such info . This is very confusing.

One last thing , AWS Fargate has very limited info for debugging. Here are my tips

  • You can get only possible error under Cluster>Task>{TaskName}>{container} . Under reason for error.
  • If the error is very small , I always got Error Code0 , no space between Code0. Nothing on Cloud watch as well .
  • If the error text is big (may be around 20 char or > ) you will get it both in the #1 and the CloudWatch log.

Hope it helps

like image 44
rajesk Avatar answered Oct 06 '22 23:10

rajesk


As Marc mentions, this is usually a wrong DockerHub/repo image name.

In my case it was because the image name was calculated by an environment variable, in my Docker Compose file:

services:
  web:
    image: scipilot/my-web-app:$MY_VERSION_TAG

And I'd missed setting the environment variable when using ecs-cli compose service up. So Fargate was probably trying to pull "scipilot/my-web-app:" from Docker Hub, hence an invalid format error.

like image 22
scipilot Avatar answered Oct 07 '22 01:10

scipilot


In my experience you need to add permissions in your ECR. In your repository there is a tab their called permissions. Add those permissions you think you need.

like image 27
Cliff Richard Anfone Avatar answered Oct 07 '22 00:10

Cliff Richard Anfone