Menu
I'm using a google model (binary file: around 3GB) in my docker file and then use Jenkins to build and deploy it on the production server. The rest of the code is pulled from the bitbucket repo.
An example line from the docker file where I download and unzip the file. It only happens once as this command will be cached.
FROM python:2.7.13-onbuild
RUN mkdir -p /usr/src/app
WORKDIR /usr/src/app
ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install --assume-yes apt-utils
RUN apt-get update && apt-get install -y curl
RUN apt-get update && apt-get install -y unzip
RUN curl -o - https://s3.amazonaws.com/dl4j-distribution/GoogleNews-vectors-negative300.bin.gz \
| gunzip > /usr/src/app/GoogleNews-vectors-negative300.bin
Everything works fine when I build and run the docker on my local machine. However, when I do make patch-version to push these changes to the production server through Jenkins, my build process fails at the end. The setup, build and test phases works fine. However, the post-build phase fails. (The build process pushes changes to the repo, and according to the logs, all commands in docker file also runs fine.) Something happens after that and I get the following error when I looked at the logs.
18:49:27 654f45ecb7e3: Layer already exists
18:49:27 2c40c66f7667: Layer already exists
18:49:27 97108d083e01: Pushed
18:49:31 35a4b123c0a3: Pushed
18:50:10 1e730b4fb0a6: Pushed
18:53:46 error parsing HTTP 413 response body: invalid character '<'
looking for beginning of value: "<html>\r\n<head><title>413 Request
`Entity Too Large</title></head>\r\n<body
bgcolor=\"white\">\r\n<center>`<h1>413 Request
Entity Too Large</h1></center>\r\n<hr>
center>nginx/1.10.1</center>\r\n</body>\r\n</html>\r\n"
Could it be that file is too large?
Before the addition of this file, everything with docker and Jenkins was working fine too.
I am wondering if there are any limitations in docker/Jenkins in handling a large file like this? or I am breaking something the way I am approaching it.
Update:
Increasing the client_max_body_size solved this specific error. However, I am getting another error at ssh -o StrictHostKeyChecking=no root@ipaddress "cd /root/ourapi &&docker-compose pull api &&docker-compose -p somefolder up -d"
The docker-compose pull fails here with an unexpected eof. It tries to download the image (1.6 GB) but cancel it after almost getting close to that size and then retries it which ends wit an eof error.
Which brings me to old question if large files needs to be handled different in this situation?
Update 2: The issue has been resolved. I needed to increase the client_max_body_size to 4 GB and also need to increase the time-out parameter for pulling the repository from our own repository server. Adjusting these two parameters has led to resolving the problem.
286
The problem was mainly caused due to the following reasons:
This answer is mainly in the context of this specific problem. However, the question regarding how to handle such files in a better way still remains open. Moreover, it is not clear if increasing client_max_body_size to 4 GB is a good idea in general.
Relevant docs for client_max_body_size: http://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size
194
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
