I've recently deployed my Django API backend to AWS EB to their Linux 2 system (exact platform name is Python 3.7 running on 64bit Amazon Linux 2
).
Almost everything is working as expected, but my application health status is Severe
and after hours of debugging I've no idea why.
The application's health check is being handled using the following endpoint (django-health-check
module).
url(r'^ht/', include('health_check.urls'))
100% of the requests have a status code of 200
but my overall health status is the following:
|--------------------|----------------|---------------------------------------------------|
| instance-id | status | cause |
|--------------------|----------------|---------------------------------------------------|
| Overall | Degraded | Impaired services on all instances. |
| i-0eb89f... | Severe | Following services are not running: release. |
|--------------------|----------------|---------------------------------------------------|
The strangest thing is the fact that the message Following services are not running: release.
is unique to the internet (seems like no one has had such problem before).
The other weird thing are the contents of my /var/log/healthd/daemon.log
file which are lines similar to
W, [2020-07-21T09:00:01.209091 #3467] WARN -- : log file "/var/log/nginx/healthd/application.log.2020-07-21-09" does not exist
where the time changes.
The last thing that may be relevant are the contents of my single file inside .ebextensions
directory:
option_settings:
"aws:elasticbeanstalk:application:environment":
DJANGO_SETTINGS_MODULE: "app.settings"
"PYTHONPATH": "/var/app/current:$PYTHONPATH"
"aws:elasticbeanstalk:container:python":
WSGIPath: app.wsgi:application
NumProcesses: 3
NumThreads: 20
aws:elasticbeanstalk:environment:proxy:staticfiles:
/static: static
/static_files: static_files
container_commands:
01_migrate:
command: "source /var/app/venv/staging-LQM1lest/bin/activate && python manage.py migrate --noinput"
leader_only: true
packages:
yum:
git: []
postgresql-devel: []
Does anyone have any idea how can this be resolved? The ultimate goal is to have the green OK health.
EDIT: In the end I switched to the Basic
health system and the problems suddenly went away. I am however still interested in solving the original problem as the Enhanced
health system provides some benefits
If an instance fails these status checks, it is marked unhealthy and is terminated while Amazon EC2 Auto Scaling launches a new replacement instance. You can attach one or more load balancer target groups, one or more Classic Load Balancers, or both to your Auto Scaling group.
An instance might fail the ELB health check because an application running on the instance has issues that cause the load balancer to consider the instance out of service.
The health agent monitors web server logs and system metrics and relays them to the Elastic Beanstalk service. Elastic Beanstalk analyzes these metrics and data from Elastic Load Balancing and Amazon EC2 Auto Scaling to provide an overall picture of an environment's health.
I believe that the problem you have maybe due to the ALLOWED_HOSTS settings located in your file settings.py.
EB sends an HTTP request to your application to see if its working but Django blocks any communication that is not from the specified hosts in the setting variable. But there is a problem here, EB sends the request to the private ip of the ec2 instance.
The easiest way to solve this is to allow all HOSTS like this inside your settings.py
file:
ALLOWED_HOSTS=['*']
This may lead to security issues but is the fastest way to do it. Now, to make it work dynamically, since ec2 instances can be spin-up at any time the private ip change from instance to instance.
To solve this you have to get the private IP at the beginning of the deployment process.
At the top of your settings.py
place the following functions:
import os
import requests
# Other imports ...
def is_ec2_linux():
"""Detect if we are running on an EC2 Linux Instance
See http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/identify_ec2_instances.html
"""
if os.path.isfile("/sys/hypervisor/uuid"):
with open("/sys/hypervisor/uuid") as f:
uuid = f.read()
return uuid.startswith("ec2")
return False
def get_token():
"""Set the autorization token to live for 6 hours (maximum)"""
headers = {
'X-aws-ec2-metadata-token-ttl-seconds': '21600',
}
response = requests.put('http://169.254.169.254/latest/api/token', headers=headers)
return response.text
def get_linux_ec2_private_ip():
"""Get the private IP Address of the machine if running on an EC2 linux server.
See https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html"""
if not is_ec2_linux():
return None
try:
token = get_token()
headers = {
'X-aws-ec2-metadata-token': f"{token}",
}
response = requests.get('http://169.254.169.254/latest/meta-data/local-ipv4', headers=headers)
return response.text
except:
return None
finally:
if response:
response.close()
# Other settings
The most important functions are get_token()
and get_linux_ec2_private_ip()
, the first one sets the access token and retrieves it for the second one to use it and get the current ec2 instance IP.
Once you have retrieved it, add it to your ALLOWED_HOSTS
ALLOWED_HOSTS = ['127.0.0.1', 'mywebsite.com']
private_ip = get_linux_ec2_private_ip()
if private_ip:
ALLOWED_HOSTS.append(private_ip)
After that just commit your changes and redeploy it with eb deploy
if you have set the EB CLI.
There is a package django-ebhealthcheck
that's designed to solve this problem by getting the local ip of your ec2 instance and adding it to ALLOWED_HOSTS
, it's very simple to use, you just have to add 'ebhealthcheck.apps.EBHealthCheckConfig'
to INSTALLED_APPS
Package github page – https://github.com/sjkingo/django-ebhealthcheck
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With