Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

The website doesn't load after deploying Django app in Google Cloud App Engine

So after deploying my Django App with Google Cloud App Engine, I cannot get a respone from the website, in other words the website keeps loading non stop.

The message I got after the deployment was done :

Updating service [default] (this may take several minutes)...done.
Setting traffic split for service [default]...done.
Stopping version [sam-website-358409/default/20220825t151307].
Sent request to stop version [sam-website-358409/default/20220825t151307]. This operation may take some time to complete. If you would like to verify that it succeeded, run:
    $ gcloud app versions describe -s default 20220825t151307
until it shows that the version has stopped.
Deployed service [default] to [https://sam-website-358409.ew.r.appspot.com]

You can stream logs from the command line by running:
    $ gcloud app logs tail -s default

To view your application in the web browser run:
    $ gcloud app browse

So, trying to debug the problem, I have run :

gcloud app logs tail -s default

I notice then lots of warnings that keeps showing repeatedly :

- This IS expected if you are initializing CamembertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
2022-08-25 16:33:36 default[20220825t175602]  - This IS NOT expected if you are initializing CamembertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
2022-08-25 16:33:39 default[20220825t175602]  /env/lib/python3.7/site-packages/sklearn/base.py:338: UserWarning: Trying to unpickle estimator LogisticRegression from version 0.24.2 when using version 1.0.2. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
2022-08-25 16:33:39 default[20220825t175602]  https://scikit-learn.org/stable/modules/model_persistence.html#security-maintainability-limitations
2022-08-25 16:33:39 default[20220825t175602]    UserWarning,
2022-08-25 16:33:39 default[20220825t175602]  /env/lib/python3.7/site-packages/spacy/util.py:865: UserWarning: [W095] Model 'fr_core_news_lg' (3.1.0) was trained with spaCy v3.1 and may not be 100% compatible with the current version (3.4.1). If you see errors or degraded performance, download a newer compatible model or retrain your custom model with the current spaCy version. For more details and available updates, run: python -m spacy validate
2022-08-25 16:33:39 default[20220825t175602]    warnings.warn(warn_msg)
2022-08-25 16:33:41 default[20220825t175602]  Some weights of the model checkpoint at camembert-base were not used when initializing CamembertModel: ['lm_head.bias', 'lm_head.dense.bias', 'lm_head.layer_norm.bias', 'lm_head.layer_norm.weight', 'lm_head.dense.weight', 'lm_head.decoder.weight']
2022-08-25 16:33:41 default[20220825t175602]  - This IS expected if you are initializing CamembertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
2022-08-25 16:33:41 default[20220825t175602]  - This IS NOT expected if you are initializing CamembertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
2022-08-25 16:33:44 default[20220825t175602]  [2022-08-25 16:33:44 +0000] [179] [INFO] Worker exiting (pid: 179)
2022-08-25 16:33:45 default[20220825t175602]  [2022-08-25 16:33:45 +0000] [191] [INFO] Booting worker with pid: 191
2022-08-25 16:33:46 default[20220825t175602]  2022-08-25 16:33:46.542163: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
2022-08-25 16:33:46 default[20220825t175602]  To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-08-25 16:33:48 default[20220825t175602]  2022-08-25 16:33:48.108067: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-08-25 16:33:48 default[20220825t175602]  2022-08-25 16:33:48.108101: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2022-08-25 16:33:50 default[20220825t175602]  [2022-08-25 16:33:50 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:132)
2022-08-25 16:33:50 default[20220825t175602]  [2022-08-25 16:33:50 +0000] [132] [INFO] Worker exiting (pid: 132)
2022-08-25 16:33:51 default[20220825t175602]  [2022-08-25 16:33:51 +0000] [1] [WARNING] Worker with pid 132 was terminated due to signal 9
2022-08-25 16:33:51 default[20220825t175602]  [2022-08-25 16:33:51 +0000] [144] [INFO] Booting worker with pid: 144
2022-08-25 16:33:52 default[20220825t175602]  2022-08-25 16:33:52.437860: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2022-08-25 16:33:52 default[20220825t175602]  2022-08-25 16:33:52.437899: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
2022-08-25 16:33:52 default[20220825t175602]  2022-08-25 16:33:52.437933: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (0083c9528b79): /proc/driver/nvidia/version does not exist
2022-08-25 16:33:53 default[20220825t175602]  /env/lib/python3.7/site-packages/spacy/util.py:865: UserWarning: [W095] Model 'fr_core_news_lg' (3.1.0) was trained with spaCy v3.1 and may not be 100% compatible with the current version (3.4.1). If you see errors or degraded performance, download a newer compatible model or retrain your custom model with the current spaCy version. For more details and available updates, run: python -m spacy validate
2022-08-25 16:33:53 default[20220825t175602]    warnings.warn(warn_msg)
2022-08-25 16:33:53 default[20220825t175602]  2022-08-25 16:33:53.266169: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2022-08-25 16:33:53 default[20220825t175602]  2022-08-25 16:33:53.266266: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2022-08-25 16:33:55 default[20220825t175602]  2022-08-25 16:33:55.698315: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2022-08-25 16:33:55 default[20220825t175602]  2022-08-25 16:33:55.698350: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
2022-08-25 16:33:55 default[20220825t175602]  2022-08-25 16:33:55.698380: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (2e597b32f6d7): /proc/driver/nvidia/version does not exist
2022-08-25 16:33:56 default[20220825t175602]  /env/lib/python3.7/site-packages/spacy/util.py:865: UserWarning: [W095] Model 'fr_core_news_lg' (3.1.0) was trained with spaCy v3.1 and may not be 100% compatible with the current version (3.4.1). If you see errors or degraded performance, download a newer compatible model or retrain your custom model with the current spaCy version. For more details and available updates, run: python -m spacy validate
2022-08-25 16:33:56 default[20220825t175602]    warnings.warn(warn_msg)
2022-08-25 16:34:10 default[20220825t175602]  Some weights of the model checkpoint at camembert-base were not used when initializing CamembertModel: ['lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.bias', 'lm_head.layer_norm.weight', 'lm_head.bias', 'lm_head.dense.weight']

The app is running fine localy and also in the VM instance created by the Google App Engine.

Here is the content of the app.yaml file:

runtime: python
env: flex
entrypoint: gunicorn -b :$PORT myteam.wsgi

beta_settings:
    cloud_sql_instances: sam-website-358409:europe-west9:sam-instance

runtime_config:
  python_version: 3.7

threadsafe: true
manual_scaling:
  instances: 2
network: {}

resources:
  cpu: 2
  memory_gb: 10
  disk_size_gb: 50
liveness_check:
  initial_delay_sec: 300
  check_interval_sec: 30
  timeout_sec: 4
  failure_threshold: 4
  success_threshold: 2
readiness_check:
  check_interval_sec: 5
  timeout_sec: 4
  failure_threshold: 2
  success_threshold: 2
  app_start_timeout_sec: 300
like image 495
Youssef Benhachem Avatar asked Oct 15 '25 23:10

Youssef Benhachem


1 Answers

I see that you are using BERT models that are large. You are probably running out of memory. I had a similar problem, and you can see the solution here.

Make sure to not load your BERT model during deployment. Instead, load it after deployment and before processing your first incoming request.

like image 90
gaefan Avatar answered Oct 18 '25 14:10

gaefan



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!