Prometheus (in Docker container) Cannot Scrape Target on Host

Question

Prometheus running inside a docker container (version 18.09.2, build 6247962, docker-compose.xml below) and the scrape target is on localhost:8000 which is created by a Python 3 script.

Error obtained for the failed scrape target (localhost:9090/targets) is

Get http://127.0.0.1:8000/metrics: dial tcp 127.0.0.1:8000: getsockopt: connection refused

Question: Why is Prometheus in the docker container unable to scrape the target which is running on the host computer (Mac OS X)? How can we get Prometheus running in docker container able to scrape the target running on the host?

Failed attempt: Tried replacing in docker-compose.yml

networks: 
  - back-tier
  - front-tier

with

network_mode: "host"

but then we are unable to access the Prometheus admin page at localhost:9090.

Unable to find solution from similar questions

• Getting error "Get http://localhost:9443/metrics: dial tcp 127.0.0.1:9443: connect: connection refused"

docker-compose.yml

version: '3.3'

networks:
  front-tier:
  back-tier:

services:

  prometheus:
    image: prom/prometheus:v2.1.0
    volumes:
      - ./prometheus/prometheus:/etc/prometheus/
      - ./prometheus/prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
      - '--web.console.templates=/usr/share/prometheus/consoles'
    ports:
      - 9090:9090
    networks:
      - back-tier
    restart: always


  grafana:
    image: grafana/grafana
    user: "104"
    depends_on:
      - prometheus
    ports:
      - 3000:3000
    volumes:
      - ./grafana/grafana_data:/var/lib/grafana
      - ./grafana/provisioning/:/etc/grafana/provisioning/
    env_file:
      - ./grafana/config.monitoring
    networks:
      - back-tier
      - front-tier
    restart: always

prometheus.yml

global:
scrape_interval:     15s 
evaluation_interval: 15s 

external_labels:
    monitor: 'my-project'

- job_name: 'prometheus'

    scrape_interval: 5s

    static_configs:
        - targets: ['localhost:9090']


- job_name: 'rigs-portal'

    scrape_interval: 5s

    static_configs:
        - targets: ['127.0.0.1:8000']

Output at http://localhost:8000/metrics

# HELP python_gc_objects_collected_total Objects collected during gc
# TYPE python_gc_objects_collected_total counter
python_gc_objects_collected_total{generation="0"} 65.0
python_gc_objects_collected_total{generation="1"} 281.0
python_gc_objects_collected_total{generation="2"} 0.0
# HELP python_gc_objects_uncollectable_total Uncollectable object found during GC
# TYPE python_gc_objects_uncollectable_total counter
python_gc_objects_uncollectable_total{generation="0"} 0.0
python_gc_objects_uncollectable_total{generation="1"} 0.0
python_gc_objects_uncollectable_total{generation="2"} 0.0
# HELP python_gc_collections_total Number of times this generation was collected
# TYPE python_gc_collections_total counter
python_gc_collections_total{generation="0"} 37.0
python_gc_collections_total{generation="1"} 3.0
python_gc_collections_total{generation="2"} 0.0
# HELP python_info Python platform information
# TYPE python_info gauge
python_info{implementation="CPython",major="3",minor="7",patchlevel="3",version="3.7.3"} 1.0
# HELP request_processing_seconds Time spend processing request
# TYPE request_processing_seconds summary
request_processing_seconds_count 2545.0
request_processing_seconds_sum 1290.4869346540017
# TYPE request_processing_seconds_created gauge
request_processing_seconds_created 1.562364777766845e+09
# HELP my_inprorgress_requests CPU Load
# TYPE my_inprorgress_requests gauge
my_inprorgress_requests 65.0

Python3 script

from prometheus_client import start_http_server, Summary, Gauge
import random
import time

# Create a metric to track time spent and requests made
REQUEST_TIME = Summary("request_processing_seconds", 'Time spend processing request')

@REQUEST_TIME.time()
def process_request(t):
    time.sleep(t)

if __name__ == "__main__":
    start_http_server(8000)
    g = Gauge('my_inprorgress_requests', 'CPU Load')
    g.set(65)

    while True:
        process_request(random.random())

Answer 1

While not a very common use case.. you can indeed connect from your container to your host.

From https://docs.docker.com/docker-for-mac/networking/

I want to connect from a container to a service on the host

The host has a changing IP address (or none if you have no network access). From 18.03 onwards our recommendation is to connect to the special DNS name host.docker.internal, which resolves to the internal IP address used by the host. This is for development purpose and will not work in a production environment outside of Docker Desktop for Mac.

Answer 2

For reference for people who might find this question through search, this is supported now as of Docker 20.10 and above. See the following link:

How to access host port from docker container

and:

https://github.com/docker/for-linux/issues/264#issuecomment-823528103