Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Nomad and port mapping

Nomad has three different ways to map ports:

  1. Network stanza under group level
  2. Network stanza under config -> resources level
  3. port_map stanza under config level

What is the difference and when I should use which?

like image 955
Alan Sereb Avatar asked Aug 26 '20 16:08

Alan Sereb


People also ask

What port does Nomad use for HTTP?

If Nomad allocates port 23332 to your api task for http, 23332 will be automatically exposed and forwarded to your container, and the driver will set an environment variable NOMAD_PORT_http with the value 23332 that you can read inside your container. This provides an easy way to use the host networking option for better performance.

What is bridge mode in Nomad?

Bridge mode allows compatible tasks to share a networking stack and interfaces. Nomad can then do port mapping without relying on individual task drivers to implement port mapping configuration. The following example is a group level network stanza that uses bridge mode and port mapping.

How do I assign a random port to a nomad application?

Running this by clicking the Run Job button in the Nomad Web UI gives us a successful plan: Click Run and Nomad will start up 5 instances of the application, and assign a random port to each. The port numbers can be viewed in the UI by clicking on the job name and then each allocation in turn:

How do I use host networking with Nomad?

For example: If Nomad allocates port 23332 to your api task for http, 23332 will be automatically exposed and forwarded to your container, and the driver will set an environment variable NOMAD_PORT_http with the value 23332 that you can read inside your container. This provides an easy way to use the host networking option for better performance.


1 Answers

  • First of all port_map is deprecated, so you shouldn't be using that as part of task driver configuration.

    Up until Nomad 0.12, ports could be specified in a task's resource stanza and set using the docker port_map field. As more features have been added to the group network resource allocation, task based network resources are deprecated. With it the port_map field is also deprecated and can only be used with task network resources.

    Users should migrate their jobs to define ports in the group network stanza and specified which ports a task maps with the ports field.

  • port in the group network stanza defines labels that can be used to identify the port in service discovery. This label is also used as apart of environment variable name that indicates which port your application should bind to.

  • ports at the task level specifies which port from network stanza should be available inside task allocation/container. From official docs

    A Docker container typically specifies which port a service will listen on by specifying the EXPOSE directive in the Dockerfile.

    Because dynamic ports will not match the ports exposed in your Dockerfile, Nomad will automatically expose any ports specified in the ports field.

TLDR;

So there is only one correct definition:

job "example" {
  group "example-group" {
    network {
      # Dynamic ports
      port "foo" {}
      port "bar" {}
      # Mapped ports
      port "http"  { to = 80 }
      port "https" { to = 443 }
      # Static ports
      port "lb" { static = 8080 }
    }

    task "task-1" {
      driver = "docker"
      config {

        ...
 
        ports = [
          "foo",
          "http",
        ]
      }
    }

    task "task-2" {
      driver = "docker"
      config {

        ...
 
        ports = [
          "bar",
          "https",
        ]
      }
    }

    task "task-3" {
      driver = "docker"
      config {

        ...
 
        ports = [
          "lb",
        ]
      }
    }
  }
}

Consider running this type of job file (with whatever images). Then you will get the following port mapping between a backend and containers:

for port in $(docker ps --format "{{.Ports}}"); do echo $port; done | grep tcp | cut -d':' -f 2

# Dynamic ports 'foo' and 'bar'
# 25968->25968/tcp,
# 29080->29080/tcp,

# Mapped ports 'http' and 'https'
# 29936->80/tcp,
# 20987->443/tcp,

# Static port 'lb'
# 8080->8080/tcp,

Now, if you get inside task-1 allocation/container and check env variables, then you would be able to get values for allocated ports if your tasks need to communicate with one another.

env | grep NOMAD | grep PORT

# NOMAD_PORT_bar=29080
# NOMAD_HOST_PORT_bar=29080

# NOMAD_PORT_foo=25968
# NOMAD_HOST_PORT_foo=25968

# NOMAD_PORT_http=80
# NOMAD_HOST_PORT_http=29936

# NOMAD_PORT_https=443
# NOMAD_HOST_PORT_https=20987

# NOMAD_PORT_lb=8080
# NOMAD_HOST_PORT_lb=8080

In order to make communication between services easier, it is better to use service discovery, e.g. Consul (also from HashiCorp) and to make you life even easier consider some sort of load balancer, e.g. Fabio or Traefik. Here is a nice blog post from HashiCorp's Engineer about it.

like image 107
Ilya Kisil Avatar answered Sep 28 '22 07:09

Ilya Kisil