Nomad and port mapping

Question

Nomad has three different ways to map ports:

1. Network stanza under group level
2. Network stanza under config -> resources level
3. port_map stanza under config level

What is the difference and when I should use which?

Answer 1

• First of all port_map is deprecated, so you shouldn't be using that as part of task driver configuration.

  Up until Nomad 0.12, ports could be specified in a task's resource stanza and set using the docker port_map field. As more features have been added to the group network resource allocation, task based network resources are deprecated. With it the port_map field is also deprecated and can only be used with task network resources.

  Users should migrate their jobs to define ports in the group network stanza and specified which ports a task maps with the ports field.

• port in the group network stanza defines labels that can be used to identify the port in service discovery. This label is also used as apart of environment variable name that indicates which port your application should bind to.

• ports at the task level specifies which port from network stanza should be available inside task allocation/container. From official docs

  A Docker container typically specifies which port a service will listen on by specifying the EXPOSE directive in the Dockerfile.

  Because dynamic ports will not match the ports exposed in your Dockerfile, Nomad will automatically expose any ports specified in the ports field.

TLDR;

So there is only one correct definition:

job "example" {
  group "example-group" {
    network {
      # Dynamic ports
      port "foo" {}
      port "bar" {}
      # Mapped ports
      port "http"  { to = 80 }
      port "https" { to = 443 }
      # Static ports
      port "lb" { static = 8080 }
    }

    task "task-1" {
      driver = "docker"
      config {

        ...
 
        ports = [
          "foo",
          "http",
        ]
      }
    }

    task "task-2" {
      driver = "docker"
      config {

        ...
 
        ports = [
          "bar",
          "https",
        ]
      }
    }

    task "task-3" {
      driver = "docker"
      config {

        ...
 
        ports = [
          "lb",
        ]
      }
    }
  }
}

Consider running this type of job file (with whatever images). Then you will get the following port mapping between a backend and containers:

for port in $(docker ps --format "{{.Ports}}"); do echo $port; done | grep tcp | cut -d':' -f 2

# Dynamic ports 'foo' and 'bar'
# 25968->25968/tcp,
# 29080->29080/tcp,

# Mapped ports 'http' and 'https'
# 29936->80/tcp,
# 20987->443/tcp,

# Static port 'lb'
# 8080->8080/tcp,

Now, if you get inside task-1 allocation/container and check env variables, then you would be able to get values for allocated ports if your tasks need to communicate with one another.

env | grep NOMAD | grep PORT

# NOMAD_PORT_bar=29080
# NOMAD_HOST_PORT_bar=29080

# NOMAD_PORT_foo=25968
# NOMAD_HOST_PORT_foo=25968

# NOMAD_PORT_http=80
# NOMAD_HOST_PORT_http=29936

# NOMAD_PORT_https=443
# NOMAD_HOST_PORT_https=20987

# NOMAD_PORT_lb=8080
# NOMAD_HOST_PORT_lb=8080

In order to make communication between services easier, it is better to use service discovery, e.g. Consul (also from HashiCorp) and to make you life even easier consider some sort of load balancer, e.g. Fabio or Traefik. Here is a nice blog post from HashiCorp's Engineer about it.