Menu
I'm attempting to run a four-node swarm from my host machine, with my machine being the manager, and three Hyper-V VMs as worker nodes.
I'm attempting to run a static web page as a service across all nodes, but keep receiving hnsCall failed in Win32: An adapter was not found error on my host machine, and No such image from the VMs (trying to pull the service image from private registry).
This happens when deploying a stack or using service create.
I can, however, run the static page via container instance and it works fine.
I followed this guide to set up and create my VMs, which I suspect is where my problem arises.
The Virtual Switch that I created via the HyperV Virtual Switch Manager, named thevswitch, is both external and set on the correct adapter. I initialized a swarm using the IPv4 of my host machine under thevswitch for the --advertise-addr flag, and successfully joined the VMs as worker nodes to this IP, while using the driver. Docker doesn't seem to see the virtual switch I created regardless.
PS C:\Docker\swarm-test> docker version
Client: Docker Engine - Community
Version: 18.09.2
API version: 1.39
Go version: go1.10.8
Git commit: 6247962
Built: Sun Feb 10 04:12:31 2019
OS/Arch: windows/amd64
Experimental: false
Server: Docker Engine - Community
Engine:
Version: 18.09.2
API version: 1.39 (minimum version 1.24)
Go version: go1.10.6
Git commit: 6247962
Built: Sun Feb 10 04:28:48 2019
OS/Arch: windows/amd64
Experimental: true
PS C:\Docker\swarm-test> docker network ls
NETWORK ID NAME DRIVER SCOPE
3f4030aafa56 Default Switch ics local
m1c2nzylf14x ingress overlay swarm
168954d1275d nat nat local
be3f092c32cb none null local
PS C:\Docker\swarm-test> docker node ls
ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION
4xiljuldm32avsxwo8o513uv5 * ATLL000597 Ready Active Leader 18.09.2
w2ftfwt4oz00keswt6ztmq9ec vm1 Ready Active 18.09.6
82vpc65ezml87sxhhx9mlvszk vm2 Ready Active 18.09.6
lej0a3gabyi3qfta1f4le1f3s vm3 Ready Active 18.09.6
PS C:\Docker\swarm-test> docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
vm1 - hyperv Running tcp://10.195.12.68:2376 v18.09.6
vm2 - hyperv Running tcp://10.195.12.69:2376 v18.09.6
vm3 - hyperv Running tcp://10.195.12.75:2376 v18.09.6
Host ipconfig:
PS C:\Docker\swarm-test> ipconfig
Windows IP Configuration
Ethernet adapter Ethernet 3:
Media State . . . . . . . . . . . : Media disconnected
Connection-specific DNS Suffix . :
Ethernet adapter vEthernet (thevswitch):
Connection-specific DNS Suffix . : Int.CSDental.com
Link-local IPv6 Address . . . . . : fe80::e90f:a37:5f33:87de%29
IPv4 Address. . . . . . . . . . . : 10.195.12.87
Subnet Mask . . . . . . . . . . . : 255.255.252.0
Default Gateway . . . . . . . . . : 10.195.12.1
Wireless LAN adapter Wi-Fi:
Media State . . . . . . . . . . . : Media disconnected
Connection-specific DNS Suffix . :
Wireless LAN adapter Local Area Connection* 9:
Media State . . . . . . . . . . . : Media disconnected
Connection-specific DNS Suffix . :
Wireless LAN adapter Local Area Connection* 10:
Media State . . . . . . . . . . . : Media disconnected
Connection-specific DNS Suffix . :
Ethernet adapter Ethernet 4:
Media State . . . . . . . . . . . : Media disconnected
Connection-specific DNS Suffix . :
Ethernet adapter Bluetooth Network Connection:
Media State . . . . . . . . . . . : Media disconnected
Connection-specific DNS Suffix . :
Ethernet adapter vEthernet (Default Switch) 2:
Connection-specific DNS Suffix . :
Link-local IPv6 Address . . . . . : fe80::5de7:19c9:a701:b06a%65
IPv4 Address. . . . . . . . . . . : 192.168.55.33
Subnet Mask . . . . . . . . . . . : 255.255.255.240
Default Gateway . . . . . . . . . :
Ethernet adapter vEthernet (nat) 2:
Connection-specific DNS Suffix . :
Link-local IPv6 Address . . . . . : fe80::f173:8:f6e1:123c%73
IPv4 Address. . . . . . . . . . . : 172.26.192.1
Subnet Mask . . . . . . . . . . . : 255.255.240.0
Default Gateway . . . . . . . . . :
Host machine (manager):
PS C:\Docker\swarm-test> docker inspect ATLL000597
[
{
"ID": "4xiljuldm32avsxwo8o513uv5",
"Version": {
"Index": 23501
},
"CreatedAt": "2019-05-29T16:41:17.1544224Z",
"UpdatedAt": "2019-05-29T17:35:40.2375377Z",
"Spec": {
"Labels": {},
"Role": "manager",
"Availability": "active"
},
"Description": {
"Hostname": "ATLL000597",
"Platform": {
"Architecture": "x86_64",
"OS": "windows"
},
"Resources": {
"NanoCPUs": 8000000000,
"MemoryBytes": 16971313152
},
"Engine": {
"EngineVersion": "18.09.2",
"Plugins": [
{
"Type": "Log",
"Name": "awslogs"
},
{
"Type": "Log",
"Name": "etwlogs"
},
{
"Type": "Log",
"Name": "fluentd"
},
{
"Type": "Log",
"Name": "gelf"
},
{
"Type": "Log",
"Name": "json-file"
},
{
"Type": "Log",
"Name": "local"
},
{
"Type": "Log",
"Name": "logentries"
},
{
"Type": "Log",
"Name": "splunk"
},
{
"Type": "Log",
"Name": "syslog"
},
{
"Type": "Network",
"Name": "ics"
},
{
"Type": "Network",
"Name": "l2bridge"
},
{
"Type": "Network",
"Name": "l2tunnel"
},
{
"Type": "Network",
"Name": "nat"
},
{
"Type": "Network",
"Name": "null"
},
{
"Type": "Network",
"Name": "overlay"
},
{
"Type": "Network",
"Name": "transparent"
},
{
"Type": "Volume",
"Name": "local"
}
]
},
"TLSInfo": {
"TrustRoot": "-----BEGIN CERTIFICATE-----\nMIIBazCCARCgAwIBAgIUew6Ov5cRAAXjR61XOWyX0l3evSYwCgYIKoZIzj0EAwIw\nEzERMA8GA1UEAxMIc3dhcm0tY2EwHhcNMTkwNTI5MTYzNjAwWhcNMzkwNTI0MTYz\nNjAwWjATMREwDwYDVQQDEwhzd2FybS1jYTBZMBMGByqGSM49AgEGCCqGSM49AwEH\nA0IABHiJekXiApSizb5ctC5JJ0r1EBXvuN5iXMSe3VM/d9rvnjzFozY7VRgALhHb\nAP3wxnnsVPX29JC2ZUlv6//U0LSjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNVHRMB\nAf8EBTADAQH/MB0GA1UdDgQWBBRUFwLEeWDMyZMC1QLrz1xhAyapuDAKBggqhkjO\nPQQDAgNJADBGAiEAqgyoSTtQxJ9RHnW2Zk7KsJthXSaoreN59vw+Ozp7XFACIQDB\n+tmtX3nGoeAf/np9j8oqvMNqApiPcxya8ToNSg/Otg==\n-----END CERTIFICATE-----\n",
"CertIssuerSubject": "MBMxETAPBgNVBAMTCHN3YXJtLWNh",
"CertIssuerPublicKey": "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEeIl6ReIClKLNvly0LkknSvUQFe+43mJcxJ7dUz932u+ePMWjNjtVGAAuEdsA/fDGeexU9fb0kLZlSW/r/9TQtA=="
}
},
"Status": {
"State": "ready",
"Addr": "10.195.12.87"
},
"ManagerStatus": {
"Leader": true,
"Reachability": "reachable",
"Addr": "10.195.12.87:2377"
}
}
]
Just one of the VMs:
PS C:\Docker\swarm-test> docker-machine inspect vm1
{
"ConfigVersion": 3,
"Driver": {
"IPAddress": "10.195.12.68",
"MachineName": "vm1",
"SSHUser": "docker",
"SSHPort": 22,
"SSHKeyPath": "C:\\Users\\19030430\\.docker\\machine\\machines\\vm1\\id_rsa",
"StorePath": "C:\\Users\\19030430\\.docker\\machine",
"SwarmMaster": false,
"SwarmHost": "tcp://0.0.0.0:3376",
"SwarmDiscovery": "",
"Boot2DockerURL": "",
"VSwitch": "thevswitch",
"DiskSize": 20000,
"MemSize": 1024,
"CPU": 1,
"MacAddr": "",
"VLanID": 0,
"DisableDynamicMemory": false
},
"DriverName": "hyperv",
"HostOptions": {
"Driver": "",
"Memory": 0,
"Disk": 0,
"EngineOptions": {
"ArbitraryFlags": [],
"Dns": null,
"GraphDir": "",
"Env": [],
"Ipv6": false,
"InsecureRegistry": [],
"Labels": [],
"LogLevel": "",
"StorageDriver": "",
"SelinuxEnabled": false,
"TlsVerify": true,
"RegistryMirror": [],
"InstallURL": "https://get.docker.com"
},
"SwarmOptions": {
"IsSwarm": false,
"Address": "",
"Discovery": "",
"Agent": false,
"Master": false,
"Host": "tcp://0.0.0.0:3376",
"Image": "swarm:latest",
"Strategy": "spread",
"Heartbeat": 0,
"Overcommit": 0,
"ArbitraryFlags": [],
"ArbitraryJoinFlags": [],
"Env": null,
"IsExperimental": false
},
"AuthOptions": {
"CertDir": "C:\\Users\\19030430\\.docker\\machine\\certs",
"CaCertPath": "C:\\Users\\19030430\\.docker\\machine\\certs\\ca.pem",
"CaPrivateKeyPath": "C:\\Users\\19030430\\.docker\\machine\\certs\\ca-key.pem",
"CaCertRemotePath": "",
"ServerCertPath": "C:\\Users\\19030430\\.docker\\machine\\machines\\vm1\\server.pem",
"ServerKeyPath": "C:\\Users\\19030430\\.docker\\machine\\machines\\vm1\\server-key.pem",
"ClientKeyPath": "C:\\Users\\19030430\\.docker\\machine\\certs\\key.pem",
"ServerCertRemotePath": "",
"ServerKeyRemotePath": "",
"ClientCertPath": "C:\\Users\\19030430\\.docker\\machine\\certs\\cert.pem",
"ServerCertSANs": [],
"StorePath": "C:\\Users\\19030430\\.docker\\machine\\machines\\vm1"
}
},
"Name": "vm1"
}
PS C:\Docker\swarm-test> get-hnsnetwork
ActivityId : F5EEF28D-7A66-410E-8F06-85417531C225
AdditionalParams :
CurrentEndpointCount : 0
Extensions : {@{Id=E7C3B2F0-F3C5-48DF-AF2B-10FED6D72E7A; IsEnabled=False; Name=Microsoft Windows Filtering Platform}, @{Id=E9B59CFA-2BE1-4B21-828F-B6FBDBDDC017; IsEnabled=False; Name=Microsoft Azure VFP Switch Extension},
@{Id=EA24CD6C-D17A-4348-9190-09F0D5BE83DD; IsEnabled=True; Name=Microsoft NDIS Capture}}
Flags : 3
GatewayMac : 00-15-5D-0C-C5-08
Health : @{AddressNotificationMissedCount=0; AddressNotificationSequenceNumber=5; InterfaceNotificationMissedCount=0; InterfaceNotificationSequenceNumber=0; LastErrorCode=0; LastUpdateTime=132036146558887527;
RouteNotificationMissedCount=0; RouteNotificationSequenceNumber=0}
ID : C08CB7B8-9B3C-408E-8E30-5E16A3AEB444
IPv6 : False
IsolateSwitch : True
LayeredOn : 77DEBE34-C5AF-43EF-86C1-92F7CF9CD8EF
MacPools : {@{EndMacAddress=00-15-5D-0C-CF-FF; StartMacAddress=00-15-5D-0C-C0-00}}
MaxConcurrentEndpoints : 0
Name : Default Switch
NatName : ICS85DBD8A9-EEA6-4691-9FE4-901A4445B714
Policies : {}
Resources : @{AdditionalParams=; AllocationOrder=2; Allocators=System.Object[]; Health=; ID=F5EEF28D-7A66-410E-8F06-85417531C225; PortOperationTime=0; State=1; SwitchOperationTime=0; VfpOperationTime=0;
parentId=D841CDEC-707C-421C-BC2C-155F3F378A40}
State : 1
Subnets : {@{AdditionalParams=; AddressPrefix=192.168.55.32/28; GatewayAddress=192.168.55.33; Health=; ID=F7DEEC46-10E5-448D-B890-D78D93E6D560; Policies=System.Object[]; State=0}}
SwitchGuid : C08CB7B8-9B3C-408E-8E30-5E16A3AEB444
TotalEndpoints : 0
Type : ICS
Version : 38654705666
ActivityId : 6589B251-E629-4942-847F-DEE47853D5B6
AdditionalParams :
CurrentEndpointCount : 0
Extensions : {@{Id=E7C3B2F0-F3C5-48DF-AF2B-10FED6D72E7A; IsEnabled=False; Name=Microsoft Windows Filtering Platform}, @{Id=E9B59CFA-2BE1-4B21-828F-B6FBDBDDC017; IsEnabled=False; Name=Microsoft Azure VFP Switch Extension},
@{Id=EA24CD6C-D17A-4348-9190-09F0D5BE83DD; IsEnabled=True; Name=Microsoft NDIS Capture}}
Flags : 0
Health : @{LastErrorCode=0; LastUpdateTime=132036158099689062}
ID : 07E7CCC7-187E-457B-A577-1EAFB16E9731
IPv6 : False
LayeredOn : 9108D862-FE98-4FE4-8004-2C210074F964
MacPools : {@{EndMacAddress=00-15-5D-B0-CF-FF; StartMacAddress=00-15-5D-B0-C0-00}}
MaxConcurrentEndpoints : 0
Name : thevswitch
Policies : {}
Resources : @{AdditionalParams=; AllocationOrder=0; Health=; ID=6589B251-E629-4942-847F-DEE47853D5B6; PortOperationTime=0; State=1; SwitchOperationTime=0; VfpOperationTime=0; parentId=CAEB6648-F50F-4C19-95D8-038C4E3B7346}
State : 1
TotalEndpoints : 0
Type : Transparent
Version : 38654705666
ActivityId : 7F5DFDB2-6E7E-497E-A33E-5BE5655DA57E
AdditionalParams :
CurrentEndpointCount : 0
Extensions : {@{Id=E7C3B2F0-F3C5-48DF-AF2B-10FED6D72E7A; IsEnabled=False; Name=Microsoft Windows Filtering Platform}, @{Id=E9B59CFA-2BE1-4B21-828F-B6FBDBDDC017; IsEnabled=False; Name=Microsoft Azure VFP Switch Extension},
@{Id=EA24CD6C-D17A-4348-9190-09F0D5BE83DD; IsEnabled=True; Name=Microsoft NDIS Capture}}
Flags : 0
Health : @{AddressNotificationMissedCount=0; AddressNotificationSequenceNumber=0; InterfaceNotificationMissedCount=0; InterfaceNotificationSequenceNumber=0; LastErrorCode=0; LastUpdateTime=132036143540255829;
RouteNotificationMissedCount=0; RouteNotificationSequenceNumber=0}
ID : B242C9F9-89C0-4867-B14A-8A175A5D0DDB
IPv6 : False
LayeredOn : 80208A97-455B-4273-B95A-1E73666C9C52
MacPools : {@{EndMacAddress=00-15-5D-D9-8F-FF; StartMacAddress=00-15-5D-D9-80-00}}
MaxConcurrentEndpoints : 0
Name : nat
NatName : ICSBC687A8B-1FC0-4F10-A614-68EA68A3E2EF
Policies : {}
Resources : @{AdditionalParams=; AllocationOrder=2; Allocators=System.Object[]; Health=; ID=7F5DFDB2-6E7E-497E-A33E-5BE5655DA57E; PortOperationTime=0; State=1; SwitchOperationTime=0; VfpOperationTime=0;
parentId=7507FF2E-F59E-4B2C-8194-DDA947C5B5AE}
State : 1
Subnets : {@{AdditionalParams=; AddressPrefix=172.26.192.0/20; GatewayAddress=172.26.192.1; Health=; ID=B735AB8E-A2F9-4134-84BC-598A492F976A; Policies=System.Object[]; State=0}}
TotalEndpoints : 0
Type : nat
Version : 38654705666
Dockerfile:
version: "3"
services:
Hello:
image: registry.local:5000/wintest/swarm-test:2.0.0
ports:
-"8888:80"
deploy:
mode: global
PS C:\Docker\swarm-test> docker service ps test_Hello --no-trunc
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
27iw2nyi610g971bk0meeuf1h test_Hello.82vpc65ezml87sxhhx9mlvszk registry.local:5000/wintest/swarm-test:2.0.0 vm2 Ready Assigned less than a second ago
tw7ren3nakf0ukigjsqzrnkmf test_Hello.w2ftfwt4oz00keswt6ztmq9ec registry.local:5000/wintest/swarm-test:2.0.0 vm1 Ready Rejected less than a second ago "No such image: registry.local:5000/wintest/swarm-test:2.0.0"
xmdrc9102s4qjtstakrvh3msx test_Hello.4xiljuldm32avsxwo8o513uv5 registry.local:5000/wintest/swarm-test:2.0.0 ATLL000597 Ready Rejected 1 second ago "hnsCall failed in Win32: An adapter was not found. (0x803b0006)"
ezh203iv1ntj3w18wgqzdjceo test_Hello.lej0a3gabyi3qfta1f4le1f3s registry.local:5000/wintest/swarm-test:2.0.0 vm3 Running Rejected 4 seconds ago "No such image: registry.local:5000/wintest/swarm-test:2.0.0"
n3n4nfevoadawuqlusfbspg34 test_Hello.82vpc65ezml87sxhhx9mlvszk registry.local:5000/wintest/swarm-test:2.0.0 vm2 Shutdown Rejected 4 seconds ago "No such image: registry.local:5000/wintest/swarm-test:2.0.0"
t46s5etkwgumoa6jp22au3nq5 test_Hello.w2ftfwt4oz00keswt6ztmq9ec registry.local:5000/wintest/swarm-test:2.0.0 vm1 Shutdown Rejected 5 seconds ago "No such image: registry.local:5000/wintest/swarm-test:2.0.0"
rxwqbg2wndz4cdxw71k1d5ore test_Hello.4xiljuldm32avsxwo8o513uv5 registry.local:5000/wintest/swarm-test:2.0.0 ATLL000597 Shutdown Rejected 6 seconds ago "hnsCall failed in Win32: An adapter was not found. (0x803b0006)"
as92ats2qur5sr6e8lxflz71w test_Hello.lej0a3gabyi3qfta1f4le1f3s registry.local:5000/wintest/swarm-test:2.0.0 vm3 Shutdown Rejected 9 seconds ago "No such image: registry.local:5000/wintest/swarm-test:2.0.0"
What catches my attention is that inspecting the VMs seems to show that they're not part of a swarm, but they are?
PS C:\Docker\swarm-test> docker-machine ssh vm1 "docker swarm init"
Error response from daemon: This node is already part of a swarm. Use "docker swarm leave" to leave this swarm and join another one.
exit status 1
Also, the fact that checking Get-HNSNetwork shows that thevswitch is a transparent network. From my understanding, the way I created the network should've made it show as either overlay (for the driver) or external. On a previous attempt to accomplish these tasks, I had another vswitch I had created in the exact same identical way, but that one showed up as private. I don't understand how following the same steps can produce two different, and both incorrect, results.
Here also is a screenshot of my Virtual Switch Manager: V-Switch Manager
I have read all kinds of documentation from Docker and Windows regarding networking, specifically overlay networking, and I feel pretty lost. From my understanding, this should be working, and all errors I've found online point back to the same docs. I have not seen anything that would make me think this can't be done either.
Any help would be greatly appreciated.
NOTE: First time using Docker, trying to put together a successful PoC. I know that my docker-compose.yml is not pretty to look at. I've spent more time learning about networking with docker than I have been able to learn best practices.
239
For anyone who is running into a similar issue:
The issue was NOT actually network related like these error messages implied. The problem ACTUALLY ended up being that the swarm nodes (but not my host) were all using Linux containers and trying to pull a Windows-only image. They were silently throwing architecture errors -- I only found this out when I tried to pull my image directly from a node. The errors generated while deploying to the whole swarm were highly misleading.
I did not "fix" this per say. We ended up restructuring our entire solution around this fact, and the situation no longer applies.
104
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
