While on VPN, moby can't resolve private DNS #997

favoretti · 2016-11-30T22:51:30Z

Expected behavior

Moby should use the /etc/resolv.conf of the host machine

Actual behavior

It doesn't :)
Unless I log in to moby and put private DNS server ahead of 192.168.65.* block.

Information

1502FA18-9B27-4FE6-BE34-C318CB0BA4A5

Steps to reproduce the behavior

Connect to a VPN that would modify your /etc/resolv.conf
Try logging in into a private docker registry:
docker login -u vlazarenko@ebay.com -p somethingfake https://registry.ecg.so/v1/
Observe the following:
Error response from daemon: Get https://registry.ecg.so/v1/users/: dial tcp: lookup registry.ecg.so on 192.168.65.1:53: no such host

The text was updated successfully, but these errors were encountered:

ebriney · 2016-12-02T17:15:03Z

It should work, we are relying on scutil --dns output to generate dns entries on moby.
We are investigating why it don't work.

favoretti · 2016-12-03T14:41:38Z

FWIW, scutil --dns sees newly assigned private DNS servers ok:

DNS configuration (for scoped queries)

resolver #1
  search domain[0] : lair
  nameserver[0] : 192.168.2.254
  nameserver[1] : fd00::5e49:79ff:fe2b:9da1
  if_index : 4 (en0)
  flags    : Scoped, Request A records, Request AAAA records
  reach    : Reachable, Directly Reachable Address

resolver #2
  nameserver[0] : 10.250.16.37
  nameserver[1] : 10.250.16.38
  if_index : 10 (utun1)
  flags    : Scoped, Request A records
  reach    : Reachable

favoretti · 2016-12-03T14:43:04Z

But even if I reboot moby while on VPN, the actual /etc/resolv.conf is missing them:

/ # cat /etc/resolv.conf
# Generated by dhcpcd from eth0.dhcp
# /etc/resolv.conf.head can replace this line
domain fritz.box
search lair
nameserver 192.168.65.1
# /etc/resolv.conf.tail can replace this line
/ #

rogaha · 2016-12-09T09:38:08Z

@djs55 any thoughts?

djs55 · 2016-12-13T12:13:45Z

Sorry for the delay in getting back to you. The /etc/resolv.conf in the VM points at a resolver running on the host which is part of Docker for Mac -- this means we can more easily adapt when the host network changes.

Looking at the bug report I think the following sequence happened:

the query for registry.ecg.so was forwarded only to 192.168.2.254
a "No such name" response was received from 192.168.2.254
the VM tried appending the search domain to create a query for registry.ecg.so.lair
the query for registry.ecg.so.lair was sent to both 10.250.16.37 and 10.250.16.38
more "No such name" responses were received

I suspect we've misinterpreted the contents of the SC database. It looks like your global/default DNS server is 192.168.2.254 while your VPN servers are 10.250.16.37 and 10.250.16.38. The VPN servers have SupplementalMatchDomains which include lair (but not so). The current logic in the DNS forwarder says:

if the DNS domain matches a SupplementalMatchDomains entry, send it only there
otherwise send it only to the other (i.e. global) servers

Could you confirm for me that registry.ecg.so resolves correctly via 10.250.16.37 and 10.250.16.38 using a command on the Mac like

dig registry.ecg.so @10.250.16.37

I'll investigate further. Thanks for the bug report!

favoretti · 2016-12-13T17:02:25Z

No worries about delay whatsoever, thanks for looking into it!

Yes, registry.ecg.so resolves correctly:

[none][16:28:34] vlazarenko@laptop (~)$ dig registry.ecg.so @10.250.16.37

; <<>> DiG 9.8.3-P1 <<>> registry.ecg.so @10.250.16.37
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53130
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;registry.ecg.so.		IN	A

;; ANSWER SECTION:
registry.ecg.so.	59	IN	A	10.32.160.107

;; Query time: 200 msec
;; SERVER: 10.250.16.37#53(10.250.16.37)
;; WHEN: Tue Dec 13 18:01:23 2016
;; MSG SIZE  rcvd: 49

favoretti · 2016-12-13T17:05:11Z

Just to be sure we're on the same page here: 192.168.2.254 is the 'public internet DNS'. 10.32.*.* are the private VPN DNS. Those don't have any supplementary domains, at least according to the scutil output.

ebuildy · 2017-01-03T16:24:05Z

Maybe as a workaround, will be nice to edit /etc/hosts from Docker UI or a small tutorial to explain how to deal with it.

What a shame D4M is not working in professionnal environment ;-( (I mean with VPN, custom registry etc...)

djs55 · 2017-01-03T16:35:11Z

I'm hoping to work on improved VPN support over the next few betas... this is one of the issues I'm hoping to fix. Thanks everyone for your patience so far!

ebuildy · 2017-01-03T16:55:13Z

Sure no problem! I guess you guys have a ton of work to do + manage huge community, maybe a minimalist web UI to indicate whats going on / road-map could help and prevent spam..

Do you know any good workaround or tutorial? was thinking about running a DNS (via Docker) on the machine

favoretti · 2017-01-03T17:33:13Z

@ebuildy Run screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty to get moby console, then edit /etc/hosts, then hit Ctrl-A Ctrl-D to detach. Done.

favoretti · 2017-01-03T17:34:15Z

@djs55 Feel free to use me as a guinea pig for SSL-based VPN if you want to verify whether fix works in this particular case :)

ebuildy · 2017-01-03T17:39:39Z

thanks @favoretti , this is what I am doing already, at home. But I manage also a team of 12 french devs (using windows, mac & linux), so I am searching for a very clean solution without any hacks to do.

I do already docker run --rm -v /etc/resolv.conf:/r debian:8 bash -c 'echo "nameserver X.X.X.X" > /r' to fix DNS server, will be helpfull to have more entreprise feature, such as configuration file well documented.

same issue to setup NFS share, this is crucial for pro. adoption.

At least, if Docker team could blog something about these small hacks, this could help a lot people to trust and adopt Docker at work.

djs55 · 2017-01-12T09:26:59Z

I've merged a potential fix for this into the master branch, which should be released as part of beta 37, due in a couple of weeks. Sorry for the delay -- I'll ping you again when there's something you can test.

ebuildy · 2017-01-12T10:55:16Z

Great to hear!

As a workaround, If we can configure daemon.json to setup a DNS it's ok, (for now, we cannot with the stable release) /:

Is windows version have the same problem?

Thanks you,

Vanuan · 2017-01-20T13:23:59Z

As Docker 1.13 was released, this has become critical. Is there a way to cherry-pick this fix to the stable version? There's no way to go back to Docker 1.12.

djs55 · 2017-01-20T16:17:06Z

The fix is in the master branch, which is due for release in the beta branch next week in beta 39 (hopefully). Once it has been in the beta for a week, it's a candidate for cherry-picking into the next stable update. We want to be very careful with stable updates.

If you want to experiment in the meantime, try following the instructions in this comment: #1103 (comment) -- the newer build of the networking component should be compatible with the stable release (although I've not tested it). Take a backup of the file before you replace it. Normally we don't recommend this kind of hybrid configuration, but if you're blocked then it's worth a shot. If you try it and it doesn't work, please upload fresh diagnostics and ping me -- there's still time to fix bugs before the next beta release.

remear · 2017-01-20T18:41:32Z

I'd like to add my experience with this so far as additional info in case it's useful.

I started experiencing this last night while on VPN using a beta32 (? i don't remember the exact version) and attempting to both push to and re-authenticate to my private registry . It just suddenly started responding with no such host messages after having been on VPN for a while and everything working without issue. I'm currently in the building, not on VPN, and have upgraded to 1.13.0-beta38 (15084), but I still experience this issue. Applying the slirp and restarting didn't have any noticeable effect.

scutil --dns results show what should be the DNS servers that have the entry for the private registry (see below). login appears to succeed on 1.13.0-beta38 (15084), push responds with no such host, when last night on the prevision version I only received a no such host response for both login and push.

DNS configuration

resolver #1
  search domain[0] : private-place.com
  nameserver[0] : xxx.xxx.xxx.xxx
  nameserver[1] : xxx.xxx.xxx.xxx
  nameserver[2] : xxx.xxx.xxx.xxx
  if_index : 13 (en7)
  flags    : Request A records, Request AAAA records
Reachable

resolver #2
  domain   : local
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
Not Reachable
  order    : 300000

resolver #3
  domain   : 254.169.in-addr.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
Not Reachable
  order    : 300200

resolver #4
  domain   : 8.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
Not Reachable
  order    : 300400

resolver #5
  domain   : 9.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
Not Reachable
  order    : 300600

resolver #6
  domain   : a.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
Not Reachable
  order    : 300800

resolver #7
  domain   : b.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
Not Reachable
  order    : 301000

DNS configuration (for scoped queries)

resolver #1
  search domain[0] : private-place.com
  nameserver[0] : xxx.xxx.xxx.xxx
  nameserver[1] : xxx.xxx.xxx.xxx
  nameserver[2] : xxx.xxx.xxx.xxx
  if_index : 13 (en7)
  flags    : Scoped, Request A records, Request AAAA records
Reachable

resolver #2
  search domain[0] : private-place.com
  nameserver[0] : xxx.xxx.xxx.xxx
  nameserver[1] : xxx.xxx.xxx.xxx
  nameserver[2] : xxx.xxx.xxx.xxx
  if_index : 4 (en0)
  flags    : Scoped, Request A records, Request AAAA records
Reachable

semifor · 2017-01-20T19:15:17Z

I had the same issue after an upgrade to 1.13. Subsequently upgraded to 1.13.0-beta38. Same problem. Followed the instructions at #1103 (comment). That resolved the issue for me. I'm using Cisco AnyConnect for access to a private data center.

remear · 2017-01-20T19:29:24Z

I also primarily use Cisco AnyConnect for VPN. When this started happening last night I created a VPN configuration via Network Preferences. It worked for a short while and then went back to no such host responses with no change in VPN connectivity. Subsequent attempts with the non-Cisco VPN configuration only resulted in no such host responses.

ebuildy · 2017-01-20T20:36:29Z

With new D4M 1.13, it looks ok ! good job guys, such a good new.

I use Viscosity as VPN client with split DNS.

favoretti · 2017-01-21T09:10:50Z

Thank you guys! Works with my F5 SSL VPN as well!

justincormack · 2017-01-21T09:35:12Z

Great! I will close this. We understand that there may be other VPN related issues still in different setups, but we hope this now works for most people. If you still have issues please open a new issue for the specific problem.

Vanuan · 2017-01-22T14:28:00Z

Is there a webpage listing d4m beta release history?

bigkraig · 2017-01-24T04:47:56Z

this is worse with Cisco any connect

docker-robott · 2020-06-21T03:06:18Z

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked

ebriney added status/triage status/acknowledged and removed status/triage labels Dec 2, 2016

favoretti mentioned this issue Dec 19, 2016

Docker for Mac 1.12.0 TCP DNS queries fail moby/moby#24344

Closed

djs55 added status/3-fixed area/network kind/bug and removed status/acknowledged labels Jan 12, 2017

Vanuan mentioned this issue Jan 12, 2017

DNS resolving stops working #1132

Closed

justincormack closed this as completed Jan 21, 2017

djs55 mentioned this issue Mar 3, 2017

Get https://registry-1.docker.io/v2/: dial tcp [::1]:443: getsockopt: connection refused. moby/moby#31510

Closed

docker locked and limited conversation to collaborators Jun 21, 2020

docker-robott added the lifecycle/locked label Jun 21, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

While on VPN, moby can't resolve private DNS #997

While on VPN, moby can't resolve private DNS #997

favoretti commented Nov 30, 2016 •

edited

Loading

ebriney commented Dec 2, 2016

favoretti commented Dec 3, 2016

favoretti commented Dec 3, 2016

rogaha commented Dec 9, 2016

djs55 commented Dec 13, 2016

favoretti commented Dec 13, 2016

favoretti commented Dec 13, 2016

ebuildy commented Jan 3, 2017

djs55 commented Jan 3, 2017

ebuildy commented Jan 3, 2017

favoretti commented Jan 3, 2017

favoretti commented Jan 3, 2017

ebuildy commented Jan 3, 2017 •

edited

Loading

djs55 commented Jan 12, 2017

ebuildy commented Jan 12, 2017

Vanuan commented Jan 20, 2017 •

edited

Loading

djs55 commented Jan 20, 2017

remear commented Jan 20, 2017

semifor commented Jan 20, 2017

remear commented Jan 20, 2017

ebuildy commented Jan 20, 2017

favoretti commented Jan 21, 2017

justincormack commented Jan 21, 2017

Vanuan commented Jan 22, 2017

bigkraig commented Jan 24, 2017

docker-robott commented Jun 21, 2020

While on VPN, moby can't resolve private DNS #997

While on VPN, moby can't resolve private DNS #997

Comments

favoretti commented Nov 30, 2016 • edited Loading

Expected behavior

Actual behavior

Information

Steps to reproduce the behavior

ebriney commented Dec 2, 2016

favoretti commented Dec 3, 2016

favoretti commented Dec 3, 2016

rogaha commented Dec 9, 2016

djs55 commented Dec 13, 2016

favoretti commented Dec 13, 2016

favoretti commented Dec 13, 2016

ebuildy commented Jan 3, 2017

djs55 commented Jan 3, 2017

ebuildy commented Jan 3, 2017

favoretti commented Jan 3, 2017

favoretti commented Jan 3, 2017

ebuildy commented Jan 3, 2017 • edited Loading

djs55 commented Jan 12, 2017

ebuildy commented Jan 12, 2017

Vanuan commented Jan 20, 2017 • edited Loading

djs55 commented Jan 20, 2017

remear commented Jan 20, 2017

semifor commented Jan 20, 2017

remear commented Jan 20, 2017

ebuildy commented Jan 20, 2017

favoretti commented Jan 21, 2017

justincormack commented Jan 21, 2017

Vanuan commented Jan 22, 2017

bigkraig commented Jan 24, 2017

docker-robott commented Jun 21, 2020

favoretti commented Nov 30, 2016 •

edited

Loading

ebuildy commented Jan 3, 2017 •

edited

Loading

Vanuan commented Jan 20, 2017 •

edited

Loading