Minikube with DNS on ARM-based Mac

Problem:

Currently as of June 2023 there seem to be no way to use minikube's ingress-dns addon on an ARM-based Mac.

Reading on https://minikube.sigs.k8s.io/docs/handbook/addons/ingress-dns/ it seems that this should just work. However, a small repro:

minikube start --driver=docker --addons=ingress,ingress-dns --install-addons=true

kubectl apply -f https://raw.githubusercontent.com/kubernetes/minikube/master/deploy/addons/ingress-dns/example/example.yaml

minikube tunnel

# Other terminal:
🐟 nslookup hello-john.test (minikube ip)
;; connection timed out; no servers could be reached

After a while I found this posting https://github.com/kubernetes/minikube/issues/12876#issuecomment-1023970717

with a reference to the docker driver limitations https://minikube.sigs.k8s.io/docs/drivers/docker/ where it clearly said

>The ingress, and ingress-dns addons are currently only supported on Linux. See #7332

Well, the minikube ip isn't actually reachable with the Docker driver on Mac. So trying to skip docker driver and go straight to hyperkit it turns out it's not supported on Apple Silicon:

minikube start --driver=hyperkit

Exiting due to DRV_UNSUPPORTED_OS: The driver 'hyperkit' is not supported on darwin/arm64

So what to do?

Solution

Found this post https://github.com/kubernetes/minikube/issues/7332#issuecomment-1164452857

with a reference to https://github.com/chipmk/docker-mac-net-connect

Complete working example:

# Start minikube with ingress-dns
minikube start --driver=docker --addons=ingress,ingress-dns --install-addons=true

# Install and enable WireGuard tunnel so that minikube ip is reachable from host machine
brew install chipmk/tap/docker-mac-net-connect
sudo brew services start chipmk/tap/docker-mac-net-connect

# test that dns works with minikube as dns server:
kubectl apply -f https://raw.githubusercontent.com/kubernetes/minikube/master/deploy/addons/ingress-dns/example/example.yaml
🐟 nslookup hello-john.test (minikube ip)
Server:		192.168.58.2
Address:	192.168.58.2#53

Non-authoritative answer:
Name:	hello-john.test
Address: 192.168.58.2


# Add minikube to host machine dns lookup
cat >/etc/resolver/minikube-test <<EOF
domain test
nameserver $(minikube ip)
search_order 11
timeout 5
EOF

# Test to actually reach the ingress without specifying dns
✗ curl hello-john.test
Hello, world!
Version: 1.0.0
Hostname: hello-world-app-f4cd6696d-l9c88

This example is to show everything working with as few commands as possible. I'm however using nix-darwin to set most of this up.