In my previous post I have explained how I have set up Authelia to protect my different services with unified two-factor authentication. Though most of my web services only support proxy header authorization, one of the main benefits of chosing such a solution is to use Authelia as a unified OIDC provider. Initially, I had the requirement to use OIDC since I wanted to move my Tailscale mesh network off the official Tailscale coordination servers, and toward a self-hosted coordination server based on headscale. Some other changes to my network made me change my mind over this approach, though.

Testing out Headscale

Installation of Headscale as an alternative coordination server itself was a breeze thanks to great openly available documentation. I have followed this tutorial including a docker-compose file, which even bundles a UI to see the different devices in your tailnet and apply tags to them without the CLI. After also setting up Authelia as an OIDC provider, I struggled with an error message in the log-files telling me OIDC setup failed. After some troubleshooting I noticed that the error was with the error message itself; there is a GitHub issue documenting this bug. Even though the log-files claim differently, authentication with OIDC worked right away. The alternative to using OIDC for Headscale is to authenticate clients against the coordination server with a pre-shared key - this requires CLI access each team and also makes for a bad user experience, especially on mobile. Using OIDC, the Authelia log-in page can be used to onboard a new device, negating the need for Google SSO in the case of official Tailscale.

Onboarding a new Linux device is now as easy as

tailscale up --login-server YOUR_HEADSCALE_URL

To change the log-in server in mobile apps, the three dots on the start page of the Tailscale app have to be opened multiple times, until the option to change the coordination server appears. This is unfortunately not yet possible on iOS.

On my Chromebook, I had initially installed the Tailscale app from the Play store. This has worked fine with the official coordination server - when I changed that to my locally hosted Headscale instance, the app crashed however. To solve this problem, I had another idea though. Instead of using the Play store app, I used the command mentioned above in my Linux terminal on the Chromebook and installed the Linux version of Firefox on that same VM. This gave me a huge advantage: I can now run my default Chrome browser with BeyondCorp authenticating me against all company websites, while having a sandboxed Firefox browser tunneled with my home network at all times, giving me the best of both worlds. With the Play store app, I had to decide whether to use BeyondCorp or Tailscale at any given time.

While this setup worked great, I did have some doubts. For once, Headscale was still using official DERP relay servers in case a direct connection between peered devices could not be established. I was already about to start troubleshooting this, when a second doubt got to me: My personal server at home was becoming more and more a single point of failure for my whole IT infrastructure. I was already struggling with AdGuard Home as my DNS server running as a container on my server - whenever my server had to reboot, DNS would stop working for all devices at home, frustrating not only me but my wife as well. If the connection between any of my devices would now also rely on my home server, I would have no fallback option to get into my home network if the server went down (e.g. through my Raspberry Pi running OctoPi). Right at this time I noticed a blog article about NextDNS on the Tailscale blog, which made me consider changing my setup yet again. When Tailscale furthermore announced the introduction of tailnet lock, I abandoned the idea of a Headscale server and went back to the official implementation. I might even explore further integration with Caddy in the future, either be letting Caddy manage Tailscale HTTPS certificates or (even cooler) letting my web services join my tailnet using a Caddyfile directive.

Moving to NextDNS

As mentioned, I was not too happy about my home network relying on AdGuard Home as a network-wide ad blocker. While this was working fine most of the time, server maintenance meant that I would either have to change the DNS server in my router’s DHCP settings, on individual devices or live without domain name resolution (i.e. internet access, basically). Also, ad-blocking was not working from mobile devices away from home. I might have been able to create a connection, but might have suffered consequences concerning request latency. Therefore, I started reading up on NextDNS. While there are multiple alternative like AdGuard DNS as a hosted service, or other free players like dnsforge, DeCloudUs or blahdns, I was really impressed by the fine-tuned mechanics that could be applied via different profiles with a NextDNS account and decided to give it a try.

DNS should be available to all my network clients at home and be encrypted via DoH, so I went and installed NextDNS directly on my Unifi USG via SSH. Since I am running a guest network at home in a different subnet, I was able to pass different profiles I had created on the NextDNS web app directly to these subnets via my DHCP server:

sudo nextdns config set -config 10.10.10.0/24=abcdef -config 123456

The example above is for my guest network being in the 10.10.10.0/24 subnet with a dedicated profile abcdef, whereas all other VLANs should get the default profile 123456. The settings I have used for these profiles follow the recommendations posted on this GitHub page very closely. The last thing to do was to renew the lease of all DHCP clients so they would update their DNS settings, and apply my NextDNS profile to my Tailscale config (overriding local DNS settings) as well. As long as I am connected to Tailscale I am now using NextDNS whereever I go - at home I am using NextDNS for all devices, not only those connected to Tailscale.

Implementing Cloudflare Tunnels

While I was happy with the change of my DNS provider to NextDNS there was one additional change I made to my home setup recently. In the past, I had exposed ports 80 and 443 of my residential IP via this domain. While I already took some measures to protect myself against DDoS attacks by having requests proxied through Cloudflare, this only worked when attackers tried to target my domain name. For attacks against my IP address, my own server would have to do the defending. I had considered solutions like Fail2ban in the past, but another solution really caught my interest: Cloudflare tunnels. Cloudflare tunnel would allow me to create a private connection between my home server and Cloudflare using a single Docker container, letting me close all my Firewall ports and have all traffic go through Cloudflare without an external IP.

To set this up, I used Cloudflare’s web UI to create the connection to my Docker container with the support of this tutorial; the docker container itself was available as an app template from Unraid’s community store, but can just as easily be launched via docker run or docker-compose. Since I wanted to keep using Caddy as a reverse proxy handling all incoming requests, I decided to use a Wildcard DNS entry on Cloudflare. To make this work, I had to slightly adjust my Caddyfile following this forum thread, using the handle-directive to route the different origin domains within my Caddyfile. Cloudflare itself does not populate the DNS entry for a Wildcard tunnel automatically,so I created the corresponding CNAME record manually using the format tunnel-id.cfargotunnel.com for all subdomains (*). I furthermore rebuilt my Caddy image including the cloudflaredns plugin, to have this entry populated automatically just in case I change my setup in the future and lose the DNS entry.

Conclusion

Implementing these changes allowed me to lock down my network by closing all firewall ports - external facing web services are protected through Cloudflare tunnels, internal services are available from my own devices via Tailscale. In case my server goes down, I can still reach my all other devices via Tailscale’s external coordination server, and domain name resolution still works according to my rules defined in NextDNS, both at home and outside.