Building a FreeBSD pf Router behind XGS-PON

Published on

FreeBSD 15 pf XGS-PON Networking Homelab

Why Build It Yourself?

Consumer routers run a Linux kernel from 2017, a vendor-modified userspace, and a web UI that lies about what's actually configured. OPNsense and pfSense are excellent, they're both FreeBSD-based, in fact, but their abstraction is also their burden: the GUI eventually doesn't expose the knob you need, and you end up editing a config file that the GUI may overwrite tomorrow.

Running plain FreeBSD as your router gives up the GUI and gains everything underneath: a stable kernel, the OpenBSD-derived pf firewall, jails for service isolation, ZFS for storage and rollback, and a documentation tradition that takes itself seriously. This post walks through how I build one.

Hardware Pick

I'm boring on purpose, with one specific upgrade over the usual 4-port mini-PC recipe: dual SFP+ 10GbE so the WAN can be an XGS-PON SFP module instead of an ISP gateway.

Avoid Realtek NICs unless you enjoy writing forum posts. Intel chips are boring and that's the highest praise you can give a router NIC.

Install Media

Grab the latest FreeBSD 15.x memstick image and write it to a USB drive: 

# From a Linux/macOS box
curl -OL https://download.freebsd.org/releases/amd64/amd64/ISO-IMAGES/15.0/FreeBSD-15.0-RELEASE-amd64-memstick.img
sudo dd if=FreeBSD-15.0-RELEASE-amd64-memstick.img of=/dev/sdX bs=1M status=progress conv=fsync

Plug it in, boot the mini PC, and at the loader prompt drop into a serial console if your hardware supports it (most of these boxes do, via a console port on the front). Working over serial means you can recover from your own mistakes later.

Install: ZFS Root, Auto, with One Tweak

bsdinstall is genuinely good. Walk through it normally and pick:

Reboot, log in over SSH from a workstation cabled to one of the LAN-side NICs, and don't touch a thing on the WAN side until pf is loaded.

Naming the Wires

Before any configuration: figure out which physical port maps to which kernel interface name. ifconfig shows you the names; the labels on the case tell you which is which. With this build the SFP+ ports come up as ix0 and ix1, and the four 2.5G copper ports come up as igc0 through igc3. 

$ ifconfig -l
ix0 ix1 igc0 igc1 igc2 igc3 lo0

$ ifconfig ix0 | grep status
        status: active

Convention I use:

Before you cable up the WAN, make sure the X-ONU-SFPP is configured for your ISP per the pon.wiki guide. The SFP module does the PON-side authentication; FreeBSD just sees an Ethernet link with DHCP behind it.

/etc/rc.conf: the One File Most Routers Need

hostname="homefw"
zfs_enable="YES"

# Forwarding both IP versions
gateway_enable="YES"
ipv6_gateway_enable="YES"

# WAN: SFP+ port hosting the X-ONU-SFPP XGS-PON ONT
ifconfig_ix0="DHCP"
ifconfig_ix0_ipv6="inet6 accept_rtadv"
background_dhclient_ix0="YES"   # XGS-PON DHCP can be slow at boot

# LAN trunk and VLAN 20 (UniWork)
ifconfig_ix1="inet 10.0.0.1 netmask 255.255.255.0"
vlans_ix1="20"
ifconfig_ix1_20="inet 10.20.0.1/24"

# Firewall
pf_enable="YES"
pflog_enable="YES"

# DNS resolver (unbound from pkg)
unbound_enable="YES"

# DHCP server (from pkg) on both LAN segments
dhcpd_enable="YES"
dhcpd_ifaces="ix1 ix1.20"

# Time
ntpd_enable="YES"
ntpd_sync_on_start="YES"

# SSH (lock it down with pf, but enable here)
sshd_enable="YES"

Two lines in there are load-bearing in non-obvious ways. background_dhclient_ix0="YES" exists because the XGS-PON ONT can take a few seconds to settle and hand out a lease at boot; without backgrounding the dhclient call, the boot will block waiting for it, and that delay cascades into ntpd, unbound, and (via missing host keys on first boot) sshd. vlans_ix1="20" plus ifconfig_ix1_20 create the ix1.20 interface during boot. If that interface doesn't exist before pf loads, anything in pf that references it silently breaks, and the VLAN clients have no internet even though the rules look fine.

Apply piecewise: 

$ service netif restart
$ sysctl net.inet.ip.forwarding=1
$ sysctl net.inet6.ip6.forwarding=1

Bootstrap pf with a Safety Net

Don't start pf with an empty ruleset and rely on default-pass. Don't start it with a deny-all and lock yourself out either. Start with the smallest ruleset that keeps SSH and the LAN working, then iterate. 

# /etc/pf.conf, bootstrap, replace with the real ruleset later
ext_if = "ix0"
lan_if = "ix1"
vlan20_if = "ix1.20"
lan_net = "10.0.0.0/24"
vlan20_net = "10.20.0.0/24"

set skip on lo0
scrub in all

# NAT outbound from both internal segments
nat on $ext_if from { $lan_net $vlan20_net } to any -> ($ext_if)

# Default deny inbound on the WAN
block in log on $ext_if all

# Pass internal traffic outbound, stateful
pass in on $lan_if    from $lan_net    to any keep state
pass in on $vlan20_if from $vlan20_net to any keep state
pass out all keep state

Validate before you load it: 

$ pfctl -nf /etc/pf.conf
$ service pf start
$ pfctl -s rules

Order of operations matters here: the ix1.20 interface has to exist before pf parses this file, otherwise the macros referencing it fail to resolve and the rules don't load. The rc.conf above brings the VLAN up during boot, so a normal boot is fine; the gotcha appears when you create the VLAN by hand later and forget to bring it up before reloading pf.

See the pf.conf design article for the production ruleset I actually use.

unbound: Local Recursive DNS

I run the full unbound from pkg rather than the base local_unbound, mostly so its config lives in /usr/local/etc/unbound/ next to the rest of the pkg-managed services. Either works. Have it listen on the LAN and the VLAN: 

# /usr/local/etc/unbound/unbound.conf (excerpt)
server:
  interface: 10.0.0.1
  interface: 10.20.0.1
  access-control: 127.0.0.0/8  allow
  access-control: 10.0.0.0/24  allow
  access-control: 10.20.0.0/24 allow
  access-control: 0.0.0.0/0    refuse

  hide-identity: yes
  hide-version: yes
  qname-minimisation: yes
  harden-glue: yes
  harden-dnssec-stripped: yes
  prefetch: yes

Restart: 

$ service unbound restart
$ drill -u google.com @10.0.0.1   # check DNSSEC validation

Operational note from experience: a symptom that looks like a firewall problem ("ping works, web pages don't load") is almost always DNS in disguise. Check unbound first, pf second.

dhcpd: Leases for the LANs

# /usr/local/etc/dhcpd.conf (excerpt)
default-lease-time 3600;
max-lease-time     86400;
authoritative;

subnet 10.0.0.0 netmask 255.255.255.0 {
  range 10.0.0.100 10.0.0.200;
  option routers 10.0.0.1;
  option domain-name-servers 10.0.0.1;
  option domain-name "lan";
}

subnet 10.20.0.0 netmask 255.255.255.0 {
  range 10.20.0.100 10.20.0.200;
  option routers 10.20.0.1;
  option domain-name-servers 10.20.0.1;
  option domain-name "uniwork";
}

SSH: Belongs on the LAN, Not the WAN

Three things every router SSH config needs: key-only login, no root login, and a pf rule that limits SSH to the trusted LAN. 

# /etc/ssh/sshd_config
PermitRootLogin no
PasswordAuthentication no
ChallengeResponseAuthentication no
AllowUsers admin
ListenAddress 10.0.0.1

ZFS Boot Environments: Cheap Insurance

Before you change anything important, snapshot the boot environment so a single reboot reverts you: 

$ bectl create pre-pf-tightening
$ bectl list
BE                NAME      Active Mountpoint Space   Created
default                     NR     /          12.4G   2026-05-01 09:14
pre-pf-tightening                  -          1.04M   2026-05-03 17:42

If a pf change locks you out and you have console access: 

$ bectl activate pre-pf-tightening
$ shutdown -r now

Smoke Test

From a workstation on the main LAN (downstream of the Sodola switch on ix1): 

$ ping -c 3 10.0.0.1              # router LAN address
$ ping -c 3 1.1.1.1               # outbound IP routing
$ host www.freebsd.org            # outbound DNS
$ traceroute www.freebsd.org      # full path

Then from a device on the UniWork SSID (VLAN 20, 10.20.0.0/24), repeat. Both should reach the internet; neither should reach into the other's subnet without a deliberate pass rule.

If all of that works, you have a working FreeBSD edge router with the AT&T gateway out of the path.

Heads-up: This is the bare metal. The real work, clean pf rules, jails for services, monitoring, IPv6, lives in the homelab tour and the rest of this series.

Next Steps

Building a router along with this guide? Send me your rc.conf. I always learn from how other people draw the lines.

$ subscribe --to newsletter

FreeBSD, pf, and SRE notes, straight to your inbox. No spam, just signal.

Powered by Buttondown. Unsubscribe anytime. ~2 emails/month. Or grab the RSS feed.

Related Posts

← Back to Blog