I’ve been wondering about putting in a 10GB NIC instead of one of the Intel quad gigabit cards so that I can get a 10GB connection from my pfsense box to my UniFi 16-XG. Since the 16-XG is only an L2 switch and I’d like to be able to communicate across some of my VLANs (e.g. between my main secure LAN and my IoT and media server VLAN) I thought the best way might be to just eliminate the 1GB bottleneck so all of the main devices connect via 10GB.
The build I started is based on the Intel DQ77KB board with an Intel Core i5-3470T CPU and 8gb ram. I just realized though that the DQ77KB only has an x4 PCI-e slot and all of the generally pfsense-recommended 10gb cards are x8. Wondering if anyone here has an x4 10GB (I’m open to either SFP+ or RJ-45) card that they can recommend for pfsense?
I found the Asus XG-C100C and XG-C100F which were promising (I really only need one 10GB port), but I’ve read some older reports that they don’t play well with pfsense… I’m unsure whether that’s still the case.
Thanks JDM_WAAAT - love the site, just getting my feet wet here and starting some builds (which I’ll share soon).
No, this would purely be for the LAN port on the pfsense. It occurs to me though that maybe I’m misunderstanding the L2 limitations on the 16-XG. I was thinking that communication across VLANs would be bottlenecked at 1GB on the 16-XG.
Example:
pfsense LAN — UniFi 16-XG — VLAN1 (private secure LAN for servers, PCs, phones, etc)
pfsense LAN — UniFi 16-XG — VLAN2 (semi-secure LAN for IoT, media/streaming devices, TVs, etc)
I was thinking that, in the above example, the 16-XG would pass any cross-VLAN routing up the chain since it’s an L2 switch, in which case communication between VLAN1 and VLAN2 would be bottlenecked at 1GB if the pfsense box had a 1GB LAN downlink.
I’m also going to be adding a UniFi PRO 24 PoE to my setup, which is an L3 switch… if that’s connected 10G to the 16-XG, would 10G cross-VLAN communication be routed there?
Thanks @JDM_WAAAT. I appreciate you chiming in! I would certainly prefer to keep all of the inter-VLAN off the pfsense, so hopefully there’s a path there that someone can point me towards with this setup.
You’re correct that since the US-16-XG doesn’t do L3, inter-VLAN traffic needs to go via the router. You have a few options, though:
(1) Streaming media across VLANs probably won’t saturate the gigabit link; check metrics under your expected workload and see if you can just postpone network upgrades.
(2) Sell the UniFi switch and get an Aruba S2500, Mikrotik CRS317, etc. (depending on how many SFP+ and 10GbaseT ports you need).
(3) Get an Intel X520 (-T2/-DA2), MCX312, SF7002N, etc. for PFSense.
My interest in inter-VLAN 10GB connectivity is really more about file transfer speeds from several of the computers in my network to my NAS and media server (as well as a bit of future-proofing in prep for potential house purchase next year, where I’ll almost certainly be running some 10GB fiber backhaul connectivity to various parts of the house for different things).
That Aruba S2500 looks like quite a steal! I hadn’t seen that before… wow. Although I’ll probably come close to filling all of the US-16-XG ports once I move (my current apartment setup will fill about half of it). I do have a USW-Pro-24-POE switch on the way from Ubiquity - would the L3 functionality in that work to provide inter-VLAN 10GB connectivity to the US-16-XG and eliminate the need to add 10GB to the pfsense?
Yes, the USW-Pro-24-POE can handle L3 routing between VLANs. But to do so at full 10Gbps would mean reserving both of its SFP+, as well as two of the US-16-XG’s SFP+, for stacking/uplink – and 10GbE ports are scarce resources. A single L3-capable switch with enough 10GbE ports could do it all over its internal backplane.
The simplest way to upgrade to 10GbE as a drop-in replacement for gigabit is with RJ45 10GbaseT and cat6. But the switch will be $400 (8 ports) to $800 (12-16 ports). If you’re planning on using fiber anyway, SFP+ switches are much cheaper – $100 for 4 ports on the Aruba, up to $200-300 for 24 ports on an LB6M/TI24x. SFP+/RJ45 transceivers are about $40-50 each, so if you go with an SFP+ switch, try to avoid 10GbaseT. There are not many switches like the US-16-XG that have both.
My advice is to prioritise which links could really make use of the 10GbE, vs which links would be ok doing bulk transfers overnight. E.g., video editing on your workstation with files on the NAS is a great use-case for a 10GbE link there. But your HTPC doesn’t usually need 10GbE as it’s mostly just streaming media. If in fact you only have a single link that needs 10GbE, you can skip the switch for now, and just direct-connect the two, assigning static IPs on each end in a subnet different from your LAN.
My USW-Pro-24-POE arrived today. For now I’m going to lean on it’s L3 functionality and run the two DACs from the USW-Pro-24-POE SFP+ ports to the US-16-XG as you mentioned, since for the time being I’ll have the SFP’s to spare. I can then re-assess next year after moving and seeing just exactly what my needs for 10GB will be across the house.
Many thanks for all of the detailed answers and suggestions!