Today you can get 10Gbps network cards for less than $20 in the US. There are two popular models, Mellanox ConnectX-2 EN and Chelsio S310E-CR.
Switches are still expensive though. Even switches with only four 10G ports, and they may be noisy. Luckily you can grab two dual port Mellanox MNPH29D-XTR ConnectX-2 EN for around $150 and create your own “switch” easily with FreeBSD.
You need a motherboard with two or more PCIe slots(8x or 16x slot sizes). If you want a 6 port switch you need three PCIe slots, unless you get a network card like Chelsio T440-CR that has 4 ports.
FreeBSD 10.2-RELEASE ships with the Mellanox drivers, but they’re not ready to use immediately. You will need to compile and install the drivers first.
As root:
cd /usr/src/sys/modules/mlxen && make && make install && kldload mlxen
Now you are ready to create a network bridge that will act as a switch. First step is to list your device names:
ifconfig -l
mlxen0 mlxen1 mlxen2 mlxen3
Second step is to create the bridge adapter:
ifconfig bridge create
Add the ports to the bridge:
ifconfig bridge0 addm mlxen0 addm mlxen1 addm mlxen2 addm mlxen3
Make sure links are up/enabled:
ifconfig mlxen0 up
ifconifg mlxen1 up
ifconfig mlxen2 up
ifconfig mlxen3 up
Finally set the ip address for the bridge adapter:
ifconfig bridge0 inet 192.168.0.10/24
That is all. You now have a basic software based switch, connect your other systems, give them a proper ip-address and see them talk together just fine. However, wait! A bridge adapter should be much slower than a traditional switch? Yes, that is true, but for only 4-6 ports, it should not be that significant.
So I had to benchmark this, four servers with Ubuntu Linux, two of them as iperf servers connect to one dual port NIC (mlxen0 & mlxen1) and two as iperf clients connected to the other NIC (mlxen2 & mlxen3). I ran all of them at the same time and made sure all of the traffic had to go through the PCIe bus for five minutes.
The result was an average speed of 17GB/s, 8.5 GB/s each, with no tuning and the default MTU set to 1500. That should be plenty for everyone doing 10Gbps on small scale. If you need more performance, you have plenty of room left by spending endless nights of reading about network tuning in FreeBSD :-)