Homelab – November 2022

Again I did some updates on my Homelab. This time I replaced one of my compute nodes with a multi-node-server, I added a 10GbE Switch to increase my number of usable 10G ports and I created an ISCSI SAN.

The last couple of months I had some issues with the old main compute node with internal storage. I lost one RAID Array completely and the other two ones were degraded. I lost 4 disks at one time. So far I think it was some issue with the controller itself and not with the disks but after troubleshooting it and after replacing it with some intermediate solution I had some serious trust issues and I wanted to get some other solution.

So I went back to my „drawing board“ and thought about some opportunities. I simply could replace the host 1:1 to a more modern one and thats all. But I had a chance to get my hands on a Fujitsu CX400 S2 multi-node-server and I decided to use this one for the future. With this decision I had another problem: Storage. The nodes doesn’t support direct connected SAS drives and there was no build in RAID controller. I decided to build a ISCSI SAN with a TrueNAS host as storage provider. For this I did some performance tests to figure out what combination of discs, protocol (ISCSI, NFS…) works best for me.

As usual I will begin from top and end at bottom of the rack, starting with the front:

First you can see is a Lenovo Tiny. This is a small W10 client and is used to connect external drives to it to copy data from A to B. As this rack is not located at my home I often need to transfer large amounts of data. For that I use simple external discs and move them between the locations. (Internet speed in Germany is not fast enough to copy things online).

This is followed by my first compute node. It is a Fujitsu Celsius 620 (E3-1280v2, 32GB, Nvidia Quadro M2000) with ESXi on it. It is some kind of special for me as it hast a GPU in it for hardware acceleration. Main use case is Plex/Emby but I didn’t had the time to do a new install of Plex with a GPU VM.

Next is a blanc space, maybe I will add a dedicated Nextcloud machine there.

The next server is my SAN host. It is a Dell R320 (E5-2430, 48GB RAM) with 6x 1TB Disk Raid Z1 and 2x 2TB SSD mirror. The OS is TrueNAS and this host provides ISCSI shares to my ESXi nodes in my CX400.

After SAN follows compute and this is the biggest change in this revision of my homelab. I replaced my Quanta S210-X22RQ with a Fujitsu CX400 S2 with four CX250 S2 nodes. Currently I’m using two of them and maybe I will use the oder two for redundancy but as power is expensive I will use just two of them. One node is for general purpose with two E5-2680v2 (10×2,8GHz) and 256GB RAM, the second one uses two E5-2637v2 (4x 3,5GHz). The second one has way less cores but a higher frequency and higher boost. This is for applications that need higher performance cores but less cores in total (e.g. Minecraft Servers). Both have a 10G NIC with two ports, one for data traffic and one dedicated for SAN. Currently they’re running ESXi 7.0.x.

Last one is Storage but this time file-storage. I merged my two storage servers into one. Some of the drives are just moved into the new chassis, some data is migrated. Mainly my media share was moved from a striped 6x 4TB Raid Z2 with two arrays to a 4x 16TB Z1. Less disks, less power consumption and higher capacity.

On the back site it is getting a little bit more spaghetti, but that’s where all the connectivity is.

ToR is still my Netgear GS752TXS for 1G connectivity and 10G Uplinks but now it got support by my „new“ Brocade VDX6730-32 for internal 10G ports.

Now all my servers have one or two 10G ports available. As mentioned the CX nodes have two 10G Ports for separated data and SAN traffic. All SAN Ports are working with an MTU of 9000 and the rest standard 1500. Everything should have enough bandwidth now. It try to get a better picture of the back as soon as I’m onsite and don’t forget it.

Another thing I prepared this time is some switch-port documentation. In the past the environment was more dynamic and I did more changes in a shorter timeframe but now it gets less dynamic and a more static. With that it happens that I’m forgetting some port configuration and have to lookup the config more often. To get a short overview of my ports I started a documentation for my ports to see what is connected and what configuration is active on my free ports. Saves a lot of time in case I must want to add a cable. Unfortunately I didn’t found the correct shape file for my exact model for the Netgear so I needed to use a similar switch model. For the Brocade I found some shapes but they were not really usable so I decided to just use an image of the front. Not perfect but it works.