What should I do with my life?

[DireWolfM14]

Sorry if this barely on topic here, but I don't want to post this in the Off-Topic board because that board is a shitshow.  Besides, my Bitcoin node is affected by these decisions and I want to hear from the technically astute members who peruse this board.

My current situation:
I have 3 EOL enterprise grade Dell servers; 2x PowerEdge R630 and one R730xd.  All are running Proxmox as the base and each has a RAID-5 array of 3x 256GB SSDs for the OS and VM storage.  All have PERC H730 RAID controllers, which are native hardware RAID and can only emulate HBA mode (not recommended for ZFS.)  I chose them because they have battery backed NVram to protect against power outages during data migration, and tend to get great reviews for being very reliable and robust.  It's worth mentioning that I've never had any issues with them.  I also have in my possession three PERC H330s, one for each server in case I want to switch one (or all) of the servers to pure HBA (or IT, or JBOD, or whatever you want to call it) mode.  The servers are clustered together so I can manage them from a single Proxmox web interface, but do not share pooled storage.

The R730xd is the file server, currently running a Debian VM for Samba, and another for the Nextcloud Docker container.  It has a RAID-6 array of 8x 2TB SSDs for shared data, and a RAID-6 array of 8x 1.2TB HDDs for a shared backups location.  One of the R630s has a RAID-5 array of 4x 1TB SSDs for blockchains, and is running one VM that includes Bitcoin, Fulcrum, LND, and Monero, and anther VM for the Mempool Docker container.  The last R630 is running various VMs and LXCs for WireGuard, media servers, torrent nodes, and sandboxes.

I'm a relatively experienced Linux user, and feel very comfortable with root level CLI.  Recently, I've been reading a lot about Proxmox's CEPH ZFS and it's data integrity features and clustering the Proxmox nodes with pooled storage, which then led me to reading more about TrueNas.

My question is posed to those of you who have experience administering systems; what do you thing are the advantages of my current setup, and do you recommend migrating to the ZFS by using Proxmox?  Also, any experience with TrueNas, and would it be worth migrating my file server away from Proxmox in favor or TrueNas?

[Mia Chloe]

Honestly, your setup already sounds better thought out than a lot of homeland actually for me I think the biggest gains from moving to ZFS on Proxmox VE would be things like checksumming, scrubs, snapshots, and maybe if you wanna replicate but you'd also be giving up a setup that is already proven and stable in exchange for something more complex and you're still gonna be migrating.

As for TrueNAS I'm barely familiar with it but I think it really shines when the machine's main purpose is storage In your case keeping everything under Proxmox actually feels like  better choice except you're specifically chasing ZFS features or planning to make your storage larger

[achow101]

I don't think TrueNAS will be doing anything for you if you're already using Proxmox. A couple years ago, I actually migrated off of TrueNAS to Proxmox because TrueNAS was such a royal pain in the ass for self hosting any services that didn't have premade charts (service configurations) already.

My storage configuration uses ceph for storing VM images, and I have a separate NAS doing a crazy High Availability thing with ZFS. I would not recommend this configuration. ZFS is still nice though for its snapshotting capability as that makes backups very simple. But it doesn't work too well in a high availability system. I would say though that it is better than doing hardware RAID since the RAID is configured within ZFS itself so it does not depend on the survivability of your RAID card.

Ceph for VM images quite nice. Since it is erasure coded, before I take a node offline for maintenance, all of the VMs can be quickly and easily migrated to another node with no risk of data loss. I think this is better than RAID or z-RAID as it can handle if a node suddenly goes offline. But it does also use the storage less efficiently.

[NotATether]

ZFS is cool but be careful with OS support. Some distros dropped support for ZFS so if you insist on being able to use it, it may have to be kept on bare metal.

I'm of the belief that hardware never truly "dies" (until it can't POST, that is). So since you're not planning to migrate away from these units, I guess leave things the way they are? Unless you are happy with rebuilding after backing up the affected disks and partitioning them with ZFS. A guy like you probably has even more arrays just for disk imaging, anyway. 

[DireWolfM14]

I don't think TrueNAS will be doing anything for you if you're already using Proxmox. A couple years ago, I actually migrated off of TrueNAS to Proxmox because TrueNAS was such a royal pain in the ass for self hosting any services that didn't have premade charts (service configurations) already.

I've read others say similar things on r/homelab, but they seem to be the minority.  Most of the folks there sing TrueNas's praises.

My storage configuration uses ceph for storing VM images, and I have a separate NAS doing a crazy High Availability thing with ZFS. I would not recommend this configuration. ZFS is still nice though for its snapshotting capability as that makes backups very simple. But it doesn't work too well in a high availability system. I would say though that it is better than doing hardware RAID since the RAID is configured within ZFS itself so it does not depend on the survivability of your RAID card.

I have no experience with CEPH, but I assume it would still allow me to pick specific disks to assemble arrays the same way the PERC does.  Once those arrays are assemble in CEPH, can I still pass them through to VMs?

I'm not concerned about the survivability of my hardware.  Dell replacement parts are easily had for very cheap on eBay, and I have a stash of the critical components just in case of failures, including a spare perc H730.

Ceph for VM images quite nice. Since it is erasure coded, before I take a node offline for maintenance, all of the VMs can be quickly and easily migrated to another node with no risk of data loss. I think this is better than RAID or z-RAID as it can handle if a node suddenly goes offline. But it does also use the storage less efficiently.

I think this might be the only real benefit I would enjoy.  It would reduce downtime and speed up migration in case I want to take a node down for repairs or maintenance.  The only node that really wouldn't benefit from this is the DPR730xd, since it physically houses the file server drives.  And in all honesty, that's the one I worry about the most.

Some weeks ago I asked for some help on r/homelab, and that's when I started down this rabbit hole.  Maybe I just need step away from the internet for a few days and this existential crisis will sort itself out, lol.

Anyway, thanks for sharing your insight.  This is the exact kind of experienced knowledge I was seeking.

I'm of the belief that hardware never truly "dies" (until it can't POST, that is).

Even then, it's probably just a graphics controller, a CPU, or a RAM module that kicked the bucket.  Worst case scenario, it's the motherboard, and everything else can be recycled.  I've been tinkering with hardware for almost 4 decades, so yeah, there's not much that scares me anymore.

So since you're not planning to migrate away from these units, I guess leave things the way they are? Unless you are happy with rebuilding after backing up the affected disks and partitioning them with ZFS. A guy like you probably has even more arrays just for disk imaging, anyway.  

I have an old nas with RAID 1 spinners tucked away in my gun safe that I use for redundant backups, so the migration would cost me nothing but time.

[achow101]

I think you are thinking that Ceph and ZFS are the same thing, or somehow intertwined with each other? In actuality, Ceph and ZFS are 2 very different things. Ceph is really a data storage system, not a filesystem in the traditional sense of a filesystem. My understanding is that when you run Ceph, it takes over entire devices and writes things to them at the device level. I believe there's an overlay that can make it also present traditional filesystem things, but I've never used it.

On the other hand, ZFS is actually a much more traditional filesystem like ext4, btrfs, etc. With ZFS, you partition a drive as you would for a normal install, then make put ZFS onto the partitions. It does additional stuff with datasets and volumes, but that is all presented to you as essentially more partitions and in very much a typical filesystem manner.

Proxmox uses both ZFS and Ceph. IIRC Proxmox by default installs itself onto a ZFS filesystem. Then if you want to use Ceph, you have to have additional unused drives that can be assigned to Ceph.

I have no experience with CEPH, but I assume it would still allow me to pick specific disks to assemble arrays the same way the PERC does.  Once those arrays are assemble in CEPH, can I still pass them through to VMs?

Since Ceph is not a traditional filesystem, I don't think you would be doing it like that. What you would actually end up doing is creating virtual machine disks, or possibly virtual block devices, that can be passed into a VM. But these would all live on top of Ceph, not passing the devices themselves directly into VMs.

With ZFS, you could pass through the devices directly after partitioning them and setting up zpools and datasets.

But also with ZFS,  you could create ZFS volumes that present as block devices (i.e. themselves have a different filesystem and partitioning) that can be passed through to VMs, kinda like what I described with Ceph (I think). But I've also never done that.

[Cricktor]

What a strange "Subject", yellow-press click-bait style, sheesh. I don't like this, specifically because nobody searching e.g. with keyword "Proxmox" in titles would find this here. Anyway, just saying, "Subject" titles sometimes are really terrible...
I know you can do much better. But, hey, not really judging!

My first thoughts were: your hardware raid controllers could be a single point of failure but then later you wrote you have apparently enough spare parts or they're easy to get from ebay. As long as you're not out of stock of those, it should be just fine when they're good and worth it.

I just recently started tinkering with Proxmox, so no expert yet. Working on it...  

My experience with ZFS is pretty limited also. It has some interesting features, but is memory hungry^{from what I've read so far} with some of them, if you still want decent performance from it. And ZFS seems to have endless tuning knobs which could make it hard to optimize.

And Ceph is another beast, it seems (to me). I perceive it as a software-defined distributed cluster storage system, able to run on standard low-cost servers, no need for proprietary storage array hardware to scale it to your needs. Can't say to know much about it, either, mainly because I didn't need it and wasn't bored enough to chase it. (Well, well, too many other toys to tinker with...)

My question(s) regarding your setup are: are you missing something or do you feel something isn't working good enough? What's the itching that you're looking for new tinker areas?

I know too well how a homelab is a constant play field and one is rarely satisfied even when all is running pretty smooth.

Excuse me that I've more questions than answers, but sometimes questions are better than answers because they fuel more to think about stuff.