Yeah, my Proxmox cluster hasn’t even been fully operational for 48 hours yet, and I’m already writing an article titled “Mistakes Were Made.”
I genuinely wish I had known some of this beforehand, because it would have saved me time and, quite possibly, money.
Storage
I am already running out of storage. Yes. Not even 48 hours in, and I am actually running out of storage. HA storage, to be exact.
“But you’re only at 50%! There’s loads of room!”
Yeah. No.
I still have a huge VM left to migrate. That VM has a data disk of just over 1 TB. And because this is Ceph with replication enabled, that single terabyte does not consume 1 TB of storage. It consumes roughly 3 TB.
As the screenshot shows, that one remaining migration alone would push the cluster straight into uncomfortable territory.
“But then the drive would still have about 20% free space. That’s huge!”
On a regular disk, sure.
On a Ceph cluster, absolutely not. Twenty percent free space is not breathing room; it is a warning sign. Ceph needs headroom to rebalance data, recover from failures, and generally keep itself healthy. Once you get close to that range, things stop being comfortable very quickly.
So yeah. I made one fairly big mistake here.
I genuinely thought I had more than enough headroom with 6×1.92 TB. On paper it looked fine. In reality, once replication, HA requirements, and one large remaining VM entered the picture, that headroom evaporated much faster than expected.
Now what?
First, let’s get the obvious joke out of the way.
Kioxia, Samsung, or any other enterprise SSD manufacturer: you could really lend me a hand here.
But more seriously, I really only have a few options:
- Accept that my Nextcloud instance won’t be running on my HA platform anytime soon.
- Buy 3x 1.92 TB drives and add them to the Ceph cluster, effectively increasing usable capacity by roughly 1.92 TB.
- Buy 3x 3.84 TB drives and add them to the Ceph cluster, effectively increasing usable capacity by roughly 3.84 TB.
And that is about it.
And adding three more drives introduces yet another problem.
Both NOX cases are already full. There is simply no room left for an additional drive.
Which means there’s only one conclusion left: another case it is.
How am I going to fix this rather fundamental issue?
Not by rushing it, that’s for sure.
Brain out