Moving from VMware to Proxmox for Game Studio - Ceph Storage Design Feedback Needed

Hey r/proxmox,

Like many others, we're finally making the jump from VMware to Proxmox for our small video game studio but want to make sure we're setting things right.

Our primary need is a robust and performant storage solution for our production environment, which currently hosts around 40 VMs and requires roughly 40TB of usable storage. We're planning on using Ceph for this.

I've put together a potential hardware configuration and would love to get your feedback and advice. We're open to any suggestions or changes you think would be beneficial.

Here's what I'm thinking:

• Nodes: 3 x SuperMicro AS -1115CS-TNR
• CPU: AMD EPYC 9124 (16-Core) per node
• RAM: 8 x 32GB DDR5 5600MHz ECC RDIMM per node (256GB total per node)
• NVMe (OS): 2 x 480GB M.2 7450 PRO NVMe PCIe 4.0 per node
• NVMe (Ceph OSDs): 5 x 7.68TB 2.5" CD6-R NVMe PCIe 4.0 SSDs per node
• Networking: 1 x Supermicro 10-Gigabit X710-BM2 per node (for Ceph cluster) + 1 x AIOM: Supermicro 1-Gigabit i350 2 x RJ45 (for VM traffic)

My main questions and concerns are:

• OSD Count: Is 5 x 7.68TB NVMe drives per node a good starting point for our 40TB usable target?
• NVMe for OSDs: Is the CD6-R a suitable choice for Ceph OSDs? Any other recommendations for high-performance NVMe drives?
• Overall Architecture: Does this setup seem balanced for our needs, or are there any potential bottlenecks or areas for improvement?

Any insights or experiences you can share would be greatly appreciated! Thanks in advance for your help.