Your system ~3+ years old or so? Your story screams DMI 3.0 or similar PCH/PCIe switch saturation issue. DMI 3.0 caps at ~3.5GB/s but about 1.5-1.7 when bidirectional, such as drive to drive. If 100% reads hit about 3.5 then you’ve all but confirmed it. ~1.5GB/s bidirectional is a super common issue for a super common hardware combination.
It’ll happen if your U.2 ports route through DMI 3.0 PCH/Chipset/PCIe switch rather than directly to the CPU PCIe lanes. Easiest to just check motherboard manual, but you can use hwinfo to inspect the PCI tree and see if your U.2 ports are under a “chipset” labeled node. You might have different ports on the mobo that are direct, or possibly bios changes to explore. Sometimes lots of options, sometimes none. Worst case a direct PCIe adapter will resolve it.
Actually more than 3y old. My servers are EPYC Gen 2 and Gen 3 mostly.
But I don't think it is a hardware thing, as I see in CrystalDiskMark (which I believe bypasses the Windows write cache) performances that are close to the SSD specs. But it is when I do windows copy, the performance goes down the drain.
And it's not just parallelism. If I copy one file from a fast nvme to fast nvme, it caps at about 1.5GB/s. If I copy two files in parallel, they seem to split that speed, even if file1 is copied from diskA to diskB while file2 is copied from diskC to diskD.
Ahh. EPYC doesn't use DMI, so there’s an easy elimination, and they have enough PCIe lanes so that switches only come up with boards supporting oodles of drives. It’s still worth checking your mobo manual to make sure there isn’t a specific mention of port selection related to PCIe switches, or a silly bios option.
There might be some confusion that’s holding you back on diagnostics though. If a PCIe switch was in the path and causing the issue then there’s no meaningful difference between “parallel” and “bidirectional.” The nature of the switch is that all the traffic goes through it and it has a top speed and that’s split among the operations. Read or write to a single drive gets full bandwidth, but copying from one to another is read and write, so each part gets 50%. Even on the same drive, write/read and write/read also get 50% each. Include other drives on the same switch and divide it again. “And” = divide.
Your platform makes that specific issue less likely, but hardware can be a bit more quirky than you might be giving it credit.
Of course you’ve turned off windows defender real-time scanning, but otherwise I can’t think of an on-by-default reason for Windows Server to cause that issue, but it isn’t inherent in the OS. I’ve used multiple versions in dozen GB/s arrangements - 1.5GB/s was beatable 20 years ago. There’s something going on with your settings or hardware. Good news is it might be fixable, bad news is you’ll have to go trekking around to find the issue. :/
No PCIe switch involved, every SSD is directly connected to the PCIe ports, the MB supports bifurcation. Observed on H11SSL-i and H12SSL-i motherboards with various models of SSDs (mostly Intel, Micron and Samsung). Windows defender switched off.
Hyper-V is on though. And I understand that when you switch on Hyper-V, even the host OS is really a VM on top of the hypervisor. So I wonder if that's not contributing (but disabling Hyper-V is not an option).
When you say 1.5GB/s was beatable, how do you measure it? Windows copy (or xcopy) or some benchmarking software?
It’ll happen if your U.2 ports route through DMI 3.0 PCH/Chipset/PCIe switch rather than directly to the CPU PCIe lanes. Easiest to just check motherboard manual, but you can use hwinfo to inspect the PCI tree and see if your U.2 ports are under a “chipset” labeled node. You might have different ports on the mobo that are direct, or possibly bios changes to explore. Sometimes lots of options, sometimes none. Worst case a direct PCIe adapter will resolve it.