Assuming each of the GPUs runs at x16 lanes, I'm a little surprised there's enough PCIe lanes for all of that hardware. I would think Epyc would make for a more logical and cost effective choice in this particular setup. A single Epyc could handle 8 GPUs with x16 lanes. I know server-grade hardware tends to only use x8 lanes, but I'm sure these GPUs can soak up a lot of bandwidth.

They use NVLink, and not PCIe lanes. The only PCIe lanes are for those expansion slots (10gigabit network and such)

Nvidia had the original dgx now they have the dgx2. I'm guessing it would compete with that. However, unless it's considerably cheaper I would go with the dgx2 mainly because their solution is prob a bit more mature and works with their Tesla v100 gpu better. Cuz when I may looked the new dgx2 with 16 x v100 with 32 gb of hbm2 memory is about $400,000