THE DEATH&TAXES PROTOCOL :
30A breaker, 10AWG branch line, and a NEMA L6-30R outlet (240V, 30A, locking). Cheap used datacenter PDU with a NEMA L6-30P plug. "Good for a solid 5.76KW."
We're looking at setting up 2-3 server racks worth of GPU rigs, and powering them all off 220. Each rack will be using ~8kW (possible more), which will be 35+A if we use 220.
So lets say we use 9kW per rack. We'll just do two of your setups? 6x 30A 220 breakers. 10ga wire run ~100ft to our room, and terminate to 6x L6-30R outlets. 2x Compaq 24A PDUs with L6-30R plugs per rack. Then each PDU will power 3 rigs, with 3 of these
, and 3 of these
. Each rig will use an ATX PSU, and a custom server PSU.
The problem I see is that the breaker is 30A, but is being limited by the PDU to 24A, which if run at 80% load is only ~20A. Is it a better idea to install a single 50A 220 breaker per rack, run 8awg wire to 2x L6-30P, and then run both of the 24A PDUs per rack to those? The actual draw through the PDUs wouldn't change, would still be at ~20A each, but now you've got 8awg wire carrying 40A to a 50A breaker, rather than 2x 10awg carrying 20A each to 2x 30A breakers. Does this make sense?
Although I suppose if we did that, we're really limited to 40A per rack. If we went with the 2x 30A, we could upgrade the 2x 24A PDUs with 4x 16A PDUs. As long as each of the 16A PDUs were kept below 13A, the draw per breaker would be ~25A, and the total draw per rack could be increased to 52A, or 11kW.
So if we're only pulling 8-9kW per rack, we could get away with a 50A breaker and 2x 24A PDUs? But if we wanted to pull more, splitting it up between multiple breakers would give us more headroom?
I also realize that we're going to have to run another breaker for our AC equipment, but that's a topic for another day. And I'm not really an electrician, I'm just trying to figure this all out. Any suggestions welcome!
Edit: you may already know some of this the post became more of a general guide as I kept type.
So a couple things. The first is that in the use despite the common usage of "110" or "220" nominal voltage is 120 & 240. You should however stick a multimeter into an outlet to find out. You can test a 120V outlet, the 240V is going to be double that. Unless you are at the far end of distribution the measured voltage at the outlet is probably closer to 120/240 then 110 /120. Still you should adjust the power capacity numbers based off your actual voltage.
The second is that the 24A per PDU already includes the derate.
The circuit (30A breaker, 10AWG wiring, NEMA L6-30R outlet) is rated for 30A peak load. Continual load is derated to 80% of that so 30A * 0.8 = 24A. A code compliant circuit using a NEMA L6-30R receptacle (proper name for "outlet", outlets are actually on the PDU) is good for 24A continual or 30A peak. That is why the PDU lists 24A, you don't have to further derate that. Putting it together with 240V means each PDU is good for 5.76KW of power per pdu.
Third. I am sure you have double checked but most PDU don't use C19 connectors. I assume your server PDU is "beefy" and pulls more than 12A? Just don't want someone ready briefly to be confused. Almost all PSU have a C13 connector. PDU have a C14 connector so the cable most people will need is one C13 to C14 cable per device. Are there other options?
Well there are but I think you find any other option is either more expensive or not up to code. 240V, 30A circuit just comes down to the most economical choice. The used PDUs are dirt cheap, the outlets are nothing "exotic" (even home depot probably carries a NEMA L6-30R is look hard enough) and while 10AWG isn't "fun" to work with it is still a lot easier to manage then the larger cables. For example the standard connector for a 50A circuit is a NEMA L6-50R which doesn't exist except on paper, as safer pin and sleeve solutions are used. One common alternative is the California standard (become widely used in movie industry) http://www.stayonline.com/detail.aspx?id=6272
Yeah that is the plug only. $70. A matching outlet is another $50 to $70. To make a custom cable which converts a 50A connector into 2x NEMAL L6-30 plus the 50A outlet is probably going to run you $250+.
So I am not saying "the 30A route" is the only possible way to deliver power, I am just saying trust me I did the research and cost per KW you aren't going to find a cheaper, code compliant solution then using 240V 30A circuits and 30A (derated to 24A) used PDUs.
So if you need <5.76KW of power. Install one 30A breaker, make one 10AWG run, install one NEM L6-30R outlet, and get one matching PDU*.
So if you need <11.52KW of power. Install two 30A breakers, make two 10AWG run, install two NEM L6-30R outlets, and get two matching PDUs*.
So if you need <17.28KW of power. Install three 30A breakers, make three 10AWG run, install three NEM L6-30R outlets, and get three matching PDUs*.
So if you need <23.04KW of power. Install four 30A breakers, make four 10AWG run, install four NEM L6-30R outlets, and get four matching PDUs*.
* I recommend using the APC AP9571. There is nothing magical about it but APC made millions of them so there are tons of used ones on ebay for dirt cheap.
An ebay listing may say 208V but it works fine at 240V as well (datacenters are usually 208V which is why it lists that).
30A, 24A derated (5.76KW usable)
NEMA L6-30P plug to match your economical NAM L6-30R recptacle.
Has twelve C14 outlets.
Just get some C14 to C13 cables from amazon or monoprice (<$5 ea) to connect from the PDU to each PSU.
If the distance from your breaker panel to where your rigs (and NEMA L6-30R receptacles) is long you may want to consider a subpanel. How this would work is that in your main panel one or more high current breakers is used and a single run with large gauge wire is run to a sub panel located near the rigs. The sub panel then has say 6 30A breakers and short 10 AWG runs to NEMA L6-30R receptacles.
If you go another route just remember you can pull the full rated current of the PDU (as it has already been derated). However the circuit (receptacle, wiring, and breaker) has to be designed such that the continual load is only 80% of the circuit peak capacity. Some PDU labels will list both capacities (30A, 24A derated). This is why you will find PDUs generally are 12, 16, 24, or 40 Amps.
12A PDU is 80% of a 15A circuit (15A outlet, wiring, and breaker)
16A PDU is 80% of a 20A circuit (20A outlet, wiring, and breaker)
24A PDU is 80% of a 30A circuit (30A outlet, wiring, and breaker) <- The sweet spot in terms of $/KW installed
40A PDU is 80% of a 50A circuit (50A outlet, wiring, and breaker)
Another piece of advice is to start small (i.e. start with enough gear to fill one 5.76KW "unit"). Speaking from personal experience when you start getting into the tens of KW of power range everything becomes more of a challenge especially cooling. It is much easier to design a single efficient layout (say half a rack) and then scale out, rather than buying gear needing 30KW+ and then realizing the engineering, power, and cooling challenges are no joke.