Request for Discussion: proposal for standard modular rack miner

[chiguireitor]

with >60% humidity points here on my country, high density only happens with water blocks).

 I don't understand that statement, heat sinks don't care about humidity to cool, they care about temperature differential. They do NOT use evaporative cooling like humans do.

[...]

Humid air increases static pressure on fans, lowering considerably the air pushed through the heatsink (that means CFMs go down by a whole lot).

Waterblocks OTOH don't rely on changing ambient conditions (if you're going closed loop).

There's also the possibility of open loop cooling with chillers and such, but i'm not a fan of that kind of cooling tech.

[1715143491]

1715143491

[1715143491]

1715143491

[1715143491]

1715143491

[sidehack]

It also makes a difference if you want to use evaporative coolers to knock your 37C ambient down a bit.

The real point he's making is, for approximately equatorial locations air cooling is not easy to accomplish so some provision for waterblock installation is probably necessary.

If you have a string design with a dozen local ground planes all at different absolute potentials, you do not want a heatsink spanning them without isolation. If that's not the case, sure using powerpegs is probably great. I wouldn't design a standard heatsink to require them, but if you can use 'em affordably without catching stuff on fire I wouldn't rule it out.

If you're taking power in through cables instead of a backplane socket, that's not another argument in favor of using a backplane.

[NotFuzzyWarm]

My thoughts,
1) Make the controller aware of power quality via the signal all psu's provide.
2) Since most server PSU's provide +5v control power, buffer that with a small supercap & use it to power the control board. Size the supercap to allow the controller to run for say 10-15sec and use that time to do a controlled shutdown and possibly a restart if the power come back fast.
3) Is it possible to just use a internal LAN via a multi port switch? Seems to me it would be a lot easier for addressing the boards & data xfr vs the common SPI bus from board to controller and there are chips galore out there made just for internal LAN coms.

[Meech]

I know copper is expensive now but aluminum retains heat for too long.  I'm sure you could reduce the heatsink size by using a utilizing a more efficient copper heatsink.  Copper may have the same properties but there must be something other than aluminum.  Could you have rear pull fans and  fans on the heatsink pulling of it's heat directly?  That would limit how many boards you could place in the case but it would be quieter and possibly allow for more dense placement of chips on each board.

[-droid-]

yeah the modular design would solve so many issues.. something similar to the S2 backplane with interchangeable cards not depending on manufacturer

[chiguireitor]

I know copper is expensive now but aluminum retains heat for too long.  I'm sure you could reduce the heatsink size by using a utilizing a more efficient copper heatsink.  Copper may have the same properties but there must be something other than aluminum.  Could you have rear pull fans and  fans on the heatsink pulling of it's heat directly?  That would limit how many boards you could place in the case but it would be quieter and possibly allow for more dense placement of chips on each board.

Copper has WAY better thermal conductivity than Aluminium, almost double, but the weight is a great deal too, so i doubt anyone is going to do big heatsinks with them.

[sidehack]

So. The day's discussions.

Chiguireitor, we spent most of today figuring out how to do things well with S1-spec heatsinks. If we maintain the 4U case size, we can comfortably fit seven heatsinks of S1 dimension and screw pattern in with room alongside for power supplies. Boards would be too tall to allow for PSUs mounted at the top how we originally planned.

This would allow us (and others) to build a single standard S1-sized board which could act as an upgrade for S1 miners or fit into the rackable machine. It also has the side benefit of being able to build S1-formfactor standalone miners, since we'd already have boards and heatsinks. All that remains to acquire is fans, controllers and a bit of framework.

For GekkoScience specifically, it would mean basically merging the Spec1 and Spec2 designs into a single board. The intent for Spec1 being a quarter S1 instead of a half S1 was that the board could be run standalone as its own 50-150W machine. The Spec2 was almost from the beginning intended to be rack-buildable. If we divide the market into roughly three sectors using Bitmain products as examples, we have the U3, S1 and S2. Our Spec1 would have fit U3 and S1 sectors and the Spec2 fit S2. However, if we make this change, we'd have the Spec1 as a single 30-chip board meeting S1 and S2 sectors and design a different product for the U3 sector. I think we're okay with that plan.

Designing boards for the rack-standard as based on S1 standard also helps cement and maintain S1 as a standard for its own market sector, which I don't think anyone is going to really argue against. This also helps maintain driver compatibility, as a single driver per board design would work for both S1-refit and rack-miner installations.

Being as a plethora of waterblocks already exist for S1 standard, an S1-standard-derived rack machine could be refit with S1 waterblocks pretty readily.

The problems we're coming up with are geometric in nature. There's plenty of room widthwise to fit seven boards. We could probably do eight, but seven gives better power headroom and more efficient per-board cooling off 2400W of available power. The problem comes in when we want to fit supplies and fans together. The width of a pair of DPS1200 supplies plus three 120mm fans will not work in a 17.5" OD rack case. It could be done if we switched to something like the Emerson 1200 that Spondoolies uses, which are also fairly expensive and still pretty tight.
If we want to keep a DPS1200 (whether that exact supply, or something with its dimensions) we've got three choices, far as I can tell.
1. Put the supplies at the front. This makes the power cord readily available but also gets your PSU cooling air blowing out the front. At full power you could be venting over 200W per machine out into your cold-aisle space.
2. Recess the supplies inside the machine. This makes them inaccesible from the outside for replacement, which shouldn't matter to anyone except Spondoolies fanboys since nobody else has ever built a machine with ready-swappable PSUs. It does also, however, make plugging in the supplies more cumbersome - we either let you have to thread your cord carefully into the socket about 2" inside the machine, or we put on an external socket wired to an internal plug into your PSU. One option removes convenience and the other adds a fair amount of cost.
3. Use a matrix of 80mm fans across the back instead of 120mm fans. Putting in 2x4 80mm fans instead of 3x 120mm fans will give us an additional ~1.5" of horizontal room to play with, making space for PSUs and allowing some gap/play between fans. Small fans will have to spin faster and so will make more noise. Sourcing more fans will probably also increase cost.

Currently I'm in favor of using 80mm fans across the back, because it's the least cumbersome option and, though it results in more noise, the purpose of rack gear has never been "silent running".

[NotFuzzyWarm]

On the psu's:
Or - dedicate a couple U of rack height and make/sell a PDU case for plugging in either the DPS1200 or better yet the IBM 2kw+ psu's either independently powered or wired 1+n to power rack miners above and below each PDU.... Please please please

For a 2400w load 3 of the HP's wired for current share would be marvelous allowing true hotswap of a deunct supply (haven't had one yet though) while the other 2 take up the load. With all 3 in, gives a nice margin for the supplies right around the 80% load butter zone.

[chiguireitor]

So, if i'm getting it right, the case would host things like these?

(Removed front and top face to make it clear what's hosted inside)

Front view


Rear view


Legend:

• Light grey: Case
• Green: Hashboards
• Violet: Heatsinks
• Blue: PSUs
• Yellow: Controller
• Black: 80mm Fans

As i see it, it only fits 6 S1 sized boards, so it could be tops 3.5 Kw of heat to be dissipated, with 80mm fans you could move enough air for them, but it has to be calculated as the case would be really tight.

Am i right on this one, or did i get it all wrong? I could do a fluid model to see if this thing would work.

[sidehack]

I was thinking something like a 2U with pairs of DPS2K would be good for someone wanting to run dual-sided waterblocks. With BM1385 boards you could push about 30TH from 11U like that. I'd like to insist on internal supplies the default option but it shouldn't be difficult to circumvent them for a slightly more custom setup like that.

Chig, I can honestly say there's not a single thing about your 3D model that is correct. Either I suck at describing things or your English needs work - probably both.

The boards are mounted vertically. Their heatsinks are screwed to the bottom and stand up with their fins horizontal. This gives room for potentially 8 blades but seven is a bit better for power and space. These blades would take up approximately 14 inches of the 17.5 inch width of the inside of the case; the remaining space would have PSUs at the back and controller at the front. That's the current plan.

The original description from the first post had 8 blades spanning the full width of the case and occupying the bottom 5.5 inches. The top 1.5 inches had a pan across it with PSUs, controller and all cabling on this pan.

[chiguireitor]

I was thinking something like a 2U with pairs of DPS2K would be good for someone wanting to run dual-sided waterblocks. With BM1385 boards you could push about 30TH from 11U like that. I'd like to insist on internal supplies the default option but it shouldn't be difficult to circumvent them for a slightly more custom setup like that.

Chig, I can honestly say there's not a single thing about your 3D model that is correct. Either I suck at describing things or your English needs work - probably both.

The boards are mounted vertically. Their heatsinks are screwed to the bottom and stand up with their fins horizontal. This gives room for potentially 8 blades but seven is a bit better for power and space. These blades would take up approximately 14 inches of the 17.5 inch width of the inside of the case; the remaining space would have PSUs at the back and controller at the front. That's the current plan.

The original description from the first post had 8 blades spanning the full width of the case and occupying the bottom 5.5 inches. The top 1.5 inches had a pan across it with PSUs, controller and all cabling on this pan.

I think vertically you cannot fit 7, as the default heatsink of the S1 takes effin 3.5cm

Here's a model with vertical mounted blades



Also, i corrected the default heatsink width a bit, as i used 3cm off the top of my head, but just measured it and it is 3.5cm

[chiguireitor]

Btw, i'm assuming 19in rack width and 4U height (which seems like the minimum for the S1 form factor vertical mounted blades).

IMHO a side-mounted-hack of the blades would be better spacewise for the case, however, it is yet to be seen if that much heat can be handled efficiently by the case.

[chiguireitor]

You could also go the shorter way, and put the blades horizontally with centered PSUs with the sturdy Dell 750w models, and one RPi on top controlling the whole affair like this:



But then, you might want to replace the 80mm fans with with two badass 140mm fans to end up with something like this:

[sidehack]

The current width of the S1 heatsink doesn't matter, as we won't be using them.

The render with vertically-mounted blades is closer but still not right. Seven blades, heatsinks sized to fit, with about 3.5 inches to one side for PSUs and controller.

[chiguireitor]

What width will have the heatsink? to render a model

[Finksy]

Here try this out:



Same legend as Chig.  Had to make it 18" width in order to fit 7 hash boards with the dual heat sinks as spec'd, in practicality you might be fine with 17.5".  All fans 120mm, I wouldn't personally bother with 140mm fans. Keep it simple, buy in bulk.  

The only way I'd practically see you with 8 hashing boards @ ~300W each would be to make it taller than 4U to accomodate a single 2880W PSU up top. Even with 2x DPS 1200 stood up with a breakout board/backplane at the rear of the PSU shell, you won't have enough room.  

7 Hashboards of BM 1385's @ 300W each will still be ~9.7 TH/s and around 2230W at the wall.

Edit: Don't mind the crappy render, it was my first time.

[chiguireitor]

Omg it seems i was blind or something, just read it was 7-8 Blades, not 7-8 S1s

That gives a LOT better disposition... Will try a jab at it tomorrow

[sidehack]

If we go exactly S1 dimensions, we probably aren't going to be able to double-side heatsinks. Individual boards could still have chipsinks on 'em (like the S5+) but with the limited room above the heatsink and no extra room past the end of the heatsink (the original 2" extra at the front) rules out a lot of what could be done with a single-sided board and still have room for full heatsinking on both sides.

Two DPS1200 stacked vertically are pretty much exactly 7" tall, so can't fit inside a case whose outer dimension is less than 7". The only real way to do it is to set them horizontally one on top of the other, or vertically side-by-side. This takes up between 3.25 and 3.5 horizontal inches depending on which option you take.

The outside width of the case should not exceed 17.5 inches to ensure it fits comfortably in and out of racks, where I believe the defined standard is 17.75 inch inside. 18AWG steel knocks off about fifty thousandths per layer. If we assume we get 3.25 inches for PSU and 5 layers of steel we have 14 inches. This gives us 2 inches of width per board and heatsink. I figure 1.5 inches for heatsink will give half an inch for PCB, components and gapping to get boards in and out.

Unfortunately, 3x 120mm fans is more than 14 inches. There's room across the front to do it since the PSUs don't go all the way up there. 4x 80mm PSUs wide covers 320mm or 12.6 inches, so there's room for play at the sides and inbetween. Two fans high is 6.3 inches, plenty of room left over in the 7-inch height (or, if we say 3 layers of steel and 2mm under 7 inches for shelf play in the rack, 6.75 inches inside) for fans to be comfortable. With that much fan on the backside you might not even need front fans to push.

Finksy's model is pretty close to what I'm thinking. Maybe tomorrow I'll run out a sketch and see if our scanner wants to work on my machine.

If we went to an S1-formfactor board, we'd probably shift the data cabling back to stock USB - no sixth wire to control link light. Sounds like software fan control is the best way to go, so the internal hub board would have 8 devices, where seven are boards and the eighth is a built-in controller that handles link lights and fans driven by cgminer integration. We'd probably design boards to have holes/pads for both USB jack and a pin header, the intention being to use the pin header for USB signals inside the rack case but populate the jack instead for boards intended for S1 use, with generic hubs and whatnot.

[QuintLeo]

I have 2u "server" (no external drive option) cases sitting behind me with 4 80mm fans - about an inch on either side of the outside fans of clearence, quarter inch or so between each fan, and rack handles, so ballpark 3" of horizontal clearence that's not got fan in the way.

 I don't think it would be an issue to have 1" or so of one fan blowing on the PS though, don't see why 3x 120s would be a problem.

 Getting lots of airflow out of 80s is a VERY LOUD proposition - highest airflow I know of from an 80mm is the Delta 80CFM unit, which were COMMONLY known as "screamers" (6800 rpm or some such off the top of my head, those things make the 120s on a S5 sound QUIET) back when they were commonly mounted on Swiftech 370 or Alpha 8045 heatsinks on Athlon/Athlon XP CPUs.

 I don't have any of those, but I had a couple of the "next step down" 68CFM units - which were clearly audible THROUGH internal doors and the cases they were in.

 
 I'm wondering about specifying those DPS1200 - the specs I'm finding on those come out to 750-900 watts each at 110VAC depending on manufacturer, so you'd need 3 or you'd have to go 220v to power them (where they spec 1200 watts, but at that point might as well spec *1* of the IBM Bladecenter 2880 watt units instead). Is there a different DPS1200 than the ones I've been finding?

[sidehack]

I'll say this a third time. The problem is physical dimensions. There's only two ways a DPS1200 would fit in a 4U case, and it takes up either 3.25 or 3.5 inches of horizontal space. This leaves less than 14 inches of width for fans, and 3x 120mm fans are more than 14 inches long. Therefore, impossible. It's not that 1" of fan would be blowing on the PSU. It's that 1" of fan would be coexisting in space with a portion of the PSU. If you want 3x120mm fans, you cannot mount your PSUs in a way that they're accessible from the outside - hence the three alternatives I posted.

110V on the DPS1200 gets you 900W, sure. That's still 3/4 power if you only have 110V available. But if you only have 110V available, why are you buying a 2400W rack-mount miner? I'm more than willing to ignore that small percentage of potential customers.