Request for Discussion: proposal for standard modular rack miner

[dogie]

1) With 3x140mm the entire front face is pretty well covered by fans. Looking at bladed miners like the S2, you can tell exactly which boards are between fans by looking at the per-board temps. I'd like to avoid poor-coverage zones like that if possible. The rear fans were dropped to 120mm because of height requirements for fitting in PSUs. If that ends up actually restricting the airflow, we remove them and put in a grill instead. I'm not a fan expert and these points need to be evaluated by someone who is.

Think of the SPXX (all of them ), they used the absolute perfect cooling layouts. You don't need to ram air in because it really doesn't do anything, as evidenced by swapping out SP120s on an S4's front fans not changing the temperatures pretty much at all. You then suck all the air out, in a long, smooth and continuous column. This column can be wedge shaped if your intake is larger than your exhaust and it works fine - air is a wonderful fluid.

Its all about them rear fans, and you certainly don't want grills on your exhaust (see above).

2) The hashboard volume occupies the lower 3U (approximately) of the 4U case. The top 1U is separated from them with an internal panel above the hashboards (which mostly seals them off and keeps main cooling airflow restricted to between the heatsinks and boards) and is where the PSUs, controller and all cabling live.

I think you're going to be super close (see above again) and it might fit on CAD but will probably be prohibitively tight to use IRL.

3) Yes, 1U fans are really annoying. If I'm thinking right, the DPS1200 fan is quieter than the thing on the SP supplies but it's been a while since I fired one up. I prefer server supplies pretty much universally to ATX and would consider installing an ATX supply to be a waste of space, a waste of cost, and really asking for failure.

The fan is far, far far louder than the SP35 which can run 40% fans and its the thing you can hear through walls, floors and really gets to you. It was similar on the SP10 although that PSU was overloaded and the main fans were only 50mm. It was such a shame on the SP35 as the entire unit was quieter than an S5 if you underlocked it enough so that the PSU fans wouldn't spool up to max.

4) Yeah, temp control hadn't been thoroughly discussed hence why I made brief suggestions for both hardware and software control. This point needs to be ironed out.
Regarding characterising ambient intake versus chip temps, how do you suggest we measure ambient intake if not by the same means as measuring exhaust? Additionally, as Novak pointed out, since the board design is deliberately unspecified there is no specific requirement (at present) for this to be true.

a) Put it on a test board in a test case, measure it, see what the rough delta is tracking. Use that in software. It might vary between chip type but you'll still be able to get the delta from your test boards.
b) Ignore ambient and just fan control off chip temp, which is really all you care about. Again as I said above you don't care if its a 60C ambient as long as the chips are happy.

5) There are no screw patterns specified yet, but it seems likely that the outer two mounting holes will be very close to the edges of the heatsink.
I'm no expert in thermal management so the heatsink point should be evaluated by an engineer knowledgeable in that field, make sure we can safely dissipate the heat spec in the volume provided with the expected mass flow of air.

Thermal expansion is a consideration and not an issue. Its countered by using clearance holes to mount the heatsink to the PCBs, so its sliding on the bolthead rather than loading your PCB.

7) The hashboards are specified as 5" tall, which is about 6mm shorter than 3U. If the upper room (cable tray, whatever you want to call it) is at most 1U high, we should come in under this. If that's not good enough (which can be determined without a lot of trouble), I'd probably convert inch-measure specs to metric with a fixed 1" = 25mm conversion which gives another 2mm height reduction to the boards. If that's still not good enough I reckon we'll have to do some tweaking.

In your current layout I don't think things will fit nicely (discussed under 2)), but I was actually referring to external being too large. U racks are explicit in that 4U = 4U, 1mm over and you're impinging on the next 4U of space and pretty soon screw holes don't line up. Which is why I suggested designing for external of <7" rather than =7".

Regarding horizontal width - have you measured many rackable miners for width, or just servers? Most miners I've seen aren't too concerned with rails, which eats about half an inch off each side.
A custom case design is exactly what we'd need anyway, I think, so we should be able to specify our dimensions any way we want that doesn't violate rack requirements of maximum height and width.

Yeah, dimensions are in each individual guide. The majority are thinner than 19".

[sidehack]

I don't plan on doing much with CAD. I prefer to actually build something, so if I test the dimensions it'll be a quick calculation on paper and then seeing how it works when I build a mockup. Hopefully I have time to do some of that in the coming weeks. I agree, it is probably going to be tight and will probably require some adjustments. You're right, maximum outer dimension height of seven inches (and not a millimeter more) must be maintained. Ideally we'd come in 1-2mm under that in case the units are shelved instead of stacked, since there probably won't be horizontal room for rolling rails.


If we want software fan speed control, which I'm not inherently opposed to, we'd need to see about writing a cgminer module that talks to a controller on the USB board. All driver code for the various boards made would have to talk to this module, which would unify all that data and send off to the controller with PWM commands. I'm not sure how hard or easy that'll be to integrate with the existing cgminer framework. Someone with more experience with the guts of it will have to weigh in.

I think the fan dimension point you're arguing with Novak is irrelevant. Removing a fan to fit in the PSUs does not apply, because the PSUs aren't in the same space as the hash boards. The hashboards take up the entire width of the bottom 3U, so you need that entire width with an unrestricted front-to-back airflow path. The PSUs are laid flat in the top 1U, which is almost entirely isolated from the bottom 3U by means of the horizontal separator panel forming a "ceiling" above the hashboards.

Additionally, the S4 reference picture is irrelevant. If we have rear pull fans, there would be no interstitial grilles. The rear would only be grille'd if the only fans were the front fans.

My office is right next to a room with about 30 various Spondoolies rigs in it (at least one of everything they made). I don't need to be told how annoying they can get.
Also, if you know me at all, you know I pretty much despise most of the engineering on SP's rack gear. This includes having a two-foot-long airpath which pretty well bakes the rearmost chips. I'm even a little worried ten inches will cause problems. But that's not really what you were talking about, I know; you mean the suction from rear fans instead of push from front fans.

If someone who knows the mechanics of air cooling can weigh in with a quantitative consideration of front push fans versus rear pull fans versus both would be nice.

[el_rlee]

I don't understand why not a single hardware manufacturer standardizes it's PCB's so the heatsinks and housings can be re-used...

[dogie]

If someone who knows the mechanics of air cooling can weigh in with a quantitative consideration of front push fans versus rear pull fans versus both would be nice.

Its easiest to show you. If you've got an S4 or S4+ around, remove one of the pair of fans and see, remove the other and see. The one with rear fans will do just about fine, the one with front fans will overheat. If you're really determined, you could try reversing the direction of an SP3X's fans and watch it overheat.

[sidehack]

I do have access to an S4+ which could be set on fire and nobody would really complain. Maybe I'll play around with it in a few days when I'm not busy. All the Spondoolies gear is hosted.

[dogie]

I do have access to an S4+ which could be set on fire and nobody would really complain. Maybe I'll play around with it in a few days when I'm not busy. All the Spondoolies gear is hosted.

The S4+ runs pretty close to its readline at 35-40C so you'll get better data whacking its frequency down first.

[sidehack]

So, here's a thought.

Two boards from this machine could fit into an SP20-scale miner with a single 120mm fan and hook up to an external 650-750W PSU. Opinions?

[chiguireitor]

Ok chiming in for the first time....

Sidehack, you know i love all the products you do, but in terms of heat dissipation we have our differences, so bear with me

It could be nice if the case could also fit water cooled blocks, as this is a need for most tropical countries, and would make heat management a lot better and let home miners mine without the noise they make (currently writing this with 6 S5 at my back).

Also, how about starting with the de-facto standard for boards that is the S1/S3/S5 hashboard? That would make adoption fairly easy, and the hole design on them allows for several individual boards screwed to a standard set of holes or having several different boards.... lemme explain myself with an image:



The white (and yellow) outline is the S5 board dimension, the red circles are the holes preexistent on the boards, and the green lines are the "logical" separations on the boards.

You could have a case with vertical mounting spacers, which could host 4 Small boards, 2 Mid boards or 1 Large board. The boards themselves could have a nomenclature so you can know how much you can fit of them on the case (S, M, L) and have them mounted with atx spacers.

Also, have some cable managemente in the inside, so you can have them orderly without a mess.

I can make a model if you want (i know you're more of the hands-on type, but having a model doesn't hurts anyone).

[sidehack]

Changing board sizes greatly changes cabling requirements. If you make quarter-sized boards now you have to provide for 32 internal data and power connections. The provision for having several different boards is already allowed in that there are eight separate heatsinks. Buying 16 or 32 small boards instead of 8 big boards is about like trying to make a large-scale miner out of USB sticks. You end up with a lot of needlessly repeated parts. Small boards also limits your topology choices, which limits your efficiency overhead options. I can give you more details about this point with reference to some of our internal board design decisions if you like.

I would prefer to stick to a design based around air cooling. If you can't fit a waterblock in the same space as a two-inch-tall heatsink there's problems with your waterblock. If the rear panel is removable could provide ready access for hoses and such.

Not sure what ATX spacers would be doing anywhere?

[chiguireitor]

Indeed, the nice thing about it is that you could design a big board mix it up with a plethora of different architectures inside them.

A removable back plate would indeed solve most of the watercooling issues with a air-only solution (i understand your air only preference, but with >60% humidity points here on my country, high density only happens with water blocks).

Also, i was thinking about the thermal protection you proposed, it would be nice to have the thermal watchdog in a separate process NOT related to cgminer, as you can see what happened to the S5s with that option.

I was thinking that the boards should go with atx spacers to have them mounted like standard motherboards on the case (but in this case on the vertical mounting planes), with some separation of the PCB backside to let the heat flow thorugh the ground pads (not unlike the S5).

I really should get a 3D model because i don't seem to make sense with words today (had my birthday celebration yesterday and i'm a little verbally impaired atm )

[sidehack]

Yeah, a standard miner shouldn't ignore watercooling requirements since a lot of folks would like that option. You'd still be limited by power and data connections, so density wouldn't be as good as some other options (like fitting twice the power from a C1 as from an S3). I think making the rear panel removable shouldn't be too difficult.

The vertical mounting planes are themselves the primary heatsinks, screwed to the bottom of the case. If you space the PCBs off things, you're removing their contact with heatsinks and then things catch on fire. The boards do not mount to the case at any point. It's like how boards are installed in a Dragon.
The separation between PCB and heatsink on the S5 is actually the chips themselves, since most of the heat in a BM1384 comes out the top. If the ground planes on that miner ever interacted with the heatsink they'd short out and break stuff - like the Prisma did.

I think what happens with S5 fan control is more the fault of whoever wrote the driver code than an inability of cgminer to do things right. A separate control software is an option, but that removes the single-point control which an end user might appreciate.

[Meech]

I like the 4U case, plenty of possibilities with cooling ala S2 and a controller like the S4 would be nice.  Massive heatsinks seem to be popular and cheaper but new cooling needs to be developed.  Companies should stick to a standard case and board setup, it saves them money and us to just ship new boards.  Saves us from ditching rigs altogether.

[AJRGale]

10 inches long contiguous aluminum heathsink?

The thermal expansion of such a slab of aluminum will be literally ripping the chips off of the PCB.

Either the heathsink or the PCB needs to be partitioned into sectors.

"rip the chips off" ? are you glueing on the HS? most heasinks have thermal gel, or thermal pads between the chip and sink, then its bolted down to a flex plate or secondary heatsink, or sometimes just machine screws holding onto the PCB.

I have an old heatsink that held onto 10 audio amps and is 30cm/12in long. it never warped due to the 80 odd degrees Celsius of thermal input.
the only way them heatsinks warp, is incorrect installation. Using the PCB to hold 1KG of heatsink whilst dangling it in-font of a fan board flexes off the sink. I'm thinking of the RK-Box that did this.

also still surprised no one wants to use the powerpeg style heatisnk..

Can you talk a little about why USB was chosen over saaayyy a PCI-E bus (a la Block Erupter Blade backplanes)? My thought would be that a USB driver is easier to work with but PCI-E is pretty cool and very modular...

the PCIe hardware is only designed for 175W. and I have a pet peeve of companies using "Standard" hardware with non-standard layouts. There will be a stupid person attempting to plug a video card in and go "LOL mah vidz card makes fire! I sue yooou!".. and also could lead to the PCISIG suing the company misusing their hardware.
 
now if there was data throughput via PCIe lanes, thats a complete different dev path again..
Hell, I'm not sure if it was just the mini-PCIe (laptop card slots) that only had USB lanes, or if the full sized PCIe does as-well.

[QuintLeo]

If you're talking powerpeg like the Alpha 3085 or the Swiftech 370, those things were EXPEN$$IVE for a reason - high cost to MAKE that style of HS, though they worked well. They don't work better than fins for crossflow cooling though, they were intended for updraft/downdraft specifically.

 If not, you'll probably need to explain what you mean.



 PCI-E hardware can use a LOT more than 175 watts. Look at ANY of the Radiion "x2" cards, typically in the 400+ watt range, for examples. Just have to use enough power connectors.

 Standard PCI-E does NOT use USB in any way shape or form.

[SerialLain]

...
A removable back plate would indeed solve most of the watercooling issues with a air-only solution (i understand your air only preference, but with >60% humidity points here on my country, high density only happens with water blocks).
...

Out of curiosity, what constitutes as high density? in kW per volume.

I've been doing some custom watercooling stuff for fun, wondering how thin the cooling blocks should be. So far I've only gone down to (in total) 7 mm thick blocks and they have had problems with flow.

As for liquid cooling it should generally require little (or no) servicing, especially since miners aren't kept running for long.

[sidehack]

SerialLain, have you done any... experiments... lately?

Our design philosopy is simple, durable, reliable. Given two options to provide the same function, we will always pick the one that does so more simply and more durably. PCI-type socket backplane is a nifty idea for yanking cards in and out quickly and easily, but:
- the backplane burns about an inch of vertical height, reducing hash density
- the backplane PCB is very large and built in low quantities (compared to PCBs) and is therefore comparatively expensive
- means of securing PCBs within the case is cumbersome and unreliable (especially if PCB is secured with a hanging heatsink, versus a secured heatsink with a hanging PCB)
- increases the number of breakable plastic parts
- requires edge-connector fingers on every PCB, which adds to cost
- if something breaks, zero modularity makes repair or replacement difficult or expensive

I'm heavily in favor of using USB 2.0 protocol, which keeps the board-level hardware interfacing very simple. The designer can leverage any number of protocol converters like CP2102, MCP2210 and so on, or tie into a USB-enabled microcontroller. This also simplifies coding at the controller end, as cgminer is already quite good at talking USB. This also simplifies hardware requirements for the controller, as it's trivial to find a decent minicomputer board with USB jacks.
Using a PCI-hardware backplane, even if the signal is still USB, really isn't any better than a PCB with a securely-mounted heatsink and a fifty-cent cable. By my consideration, it's substantially worse based on cost and longevity.


Also, PCIe standard (last I checked) provided for a total device power dissipation of 300W. There are nonstandard devices which exceed this, however, but they aren't labeled with the PCIe standard logo. The PCIe standard allows for one 8-pin jack (at 150W) and one 6-pin jack (at 75W) and 75W through the socket.

[QuintLeo]

with >60% humidity points here on my country, high density only happens with water blocks).

 I don't understand that statement, heat sinks don't care about humidity to cool, they care about temperature differential. They do NOT use evaporative cooling like humans do.

 Ignore "heat index", that's only an estimate of how hot a HUMAN feels due to humidity level reducing the ability of a human body to cool itself, NOT the same mechanics as for an item cooled by heatsink-to-air heat transfer.

Using a PCI-hardware backplane, even if the signal is still USB, really isn't any better than a PCB with a securely-mounted heatsink and a fifty-cent cable. By my consideration, it's substantially worse based on cost and longevity.

 Cost more, definitely so when you include the cost of a backplane - though not as much as you think, passive PCI backplanes do exist and have been used for a long time in some hardware and aren't exactly rare. No need to reinvent the wheel there.

 Mount really isn't any more complex than mounting those heatsinks to the case or however you're planning to mount them.

 Longevity, IME PCI connections tend to last longer than the hardware they are being used by, Heck, I've got "ancient" ISA based gear that still connects reliably after 20+ YEARS of usage. I do NOT see a longevity advantage for the typical cheap connector used on any USB setup - though I doubt it would average much if any worse, BOTH will probably outlast Bitcoin mining.

 The size disadvantage I can see POSSIBLY being an issue, especially since you're trying to limit board length for a better balance on front-to-back cooling.


 Just had a thought - but I can see mounting issues getting "interesting". Make the hash boards horizontal, instead of vertical, then mount the power supplies to one sire of the case. Would probably need a subframe mounted inside the case, or spacers between the boards, to keep the boards from flexing too much. Would give fewer but larger boards, so would be a bit less "flexable" about incrimental upgrades. It WOULD make the cooling issues on the hash boards easier to manage.

 Second thought - why limit it to 4U? As I recall BitFury and Avalon both made rack-mount 6U miners, which would make space management a LOT easier inside the case.
 6x 120mm fans on the front would generally be more airflow per square inch than 3x140 too while using a LOT more common size of fan with a LOT more options available.

 Delta, for example, does not list ANY 140mm 12V fans on their website, but they have a TON of 120mm 12V options (the 140mm fans they DO list are 24V and UP).

[dogie]

Also, PCIe standard (last I checked) provided for a total device power dissipation of 300W. There are nonstandard devices which exceed this, however, but they aren't labeled with the PCIe standard logo. The PCIe standard allows for one 8-pin jack (at 150W) and one 6-pin jack (at 75W) and 75W through the socket.

I don't know about that, my HD7990s had 3x8-pins = 525W and they were official enough.

[chiguireitor]

...

Out of curiosity, what constitutes as high density? in kW per volume.

I've been doing some custom watercooling stuff for fun, wondering how thin the cooling blocks should be. So far I've only gone down to (in total) 7 mm thick blocks and they have had problems with flow.

As for liquid cooling it should generally require little (or no) servicing, especially since miners aren't kept running for long.

High Density varies a lot depending on the nature of your deployment, but it can go as high as 18Kw/m3 depending on the cooling you have.

However, i tend to go a little more low there, the densest deployment i have is about 2.7 Kw/m3.

Watercooling isn't only for density, though, it is also for ultra-high humidity deployments, like my country's usual >60% (even near 98% 4 months a year) relative humidity.

7mm block is too small, i'm designing mine with 2cm blocks at least.

[AJRGale]

If you're talking powerpeg like the Alpha 3085 or the Swiftech 370, those things were EXPEN$$IVE for a reason - high cost to MAKE that style of HS, though they worked well. They don't work better than fins for crossflow cooling though, they were intended for updraft/downdraft specifically.

 If not, you'll probably need to explain what you mean.

Powerpeg: http://tem-products.com/index.php/thermal-connectors/power-peg.html
its basically a round bit of copper, that sits though a 2.5mm hole in the board, that the ground/thermal pad of them QFP can solder onto, and since its a full chunk of copper over via holes thats been plated with 2uM of copper, to pass heat though and you can screw a heatsink directly to it. its been around for a few years now, I even pulled it up in the hardware section here to ask why no one thats manufacturing miners with thermal vias are using it.

PCI-E hardware can use a LOT more than 175 watts. Look at ANY of the Radiion "x2" cards, typically in the 400+ watt range, for examples. Just have to use enough power connectors.

 Standard PCI-E does NOT use USB in any way shape or form.

i would assume you are using power wires off the PSU directly.

Mini-PCIe in laptops do, many wifi cars use the usb protocol over the pcie bus. hell, used it for my old EEEPC701 mods, put 32GB of flash memory on it (via hub and 4x8GB drives).

I was wrong with the power throughput Via the bus only, its 75W, not 175W. about page 35 has the power requirements on the bus.
http://read.pudn.com/downloads166/ebook/758109/PCI_Express_CEM_1.1.pdf