Request for Discussion: proposal for standard modular rack miner

[NotFuzzyWarm]

heh heh heh, ja I still like an external power bank but if they fit, well why not keep them(semi) internal. As for converting to using an external power rack, would be just as I do now -- run heavy gauge power feeds to split into short PCI cables near the cards. No use of the PSU adapters needed. Since the cards use PCIe for power in - is a non-issue to me at least.

As for mis-matched supplies, THAT is a problem but should not necessarily be yours... Drop-in replacement DSP1200 supplies abound with many sellers doing Amazon next-day Prime. Or, just have a few from a multi-pack buy  kept as spares. One has to draw a line somewhere.

[Finksy]

IBM 2880W!!!!!!  1 power cord, 1 breakout board, 1 PSU, 80+ platinum, all for ~$50-70 with fan packs.  I realize I am biased because I sell boards for them, but I am biased for a reason.  They are the best...

[NotFuzzyWarm]

Can't disagree with that. Aside from the redundancy aspect not sure why Sidehack is hell-bent on the DSP1200's but that is what is currently on the board so we follow his lead.

[NotFuzzyWarm]

As it stands it looks like the PSU's will be sliding into a sleeve spot welded onto the case lid so it should be an easy change to use other supplies... And again, since the hashboards use PCIe power connectors, this should not be a sticking point.

After all, gotta leave _something_ for folks to tinker with

[sidehack]

If you want to take the machine apart and run your own cabling, taking cables in through a PSU slot directly to the boards would certainly be possible. If you didn't, a simple insert that plugs into the PSU socket on the backplane and has either screw terminals or PCIe jacks accessible sticking out the back of the case would do nicely.

I would rather see common-rail redundancy be not an option at all than see it be the only option. It'd be a good feature to have, and there are certainly situations when you'd see improved performance, but it leaves at least as many situations when split rail is desirable. With a bit of thought on flexible internal connections, any number of configurations of 1 to 3 PSUs in common or separate rails is possible from the same simple hardware.

The most recent render actually fairly closely resembles our original idea, except that PSUs are now behind the cards in a deeper case instead of right over top. Having that top chamber with a fixed separation does require more disassembly in order to get cards in and out. Hm...

What could be done with keeping about 2 inches of width at one end of the case wherein resides the controller and such? You could mount your cabling interconnects at the top of this, and your backplane terminates at that point. You could put any required tall parts on that end of the backplane board so there'd be no restrictions at all on hanging parts in the airpath of an inverted supply. Simple cabling interconnects between backplane output and main cabling input will give you the ability (as previously described) to combine or isolate rails as desired. Something like that could work.

I'm not hell-bend on DPS1200 so much as on a fairly standard 1Ux2U server PSU. Please note every time in the last five pages I've given consideration to a design change requiring accomodation for three different common models in that approximate dimension. I'm fairly set on 4U height, which takes the IBM 2880 out of consideration as an internal supply (and there are numerous other reasons already documented) but also note that every time in the last five pages I've given consideration to a design change I've also required accomodation for interfacing to external PSUs including the 2880. The first post mentions we're working around the DPS1200 because that's what we have, but from the outset it was intended that provision for other PSUs, internal and external, was essential.

[NotFuzzyWarm]

<snip> Having that top chamber with a fixed separation does require more disassembly in order to get cards in and out. Hm...
<snip>

Not if the chamber is part of the top of the case.

To me easiest way to hold the PSU's is in a sleeve that is part of the case and welded to the rear panel. That sleeve could also serve as a mounting point the power backplane. Thinking on it, having the sleeve as part of the top as I mentioned earlier means longer cables from the PSU's to the boards and not good...

[Witrebel]

I think the sleeve idea is very much what we are thinking, at least as you describe it and I picture it, I can't speak for sidehack.  

Regarding the 2 inches or whatever.  I think I see what you are getting at, I will play around with configurations like that next.

Again, it would be helpful to know how much flow you want the tops of the hashing cards to see, the part where there is no heatsink.  Does this need to be entirely open to to the hashing space airflow with little/no obstruction? There is ALOT of free space if you drop the hashing area flowpath to just about height of one fan and leave everything above it as fair game for circuits, cabling, or tall parts the don't need significant heat dissipation.

Now its probably a little late in the conversation for this, but I do want to ask.  What specifically is the reason for the S1 compliance?  As I understand it you would need to at a minimum replace the hashboards and controller.  Do you envision the actual S1/3/5 heatsinks being used on your upgrade hash boards?  If not, then you are basically building all dimensions of this rack unit around the ability to strap the same hash cards on an existing frame and fan unit.  Is that really worth whatever design sacrifices you make to achieve it?  Just playing devils advocate here, thinking that if we are trying to define a long standing form factor that will be applied to both 2 card standalone miners and larger rack mounted multi card miners, then we really ought to be certain that we define it right.  My understanding of the argument is basically backward compatibility = more adoption/sales? And perhaps the fear that existing waternblocks couldn't be designed into a new non S1 compliant frame standard.

[sidehack]

Fuzzy, yeah a sleeve mounted to the back panel and used as structure to hold the backplane is exactly what I'm thinking. The top of the case should be an entirely independent panel, I think, for ease of digging into the works.

The top of the hashcard above the heatsink would probably be home to all the tall parts. Any through-hole or tall SMD caps, power jacks, interface or control chips, if you got VRMs in your design (that aren't super-large) they probably go up there. With a bottom-cooling chip your PCB is right up against the heatsink and you have clearance to adjacent for tall parts, but if they're at the top you have your ~2" clearance to the next board instead of the ~0.5" clearance to the next heatsink. I'm assuming you won't need a lot of airflow up there, and somewhere between 0% and 10% of your heat would be generated there.

Conversation about rolling with S1 dimensions is probably around page 2. Making a single board that works for both rack and small units means someone could design one product instead of two and fit both markets. There's already a lot of S1/3/5 chassis out there waiting to be messed with, and waterblocks for that formfactor are also pretty common.  The S1 design has proven itself pretty well, from almost silent running in the S3 to pushing pretty good power density with overclocked S5. I'm comfortable considering that board size and heatsink layout as a decent home-miner standard, and because of the opportunity for compatibility both from boards and waterblocks, I'm comfortable using that board size in a rack machine as well. The taller board certainly requires changes to case layout, as we've seen in the last several pages, but I think the long-term benefits of compatibility are worth the hassle now.

[Witrebel]

I took a stab at your 2in side channel idea.

It definitely has some layout advantages but you do sacrifice some heat sink to get it.  I ran three sets of numbers, all assuming a straight 2 inches of width wall to wall inside that side channel for controllers.

If you leave 0.50 inches between each circuit card/and the next heat sink or wall, you can have a maximum PCB - heat sink height of 1.29".
If you go with 0.475 inch clearance between cards, you can get 1.32in of heat sink
0.40 inch clearance gets you 1.4 inches of heat sink.

So you could definitely play around with the actual width of that side section, card clearance, and sink height, and get something workable.

This is what it looks like as I interpreted your idea.  Shown is the heat sinks at 1.32" tall with 0.475 card spacing, for what its worth.





Fair enough on the S1 compatibility. 

[QuintLeo]

It's only common at the breaker box and in the main supplying the box, NOT in the home itself.
 NOT the same thing.

 Having does more than a little rewiring over the years, I was FULLY aware of how power commonly arrives at the breaker box - but you can't plug a miner into a breaker box much less a main supply.

And how many typical homes have rooms wired for a >24A load between 3 circuits using standard 110/120V outlets?

 Common in kitchens. Not so common elsewhere. Also, I'll point out that 24A at 110 is easily handled by *2* circuits, most common US house outlets being on at least a 15A circuit. *2* circuits in a room are more common, even some of the MOBILE HOMES I've owned came with a seperate circuit on each side in at least the living room, going back into the 70s when I started paying attention to that sort of thing.

My point is that 240V can be had for those serious enough

 Never disagreed with that point, just pointing out it wasn't common in homes without rewiring work.
 Kinda redundent debate at this point, sidehack having mentioned he has no interest in this being intended as a home miner.



 

I focused on trying to maximize the PSU intake tract

 1u x 2u = 1.75" x 3.5" (actually a hair less due to case thickness). Figure if your "track" is 1" before you widen it, need 5.25" or so worth of width to feed each PS.




 Load balanced redundancy is a PITA to try to set up with PS that are not specifically designed for it. Takes a fair bit of additional circuitry to make it work reliably. I'd skip the whole idea.

[sidehack]

Is the panel above the hashboards necessary? I can see how it'd be undesirable to suck in hot air from the hashboard exhaust into your PSU for "cooling". I ask because having a panel there will make accessing cards more difficult. The panel would have to be cut short or punched out for power and signal connector access, but now you have a lot more disassembly required in order to get to your hashboards for installation or maintenance. If it's necessary it's necessary (and given the kind of heat those cards will be generating, it's probably necessary) but that's going to make other things more difficult.

Could we make a kind of duct enclosing the backplane - maybe even open at the top for ready access but more-or-less sealed by the case lid - through which the PSUs can draw fresh air in through that two-inch side space instead of from the hashboard airflow? You could isolate the PSU intake from the hashboard heat without requiring a separator panel above the hashcards, which will keep access to them pretty easy. You wouldn't have to worry about getting your cabling out of the way before removing the separator to get to your hashboards, which might be handy if you need to do something to one board without shutting down the other seven. You also aren't limited to keeping your PCIe jacks in a predefined area of the board to be accessed through fixed punchouts.

That two-inch controller space can probably be shrunk without causing a lot of problems. If a small SBC were mounted vertically and the USB/interface board put above it (basically, both boards flat against the side wall) that space could probably be more like one inch. Some room for ventilation would need to be left open if this was providing cool air intake for the PSUs.


Load-balancing redundancy is indeed a pain for supplies not designed for it. If it's possible to allow load balancing for supplies with internal support for this, without making the option required (which means also allowing split-rail operation of non-balancing supplies) and also without requiring a lot more expense or complexity, I'd like to keep it possible. If it's too cumbersome to rig up a setup allowing for both configurations, I think split-rail should be kept instead of mandating a match set of load-balancing PSUs. However, probably some refinement on the two-point cabling idea presented earlier should allow the end user to decide between split and common rails without requiring complex circuitry.

[philipma1957]

Okay the ant miner s-7 dropped today.

Specs are unreal 4800 gh at 1200 watts

Box looks like 1/3 of an s-5+

I will suggest everyone look at it closely then come back to this thread.

This  s-7 is like the size of a s-5 and do more then 4x the hash. For 2x the power

[sidehack]

Okay. Bitmain built another overpriced jet turbine and now there are even more perfectly good S[odd] chassis about to be retired.

My opinion is, 2x the power in S5 volume is asking for trouble - especially if it has the same firmware bug of excess heat generation on network dropout. Instead of melting a bit it'll probably go full Prisma.

What do you suggest we do differently in a rack machine because Bitmain built a 1.2KW "home use" miner with no innovation, dual 4200RPM fans and three hundred tiny glued-on heatsinks? Overall, I'm fairly disappointed.

[Witrebel]

Looking at the flowpath, I would advocate for at least the rear portion of the ducting, perhaps including the angled bit, and ending just at the back of the hashing cards, or maybe extending an inch or so over them.   I feel like this would help the flowpath significantly, but this is totally based on intuition and opinion, and I would strongly recommend we attempt some actual CFD simulations before settling on the exact flowpath layout.   High level though, I would say you could loose the panel over the cards.  I don't know how much those PSU's need to breathe, so its hard to say if they could pull all their intake through the side space, but it doesn't seem impossible. 

I also think it depends what sort of negative pressure the 3 120mm fans end up creating inside.  You wouldn't want that negative pressure fighting your PSU fans too much, but with the entire front face meshed and open, I doubt that would be a serious issue. 



As for the S7, the overpriced part is true, and while the jet turbine part is also true, I think it depends who you are building this miner for.  And lets not forget the fact that you aren't just designing a miner, you are designing a standard/form factor that is to be backward/forward compatible.

If you are designing for the home mining guys who want a few of these in the room or a room over from where they sleep, with wives and such, then yeah, jet turbine solutions don't cut it. 

But if you are designing this towards the economies of profitable bitcoin mining, then I think some allowances need to be made in regards to noise.  I am beginning to see the light about denisty, in terms of hash power per unit of volume. Chip efficiency held constant, the only way to increase this is increase the thermal capacity of the unit.  Which means either waterblocking, immersion cooling, or more airflow/better heat sinks.  Not saying that 2.2kw in a 4U case isn't dense enough.  But it certainly isn't as dense as the competition is trending towards. 

[sidehack]

I tend to think of Spondoolies first when I think of power density, and I've never had one run nearly as well as advertised because they were so finicky about heat. Add the order of magnitude complexity increase because of their 100W BGA idea to insane internal temperatures and they're actually sorta terrible. There comes a point where density murders reliability, and reliability is one of my primary design requirements.

If we can safely put more heat into this machine (on pure aircooling) without losing reliability or becoming a fire hazard, I'm okay with that. Waterblocking is a secondary concern, but also one of the reasons to go with S1 board size - any C1 waterblock or aftermarket watercooler would drop right in and not require a new design there. That product already exists and is pretty good. Home mining is also a secondary concern so noise is a secondary concern - but it shouldn't be deafening either. 80dB fans is kinda stupid any way you look at it and pushing mechanical components to that extent probably sorely limits lifetime.

If the portion of panel over the rear of the cards is shorter than the gap between the cards and front fans, a board could be unscrewed, slid forward and lifted out without requiring anything else to be moved or disassembled. That's great. Keeping that rear portion in place (the angle and under the heatsinks) makes sense too as it keeps airflow pulling toward the fans without a lot of turbulence around the supplies.

[Finksy]

Common in kitchens. Not so common elsewhere. Also, I'll point out that 24A at 110 is easily handled by *2* circuits, most common US house outlets being on at least a 15A circuit. *2* circuits in a room are more common, even some of the MOBILE HOMES I've owned came with a seperate circuit on each side in at least the living room, going back into the 70s when I started paying attention to that sort of thing.

How do you propose to power 3x PSU's evenly between 2 15A circuits?  2x PSU's on one circuit would make for 16A, and I'm not aware of a legal/safe way to balance 3x loads between 2x 110V circuits.  Not that this is relevant, but for discussion's sake.

[sidehack]

You're right, it's not relevant. It's for the customer to figure out.

[NotFuzzyWarm]

My take on the s7 and then 'nuff said about it:
Ya I will get at least 1 after they start delivering and we hear more about them. As to their pricing... well my first s2 was around $1,250 as I recall and the 1st s4's the same so, not too terrible for first release.

I will probably finally retire my last s2 that's on-line and few S3's to free up power for the s7(s). Also have 2 s4's that are starting to act up by dropping boards from time to time...All in all it is a very good balance on power/replaced hash rate but kills me to take ANY perfectly good miners offline

On the hot turbines -- that exhaust fan will need to to watched as most are NOT happy in the hot side and their bearing life tables reflect that. They should start looking at selling the s7 in pairs and use a dual-squirrel cage blower for them. I happen to have one in the shop and will do just that
We now return to our regular programming.

[dogie]

I tend to think of Spondoolies first when I think of power density, and I've never had one run nearly as well as advertised because they were so finicky about heat. Add the order of magnitude complexity increase because of their 100W BGA idea to insane internal temperatures and they're actually sorta terrible. There comes a point where density murders reliability, and reliability is one of my primary design requirements.

I don't believe that any of their problems are due to heat at all, if ambient is high the fans scale or the frequency / voltage scales down. The real problem is the custom controller's high failure rates out of warranty, the entire setup is crazy complicated.

[sidehack]

They're so dense that it's difficult, if not impossible, to keep one running at top speed - which is to say, advertised rated speed. The overall complexity of their designs is definitely pretty bad. I mean I like that they give you plenty of information on what power is going where, but 200A 4-phase bucks into BGA chips with a custom protocol in an FPGA on the control board is not a recipe for reliability even if your stuff doesn't run 110C.

That's why this proposal intends to keep things relatively simple. Increasing complexity simultaneously increases both the odds of failure and the difficulty of maintenance - not to mention the initial cost.