Request for Discussion: proposal for standard modular rack miner

[Finksy]

I feel that a large part of that problem is their rating as well as the design.  Bitmain rates miners at a reliable level of clocking, which can be maintained more or less indefinitely.  SP-Tech -especially notable with their SP20- have ratings that for all intent and purpose were un-achievable, and completely high strung.

[1713273529]

1713273529

[1713273529]

1713273529

[1713273529]

1713273529

[sidehack]

I do a bit of advising for a guy in a somewhat equatorial region, who bought several SP35 earlier this year. He set them up in a climate-controlled room with AC venting directly into their intakes and still can't sustain more than about 90% rated speed from the lot. Meanwhile it's not too difficult to get 20% over rated speed out of an S5 without any problem. Yeah I'd say there's something wrong with their rating system.

For the machine design being proposed here, I'd like to see at least 260W per blade stock rating sustainable in 30C ambient. It'd be nice if it could handle 320W per blade without a lot of danger. I know some of the danger is going to depend on the board design itself, low-resistance chip-to-sink connection and such, but a lot of it is also the design of the heatsink itself. I'm looking forward to seeing some of Witrebel's simulations. If we can get the basic layout hammered out, the next step is building a sturdy heatsink which can perform inside the specified dimensions.

[Witrebel]

Work has picked up a bit as I get ready to enter port and pack up all our gear. 

I will have to investigate what simulation packages I have access to through our license pool, but I have a feeling to do this right I might need to get some third party software like COMSOL or FLUENT and export our models geometry into one of those multiphysics packages.  I really hesitate to post anything I have from the "express" simulation packages I have on board as I doubt their completeness (I get outputs like average velocity  over the entire flow path, as opposed to seeing turbulent pockets etc). 

I just don't want to promise the world regarding simulations and then not deliver, as there is both the issue of gaining access to a reputable CFD/multiphysics package, as well as the learning curve associated with getting useful data out of it. 

I will attempt to get another model rendered up reflecting the latest ideas before I get into port tomorrow. But it will probably be a week at least until I have a better handle on what I can actually produce simulation wise. 

As for the density issue, I agree with you sidehack.  Reliability is key, and furthermore, I like 10 x 10watt chips alot better than 1 x 100watt chips.  On that note if a chip goes down in string topology, does that mean the entire string MUST go down?  Or are their any possible failure modes that don't result in loosing the whole string? 

[sidehack]

One thing that could be implemented to maintain a string is I think something ASICMiner attempted on the Prisma - basically, FET bypass. You could put basically an analog voltage-detection circuit on each node that monitors the chips' core voltage per node and drives a bypass FET in parallel with the chips to maintain that core voltage. I'd probably use a simple op-amp circuit with a threshold reference of about 1.1* an externally-generated Vcore (basically, a fixed resistor divider across your total voltage giving you expected per-node voltage plus ten percent) driving the FET. If a chip drops out, the current through that node drops and the voltage spikes briefly - it'll be partially buffered by your node-level caps. If the op-amp detects the node-level voltage goes up to 110% of expected value, it'll kick on the FET which will start to draw excess current through it and buffer back down. The issue with this is your bypass FET is now sinking the entire power typically burned by an ASIC, meaning you're still running the same amount of power but doing 1 chip's-worth less work.

That system worked for ASICMiner because of how the comms worked on the BE200. Each chip was on a buffered parallel bus with individual addressing. One chip turning off didn't affect the operation of any other chip, at least directly. It wouldn't really work as well for BM1385, depending on the failure condition. BM1385 (if I understand correctly, and it was the case for BM1380, 82 and 84) relay comms, so the first chip talks to the second which talks to the third and so on. If one chip drops out in a way that it stops relaying comms, everything downstream from it will also turn off. A bypass FET to pick up the slack of a downed chip is okay if the chip is still talking but not hashing; in the event of a downed chip not talking, everything downstream will stop hasing and every node will be in full bypass mode - every FET will have to sink the power of all the chips on that node, and probably burst into flames pretty quickly.

That's one reason I'm hoping, if PlanetCrypto can get a chip dev project going, he'll go with parallel comms instead of relayed. It'll require a bit more work for the node-level comms shifting, but by fully parallelizing everything you increase overall reliability. If you were running an S1 (definitely not string) and you blew the first VRM, everything would stop hashing because the first 8 chips turned off and now you couldn't communicate with the last 24 which were still working perfectly. Compare that to an AM Tube, where you can smoke as many VRMs as you want and everything else keeps going.

You might ask PlanetCrypto what he has access to for simulation packages. Novak's last job centered around doing CFD for jet turbines so I know he knows a thing or two but I really don't know anything myself.

[Witrebel]

Interesting...

So help me get on the same page regarding the basic power distribution of these systems.  You have your PSU, which outputs some voltage, nominally 12v.  Some supplies may have the ability to tweak this voltage slightly?

On an S1 this 12v goes to a VRM (a buck converter?) that drops the 12v to your core voltage, which is then seen by 8 chips in parallel, so if any one goes down, the other 7 chips would still see power, unless the chip fails in such a way that it short out the supply, and then you loose all 8 chips.

In a string topology, I assume the chips are powered in series? And this is where your "total voltage" comes from, which is where you would use the voltage divider to get your 1.1*v_core_excpected?  Is the on resistance of a MOSFET significant at these levels and hence the power loss through the MOSFET?  I guess I have a hard time understand why you can't just use your voltage control to lower the total voltage across the string by 0.7v or whatever node voltage.  Then you bypass the chip and the rest of the string keeps working?  Please pardon my lack of knowledge here, I'm just trying to wrap my head around the whole thing. 

[sidehack]

I guess the bypass FET only works if the chip fails open or stops hashing - conditions where current passing through the chip are substantially reduced. If the chip burns up and fails short, the voltage across that node is effectively zero.

Let's use S5 voltage levels as an example. It's easy. There are 15 pairs of chips and your total voltage is 12V, so each pair (the chips in each pair are in parallel with each other) sees 0.8V (12V/15). Each chip draws 12A at 0.8V to get 22GH.

Say one chip fries short; now instead of passing 12A at 0.8V alongside its partner (also passing 12A at 0.8V), the fried chip passes 24A at approximately zero volts. You now have effectively 14 pairs, so your per-node voltage increases from 12/15 to 12/14 or 0.86V. If you detect a condition like this, and you can control your total voltage, you can lower it to compensate. If your chips don't relay comms from one to the next, the rest of the chips might keep working without a hitch.
However, with Bitmain chips, if the chips drop out enough to no longer communicate properly, they can't relay work to other chips and they'll turn off. This upsets the current balance of the whole string and it stops working.

Say one chip fries open; now instead of passing 12A at 0.8V alongside its partner, the fried chip passes 0A at 0.8V. The circuit as a whole still needs to pass 24A for the other chip pairs to work, so the voltage at the damaged node will increase until the one functioning chip is passing 24A - either because now its core voltage is so high it's actually drawing that much current to operate, or (more likely) because it fries. If you have a bypass FET, when it sees the node voltage increase (because the apparent resistance suddenly doubled by taking out one chip, remember V=IR) the FET will kick on and start sinking 12A. This will bring the total current back up to 24A without breaking any more chips, and the node voltage will return to the 0.8V it's supposed to be at. Instead of two chips each hashing at 12A, you have one chip hashing at 12A and one as a dummy load at 12A. Again, if your chips don't relay comms the rest of the chips will keep working, but if your down chip can no longer relay, all the downstream chips it should be feeding work to will turn off and the string stops working.

The on resistance of the MOSFET is actually the part being actively controlled by the op-amp circuit. It'll be operating in the linear region (instead of switching between off and saturated) as a high-power variable resistor. The power loss through the FET will be the same as the power loss from a working ASIC. It has to be so, because for the rest of the system to work properly, there needs to be a chip there sinking 12A at 0.8V - either an ASIC, or a dummy load.

If you can actually reroute chained comms around bad nodes entirely, you could use a switched FET to draw all the current through a node and drop its node voltage to near zero (Inode*Rdson) which would basically turn off both chips whether one still worked or not. You'd then need to drop your total voltage by one node's worth.
For example, say you have six chips in three nodes: A+B, C+D, E+F. Comms go A->B->C->D->E->F. If chip D suddenly drops out, you can bypass the entire node by shorting it out (which turns off C+D) and redirecting comms so you now have A->B->E->F. It's definitely possible, but not really easy. You'd have to have a way of very rapidly determining which node wasn't functioning right (which I guess could be done with node voltage threshold measures - if it gets too high or too low, disable the node and latch it off), and then rig up multi-channel switches on every node. If bad-node voltage was almost zero, the level shifters from the last working node would still work for the next node up, but you'd need a set of dual-throw switches on every node. Possible, but cumbersome. Course now I'm interested in actually designing it and seeing how bad that would be. If it's only a matter of a couple bucks per board it might be worth putting in, since you'd be chopping a lot of the more probable board-failure conditions to reduced-capacity condition instead.

I guess the feasibility of the system is entirely dependent on the probability of chip failure which disables the entire board without the system. Economically, if the percent increase in the cost per board is greater than the probability of the failures it prevents, it's actually a net loss for the customer. On average.

[Witrebel]

So I assume the 2 chip per node layout is used to reduce part count on the VRM side?

I wonder if doing 1 chip per node and some fet protection circuitry on every chip would increase costs substantially?

Furthermore, it sounds like this is pretty much only relevant to a new chip design with parallel comms.  Unless of course you ran multiple strings on one board and called each string a separate device, again raising complexity and parts count. 

Any idea what the probability of failure actually looks like for these chips?

Lastly, not sure on the cost difference and board layout implications, but you could also offer this as a manual repair solution.  Simply place bypass jumper headers near each chip.  In your example, with 2 chips per node, failing closed would result in the auto voltage reduction at the source, but failing open would take down a string/board.  You could simply pull the board, adding a jumper to force the node closed.  Low tech but it gets you hashing again quickly.

[sidehack]

Could work. If you're already doing board-level soldering, all I'd really need is jumper pads to reroute comms. It'd be best to pull the chips so you don't get comms interference, and once that's done you jumper from VCore to GND pads where the chip used to be. You could also lift the big node-level cap and jump across it. Pads are pretty much free, after all.
Actually, my two-chip "L board" for testing BM1384 is set up a lot like that. I have two power-input jacks, one for Vcore and one for 2*Vcore, and a set of five jumper pads to take comms from either a second chip at the same node (on Vcore) or a second chip at another node (at 2*Vcore). Heck, it's hooked up and running right now (http://eligius.st/~wizkid057/newstats/userstats.php/1BURGERAXHH6Yi6LRybRJK7ybEm5m5HwTr should be seeing 22GH, two chips in a string at 200MHz) so I know the concept is sound.

You'd have to remember to take your core voltage down by one node's worth or your remaining chips will run pretty hot.

On the S5, there are no VRMs at all. 2 chips per node gets them the desired power consumption and hashrate; it's a 30-chip board, just like the S1 had a 32-chip board. Using more chips per node increases your total current (which is one reason the S7 pulls around 400W per board versus the S5's 250W per board, because it has 3 chips wide instead of 2) but you also get better balance. If one chip is running a bit high but another on the same node runs a bit low, they kinda cancel each other out. It's easier to buffer out brief transients with a wider string because any one chip's ripples will be absorbed by the other two chips.
Wider nodes also reduces the number of level shifters required compared to the number of chips. Each node needs a level shifter to bring comm data up to its local ground reference (and other node-level parts for IO voltage and such). The S5 has 15 nodes and 30 chips, so 1 shifter per 2 chips. The S7 has 18 nodes and 54 chips, so 1 shifter per 3 chips. When considering the cost of parts that aren't directly increasing your hashrate, you want to maximize the ratio of ASICs to non-ASICs. This means more ASICs per node.

The optimal from that criteria would be to have all chips in one node, but then you have a VRM design and you start factoring in the relatively high cost of VRMs. The more chips you have per VRM the better, so things like the S1 were okay. The standard chip for VRMs has been the TPS 53355 which has a maximum current output of 30A, which is great for higher-voltage lower-current chips like the BM1380 on the S1, but not so great for the low-voltage high-current BM1384. At top clock, a '55 could power two chips. At midrange setting (say, 275MHz - 15GH/5.4W per chip) you could just barely run four but it'd probably catch on fire if your ambient was warm. The S5 would have needed 15 VRMs (at probably $5 each minimum) per board to run the same hashrate and those VRMs would have decreased the system efficiency by between 10 and 15 percent. By going string on the S5, Bitmain ended up saving $50 in VRMs by adding about $15 in additional node-level parts, and increased the board-level efficiency by at least 10%.

If you can keep the chips in recommended temperature and voltage range, I'd think the odds of failure would be pretty low. If an auto-rerouting system cost $10 for an otherwise $150 board, you'd want a probability of board failure greater than 10/150 or 6.7% for it to really be feasible in the long run. I highly doubt the odds of board failure are that high or we'd be seeing a lot more threads yelling at Bitmain. I'd be surprised if the odds of Prisma board failures was even that high, and those were famous for spontaneous (and often dramatic) death. I ran 44 boards for six months without any failures.

[Witrebel]

Initially I was thinking through hole style jumpers or zero ohm resistors with sockets or something physical that a non board level type person could manipulate, but why have a crapload of parts that you hope you never need to use. 

I mean realistically, you could just make a small "re work" section in the PCB art next to each asic that had comms lines readily available for an x-acto knife, and then just a couple of pads set up for a shorting Vcore to ground and rerouting the comms with a nice some nice crude solder blobs

Do the chips have any on board voltage regulation at all for the core voltage? Or do you need to drop your total input voltage, say from 12v to 11.2v?

[sidehack]

Chips have zero voltage. It would be possible to set a "cut here" zone to disconnect local comms and solder pads to jump to the next node could be populated by either solder blobs or 0-ohm resistors. That's pretty much how my L-board works - except instead of "cut here" I just don't put a chip on the second pad in that node so there's nothing for it to talk to anyway. With a "cut here" you wouldn't have to worry about pulling chips, which reduces the "tools required" from a hot-air station to a $5 firestarter.
Leaving big fat pads to short around can get dangerous, because if you accidentally short them when you don't want to things could get screwed up ala Prisma. If you've got a soldering iron, knock off the node-level caps and short across their pads. Wouldn't take but a minute.
Total increase in PCB cost plus parts cost plus additional assembly, $0 - sounds like a win to me.

The chips have zero means of regulating voltage. The thing that keeps the node voltages at about 0.8V is that all the chips are operating at the same frequency and doing the same mount of work - they're all pulling the same amount of power, so the same amount of current is going through each node, so the voltage divides evenly just like it was a string of equal-value resistors. If one chip starts to misbehave, you can think of its resistance changing (either up or down depending on the error) which means the voltage across it changes and that affects how much voltage is left over to divide among the others in the string.

If you bypassed one node, you'd have to drop your board voltage by one node's worth. If you went from 15 nodes to 14 but didn't change your voltage, now your nodes get 0.860V instead of 0.800V and they start to cook a bit.

[RichBC]

I am far from certain that the complexity of adjusting the supply sting voltage and bypassing the data string is worth cost? What I do think is worth doing would be to detect a chip that is not responding and to shut the board down to prevent further damage.

What would then be useful would be to identify which chip has failed. I do not know if the chips identify and are allocated addresses during power up, but assume this to be the case?

Then perhaps some pads  & cut areas that would enable comms to bypass the failed chip and to attach a suitable value of resistor for the power if chip replacement was not possible?

Rich

[sidehack]

What you're talking about would be even more expensive. The material cost of a "cut here" bypass is zero dollars. The complexity of adjusting the supply voltage is already on the board; I refuse to build a fixed-voltage miner.

If you want to replace your chip with a fixed-value resistor, you need to replace that resistor every time you change your clock or voltage settings. It's not practical. Also, .05ohm 10W resistors probably aren't cheap.

Since now we're at board-level discussion, this should probably be moved to a board-level discussion thread. I think we should get back to the machine specs.

[NotFuzzyWarm]

Chips have zero voltage. It would be possible to set a "cut here" zone to disconnect local comms and solder pads to jump to the next node could be populated by either solder blobs or 0-ohm resistors. That's pretty much how my L-board works - except instead of "cut here" I just don't put a chip on the second pad in that node so there's nothing for it to talk to anyway. With a "cut here" you wouldn't have to worry about pulling chips, which reduces the "tools required" from a hot-air station to a $5 firestarter.
Leaving big fat pads to short around can get dangerous, because if you accidentally short them when you don't want to things could get screwed up ala Prisma. If you've got a soldering iron, knock off the node-level caps and short across their pads. Wouldn't take but a minute.
Total increase in PCB cost plus parts cost plus additional assembly, $0 - sounds like a win to me.

The chips have zero means of regulating voltage. The thing that keeps the node voltages at about 0.8V is that all the chips are operating at the same frequency and doing the same mount of work - they're all pulling the same amount of power, so the same amount of current is going through each node, so the voltage divides evenly just like it was a string of equal-value resistors. If one chip starts to misbehave, you can think of its resistance changing (either up or down depending on the error) which means the voltage across it changes and that affects how much voltage is left over to divide among the others in the string.

If you bypassed one node, you'd have to drop your board voltage by one node's worth. If you went from 15 nodes to 14 but didn't change your voltage, now your nodes get 0.860V instead of 0.800V and they start to cook a bit.

I have been wondering for a long time how the string setup worked reliably without any signs of a shunt regulator across them. So it is only because all chips are more or less doing the same operations at the same time that keeps the voltage divided evenly... Elegant solution to eliminating the need for VRM's provided all the chips have identical tested specs or apparently at least fairly close ones. Is also a dicey solution in that all must chips be doing the same operations so their loads are identical. I do hope that Bitmain is binning the chips to keep all in a chain reasonably matched!

I like your idea on adding mosfets to do just that for when a chip goes down. I assume that the bypass FET will be switched on hard for minimum losses? As you said only problem is that when a chip goes down hard ya lose coms from that point. In hardware mode, as you said - give the associated cap a push to pop it off and jumper across the pads. Perfect solution (until too many get tweaked).

Along those lines.... Looking at Bitmains spec for the S7 they call for 12vdc OR MORE. Up to 5% more.
With the s5 string setup I wonder how much neg margin we have vs stock clock speed? Bitmain had said the S5 could be fed as low as 9.7v when underclocked. On most of my rigs I usually read 12.1x at the supply and 11.92v at the PCIe connectors. The HP/IBM psu's do have a voltage trim/remote sense sooo.
Do you know how low the supplies can be set (for underclocking/volting) and how high when compensating for drop across power leads?
What pin terminal is the sense/trim input?

[sidehack]

I've seen S5 node-level voltage variations of about +/-4% from nominal. The S5 init code ramps the chip work up slowly from zero to full-speed so there are no substantial differences in workload from one chip to another. The Prisma had node-level FET buffering but they got rid of it on BE300 test boards because they weren't having any of the balance issues they saw on Prisma. I don't know what chip-level differences there are between chips designed with string in mind versus parallel/VRM. I do know the 1384 appears to be internally shunted to about 1.2V, which might be a deliberate design choice to keep current at least close to balanced in a failing sring so you don't end up blowing caps and catching things on fire.

[NotFuzzyWarm]

<snip> I do know the 1384 appears to be internally shunted to about 1.2V, which might be a deliberate design choice to keep current at least close to balanced in a failing sring so you don't end up blowing caps and catching things on fire.

It is a certainty that process variations will result in chips that have slightly different electrical operating specs. How much different - dunna know but it can be substantial ergo my hope that they are testing/binning for best matches in a string or at least the chip pairs. Nice of them to think of the need to clamp Vcore. Passive shunt using 2 silicon diodes or zener?

[NotFuzzyWarm]

A good discussion about DC power connectors and the process of selecting them. The course is 49min total http://www.techonline.com/asset/download/4437224/course
Some timestamps for it
A--Slide 1: Introduction and setting the stage 00:00
Part B--Slide 8: Misconceptions, examples, and basics 04:20
Part C--Slide 15: Starting the process 10:00
Part D--Slide 26: Selection process: theory & reality 18:20
Part E--Slide 41: What are other concerns?  33:20
Part F--Slide 46: Working with your supplier 39:10

And one from Intel on PCB design http://www.techonline.com/electrical-engineers/education-training/courses/4000356/Fundamentals-of-PCB-Design

[NotFuzzyWarm]

You seen this about connecting hash boards via debug pins that are on them? https://bitcointalk.org/index.php?topic=889206.msg9902302#msg9902302
It's a simple 3-wire serial port meaning that with a port-selection method of addressing the boards eg COM1, COM2, and such it could be translated with a multi-port rs232/USB switch(s).
Hmm. Easy way to get away from a shared SPI com bus?

[sidehack]

Isn't that post talking about firmware on the controller, or am I reading wrong? Also I don't quite know how anything in your last couple posts has anything to do with anythign.

Especially if we're wanting to steer back to the actual topic of discussing mechanical design instead of board-level, which excepting the mechanical constraints of the PCB is completely outside this thread's intended scope.

[NotFuzzyWarm]

Ja is rather OT from the mechanicals I guess. Figgered the connectors vid could apply to PSU backplane connections though, the debug maybe as alternate way to address chips 
And now back to regular scheduled programming.

[sidehack]

Ah, okay. I see what you're saying on the power backplane.
Chip addressing shouldn't really be an issue how I see it, since each board would enumerate as its own device in cgminer if they were USB-connected. No one board should overlap with another. Or are you thinking a multiplexer on each board that talks to each node independently?