Diagnosing and repairing the common Mini-Rig

[Inspector 2211]

So my Mini-Rig was scheduled for delivery on Tuesday, but when I arrived at my mining office at 10:40, it turned out that I had missed the FedEx guy by less than half an hour. Another 100 bucks down the drain. Having once owned an airplane, I've acquired a very high tolerance for wasted Benjamins, however, and merely made sure that I won't miss him on Wednesday, by explaining the situation to my manager and taking the company-issued notebook with me when I left that evening. For the first time ever, I would work in my sauna-like mining office - only until the delivery, of course.

After giving the FedEx guy a Coke as a tip and savagely ripping open the cube-shaped package, I found the Mini-Rig upside down, its tiny rubber feet in the air. After wrestling the beast into an upright position, and noticing (but ignoring) something rattle inside, I plugged both of its USB cables into a prepared Netbook running cgminer 2.6.4 on Ubuntu 12.04. I plugged in the power cable, and - bingo - all 17 miners came up. Time to go to my daytime job.

This could have been a Hollywood-style happy ending, but, as it turned out, my joy was premature.

Upon arriving at the daytime job, I immediately saw that the hash rate of the Mini-Rig was not the advertised 22 GH/s, but just 16 GH/s or so.
Facing a deadline on Friday, a last-minute feature request which involved generating an .rpm on Ubuntu (!) and integrating this into my employer's build system, which is NOT based on "make", but on something quite esoteric, I could not spare the 20 minute round trip time and drive to the mining office. Diagnosis and repair would have to wait till the evening.

Back at the mining office, which is 10 minutes from my daytime job, I immediately saw what's going on: 6 of the 17 boards inside the mini-Rig were offline. Restarting the rig resolved the situation only for a very short time. Soon, it was back to 6 failed boards.

Shipping the rig back to BFL would be the coward's way out, and would cost me several hundred dollars in lost mining revenue. So I decided to open the case and have a look.

The U-shaped top/front/back is secured to the more massive, also U-shaped bottom/left/right by an impressive number of hex screws, probably around 30 in total. The last six are easy to overlook as they sit at the bottom of the case.

Lifting the top/front/back part up and turning it upside down before placing it on top of the main chassis in a perpendicular fashion revealed the guts of the mini-Rig. There are 3 populated "floors" housing 6 circuit boards each, one floor - hard to see which one - obviously missing a board, for a total of 17 boards.
The top floor is not populated, and thus someone with 4 mini-Rigs could turn them into 3 fatso-Rigs and host them at Inaba's data center for $900, saving $300 per month.

The LCD in front only displays static text - I assume that grand dreams of a CGminer feedback path to this display had been sacrificed on the harsh altar of software engineering reality months ago.

I also identified the source of the rattling: One of the 17 circuit boards had worked loose during shipping and was merely resting on its standoff, instead of being secured to it. Quite easily the board could have come to rest skewed or even diagonally, with one of the standoffs short-circuiting and frying the board. I was incredibly lucky that all 17 boards still worked - but 6 of the 17 would always drop off after a little while. Or would not come up in the first place.

I went to one of my other, BFL Single-occupied, mining offices, where one netbook was handling 9 Singles, but another netbook only one Single (don't ask). For the sake of science, I unplugged the lonesome Single and carried the netbook to my mini-Rig office. There, I plugged each of the two USB-cables of the mini-Rig into a different Netbook, power-cycled the mini-Rig and observed the outcome. Would this resolve the problem? Not fully trusting Netbook hardware nor CGminer stability, I think a $100-a-day device like the mini-Rig deserves to be plugged into two netbooks, anyway.

It did *not* resolve the problem. But it shed some more light on it.

One netbook had 8 boards mining like a charm.
The other netbook only saw three working boards - six boards going dead instantly or after a little while.

So, at least I had bisected the problem and immediately proceeded to bisect it again. Why are 3 boards working, but six are not?

I looked at the USB hub. The mini-Rig sports two 10-port USB hubs - I know them very well, as I have about ten of them myself. In fact, I use them to hook up my Singles to various netbooks and notebooks. None of the USB hubs in the mini-Rig is externally powered - something which would be quite easy to do, as a PC power supply, which is used in the mini-Rig, does sport 5V wires.
As the old saying goes, for want of a few butt connectors (they are really called butt connectors, I'm not making this up), a mini-Rig was lost. Or 35℅ of it, at least. But I'm jumping ahead.

Each of the 10 ports has an activity LED, or maybe it is a power LED. I immediately saw that while 3 of these LEDs, the ones closest to the USB uplink, were steadily on, the other 6 were frantically blinking in blue. Blue as in "Code Blue". I walked to one of my other mining offices where I keep the stash of 10-port USB hubs, grabbed one of them, and walked back to the mini-Rig office. Who says that Bitcoin mining doesn't provide exercise...

I then plugged the new 10-port hub into another USB port of the netbook with the 3 working and the 6 sick miners, pulled out one of the Mini-Rig cables of the code-blueing BFL hub, plugged it into my own hub and hit 'q' on the CGminer console.
I run CGminer from a trivial shell script with an endless loop, so 'q' really means "restart" here and - bingo - I had 4 working boards and 5 non-working ones on this netbook. BFL hot-glues the USB plugs onto the hubs, but it's cheap glue and cannot stop a determined Bitcoin miner. Soon I had 5 working boards and 4 non-working ones, I worked loose the third cable, plugged it into the new hub, hit 'q' on CGminer and - I had 9 working boards, in addition to the 8 working boards on the other netbook. Unplugging 3 cables had resolved the overload condition and the sick hub had recovered, now successfully handling 6 USB cables.

Since then, the mini-Rig is running at approximately 24 GH/s, and I'm a happy camper.

However, I still have a few recommendations for BFL.

1. When fastening a board to standoffs, or, in fact, anything to anything at all, please use a thread-locking adhesive. It will prevent a screw from working itself loose during shipping. I have received Singles with loose screws rattling around inside, and now the mini-Rig with a whole board rattling around inside. Stop the rattling, and I promise I'll stop the tattling.

2. When a cheap hub from China boasts it's a 10-port hub, that doesn't make it so. Take this spec with a grain of salt, and ship a mini-Rig with three hubs, loading each one with 6 plugs only. Even netbooks have three USB jacks. Three cables coming out in the back is still OK.
Or, alternatively, power the hubs. 5V cables are available a few inches away. Use them. Two words: "Butt connector". Two more words: "Crimping tool"

3. There are horizontal aluminum plates or steel plates, intended as fan baffles, inside the mini-Rig and they just rattle around inside. They are not bolted down. This is very sloppy, as these plates could damage something during shipping, for instance one of the boards in case a board comes loose (as it did). As I said, my mini-Rig arrived upside down.

Regards,

Harry C.

[1714176283]

1714176283

[1714176283]

1714176283

[1714176283]

1714176283

[1714176283]

1714176283

[1714176283]

1714176283

[Carlton Banks]

Lol, post reminded me of that photo on the BFL site of a sweet little old lady (squinting through her spectacles and smiling), captioned "Quality control ninja"

[BFL]

So my Mini-Rig was scheduled for delivery on Tuesday, but when I arrived at my mining office at 10:40, it turned out that I had missed the FedEx guy by less than half an hour. Another 100 bucks down the drain. Having once owned an airplane, I've acquired a very high tolerance for wasted Benjamins, however, and merely made sure that I won't miss him on Wednesday, by explaining the situation to my manager and taking the company-issued notebook with me when I left that evening. For the first time ever, I would work in my sauna-like mining office - only until the delivery, of course.

After giving the FedEx guy a Coke as a tip and savagely ripping open the cube-shaped package, I found the Mini-Rig upside down, its tiny rubber feet in the air. After wrestling the beast into an upright position, and noticing (but ignoring) something rattle inside, I plugged both of its USB cables into a prepared Netbook running cgminer 2.6.4 on Ubuntu 12.04. I plugged in the power cable, and - bingo - all 17 miners came up. Time to go to my daytime job.

This could have been a Hollywood-style happy ending, but, as it turned out, my joy was premature.

Upon arriving at the daytime job, I immediately saw that the hash rate of the Mini-Rig was not the advertised 22 GH/s, but just 16 GH/s or so.
Facing a deadline on Friday, a last-minute feature request which involved generating an .rpm on Ubuntu (!) and integrating this into my employer's build system, which is NOT based on "make", but on something quite esoteric, I could not spare the 20 minute round trip time and drive to the mining office. Diagnosis and repair would have to wait till the evening.

Back at the mining office, which is 10 minutes from my daytime job, I immediately saw what's going on: 6 of the 17 boards inside the mini-Rig were offline. Restarting the rig resolved the situation only for a very short time. Soon, it was back to 6 failed boards.

Shipping the rig back to BFL would be the coward's way out, and would cost me several hundred dollars in lost mining revenue. So I decided to open the case and have a look.

The U-shaped top/front/back is secured to the more massive, also U-shaped bottom/left/right by an impressive number of hex screws, probably around 30 in total. The last six are easy to overlook as they sit at the bottom of the case.

Lifting the top/front/back part up and turning it upside down before placing it on top of the main chassis in a perpendicular fashion revealed the guts of the mini-Rig. There are 3 populated "floors" housing 6 circuit boards each, one floor - hard to see which one - obviously missing a board, for a total of 17 boards.
The top floor is not populated, and thus someone with 4 mini-Rigs could turn them into 3 fatso-Rigs and host them at Inaba's data center for $900, saving $300 per month.

The LCD in front only displays static text - I assume that grand dreams of a CGminer feedback path to this display had been sacrificed on the harsh altar of software engineering reality months ago.

I also identified the source of the rattling: One of the 17 circuit boards had worked loose during shipping and was merely resting on its standoff, instead of being secured to it. Quite easily the board could have come to rest skewed or even diagonally, with one of the standoffs short-circuiting and frying the board. I was incredibly lucky that all 17 boards still worked - but 6 of the 17 would always drop off after a little while. Or would not come up in the first place.

I went to one of my other, BFL Single-occupied, mining offices, where one netbook was handling 9 Singles, but another netbook only one Single (don't ask). For the sake of science, I unplugged the lonesome Single and carried the netbook to my mini-Rig office. There, I plugged each of the two USB-cables of the mini-Rig into a different Netbook, power-cycled the mini-Rig and observed the outcome. Would this resolve the problem? Not fully trusting Netbook hardware nor CGminer stability, I think a $100-a-day device like the mini-Rig deserves to be plugged into two netbooks, anyway.

It did *not* resolve the problem. But it shed some more light on it.

One netbook had 8 boards mining like a charm.
The other netbook only saw three working boards - six boards going dead instantly or after a little while.

So, at least I had bisected the problem and immediately proceeded to bisect it again. Why are 3 boards working, but six are not?

I looked at the USB hub. The mini-Rig sports two 10-port USB hubs - I know them very well, as I have about ten of them myself. In fact, I use them to hook up my Singles to various netbooks and notebooks. None of the USB hubs in the mini-Rig is externally powered - something which would be quite easy to do, as a PC power supply, which is used in the mini-Rig, does sport 5V wires.
As the old saying goes, for want of a few butt connectors (they are really called butt connectors, I'm not making this up), a mini-Rig was lost. Or 35℅ of it, at least. But I'm jumping ahead.

Each of the 10 ports has an activity LED, or maybe it is a power LED. I immediately saw that while 3 of these LEDs, the ones closest to the USB uplink, were steadily on, the other 6 were frantically blinking in blue. Blue as in "Code Blue". I walked to one of my other mining offices where I keep the stash of 10-port USB hubs, grabbed one of them, and walked back to the mini-Rig office. Who says that Bitcoin mining doesn't provide exercise...

I then plugged the new 10-port hub into another USB port of the netbook with the 3 working and the 6 sick miners, pulled out one of the Mini-Rig cables of the code-blueing BFL hub, plugged it into my own hub and hit 'q' on the CGminer console.
I run CGminer from a trivial shell script with an endless loop, so 'q' really means "restart" here and - bingo - I had 4 working boards and 5 non-working ones on this netbook. BFL hot-glues the USB plugs onto the hubs, but it's cheap glue and cannot stop a determined Bitcoin miner. Soon I had 5 working boards and 4 non-working ones, I worked loose the third cable, plugged it into the new hub, hit 'q' on CGminer and - I had 9 working boards, in addition to the 8 working boards on the other netbook. Unplugging 3 cables had resolved the overload condition and the sick hub had recovered, now successfully handling 6 USB cables.

Since then, the mini-Rig is running at approximately 24 GH/s, and I'm a happy camper.

However, I still have a few recommendations for BFL.

1. When fastening a board to standoffs, or, in fact, anything to anything at all, please use a thread-locking adhesive. It will prevent a screw from working itself loose during shipping. I have received Singles with loose screws rattling around inside, and now the mini-Rig with a whole board rattling around inside. Stop the rattling, and I promise I'll stop the tattling.

2. When a cheap hub from China boasts it's a 10-port hub, that doesn't make it so. Take this spec with a grain of salt, and ship a mini-Rig with three hubs, loading each one with 6 plugs only. Even netbooks have three USB jacks. Three cables coming out in the back is still OK.
Or, alternatively, power the hubs. 5V cables are available a few inches away. Use them. Two words: "Butt connector". Two more words: "Crimping tool"

3. There are horizontal aluminum plates or steel plates, intended as fan baffles, inside the mini-Rig and they just rattle around inside. They are not bolted down. This is very sloppy, as these plates could damage something during shipping, for instance one of the boards in case a board comes loose (as it did). As I said, my mini-Rig arrived upside down.

Regards,

Harry C.

If you're not getting the minimum 25.2 GH/s, contact josh@butterflylabs.com for guidance.

[Inspector 2211]

Today, I took another look at the mini-Rig.
Why is it not getting the advertised performance of 25.2 GH/s?
I counted USB plugs, took a flashlight and peeked into all three "floors" of the mini-Rig and was stunned to find out that the mini-Rig actually has 18 FPGA modules, not 17 as some people had reported.

But the sum of the FPGA modules seen by two instances of CGminer on two netbooks was only 17, not 18.
Did I have a DOA FPGA module on my hands?

By tracing the one USB plug which always resulted in a dark LED, something I had overlooked the day before,
I arrived at the conclusion that the faulty FPGA module resides in the middle floor/tray of the mini-Rig.

I decided to remove it for RMA purposes and also fasten the FPGA module at the bottom floor, which had worked loose during shipping.

On top of the rig's interior, there are two crossbeams of unknown purpose, maybe they serve to brace the top of the case so that one can safely sit on it. Or put mini-Rig on top of mini-Rig on top of mini-Rig. The crossbeams are attached to huge standoffs by large Phillips screws, but it was not particularly hard to remove the 4 screws and the 2 crossbeams.

The standoffs, not your typical toy size standoffs but huge, thick ones, proved a more difficult challenge. They are round standoffs, not hex standoffs, so my ample collection of nutdrivers was of no use to me. (I once had an agitated discussion with a TSA agent in Burbank airport, explaining to her that a nutdriver is not a weapon, because in contrast to a screwdriver, it is completely blunt. But she just kept repeating "no tools over 7 inches", whereupon I finally gave up, dumped the nutdriver in a garbage bin and counted my blessings that this standoff had not devolved into a strip search and that her name was not Lorena Bobbitt.)

In the end, I chose to attack the huge round standoffs with a full-size pair of pliers, and one by one, all four of them could be twisted off. Which allowed me to remove the unpopulated 4th floor, which doubles as the ceiling of the 3rd tray level.

On top of fans of the 6 modules on the 3rd floor there is a aluminum baffle plate, whose purpose is to prevent hot air from circulating back down to the PCBs. It has 6 circular holes for the fans, but is otherwise not secured vertically - in case the mini-Rig is flipped over, this plate will just rattle around. It has about 1 inch or 1 1/2 inches of play.
Since it is not secured, all I needed to do is just take it out.

Now tray #3 was exposed in all its glory. But tray #3 was working just fine, it neither had a loose FPGA board nor a dead FPGA board. So I unplugged both of its power connectors, lifted it out and set it aside.

I repeated the "remove 4 giant standoffs" and "remove the rattling baffle plate (baffling rattle plate?)" actions, and had a clear view of tray level 2 with its dead FPGA module. However, there was no outward sign as to why the FPGA module should be dead. It had power connected to it, its LEDs were on, and its USB cable was plugged in. I unscrewed both of the screws which attach the FPGA module to the floor #2, then put it on its side, removed the 90 degree USB mini connector and plugged in a known-good USB cable, before reconnecting power to the mini-Rig, wondering whether the cause of the dead module was merely its USB cable. Alas, no such luck. It was a dead module allright. I loosened the two screws on its power connector and removed the power cables. Then, I put the defective module on my workbench.

But I still wanted to reattach the FPGA module on floor 1 which had worked loose during shipping, so I unplugged the two power connectors of tray 2, then lifted out tray 2 and set it aside.

Standoffs. Baffle plate. Rinse and repeat.

At last, tray 1 lay exposed. Yes, the left front module had come loose during shipping, but no other one.
Miraculously, the module had survived its ordeal unscathed was one of the 17 working modules.
I stole the screws from the now-removed DOA module and secured the rattling, but working module into place.

Before reassembling the rig, I borrowed a know-good module from tray 3 and put it into the dead module's spot on tray #2.

After assembling the rig, I turned it on.
Once again, mining with 17 modules at about 24 GH/s total.
Phew.

Finally, I unplugged a Single and plugged in the suspected-dead mini-Rig module in lieu of it, restarting CGminer.
Would the board miraculously rise from the dead and work on the lab bench, if not in the mini-Rig?
As I had expected, it didn't.
I then asked BFL for an RMA number, via the contact form on their website.

Tune in next week to find out:
- How quickly will I get an RMA number?
- After sending in the DOA module, how quickly will BFL ship a replacement?
- Will I be able to reach the advertised 25.2 GH/s?

[thebaron]

Dear BFL:

Lockwashers.

Love,
thebaron

[crazyates]

Is it profitable to miss out on a week of mining at 24GH/s just to get 25GH/s in the future? I would hate getting a dead board, but Im just curious.

[MXRider]

Is it profitable to miss out on a week of mining at 24GH/s just to get 25GH/s in the future? I would hate getting a dead board, but Im just curious.

Not profitable. That's why he is sending only the broken module to BFL.

[uuidman]

Happy to see you handling the mini-rig and not selling it, regarding what you wrote some time ago. Very good feedback for BFL to improve from.

[Inspector 2211]

Yesterday (Thu.) I received the replacement module from BFL. Thankfully, they had cross-shipped. Just before midnight, I installed it in the mini-Rig and it works!

However, I noticed there is another problem module in the mini-Rig - a module which mines a little bit, then goes into an error state (as evidenced by cgminer not being able to read its temperature), then comes back and mines a little bit more, before going into an error state again. Sometimes, for instance on Thursday, it stays down, until the power is cycled.

I've asked BFL for a replacement for this sick module also - too early to expect a reply.

[Cablez]

That is worrisome news inspector, I hope BFL address it very soon.

[Inspector 2211]

Now the flaky module has completely failed and cgminer doesn't recognize it anymore. Bummer.

[JWU42]

I feel your pain.  Quality of the mini rig was not as planned.  19 units in mine, 2 totally unsecured ( no screws) and one with a power connection problem.  1 dead USB hub to start that caused 9 units to become bad flashed.

It has been running ok now for a week but the fit and finish is lacking.  I guess my expectations were too high.  Also not at 25.2 gh...

They were quick to send a replacement USB hub and a recent rma for a single was also handled very fast - kudos for trying to make it right.

I hope this does not provide a glimpse in to what the SC mini rig will be...

[arklan]

since it appaears the casings were custom made, it's a shame about the rest.

[fcmatt]

All of this and nows asics to ruin the party. I hope they are months late so you can make some bank!

[BFL_Josh]

I am working on improving the build quality and QA procedures for both the singles and minirigs... and also educating the assemblers in the use of Loctite blue and proper application of thermal paste.

Sorry to hear that you had so many problems, are you taken care of now (sorry, I just saw this thread.)?

[Inspector 2211]

Josh, I'm still waiting for shipping notification of my second replacement module.
I'm once again down to 17 modules and approx. 24 GH/s.

[Inspector 2211]

Today, I received the tracking number for the 2nd replacement module.
I appreciate that BFL cross-ships.
I plan to have only one downtime of the mini-Rig, when I remove the failed module and mount the replacement module, not two downtimes.

[Inspector 2211]

I replaced the second dud module today and will ship it back to BFL tomorrow.

The new module works perfectly.

One thing I noticed - BFL did not ship the replacement module in an antistatic bag.
These bags exist for a reason.
Every motherboard and every hard disk and every graphics card is shipped in an antistatic bag.
For fun?
To spend an extra 20 cents just for kicks?
I don't think so.

Apart from starting to use a thread-locking adhesive, this is one other opportunity for improvement Josh may look into.

[live627]

One thing I noticed - BFL did not ship the replacement module in an antistatic bag.

What? I would hope they know better than that. Was your first replacement module in one?

[Inspector 2211]

One thing I noticed - BFL did not ship the replacement module in an antistatic bag.

What? I would hope they know better than that. Was your first replacement module in one?

Nope.
Just bubble wrap.