That's very interesting. So you didn't try to program it, but you hooked up the unit. Did you try reading the code? Did you get anything at all?

The reason it's el-interesting-o is that there's a little user page for data and the official page for code and all that stuff on the Atmels. You might have cleared the data field when you tried to read the code. Now, normally that would spell fuck-ola, but what if BFL used the same code load and put the "run at lower speeds" commands in the data field. By clearing it you got a default monarch (which I think has some sort of speed control like the Single/60's had, there is no way my Monarch comes up at 699gh every time)

Hm. There is a way to change the clock speed. Which means they needed to store state. And the chipset on there does support dynamic setting of voltage points, that's in the Intersil documentation. How can I write commands out to the USB port using something as dumb as HYPERTERM or PUTTY? Is it possible to just connect to the USB serial port and say "ZCX", "ZAA", "ZAB", etc and just fuzz the damn thing until we trip over something?

Well, here's a Monarch pic for everyone.



This is a view inside the pump from a leaking Monarch that came in to me for someone. The water was leaking, and the Monarch is dead. Dead part is simple: I can see that the FETs on the left side are shorted, the gate to drain resistance is 0 (ie: short) while the gate to drain on the other one is 50 ohms (good gate).

So I took off the water block, opened it up and found it flooded. Cleaned it out, put water in the unit by taking off the right side hose, and that's what I saw.

Those bubbles are indicitive of a water leak. Looks to be right at the junction in the pump where the water pipe goes into the pump. I am not certain of course, but it looks like a defect in the pump.

Interesting. Anyone else got a monarch that's dripping? Is it coming from inside the pump housing? Which side?

C

The broken monarch came in under the "just donate the repair cost to a food bank" program :-) 500gh unit, small heat sink on the back, leaking water system, shorted out unit.

Sure enough it was leaking even on the trip over, and when I shake it I can hear the water sloshing in the radiator (which means it is not full, and ain't gonna work). Need to talk to the user about opening the loop and purging it.

However I think I see the problems: One is that it's not leaking around the pump it's leaking *inside* the pump block, specifically by the left flexible fitting. That's really odd.

And the FETs on the left side are shot. We can tell because of the 0 ohm resistance at the +12 power plugs (short) and the 0 ohm readings on the gates on one side with 50 ohm readings on the other side's gates. Simple enough, parts on the way.

Hm. I'm going to clean this up and take some pics to see if that's where the problem really is. Running with the radiator upside-down allows it to run with less than 100% water so I can see it leak.

*nod* I'd agree, the Monarch is not going to work very well in racks in a data center. The more I watch reports of failures, and the more I think about it, the more I see that the big failure here is that it's throwing heat in 3 directions:

• There's the heat coming off the radiator on the end. A lot, but well managed; it blows out and away.
• There's the heat coming from the back of the board. Also a lot, but going sideways.
• And there's the heat coming off the tops of those FETs. Just drifting off the other side.

That's probably the killer: If you put a monarch next to another monarch, the heat from the back of one set of fets blows onto the front of the one next to it. Making those fets hotter. And then next one you put down is going to get even hotter air blown on it from the neighbors. Eventually the FETs overheat, conduct, and short out. Or they throttle a lot and run slow. But with limited heat transfer ability I could see a top fet overheating before the temp sensor on the bottom picks it up.

I think that's the #1 problem with these things (technically): If you don't have some air flow, a heat sink is nothing but a metal blanket on a part. And heat sink efficiency depends completely on the difference in temp between the part and the ambient air temps. Blow hot air on a FET with a heat sink and it's not going to work. Blow air front to back, and you will just cavitate the air on the back heat sink (essentially fighting with the fan), and totally miss the tops of the FETs (due to the water block).

Top to bottom fans would be better, but then you need to move the heat into the hot aisle with baffles or something. And the bottom of the tray will limit the air flow. But in any case, the bottom FETs are going to get hotter and conduct more than the top ones. Which will result in the bottom ones conducting too much, going into thermal runaway, and boom.

The second thing is a basic problem: Those FETs are just packed in too damn close. On the Singles you had six FETs in a small area, but then you had board space for the heat to dissipate. Here you have 18 FETs on one side, the 18 on the other between two of the hottest things on the board and the FET drivers in the middle (which generate heat because they have to switch the gates under a 24-1 voltage ratio). There's just nowhere for the heat to go but out the back, and the high density means that a lot of heat needs to be moved off the board quickly.

The back heat sink doesn't get too warm, which sounds nice except that it means the thermal transfer from the board to the sink is limited. That's why the FETs went to 100c while the heat sink on the back was cool to the touch. A front fan fixed that, but still not optimal if you have more than one Monarch.

Kind of reminds me of the singles: Putting half the FETs on the ends of the board was a good idea. But putting the other half in the center of the board, between the heat sinks for the chips, was a bad idea. That's why you could either run them like banshees with full fans in the cases, or more quietly if you took them out of the case and put a single fan blowing air in from the *side* against those center heat sinks.

Ug.

I'm getting a blown one in tomorrow, shorts power supplies and the pump was leaking, I'll post what I find as I fix it. Could just be a shorted FET, if so removing the fets on that side should get half of it going and putting on a new FET should fix it. We'll see.

Actually if this one's plumbing is bad as well I might try to splice in one of the sedion water blocks I used on a jally and put that on the back of the board while purging the water loop. Then *all* of the heat would go to the radiator and out, that might do it. Or just put a water block with it's own radiator on the back and see what happens to the FETs on the front.

Hm.

Well, if they were not under receivership then we could at least ask them what was in the code, if they could open-source it, how the 4.2.0 BFG code works, etc. I doubt the receiver would know. However if they released instructions for overclocking then people would do it, blow up their miners, then come back to BFL demanding warranty work or refunds.

So it would be in everyone's best interest not to say. *sigh*

The hashing changing with a different power supply thing someone pointed out is interesting: It's possible that the Monarchs are doing a more advanced version of what the Singles and Jallies did when they powered up: They would do sample jobs and such adjusting the clock frequency to find a perfect match within a speed band. That's why they always reported slightly different speeds with ZCX when powered on.

Now on the 1850 chips you could not change the chip voltage, that was controlled by a pair of resistors in a bridge for each set of 8 chips and was static (upping it with a soldering deck would cause all SORTS of fun, Chilis did this and re-defined the meaning of "hot running" for 10+ more gh).

On the Monarch power control chips however, you can. That Intersil chip can do a lot of things....

So now I wonder if what happens is when you power the miner up, the software spends that minute before starting to has sweeping the chips with a range of clocks *and* voltage settings to find the best balance. It would make sense, since that's what the singles kind of did to the best of their ability.

That would also explain why units that are powered down hot then back up before they completely cool will throw those queue errors and make a slightly different high pitched sound: When hot the components like resistor bridges and RC circuits would react a bit differently and it could be choosing a more power/more noise solution as an optimal one. Perhaps the 4.2 code disables sweeps at too high a voltage/clock, or has error correction to clean up any noisy results on the fly.

*sigh*

C

Really? Where did I blow it? (I forgot with rewards, didn't I, LF uses relative addressing in his talking)

The last block with rewards could be solved by a single miner running on a 386sx/16 chip if there was only one miner left in the network but it would still only take that miner 10 minutes on average since the difficulty would have sunk as other miners left the network.

C

Also remember difficulty can go down as well as up. We had a difficulty change of 1% a few days ago, apparently miners left the network at the same rate as new ones came in. The last satoshi of the last bitcoin will be mined approximately 10 minutes after the next to the last one.

That's how difficulty works. As long as there are 2 people mining with 386sx computers, a block will drop every 10 minutes.

C

Got about 2 gallons of beer ready to go. I love brewing.

FYI the latest version didn't fix the Monarch errors. Luke, if you're interested I'll do a quick post of what I found. I did have a hack where it tries to do an intelligent match on the replies back to BFGMiner to compare to the queue to find dropped bytes, but it's spectacularly ugly and that 4.2 custom code doesn't do that from what I can tell.

I think they just went back to single-reply responses and single work loads. Finding exactly where these happen in the BFL code is tough for me.

C

Well, an update: Have two people sending over Monarchs for review. One leaked coolant and probably shorted a FET, the other is a unit which comes up but has no engines on both sides of the chips.

They're coming in USPS Priority Mail, will review over the weekend. Fees include donations to local food banks.

Will post thoughts and results.

C

Sure. Pack them up USPS Priority Mail, as discussed payment is make a donation to a local food bank or soup kitchen. I'll post the findings and such on my "Hacking a BFL Monarch" thread, and a summary here just to close the loop.

C

More like Helm's Deep with that little guy selling his one bitcoin--shooting his little arrow.

That will bring on the HORDE! So no HORDE, HODL!

The simpler solution is to cut the hose and plumb in a real pump with an air reservoir/trap. That would do it too, just make sure your new pump runs in the same direction as the two on the Monarch board and still run the ones on the board (I think they would work best if all were running in the circuit).

Problem is the pumps in those units can't self-prime. They're not designed to. When they pump, they expect to pull water in and push water out. If there are air bubbles they will either cavitate or not be able to pull enough vacuum to prime.

Check out item number 111241824048 on Ebay for an example. Note the water above allows the air in the system to bubble out, and provides a hell of a lot of room for expansion (and it looks like a cute little bong).

That will do it. Hm. No, I will not think of ways to plumb a Monarch into a bong. Hm.

To quote Bill Murray:

I think it sucks :-)

There are two problems that seem to happen. Both I think are related to minor noise in the data stream and how a lot of data can go to the BFL chips' very high number of engines.

The biggest problem is the annoying "data mismatch". What happens is the BFGMiner queries the Monarch in a block mode for all queued results, and the Monarch sends back all the data with a unique serial number for the block in question, followed by the results. The problem is that can get garbled between the Atmel sending it on the parallel bus, the FTDI chip that converts to serial, and of course the computer running BFG.

If an entry is garbled, all the results are thrown out and now you have a case where the Monarch thinks its' queue is empty but BFGMiner has the requests outstanding and doesn't send any more data to those engines. Eventually it times out with a huge number of errors that can sometimes crash BFGMiner and it restarts.

The other problem is sending data *to* the Monarch. Because of all the engines you can send a whole crap-ton of requests at once, I think if one of those gets garbled the Monarch never works on the job but BFGMiner either thinks it's occupied or you get the "Failed to send queue".

Couple that with the fact that sometimes if I power my Monarch off and leave it off for an hour to cool off it works perfectly with BFG 4.8. No errors even after a week of running. But turn it off and on hot and it will drop in speed to 400-500gh with occasional massive streams of errors.

BFGMiner 4.2 on the BFL site does not have this problem AT ALL. Literally I can stop 4.8 with errors and crashes, plug in my 64 bit Windows 7 laptop, run the 4.2 code (it's compiled 64 bit. AARRGGHH!) and it works flawlessly. Never a burp, problem, or anything. So it's not really the hardware, or more to the point it can be fixed in software.

Why. Why why why why why why why? I'm thinking maybe the queueing features were disabled; in the GIT code repository I see that someone made a change to the code I was fiddling with that solves the first problem well but the Filed to send queue eventually jams up the Monarch and BFG. Thus the "it's a two part thing" I think.

It doesn't help much that I know how to program perfectly in Fortran-77 and my C abilities are more Objective C from NextStep. The C++ code in the Bitforce section is really kind of... tight...

C

Ok. Banging on the code for the BFL Monarchs. Still need to fix the highly annoying error where it gets stuck, drives me nuts that the custom 4.2 64 bit code works perfectly but the 4.8 has queueing problems.

C

Quick question: Is it possible to request a higher difficulty by default in BFGMiner?

C

Remy=How cute! I could see sending her off to the Prom with her little date (yeah, I'm old).

Better to post a pic than a link to a flash site. For whatever odd and unknown reason I am not comfortable unblocking flash on a linked porn site from bitcointalk. I wonder why. :-)

Oh, remember: Water cools better than propylene glycol, but the problem is it will freeze when temp goes below 32f. So never leave it in an unheated garage or outside in the snow when not running.

You could also just ask BFL to send you a new set of water blocks and replace on both chips. If youre comfortable with AS5 that could be a quick solution.

C

Ok, thoughts:

1) Clean off the areas the water got on (it's water and propolyne glycol) with 95% isopropyl alcohol (good rubbing alcohol, at the CVS). Use a paper towel soaked with it, then q tip. That will clean off the stuff, don't knock off those little heat sinks.

2) The screws that hold on the water block to the copper plate are *under* the copper plate, so you would have to take it off the chip to tighten the right screws. When you take it off you will see them, tighten them down and make sure they are snug, but not insanely tight (or you will strip them).

When you take the block off the board and chips, you will have to clean off and re-apply the heat sink compound.

I like Artic silver 5, the grey stuff. Clean the old stuff off the sink and the chip with a paper towel with alcohol again. Do not get it on the sides of the chip, wipe it so it comes right off the chip itself. Then follow the instructions in the AS 5 thing to put new stuff on. A thin coat is all one needs, I put a thin trace on the chip, then clean my finger with isopropyl and use my finger to spread it on the chip evenly. Once again don't go over the sides of the chip but get to the edges.

Then I put a thin smudge on the heat sink plate itself and put it back on. Put the screws in loosely at first, the put them back on and tighten them in a 4 cross pattern (1,3,2,4,1,3,2,4) a little each time. Remember if you tighten down one side all the way then the other, it could crack the chip edge. Finger tight, enough so that with your fingers gently squeezing the screwdriver you snug the screws down.

Then the three or two outside screws, they provide structural stability which is nice.

Now time to fire up. Start up the unit and keep your dry finger on the back behind each chip. If it gets hot to the touch shut down *IMMEDIATELY* and purge. If it runs hashing for 5 minutes and is not hot to the touch (100 degrees is about what I would say is getting hot) then you probably do not need to purge.

If you need to purge:

Well, you have to pull one of the hoses from the radiator, then fill it with the pumps running. Bit complex. What I did was replace a hose from the pumps to the radiator. Put both the radiator and the hose in a bucket, immerse them and start up. Stuff will come out of one of them and be sucked in the other. Lift the one that it's coming out out of the water and run it until there are no bubbles. Then put both back in the bucket while it's running and complete the connection. That way you have no air going into the system and it will work.

Fun, but do-able. Make sense?

C