Bounty[PAID OUT] : a bitstream for better utilizing the Cairnsmore1 157-294.5btc

[gyverlb]

[...]
My problem might be that I can't update the controller and still use the 1.1 firmware on it but I don't have any solution for it (no Windows license here).

I would just try and download a copy of Windows 7 install it without a key, you get 30 Days trial, or borrow someones Win7 Cd and load it from that. That way you could at least get to 1.3, or 1.4 now. You don’t actually need to own the copy just get it installed and use it for 30 days.

There are 2 problems that make this impractical:

• I've not installed Windows since XP (and even then it was a one-shot because I had to for work) I'd probably spend several days on this in my free time. Last time I saw a Windows system I didn't even know where to begin, its UI is now a total stranger to me.
• I don't have any system on which to install Windows 7. The only available system I have is a test system without a harddrive, booting from an USB key. I'd have to buy a drive or install Win7 on an USB key (is it possible yet?). I thought of using virtualisation on my laptop, but I'm afraid I don't have the disk space on my SSD (I've ~15GB left and that would stretch it).

My best bet if Enterpoint doesn't release a Linux version is to find someone with a Windows install (can't think of any friend with that) and pray that I won't trash his/her system. I could sell the boards too... Ironically I wanted to buy some more, but there's no way I'd knowingly buy something that requires me to spend time on something I don't want to. Especially if it's related to mining: time is money.

[ebereon]

[...]
My problem might be that I can't update the controller and still use the 1.1 firmware on it but I don't have any solution for it (no Windows license here).

I would just try and download a copy of Windows 7 install it without a key, you get 30 Days trial, or borrow someones Win7 Cd and load it from that. That way you could at least get to 1.3, or 1.4 now. You don’t actually need to own the copy just get it installed and use it for 30 days.

There are 2 problems that make this impractical:

• I've not installed Windows since XP (and even then it was a one-shot because I had to for work) I'd probably spend several days on this in my free time. Last time I saw a Windows system I didn't even know where to begin, its UI is now a total stranger to me.
• I don't have any system on which to install Windows 7. The only available system I have is a test system without a harddrive, booting from an USB key. I'd have to buy a drive or install Win7 on an USB key (is it possible yet?). I thought of using virtualisation on my laptop, but I'm afraid I don't have the disk space on my SSD (I've ~15GB left and that would stretch it).

My best bet if Enterpoint doesn't release a Linux version is to find someone with a Windows install (can't think of any friend with that) and pray that I won't trash his/her system. I could sell the boards too... Ironically I wanted to buy some more, but there's no way I'd knowingly buy something that requires me to spend time on something I don't want to. Especially if it's related to mining: time is money.

gyverlb, sorry that things are not working out for you. but why did you purchase a hardware when you don't know how to do updates etc.

This is not the way I would buy hardware. When I know I have limited opportunities to work with hardware like this (dev board, no informations, not know software to update, forced to use only Linux etc.), I would better go for something that is better supported and documented. It's just my point of few.

But to give you also an other hint to update from the controller from Linux. Buy an simple JTAG cable, then use the xc3sprog with the -Ixc3s50an.bit option. This way Zefir managed to flash the controller from Linux successfull. Just ask him about the used cable. I think it will not cost the world (~10-20$) and it would make you also happy with your boards =o)

eb

[yohan]

[...]
My problem might be that I can't update the controller and still use the 1.1 firmware on it but I don't have any solution for it (no Windows license here).

I would just try and download a copy of Windows 7 install it without a key, you get 30 Days trial, or borrow someones Win7 Cd and load it from that. That way you could at least get to 1.3, or 1.4 now. You don’t actually need to own the copy just get it installed and use it for 30 days.

There are 2 problems that make this impractical:

• I've not installed Windows since XP (and even then it was a one-shot because I had to for work) I'd probably spend several days on this in my free time. Last time I saw a Windows system I didn't even know where to begin, its UI is now a total stranger to me.
• I don't have any system on which to install Windows 7. The only available system I have is a test system without a harddrive, booting from an USB key. I'd have to buy a drive or install Win7 on an USB key (is it possible yet?). I thought of using virtualisation on my laptop, but I'm afraid I don't have the disk space on my SSD (I've ~15GB left and that would stretch it).

My best bet if Enterpoint doesn't release a Linux version is to find someone with a Windows install (can't think of any friend with that) and pray that I won't trash his/her system. I could sell the boards too... Ironically I wanted to buy some more, but there's no way I'd knowingly buy something that requires me to spend time on something I don't want to. Especially if it's related to mining: time is money.

There is another possibility and that is to acquire a programming cable that will work with Xilinx ISE tools. Webpack versions of ISE are free, will run under Linux, and whilst they won't build for a LX150 Spartan they I believe will allow programming on these devices and certainly will for the XC3S50AN used for the controller. Some details http://www.xilinx.com/support/download/index.htm. There are several options for cables that will work with this software including our Prog2 (cheap parallel port cable) and Prog3 (dearer). Digilent, Xliinx and various Chinese clones are available as well. Don't ask me if the Chinese ones work. I have never tried them. 

[gyverlb]

gyverlb, sorry that things are not working out for you. but why did you purchase a hardware when you don't know how to do updates etc.

Because I had no idea that there still were hardware manufacturers that wouldn't support Linux by default today. Especially in this very case: addressing the Bitcoin mining community: many (most?) of us have Linux rigs some even use the RasperryPi or home routers to connect to their FPGAs. Bitcoin is open-sourced at its roots.
The only question that I asked myself is if the board could be updated via USB (this is why I didn't purchase a JTAG cable). Even the documentation of the controller programming interface would be enough. In fact no Linux support isn't the problem, having to rely on an obscure utility that only runs on some OS versions without the safety net of documentation of the interfaces it uses is a recipe for transforming anything in a paper weight mid or longterm.
This is not the first time I purchase hardware for which I have to code the driver or some tools myself: you can still grep my name in the Linux kernel for the support of some long forgotten IDE controllers. So usually I'm not especially worried, but reading that there is some IP in there that can't be disclosed to build the tools I need is worrying me.

This is not the way I would buy hardware. When I know I have limited opportunities to work with hardware like this (dev board, no informations, not know software to update, forced to use only Linux etc.), I would better go for something that is better supported and documented. It's just my point of few.

We knew very little of the details when we pre-ordered. Everyone took a risk, myself included.
But having the thing closed to me because I can't have access to the information I need to make it work the way I want is exasperating. I'm a tinkerer : if I can't hack it I won't like it and it better make itself forgotten (which it doesn't do well currently). It's already too bad that Xilinx won't sell you the right to hack the Spartan6 with it but separately. I already had difficulties wrapping my mind around this fact, seems the FPGA world is not for me.

But to give you also an other hint to update from the controller from Linux. Buy an simple JTAG cable, then use the xc3sprog with the -Ixc3s50an.bit option. This way Zefir managed to flash the controller from Linux successfull. Just ask him about the used cable. I think it will not cost the world (~10-20$) and it would make you also happy with your boards =o)

That's a good alternative thanks for the heads up.

[zefir]

My best bet if Enterpoint doesn't release a Linux version is to find someone with a Windows install (can't think of any friend with that) and pray that I won't trash his/her system. I could sell the boards too... Ironically I wanted to buy some more, but there's no way I'd knowingly buy something that requires me to spend time on something I don't want to. Especially if it's related to mining: time is money.

[...]
But to give you also an other hint to update from the controller from Linux. Buy an simple JTAG cable, then use the xc3sprog with the -Ixc3s50an.bit option. This way Zefir managed to flash the controller from Linux successfull. Just ask him about the used cable. I think it will not cost the world (~10-20$) and it would make you also happy with your boards =o)

eb

Hi gyverlb,

totally agree with you, the information on what SPIProg.exe should be disclosed. If you are a Linux kernel developer you for sure know that programming an SPI device is no rocket science, and one would basically just need to know how the Spartan3 is wired to the FTDI port to access it. There are several SPI programming tools available as open source (like this SPIProg) that can be easily extended to use the related channel over libftdi.

But in the end it is not worth the effort. I found a Xilinx Parallel Cable IV that does the job well. It is fairly outdated and slower than the USB variant (plus you might have troubles finding a parallel port at your host meanwhile), but as recovery tool for sure good enough. You can find it at Ebay for less than 10$, while the USB Platform Cable is available for around 35$.

The cable is fully supported by xc3sprog, and since it comes with the 14pin 2mm pitch cable it is plug-and-play ready (in contrast to other cable I tried first and failed).

Programming the controller is done like ebereon described: use the left-hand JTAG port (the one directly at the Phoenix connector) and run

Code:

sudo ./xc3sprog -v -c pp -Ixc3s50an.bit <controller-bitstream>

Good Luck!

[Cranky4u]

gyverlb, sorry that things are not working out for you. but why did you purchase a hardware when you don't know how to do updates etc.

Because I had no idea that there still were hardware manufacturers that wouldn't support Linux by default today. Especially in this very case: addressing the Bitcoin mining community: many (most?) of us have Linux rigs some even use the RasperryPi or home routers to connect to their FPGAs. Bitcoin is open-sourced at its roots.
The only question that I asked myself is if the board could be updated via USB (this is why I didn't purchase a JTAG cable). Even the documentation of the controller programming interface would be enough. In fact no Linux support isn't the problem, having to rely on an obscure utility that only runs on some OS versions without the safety net of documentation of the interfaces it uses is a recipe for transforming anything in a paper weight mid or longterm.
This is not the first time I purchase hardware for which I have to code the driver or some tools myself: you can still grep my name in the Linux kernel for the support of some long forgotten IDE controllers. So usually I'm not especially worried, but reading that there is some IP in there that can't be disclosed to build the tools I need is worrying me.

This is not the way I would buy hardware. When I know I have limited opportunities to work with hardware like this (dev board, no informations, not know software to update, forced to use only Linux etc.), I would better go for something that is better supported and documented. It's just my point of few.

We knew very little of the details when we pre-ordered. Everyone took a risk, myself included.
But having the thing closed to me because I can't have access to the information I need to make it work the way I want is exasperating. I'm a tinkerer : if I can't hack it I won't like it and it better make itself forgotten (which it doesn't do well currently). It's already too bad that Xilinx won't sell you the right to hack the Spartan6 with it but separately. I already had difficulties wrapping my mind around this fact, seems the FPGA world is not for me.

But to give you also an other hint to update from the controller from Linux. Buy an simple JTAG cable, then use the xc3sprog with the -Ixc3s50an.bit option. This way Zefir managed to flash the controller from Linux successfull. Just ask him about the used cable. I think it will not cost the world (~10-20$) and it would make you also happy with your boards =o)

That's a good alternative thanks for the heads up.

If you want / need to offload your CM1 collection...msg me to discuss prices   

[gyverlb]

If you want / need to offload your CM1 collection...msg me to discuss prices    

I opted for this instead (will probably get it in 3 weeks):
http://cgi.ebay.fr/ws/eBayISAPI.dll?ViewItem&item=120927703340
I hope it works as advertised, I could have bought the cheaper parallel version, but I have far more USB ports than parallel ports...

With some pains and various makomk bitstream combinations I think I can hit a ~650MH/s per board configuration which doesn't crash every day while waiting for it. I really hope the new controller firmwares are worth it!

[ShadesOfMarble]

What's the current status regarding the bounty? The way I see it, all requirements for paying out the bounty are met... (I cannot judge point 6, documentation)

[Lethos]

What's the current status regarding the bounty? The way I see it, all requirements for paying out the bounty are met... (I cannot judge point 6, documentation)

When I had it stable I agreed with many, that yes it did meet the requirements. However I've had stability problems since and I don't know what the cause is.
However they haven't really finished a polished version yet and apparently working together now, so I guess that new combined one once ready will win it.

[Glasswalker]

When I had it stable I agreed with many, that yes it did meet the requirements. However I've had stability problems since and I don't know what the cause is.
However they haven't really finished a polished version yet and apparently working together now, so I guess that new combined one once ready will win it.

Keep in mind that your recent experience with the 2 boards dropping off, that is not a bitstream issue (or rather I'm 99.9% sure it's not a bitstream issue).

For 2 physically different boards to drop off at nearly the same time, all 4 ports to vanish from the OS, but the chips keep the heartbeat going. That's not a bitstream problem. It's either hardware (board level) or infrastructure (cabling, host PC, software, driver, hub, ports, whatever) which is unfortunately outside our control. The only part of the bitstream that it could be (on a long shot) is the controller, as that interfaces directly to the FTDI and could impact the performance of the FTDI (and therefor cause this "dropping off". But the fact that 2 boards did the same thing in approximately the same time leads me to believe it was something outside the mining boards themselves.

That said I'll do my best to try and help you identify the issue, as many others will I would hope as well. Hopefully it either never happens again, or it happens again but you're able to get more detailed info from it to help diagnose.

Just wanted to clarify that the future bitstream releases will be focusing on first the dynamic clocking, and next the new hashing core. Neither of which would have any impact at all on this particular issue. So I wouldn't waste time hoping that the new versions will solve this particular issue. That said the FINAL goal of the new controller, with the completely new communications protocol (that would use only a single USB for an entire cluster of boards) would likely help aleviate this, due to an architectural change, but it wouldn't actually solve the problem (merely work around it). But that one is a long way off.

Just wanted to make sure we're not setting the wrong expectation

[Lethos]

I've had more problems than most, I wouldn't disagree there. Also I'm not blaming your bitstreams, just like I said before I don't know what is causing the problems I am experiencing. I get it working eventually, it just doesn't always stay stable, I can't pin down what is causing it yet.
Documentation is the only thing left, you guys working flat out, I don't expect a lot, It's why I do my best to write mine up.

[Glasswalker]

For sure, I just wanted to make sure I wasn't setting wrong expectations

Yeah documentation will come soon. The readme is there and it is "complete", but it is rather vague. And doesn't cover any "special" cases, or odd problems. It basically just covers hardware settings, and installing (referring out to the enterpoint instructions, or assuming access to ISE Impact and a JTAG cable).

Once we have the next version out that's the last major feature change. After that the bitstreams should remain functionally identical, just with performance improvements or internal structural improvements (without changing overall functionality at all).

Eventually way down the road more documentation will be needed for the new protocol but that's a ways off still.

Once I get the next release out and it's working, (dynamic clocking) then I'll take a bit of time to do a full documentation writeup, and hopefully the community can help me flesh it out once I get a foundation laid down.

[Lethos]

For sure, I just wanted to make sure I wasn't setting wrong expectations

Yeah documentation will come soon. The readme is there and it is "complete", but it is rather vague. And doesn't cover any "special" cases, or odd problems. It basically just covers hardware settings, and installing (referring out to the enterpoint instructions, or assuming access to ISE Impact and a JTAG cable).

Once we have the next version out that's the last major feature change. After that the bitstreams should remain functionally identical, just with performance improvements or internal structural improvements (without changing overall functionality at all).

Eventually way down the road more documentation will be needed for the new protocol but that's a ways off still.

Once I get the next release out and it's working, (dynamic clocking) then I'll take a bit of time to do a full documentation writeup, and hopefully the community can help me flesh it out once I get a foundation laid down.

If you want any help writing up documentation, you know I've tested it enough

[Entropy-uc]

I will give my opinion.  Others may think differently, there is such a broad range of hardware performance that everyone is going to have different experiences.

Below is the performance of 8 recently build boards (>04xx) after running Makomk's latest 200MHz bitstream for 4 days.

cgminer version 2.6.4 - Started: [2012-08-17 13:46:13]
-------------------------------------------------------------------------
 (5s):6241.1 (avg):6106.0 Mh/s | Q:998120  A:576799  R:366  HW:0  E:58%   U:79.1
 TQ: 1  ST: 16  SS: 0  DW: 26304  NB: 840  LW: 0  GF: 2106  RF: 8
 Block: 000005866228c10025bc465254296925...  Started: [15:04:14]
-------------------------------------------------------------------------
 ICA  0:                | 381.4/381.9Mh/s | A:38392 R:34 HW:0 U:  5.27/m
 ICA  1:                | 387.2/382.1Mh/s | A:39066 R:26 HW:0 U:  5.36/m
 ICA  2:                | 390.1/382.1Mh/s | A:39067 R:20 HW:0 U:  5.36/m
 ICA  3:                | 388.1/382.0Mh/s | A:38908 R:25 HW:0 U:  5.34/m
 ICA  4:                | 386.1/384.1Mh/s | A:38836 R:22 HW:0 U:  5.33/m
 ICA  5:                | 380.6/381.9Mh/s | A:39043 R:21 HW:0 U:  5.36/m
 ICA  6:                | 390.7/384.2Mh/s | A:38854 R:31 HW:0 U:  5.33/m
 ICA  7:                | 385.1/382.0Mh/s | A:38711 R:23 HW:0 U:  5.31/m
 ICA  8:                | 390.9/384.4Mh/s | A:39281 R:24 HW:0 U:  5.39/m
 ICA  9:                | 382.9/382.0Mh/s | A:38805 R:24 HW:0 U:  5.32/m
 ICA 10:                | 371.0/375.4Mh/s | A:15949 R: 9 HW:0 U:  2.19/m
 ICA 11:                | 371.0/375.4Mh/s | A:16269 R: 9 HW:0 U:  2.23/m
 ICA 12:                | 384.3/382.2Mh/s | A:38974 R:22 HW:0 U:  5.35/m
 ICA 13:                | 378.8/382.1Mh/s | A:38997 R:36 HW:0 U:  5.35/m
 ICA 14:                | 386.5/382.1Mh/s | A:38864 R:25 HW:0 U:  5.33/m
 ICA 15:                | 392.9/382.0Mh/s | A:38784 R:15 HW:0 U:  5.32/m
-------------------------------------------------------------------------
               

The total utility of 79.1/minute corresponds to 708 MH/s per board but this is being dragged down by one badly performing board.  If I remove that board from the calculation, I get an average performance of 740 MH/s.  That is still a hair under bounty target.

But it is close enough that I think the spirit of the offer has been satisfied. There is also some risk that the value of bitcoin will dump in the near term as the pirate dust settles (or hits the fan).  I don't want them to get shorted in such a situation. So, I intend to pay out my share of the bounty reward equally to Makomk and Glasswalker as they agreed.  I have an address for Makomk already; Glasswalker if you PM me a deposit address, I will pay out the bounty the next time I have access to my wallet.

[Glasswalker]

PM Sent.

I also want to apologize for the delays in the last few days. My day job kind of exploded (critical project went into crisis mode, and I got pulled in to deal with it, so starting this past weekend I've been working from about 8am till 1am every day. Today I'm working from 3am until god knows when due to a timezone shift with one of our partner offices... So I've had to pause my work on the bitstream for the past few days.

This insanity should be resolved by this weekend.

In the meantime, I have released an updated 200Mhz bitstream that runs at 200Mhz with almost no invalids. BUT it has some stability problems. In my absence, makomk has been working on the problem, as he began experiencing similar instability in his bitstreams as well. I thinks he has found the solution to the problem and is doing some test builds of his bitstream to see if it resolves them. If it does, we'll backport those changes to the HashVoodoo bitstream. In addition, collaboration with TheSeven has resulted in a few more improvements we plan on dropping in to the next build as well to once again re-inforce stability.

This should get us to a usable 200Mhz plus bitstream with the rock-solid stability we saw in the previous hashvoodoo bitstream (so working on ALL boards, including the older gen, sub50 serials, and any "problem" boards). At least that's my hope.

Then we're going to take a short break from releases for an unavoidable step:
We'll be building (and releasing with the source) a full testbench for use with the Xilinx simulator, allowing simulation of the bitstream before building. This will allow us to debug (in a more conventional software dev form) the bitstream, by walking through clock cycles, and seeing where things muck up. This will also allow faster problemsolving as the simulation can happen after only a partial build (basic synthesis) not requiring mapping, or placement and routing, and timing closure. So with the simulation we can test new features, ensure they work, saving build time drastically...

Making a testbench isn't quick or easy (at least not doing it properly) but I think it's the right thing to do. I've been avoiding it until now, but at the urging of TheSeven and makomk, I think it's the right thing to do to have a single solid bitstream we can all easily develop and debug. This will likely mean a week or so "pause" in any future releases (once I get back to it) so aside from this next "bugfix/stability" release using makomk's fixes, I wouldn't expect any additional releases until approximately the end of Aug, maybe first week of Sept.

After which, progress should be faster, and our ability to squash bugs will be greatly improved

Anyway, what I'm getting at is a rediculous amount of effort has gone into these bitstreams, and both the HashVoodoo, and makomk's direct fork have benefited greatly by our cooperation and collaboration since we decided to join forces. And we still have several surprises coming once we get the foundation laid and the core features settled (the new hashing core promises to present a significant boost in performance, once we've ironed the kinks out).

Development isn't going to halt on this bitstream, it's just going to keep getting better. But I also think we're lightyears ahead of where we were with the cairnsmore. We now have several "semi-stable" bitstreams in the 200Mhz range as posted by several others on this and the other thread. And for those of you having problem boards, the 175Mhz hashvoodoo bitstream (08_16_2012 release on github) has been tested thuroughly on many boards, including some of the most troublesome boards that were already packed up for RMA, and it works beautifully and 100% stable. So give that a shot to revive your lost-cause boards.

If you're having problems flashing, please read the readme in the latest HashVoodoo release. It has several tips/tricks for flashing that help in problem cases.

Anyway, just wanted to post that update (I had a few minutes waiting on an email response, so had some time to breathe between marathon work sessions). So you were all aware of where we're at.

Thanks for your support!

[makomk]

The total utility of 79.1/minute corresponds to 708 MH/s per board but this is being dragged down by one badly performing board.

Ah, that could be the bitstream bug I just fixed, though it doesn't generally seem to cause problems with the 200 MH/s bitstream. Please do reflash the badly-performing board with the new dcmwd4e bitstream when you get back home and have got a moment - this is a pain for me to test because the issue takes a while to show up and affects some boards more than others.

If I remove that board from the calculation, I get an average performance of 740 MH/s.  That is still a hair under bounty target.

Curious. Have you been getting a lot of warnings from cgminer about your pool not providing work fast enough? It's not just the U/m that's underperforming, the MH/s reported by cgminer is lower than it should be and I think that's generally caused by a slow pool.

But it is close enough that I think the spirit of the offer has been satisfied. There is also some risk that the value of bitcoin will dump in the near term as the pirate dust settles (or hits the fan).  I don't want them to get shorted in such a situation. So, I intend to pay out my share of the bounty reward equally to Makomk and Glasswalker as they agreed.  I have an address for Makomk already; Glasswalker if you PM me a deposit address, I will pay out the bounty the next time I have access to my wallet.

OK, thank you! I see Glasswalker has spotted this and replied already. Hopefully (touch wood) with the fixes above and/or the new 210 and 220 MH bitstreams I've already released performance should be comfortably over the line and you won't regret paying out. My testing board is actually at 855 MHash/s of a theoretical 860 based on the U/m reported in MPBM right now, though it's only been running for 12 hours. I believe ebereon was seeing similar speeds on his farm with the bitstream this is based on before it started falling over, so (fingers crossed) the new one ought to be able to manage that stably on a decent proportion of boards.

In my absence, makomk has been working on the problem, as he began experiencing similar instability in his bitstreams as well. I thinks he has found the solution to the problem and is doing some test builds of his bitstream to see if it resolves them.

I've actually released full-performance bitstreams (dcmwd4e) with the fix now and am waiting to hear how they work out. My own testing is looking really, really promising though.

I also want to apologize for the delays in the last few days. My day job kind of exploded (critical project went into crisis mode, and I got pulled in to deal with it, so starting this past weekend I've been working from about 8am till 1am every day. Today I'm working from 3am until god knows when due to a timezone shift with one of our partner offices... So I've had to pause my work on the bitstream for the past few days.

Ouch. No wonder you didn't have any time for bitstream development.

[Lethos]

Glasswalker: I gave your 200 and 210 versions of the hashvoodoo a try. I can attest to the stability issues, so reverting back to what is rock solid performance of your 175 version.

Hope work allows you to some down time soon, I commend you for still being able to do much else while working those sort of hours.

[Entropy-uc]

I do think the total performance issue is a function of my network performance and pool performance.  I also think a little more time tweaking icarus-timing might push it over the hump.

Thanks for the advice Makomk.  I will give the new bitstreams a try once things settle out and my brain is on the same time zone as my body.  

Paid
25 BTC to Glasswalker
25 BTC to Makomk
5 BTC to Kano

Thanks for all the hard work you've put into this guys!

[Glasswalker]

Paid
25 BTC to Glasswalker
25 BTC to Makomk
5 BTC to Kano

Awesome! Thanks Entropy-uc! We really appreciate your support! (I know I do, and I'm pretty sure I can speak for makomk and kano in that regard too)

Thanks!

[Cranky4u]

Question on stability...

I am running the 200MHs bit stream by Makomk on 2 * CM1s, series 460-ish (at work and using memory) and mining using cgminer 2.7.0 (have used 2.6.1 for a few weeks previous to upgrade).

My concern is that the CM1 boards hang several times a day with all orange lights coming on. This is normally preceded by a "lame duck" event were one of the FPGAs go offline first (normally #2 and rates drop to 2~4u/m). The red "heart beat" still continues. The only way to get them mining again is to shutdown cgminer, then disconnect power to reset the boards and fire it back up. 75% of the time I also need to disconnect the USB comms cable whilst off inorder to reset the comm port. The boards are connected to independent USB ports on my gigabyte MB which has "power boost" dedicated lines so I dont think USB brown outs are the issue.

Is it peoples opinion that the poroblem lies in;
1. the bitstream is unstable, or
2. the CM1 hardware is unstable, or
3. the independent USB lines are causing EMI/EMC issues resulting in the system being unstable, or
4. cgminer has a bug, or
5. something else.

Your help in providing the most popular line of investigation is appreciated.