The only way you could revert was if the SPI Flash had not been programmed and you had live loaded the FPGA directly. What is more likely is that they have gone into the lockup mode I have described elsewhere. Once FPGAs are in that lockup mode we think the only ways back are to either power cycle or do a live configuration. I am hoping to do some work on Rev 1.4 controller today that might partially solve the lockup issue but this unlikely to be a total answer. The proper place to fix this is in the array FPGA design and of course the twin isn't our original design and very hard to fix properly and rebuild. We do have something that is much more robust in our own design and cross fingers once that is available you won't have the issue. Meanwhile I will do my best to provide an improvement on the temporary solution in the Controller design.

On the USB hubs I would either try a switched type like I listed earlier today or go for one with an even higher port current capability i.e. more like 500mA. Also be careful of some hubs that have 1 or 2 permanently powered ports for charging Ipods etc.. These hubs generally are ok but I would avoid the ports on them that don't power down.

To be honest we have not tried cutting the USB 5V power line yet. or going back to un-powered hubs. Un-powered hubs could still have an issue if the power sequence is done wrongly i.e. before 12V is up and supplying controller power. It would certainly be worth trying with the proviso of powering the 12V first. Yesterday we happened to fit the USB hubs that have a switch and didn't turn on hub is up and stable. 10 more of those hubs arrived this morning and we will build a bigger system as we get more boards off the line to test.

We really started to get some more ideas yesterday as we were starting to get our larger model system together and we did some more sytem level thinking and testing. We had build a 10 boards system about a week ago the boards from that then went to ebereon as tested working boards. his was an additional test over what we did previously but we are now adopting that as standard. That was the previously biggest system we had to play with to see some of these sorts of issues. We think that there are a number of the issues in larger setups that are mainly a mixture of USB and CGminer issues. Like the window size problem we saw yesterday. I wouldn't have thought of that one unless I seen it with my own eyes.

The fact that CGminer isn't capable of coping with losing one processing element is basically crap. We do want to look at that but it's a trade between making forward progress on bitstream and associated items and doing this support work. We only had 1 day in reality this week on the bitstream side so it hasn't gone far this week.We do now have have a rig test that may form a basis for something better and we will probably make that test available to you and other big rig owners for system diagnostics and on-site test.

We going to see if we can also make things better in the controller for power-up and overnight lockups. One of the major issues with the twin bitstream is that if gets any charactors sent to it down the com line it will react. if that is a non-real message or a message is corrupted that usually causes a lockup/non responder. We know this can be an issue if CGminer starts sending coms before the system is ready. Harder to sort is overnight issues and that depends on whether that is the actual problem. It could be a mains power dip upsetting the USB or a coms issue. If it is a coms issue then we almost need to check messages in the controller before passing on to the array and also the other way. One other unlikely possibility and a weakness in the Icarus setup is that the coms of the second chip can interfere with those of the first chip. We don't think that should happen if you have dip switches set right and second FPGA is in reset. Here there is a possibility of simply clearing the config SPI flash of the 2 non-working FPGAs so there is nothing to interfere. Actually that gives me a thought of whether we can remove power from these 2 devices. You do need power for JTAG to work but otherwise it might work. A line parking build might do the same. I'll look at that as an option as it can be done quiclkly.




Often in bus powered systems the drop on the cabling and hubs would make the 12V derived fed win. However a powered hub might make this different. It's good and bad. First because

The board has a common "5V" rail which can be fed from either USB 5V or 5V generated from the 12V. The 2 alternative sources use in-line diodes to avoid back feeding to the other side. Which one supplies the current actually depends on which is the highest voltage. Notionally they are the same voltage but there will nearly always be a difference. Often the USB will often be lower because of cabling loss so the 12V will supply often or even share the load. However this is not guaranteed. If your hub power sits at a slightly higher voltage of course it will try to supply all or some of the current. Actually that might be an interesting thing to try. Running a hub from a bench supply at say 4.8V might allow the onboard 5V derived from 12V to always dominate.

Another way is to modify a USB cable so that 5V isn't supplied (cut the right wire maybe in a USB cable) or is regulated down to say 4.8V by an in line regulator or even a simple diode in-line to provide an extra voltage drop. Knobbling the hub supply voltage down to 4.8V might be somewhat simplier.

At the board level changing the USB side diode to one with a bigger drop would work as well.

There is another USB issue directly related to the Twin (Icarus) build. This design appears lock up if it receives a slightly corrupted message. You might get this if CGminer started sending messages before a FPGA was fully powered and one of the things we are going to try is to block messages in the Controller until we think the FPGA is fully powered and configured. This is why CGminer should not be started too early at the moment but might explain a range of things that people have observed. This won't be a problem in ur own bitstream where messages will be checked by the design before any operating action is taken.

We have started modelling a mid size mining rig so that we can firstly test boards exactly the way Bitcoiners use them but also so we can see some of the system level problems that some of you get. This is an additional test to our normal testing and we aiming put all line boards through that test. At the moment it's windows based but we will probably run a second rig on LInux as well. We have a few observations about using USB which may help you all get running.

First observation is that powering the 12V first befor plugging in USB is good way to do it.

Our second observation was using a cheap 13 way USB hub we bought to test http://www.ebuyer.com/279682-xenta-13-port-usb2-0-hub-mains-powered-n-uh1301 if we switch off using the switch on the hub until the 12V is up and settled gave good results.

Third observation is that our 20-30 unit test rig took 10-20 minutes to enumerate the USB structure and if you try tio do anything before that is complete one or more boards will get screwed up and you can not do anything to get it back short of doing the entire start up sequence again.

Out of what we saw today we have a couple of ideas to add to the Controller that may help stability with the twin bitstream and we will try those over the next couple of days. If they are of benefit a release of Controller will follow.

I only have bare details but one bug we appeared to see today on CGminer, running on Win7, was that if the dos box didn't have enough lines defined for the numbers of CM1 s running it caused a problem on the ones that could be fitted in on the bos box. So make sure you have eniough lines in the dos box if you are running a large rig.

This really depends on where the short circuit current went and that's really hard to pedict because it also depends on what paths exist in external equipment. It could be any of a number of components that have failed or at the currents you had it's possible PCB tracks have been vaporised. Can you see any obvious physical damage.

Does the Controller red led flash or come on?

If you are in any way technical and happen to have a multimeter it is possible to check if the power supplies are still working. Other than that you have the option to have us evaluate it and repair if possible. There may be a charge for this or carriage charge depending what state it is and what we think needs doing. Either way you should contact us on the bitcoin support email bitcoin.support AT enterpoint.co.uk.

There is another possible option that we can make the board work on the up/down structure when that is availkable. However that depends on what the current fault is. If a regulator has blown then that would almost certainly need replaced.

I don't think anything should have changed on the programming side but we do know if run the array programming tool from Linux directly it is an unstable process. That's why even under Linux we recommend the VM approach. It basically slows it all down and it looks like there is something like a timing loop in some of this software. We are going to lose the VM part eventually but we need our own native tool ready to do that and it's still a little while away.

You have probably already done this but check dip switch settings are those recommended for programming.

I am going to do a document for Rev 1.3 dip switch settings and hopefully I will get that done today.

There are definate differences depend in what order you fire things up in and so on and we think this is a blend of OS, drivers, CGminers issues probably more than hardware issues at least when the Controller is Rev 1.2 onwards. We saw a definate problem yesterday that is related to the power supply ramp time and that's why we did the Rev 1.3 of the controller. It solved our problem with this power up problem which is also related to the twin or Icarus build. It's possible that for other environments it Rev 1.3 isn't yet the best solution and Rev 1.2 is better.

We will contine to track down these bugs so keep reports going to the bitcoin support email.

Might be talking at cross purposes here. Array FPGAs take the 30-40 mins to do the SPI Flash. Controller is very fast and less than a minute to do it's internal SPI Flash.

Thanks. I think that is sorted now.

The process for programming the SPI Flash is very slow. Expect 30-40 minutes per board. We do have a plan to include parallel programming of the array and this will make an obvious time saving expecially if we can make it work over multiple boards at the same time. For this to happen it needs our own bitstream so we can add some necessary support features.

We have been using Win7 and XP here.

I'll put that back shortly updated. I took off the link to the Rev 1.1 version and forgot it had the only manual.
BTW, Yohan, a linux SPIProg would make flashing a few boards a lot faster, now I need to detach/attach usb cable from/to boards and windows laptop I use to flash them.

spiccioli.

We are working on improving SPI programming. Our aim is to get rid of the VM and do something more native. There should be something for Linux as well. It's really just a matter of having the time to do these items currently. Moving the bitstream forward and dealing with CGminer/driver bugs is a higher priority at the moment but these other things will come behind those things.

As part of our work on Rev 1.3 of the controller we found an unusual "feature" and is was related to a cheap USB hub we are using in one of system test setups. What we found was that this hub didn't turn power off properly when the host laptop released the port as it should. Indirectly this was causing a lockup of the array FPGAs. We have added an extra reset function to clock startup and this appears to solve the issue.

On our test setups we are now getting a good balance of hashing now on both FPGAs running the twin bitstream so I would recommend anyone with a controller on Rev 1.0/1.1 updates it to Rev 1.3.

Rev 1.3 of the controller will be on the temporary support webpage http://www.enterpoint.co.uk/cairnsmore/cairnsmore1_support_materials.html shortly. It's should sort out the clock start up problem introducted in Rev 1.2.

Rev 1.3 also has a new thermal protection feature that senses if the fan is operating in the correct speed range. This feature will turn off the array power supplies if the fan does not show as being in the correct speed range. That covers fan failures, something blocking the fan rotation, or even an extreme voltage drop on the 12V supply.

If you use a different fan other than the standard F12 or don't use a fan connection in all boards i.e. in side blow configuration this may/will force the turn off the relevant board power supplies. However there is an override using SWITCH2 of the controller dip switches. This is not recommended to use unless you really have to. It is as much a safety feature as it is damage limitation feature for the boards.

It's the first bit of SW1.

We found one minor issue with Rev 1.2 controller and there is a circumstance were the clock outputs don't start after the board is powered. You can work around this in a temporary fashion by toggling SWITCH1 off then on of the Controller dip switches. This is a reset function and the clocks should start then and be ok until you turn the board off.

Rev 1.3 is being prepared to sort this properly and should be available later today or early tomorow.

Can you send us a board by board report on email "bitcoin.support" of what you seeing and we will go through it all on the next daily support review.

SW2-5 can be any setting for Controller programming. SW1/6 as below.

Ok just for anyone that missed it we are recommending moving to the Rev 1.2 on the controller at your convenience. I still have to update the instructions for this but I will have another go at doing this later.