Cairnsmore1 - Quad XC6SLX150 Board

[gigantic]

Yohan,
Can you please write all the specifications of the board in 1 place,

everything that need to make the FGPA'(S) to work, exactly? like what exact cables are needed, how many of them, where to connect each one?etc,
i am a noob when it come to hardware, so someone will help me, but i do need to know what to buy, etc, or what cable(s) goes to each FPGA, and from where,

i am not asking you to tell me what motherboard to buy or such, but do tell what is needed to assemble,


Thank you

[steamboat]

forgive me if this has already been answered, but i've read through the thread a couple times and haven't found it.

the board is expected to do ~800-1000 mh/s yes? how much power?
if they are planning on using the icarus bitstream, will it come pre-programmed? and will i need any special software/hardware to flash to a new firmware when they become available?

[Glasswalker]

I think from reading previous posts (will have to wait for official answer from Yohan to be sure) the following might help answer some of this:

the board is expected to do ~800-1000 mh/s yes? how much power?

It's currently expected to perform identically to 2x icarus boards (perhaps a bit faster, but since no real testing on hardware has been done yet, no "official" performance numbers are released yet. Icarus currently does 380MHash/s per board, so that would put ths at 760MHash/board if it matches exactly. If it can beat Icarus performance a bit (via more efficient design/cooling and dynamic clock control) then it can maybe come in at 800+MHash using the Icarus bitstream.

As for power consumption we can safely assume these boards should use around the same power per chip as Icarus (it's the same chips, same bitstream). Their PSU circuit might be a little more or less efficient, but likely only by 5% each way at most I would expect. So "around 10W per chip" Whole board will probably come in under 50W I would expect. (40W in FPGA, 10W for the control FPGA, and other supporting circuitry at MOST). But until they've completed real hardware testing we likely won't see "actual" power usage numbers.

Also consider that in theory newer bitstreams may push the chips harder (or be more efficient), so in the end that could alter the power usage by a much higher degree than the hardware of the board itself.

I believe the chips are each being supplied by a 15A 1.2V regulator circuit, which is fairly high efficiency. So that would mean a maximum wattage delivered to the chips of around 18W (but that would likely burn out the regulators if done over a sustained period of time). So you can likely assume about 15W max sustained draw from a chip (as in that's the max the board is capable of supplying). So the board at MOST (with a super power hungry bitstream) could consume maybe 70W. But that would be a super power hungry bitstream, and realistically, I think there would be major problems cooling the chips due to the plastic packaging at that much power dissipation.

if they are planning on using the icarus bitstream, will it come pre-programmed?

Yes, on day one they are planning on using the Icarus bitstream. And from what I understand the first round of boards will ship pre-flashed with Icarus bitstreams pre-installed.

and will i need any special software/hardware to flash to a new firmware when they become available?

It's been said that the Control chip (Which is just a smaller FPGA programmed to handle all communications, and sensor monitoring, and power/clock control, it looks like a Spartan3) in combination with the USB chip they use, can drive a JTAG chain. So in otherwords you *should* be able to program it using the USB. What is currently unclear, is if this will require custom firmware on the control FPGA, or special support on the device. And if it will require custom software to do the install. (and if that software needs to be written first). So the best i can give here is you SHOULD be able to do it with the USB (no other hardware). But there is a chance the first batch of boards may not support this feature, requiring a JTAG to reflash them to add the "USB FLASH" capability to them initially. (after which USB would work fine). This is all speculation based on released info, so take with a grain of salt, and hopefully Yohan can answer more clearly in the near future.

Disclaimer: as I said, this is all speculation based on what I've seen so far in this thread, and on their site. Take everything I say here with a grain of salt, and don't consider any of it official unless it came straight from Yohan's mouth

Hope that helps

[steamboat]

awesome, thanks for the reply. I'm still new to the fpga side of things and i struggled through programming class so i'm trying to get an idea of how difficult it's going to be for me to set up an fpga farm. i've already pre-ordered one but i'd really like to get some more solid numbers on the performance so i can make a decision between this and other options.

[Dexter770221]

Wow, you really have to be borred
Power handle will be a major factor with this chips. Datasheet says of 2.2C/W Rthjc. With good heatsink and fan this will be more like 4C/W (thermal resistance adds up). So when this chip will consume 15W, core temp will increase 60C (15*4) over ambient. That means 85C core temperature (25C ambient + 60C from dissipation) and thats more than guaranteed by Xillinx for commercial grade chips. So, Rthjc is blocking higher hashrates, not power consumption. Watercooling may push this babies a little bit more...

[MrTeal]

Wow, you really have to be borred
Power handle will be a major factor with this chips. Datasheet says of 2.2C/W Rthjc. With good heatsink and fan this will be more like 4C/W (thermal resistance adds up). So when this chip will consume 15W, core temp will increase 60C (15*4) over ambient. That means 85C core temperature (25C ambient + 60C from dissipation) and thats more than guaranteed by Xillinx for commercial grade chips. So, Rthjc is blocking higher hashrates, not power consumption. Watercooling may push this babies a little bit more...

You're being pretty pessimistic on the heatsink. You can get passive northbridge heatsinks with 2.5C/W thermal resistance. With airflow you should easily be able to get Rthca to 0.5C/W or lower if you go big/loud enough. You'll have quickly decreasing gains from going much lower than that, but 2.6 or 2.7 C/W Rthja should be reasonable and that would keep the junction at 60C in a 20C room with 15W.

[nbtcminer]

It's been said that the Control chip (Which is just a smaller FPGA programmed to handle all communications, and sensor monitoring, and power/clock control, it looks like a Spartan3) in combination with the USB chip they use, can drive a JTAG chain. So in otherwords you *should* be able to program it using the USB. What is currently unclear, is if this will require custom firmware on the control FPGA, or special support on the device. And if it will require custom software to do the install. (and if that software needs to be written first). So the best i can give here is you SHOULD be able to do it with the USB (no other hardware). But there is a chance the first batch of boards may not support this feature, requiring a JTAG to reflash them to add the "USB FLASH" capability to them initially. (after which USB would work fine).

@Yohan:

Can you please let us know if we can re-program or update the bitstream via USB?

[Glasswalker]

I should add that I doubt we will see a full 15W draw per chip. The ZTEX based opensource hashing core (fully pipelined) has a fairly high switching rate as it is. Any "improved" core will likely not increase that much beyond what it is (I suspect improved cores will use a non-pipelined, iterative approach, ie: fully rolled up. Many many small tight cores, instead of 2-3 big unrolled ones). So chances are we might see 12W or 13W but I doubt it will peak out to 15W (but who knows it is possible). I was just stating what I expect the max sustained draw per chip will be based on the (admittedly little) released specs on the power supply circuits.

[Glasswalker]

Can you please let us know if we can re-program or update the bitstream via USB?

I don't think this info is known (by them) yet. They are likely doing testing on the prototype boards now. Yohan will likely come let us know "All the Juicy Bits" (as he put it) once they have completed testing and have actually hashed on a physical board.

I suspect right now they are focusing on getting it hashing. If they then have time/resources to complete/test additional features and still hit their shipping timelines, they likely will.

My understanding right now is the hardware is fully capable of it, in that the USB chip itself can directly drive the jtag chain, requiring nothing fancy on the control FPGA at all. But I don't know exactly how they have it all wired. depending on how this is implemented it MAY require something in the control bitstream to drive the JTAG (in which case it's a feature they need to deliberately add). If it's wired up in a way that allows the USB chip to natively drive the JTAG, that means the only real limitation is software for your PC that can talk to that USB chip in the appropriate way, to load the bitstream to the board. (so in otherwords the board will support it fine, but you might not be able to use the feature until someone writes said software for the PC).

Also from what I see, there are 2 JTAG headers. It appears the right-angle header next to the USB is the main JTAG chain for the 4 "Worker" chips in the matrix. The other JTAG header appears to be for flashing the Spartan3 "Control" chip. So chances are because there are 2 different JTAG chains, the USB chip can likely only drive the JTAG chain for the 4 worker chips. And the JTAG for the Control chip is probably "hands off" (unless you have a JTAG cable). So I suspect that bitstream will be fairly "fixed" and considered not user-servicable, but the bitstream for the 4 worker chips will be USB flashable by the user fairly easily.

Also of note, Yohan has mentioned that there is the possibility (but nothing confirmed, ie: hardware can do it, but Enterpoint may not have time to roll the feature out before shipping, who knows if it can be added in future). To have multiple bitstreams stored in the flash. So that you can "dual boot" the boards, or have options like a "failsafe" bitstream. So you can install new ones, and it can fail back to the "factory default" bitstream on it's own for example.

Again, same disclaimer as before. All just my own speculation based on available info. Don't assume anything until it's officially stated by Yohan or other Enterpoint "official" channels.

[Dexter770221]

Wow, you really have to be borred
Power handle will be a major factor with this chips. Datasheet says of 2.2C/W Rthjc. With good heatsink and fan this will be more like 4C/W (thermal resistance adds up). So when this chip will consume 15W, core temp will increase 60C (15*4) over ambient. That means 85C core temperature (25C ambient + 60C from dissipation) and thats more than guaranteed by Xillinx for commercial grade chips. So, Rthjc is blocking higher hashrates, not power consumption. Watercooling may push this babies a little bit more...

You're being pretty pessimistic on the heatsink. You can get passive northbridge heatsinks with 2.5C/W thermal resistance. With airflow you should easily be able to get Rthca to 0.5C/W or lower if you go big/loud enough. You'll have quickly decreasing gains from going much lower than that, but 2.6 or 2.7 C/W Rthja should be reasonable and that would keep the junction at 60C in a 20C room with 15W.

I'm not talking about Rthja but Rthjc. Thats a big difference. Rthja for FGG484 package is 9.3C/W @750 LFM. So, without heatsink you can dissipate maks 5W with ALOT of air blowing at chip. At still air Rthja is 15.8C/W. Thats 3W and you are boiling a core.

[MrTeal]

Wow, you really have to be borred
Power handle will be a major factor with this chips. Datasheet says of 2.2C/W Rthjc. With good heatsink and fan this will be more like 4C/W (thermal resistance adds up). So when this chip will consume 15W, core temp will increase 60C (15*4) over ambient. That means 85C core temperature (25C ambient + 60C from dissipation) and thats more than guaranteed by Xillinx for commercial grade chips. So, Rthjc is blocking higher hashrates, not power consumption. Watercooling may push this babies a little bit more...

You're being pretty pessimistic on the heatsink. You can get passive northbridge heatsinks with 2.5C/W thermal resistance. With airflow you should easily be able to get Rthca to 0.5C/W or lower if you go big/loud enough. You'll have quickly decreasing gains from going much lower than that, but 2.6 or 2.7 C/W Rthja should be reasonable and that would keep the junction at 60C in a 20C room with 15W.

I'm not talking about Rthja but Rthjc. Thats a big difference. Rthja for FGG484 package is 9.3C/W @750 LFM. So, without heatsink you can dissipate maks 5W with ALOT of air blowing at chip. At still air Rthja is 15.8C/W. Thats 3W and you are boiling a core.

That's without a heatsink, just blowing air at the chip. With a heatsink to get the thermal resistance from junction to ambient you add Rthjc and Rthca, which encompasses the interface between the case and the heatsink as well as the heatsink itself. When I said 2.7C/W Rthja, that was the system resistance with a heatsink with good airflow over it.

[Dexter770221]

Wow, you really have to be borred
Power handle will be a major factor with this chips. Datasheet says of 2.2C/W Rthjc. With good heatsink and fan this will be more like 4C/W (thermal resistance adds up). So when this chip will consume 15W, core temp will increase 60C (15*4) over ambient. That means 85C core temperature (25C ambient + 60C from dissipation) and thats more than guaranteed by Xillinx for commercial grade chips. So, Rthjc is blocking higher hashrates, not power consumption. Watercooling may push this babies a little bit more...

You're being pretty pessimistic on the heatsink. You can get passive northbridge heatsinks with 2.5C/W thermal resistance. With airflow you should easily be able to get Rthca to 0.5C/W or lower if you go big/loud enough. You'll have quickly decreasing gains from going much lower than that, but 2.6 or 2.7 C/W Rthja should be reasonable and that would keep the junction at 60C in a 20C room with 15W.

I'm not talking about Rthja but Rthjc. Thats a big difference. Rthja for FGG484 package is 9.3C/W @750 LFM. So, without heatsink you can dissipate maks 5W with ALOT of air blowing at chip. At still air Rthja is 15.8C/W. Thats 3W and you are boiling a core.

That's without a heatsink, just blowing air at the chip. With a heatsink to get the thermal resistance from junction to ambient you add Rthjc and Rthca, which encompasses the interface between the case and the heatsink as well as the heatsink itself. When I said 2.7C/W Rthja, that was the system resistance with a heatsink with good airflow over it.

Ok, so find me a heatsink with 0.5C/W with dimensions like 40x40 mm, size that is standard among those FPGAs. Best what i was able to find was 1.7C/W @200LFM.

[DiabloD3]

Wow, you really have to be borred
Power handle will be a major factor with this chips. Datasheet says of 2.2C/W Rthjc. With good heatsink and fan this will be more like 4C/W (thermal resistance adds up). So when this chip will consume 15W, core temp will increase 60C (15*4) over ambient. That means 85C core temperature (25C ambient + 60C from dissipation) and thats more than guaranteed by Xillinx for commercial grade chips. So, Rthjc is blocking higher hashrates, not power consumption. Watercooling may push this babies a little bit more...

You're being pretty pessimistic on the heatsink. You can get passive northbridge heatsinks with 2.5C/W thermal resistance. With airflow you should easily be able to get Rthca to 0.5C/W or lower if you go big/loud enough. You'll have quickly decreasing gains from going much lower than that, but 2.6 or 2.7 C/W Rthja should be reasonable and that would keep the junction at 60C in a 20C room with 15W.

I'm not talking about Rthja but Rthjc. Thats a big difference. Rthja for FGG484 package is 9.3C/W @750 LFM. So, without heatsink you can dissipate maks 5W with ALOT of air blowing at chip. At still air Rthja is 15.8C/W. Thats 3W and you are boiling a core.

That's without a heatsink, just blowing air at the chip. With a heatsink to get the thermal resistance from junction to ambient you add Rthjc and Rthca, which encompasses the interface between the case and the heatsink as well as the heatsink itself. When I said 2.7C/W Rthja, that was the system resistance with a heatsink with good airflow over it.

So how would this change if I used a solid copper high performance heatsink plus expensive thermal paste like AS5?

[Dexter770221]

Difference between paste one or another don't play major role. Rthjc is like 10 times bigger, so more important is how long paste can hold their parameteres. Also I wouldn't count to find a heatsink with (of that dimensions) Rthja lower than 1C/W, with decent airflow.

[MrTeal]

Wow, you really have to be borred
Power handle will be a major factor with this chips. Datasheet says of 2.2C/W Rthjc. With good heatsink and fan this will be more like 4C/W (thermal resistance adds up). So when this chip will consume 15W, core temp will increase 60C (15*4) over ambient. That means 85C core temperature (25C ambient + 60C from dissipation) and thats more than guaranteed by Xillinx for commercial grade chips. So, Rthjc is blocking higher hashrates, not power consumption. Watercooling may push this babies a little bit more...

You're being pretty pessimistic on the heatsink. You can get passive northbridge heatsinks with 2.5C/W thermal resistance. With airflow you should easily be able to get Rthca to 0.5C/W or lower if you go big/loud enough. You'll have quickly decreasing gains from going much lower than that, but 2.6 or 2.7 C/W Rthja should be reasonable and that would keep the junction at 60C in a 20C room with 15W.

I'm not talking about Rthja but Rthjc. Thats a big difference. Rthja for FGG484 package is 9.3C/W @750 LFM. So, without heatsink you can dissipate maks 5W with ALOT of air blowing at chip. At still air Rthja is 15.8C/W. Thats 3W and you are boiling a core.

That's without a heatsink, just blowing air at the chip. With a heatsink to get the thermal resistance from junction to ambient you add Rthjc and Rthca, which encompasses the interface between the case and the heatsink as well as the heatsink itself. When I said 2.7C/W Rthja, that was the system resistance with a heatsink with good airflow over it.

Ok, so find me a heatsink with 0.5C/W with dimensions like 40x40 mm, size that is standard among those FPGAs. Best what i was able to find was 1.7C/W @200LFM.

With 40x40 heatsink, 200LFM is only 3.4CFM. That's not exactly a lot of airflow, you can get a 40mm fan rated for 6 times that or more if you're willing to pay the penalty. You still have to derate the airflow for the obstruction, but putting the equivalent of 800LFM over a heatsink isn't an issue. You also are not limited to a 40x40 heatsink, anything with a reasonably thick baseplate can have a larger are than it's contact area to increase surface area. This one is 45x45 and rated at 1.1C/W with 200LPM, and it was just something I found in a minute or two on digikey.
http://media.digikey.com/PDF/Data%20Sheets/Advanced%20Thermal%20Solutions%20PDFs/ATS-61450W-C1-R0.pdf
The scaling isn't linear, but cooling drops with the square root of airflow, so increasing airflow by a factor of 4 tends to drop thermal resistance by a factor of 2. Put 800LPM through that heatsink and it should be capable of close to 0.5C/W. If you actually looked for something bigger than that, or one of those crazy heatpipe northbridge coolers it would be even better.

[Dexter770221]

Price is quiet huge. Pushing more air, as you said doesn't mean that you will be able to go low as you want. It depends on how a heatsink is made (shape), which materials was used etc, etc.  There always will be some point where it will impossible to achieve lower C/W. And no one wants to have another fan that generates 50dB of noise spinning at 10000RPMs.
Spartan6 are designed as low power devices. Things will be much more better with Atrixes7. Spec says that they will have Rthjc lower than 0.2C/W. So, even with not so big heatsink 30W from that chip will be no problem to dissipate (with total Rthja of 1.5C/W).

[MrTeal]

Price is quiet huge. Pushing more air, as you said doesn't mean that you will be able to go low as you want. It depends on how a heatsink is made (shape), which materials was used etc, etc.  There always will be some point where it will impossible to achieve lower C/W. And no one wants to have another fan that generates 50dB of noise spinning at 10000RPMs.

Of course, you eventually trend towards the point where the surface of the heatsink is at ambient and the heat dissipation is just a product of the ability of the heatsink to transfer heat to the surface. The jump from 200 to 800 LPM isn't that large though, and is well before most designs stop reacting to increased airflow. Digikey's also quite expensive, for that price you could get something like this that would probably have no trouble with that kind of spec.
http://ultimatepccooling.com/xiptponobrch.html

Again, I just said you can get a 0.5C/W or lower heatsink if you go large or loud enough. I still think 1.8C/W (4-2.2) isn't really a good heatsink/fan combination, and is pretty pessimistic.

[Dexter770221]

... Again, I just said you can get a 0.5C/W or lower heatsink if you go large or loud enough....

But no one wants to do that The whole idea about FPGAs is that they are smaller, cooler, less noiser than GPUs. Even if you achieve 2.7C/W top power draw shouldn't be more than 17W, commercial grade chips, 25W industrial grade chips at 25C ambient temp. Compared to 40W of one chip in BFL Single is damn small value. And we have to remember that propagation delays thru silicon raises when temperature raise. More power draw, faster MHz limit = less MH/s. There is somwhere a sweet spot and it's defenitevely below 17W thanks to high Rthjc... Are we agreed?

[yohan]

Ok so time for a little more info.

We have enough heatsinks arriving next week to do the first 100 boards so that is good and solves the problem for May and early June shipments. We might have a little gap when we run out before our main batch arrives. The initial heatsink will be 10mm higher than the final solution. They are both from the smae family of parts. The higher one is about 1.5 Deg/W and the smaller 1.9 Deg/W at 200 LFM. at higher air flow rates they are very close in performance and both better again. That's why we choose to run with the smaller part as standard as it did not make much difference on thermal performance with our preferred fan option. These thermal resistance numbers are about half the thermal resistance of the Spartan-6 itself so they are not much of a limiting factor in the scheme of things. We can't do anything much about the Spartan-6 thermal performance that's with the Xilinx team that designed it so we just have to work with what we have and make the best use of it.

The second part of the cooling solution is an Arctic F12 fan which is a good balance of low noise (22.5dbA), long lifetime and high air blow. There are higher flow options but much noisier and we might offer those as an option somewhere down the line. Similarly copper versions of the standard heatsink make also be offered but only offer a little more performance for a lot of weight, direct cost, and shipping cost. 4 of the best are 0.5 Kg in weight and that is expensive to ship.

We won't ship a PSU as standard as many people will want to use options other than the 2.5mm jack. The weigh of these PSU is such the carriage will cost more than the PSU. However the 2.5mm jack ties up with 60W power supplies commonly available for LCD monitors at low cost. We may offer a PSU as a cost option for those that want it to arrive ready to go but there will be the extra cost of carriage. We will attempt to test a range of models and list our results but that won't be immediately.

On the schedule we are probably 2-3 days behind our original schedule plan but we still expect the first units to ship in May. All of the array FPGAs are programming under JTAG and the local SPI flashes are working and do configure the FPGAs now. The controller we have more work to but it is running a basic build now and is generally ok so far.

The Issue 1.1 design is now frozen and that starts it's path though manufacture on Monday. We have added a fan monitoring hardware now to all 4 fan headers and that connects to the controller.

JTAG driving capability is in the controller and it would probably use a micro engine of the FT4232 to drive it but there are other options. Update of the SPI flash is also viable if support is built into the array FPGA design.

LEDs - each FPGA has 4 LEDs.

We have yet to look at Bitcoin bitstreams and the compatibility and performance of them with Cainsmore1 but I expect we will get some of that done next week.

Generally I am happy with our progress so far. We do still have some way to go yet but that is the norm for this sort of project. On the manufacturing side our component supplies are shaping up well and we now have very few problems now in area.

I think what we have achieved in the 15 days since we started is pretty good even though the team here do things like this on a regular basis. This time we didn't beat our record delivery for designing a major new board and sending some to a customer fully tested but you can't win them all. Maybe we should have run a night shift like we do on really critical project time lines and we could have shaved a couple of days off but that's all academic now.

More to come over the next week or so.

Yohan

[MrTeal]

... Again, I just said you can get a 0.5C/W or lower heatsink if you go large or loud enough....

But no one wants to do that The whole idea about FPGAs is that they are smaller, cooler, less noiser than GPUs. Even if you achieve 2.7C/W top power draw shouldn't be more than 17W, commercial grade chips, 25W industrial grade chips at 25C ambient temp. Compared to 40W of one chip in BFL Single is damn small value. And we have to remember that propagation delays thru silicon raises when temperature raise. More power draw, faster MHz limit = less MH/s. There is somwhere a sweet spot and it's defenitevely below 17W thanks to high Rthjc... Are we agreed?

Yes, definitely. I wouldn't want to get nearly that close to the limit. That being said, if someone like eldentyrell releases a bitstream that does 250MH/s on an LX150 but pushes power consumption up to 15W, there are options out there that might be worthwhile to deal with that.