Cairnsmore1 - Quad XC6SLX150 Board

[DiabloD3]

Don't know how much the price difference is in the UK, but you might want to consider using this fan:

http://www.silenx.com/quiet.fans.asp?sku=efx-12-15

More quiet, good air flow and fairly cheap. This one is thanks for Fizzisist for making me a fan (pun intended)!

Its not a Delta AFB1212. There is no reason to EVER use non-enterprise/industrial fans in a computer.

http://www.delta.com.tw/product/cp/dcfans/download/pdf/AFB/AFB120x120x38mm.pdf

They make ones that are slower/quieter than what most people are used to from Delta (although, if you're worried about noise, taking a faster than and undervolting to 5v is often superior, as is ramping voltage on chip temp; it also reduces wear greatly).

[Glasswalker]

not offended at all! thanks for your help really! I dont doubt the manufacturer at all just eager to see what these things can do!

Excellent Glad I could help...

(damn these internets and their inability to convey the subtlety of inter-personal interaction!)

[Glasswalker]

You do seem to be assuming these will ship with any bit stream at all.

If they don't get round to having a working bit stream by the first delivery slot, which is in under 2 weeks then they will be shipping without.

Yes, I am making some assumptions. But as I said, it's more a matter of testing. as far as the design is concerned there is no reason to believe the icarus bitstream won't work out of the box. And worse case it may require some tweaking of the control fpga's bitstream to adjust communications to be more "within the range" of the icarus. But ultimately the board is designed to run the icarus bitstream AS-IS. (without any alterations). IE you could flash a binary bitstream to them from icarus and it should not know it's not on an icarus board.

So yes, I assume with their available skillset and experience they can have this thing with an icarus bitstream on it for ship day. (and I could be proven wrong on that, if so, so be it )

[ice_chill]

Could a 8 or even 12 chip board bring the price down to match BFL's offering in price/performance ?

[yohan]

Until we publish the data, or somebody else does, that's an academic question. Some patience is required but not too much longer to wait now.

Yohan

[DiabloD3]

Hey Yohan,

I know this may sound like a strange question, but is there any mass produced case that would fit the quad board? It looks to be roughly the same size as a PC104 board, but I doubt any of the mounting holes match.

[yohan]

It's not a strange question and we didn't design it for any specific format. This time we just wanted to do something simple and that is what Cairnsmore1 is.

There are 2 things under consideration here.The first is that for the next generation we do design either with a specific case or racking in mind. We are also considering what we might do with the current Cairnsmore1 and we might do a simple case for these. Worst problem is then is the cost of shipping such a case.

[Garr255]

Worst problem is then is the cost of shipping such a case.

Make it optional.

[Glasswalker]

It's not a strange question and we didn't design it for any specific format. This time we just wanted to do something simple andthat is what Cairnsmore1 is.

There are 2 things under consideration here.The first is that for the next generation we do design either with a specific case or racking in mind. We are also considering what we might do do the current Cairnsmore1 and we might do a simple case for these. Worst problem is then is the cost of shipping such a case.

You could just design the components to build a rackmount case type system (backplanes, communications busses, power distribution, mechanical mounting) and work with a few well placed systems integrators to actually build systems and ship them. If you design your solution to work with existing COTS gear (for example design your backplane to support the EATX standard) then you could use conventional off the shelf rackmount cases.

Come up with LCD display solutions that clip into 5.25 drive bays and so on. sell these components, and let people buy them to assemble their own solutions, or work with systems integrators to deal with shipping completed clusters (in which case if the SIs are local, you can cut down shipping of the heavy bits)

[Fuzzy]

Is there a date for when the test data will be released? Or is it a "when it's done" thing for now?

[Lethos]

I couldn't easily find this, but what are the dimensions of the board?

Since I know I wasn't the only one curious about it's dimensions.
Yohan confirmed these details via email when I ordered.

"Board is 126.492mm x 188mm give or take manufacturing tolerance. Main
mounting holes are 5mm (x) and 5mm (y) in from the corners."

[yohan]

The initial performance numbers are high on our list now to do. This week has been a slow week and was planned that way so we could clear critical aspects of other projects and do other things that are ongoing.

We are expecting the Issue 1.1 PCB in today and once that is assembled we will do a quick basics check again and then move onto the integration and performance testing aspects of the project. It is then a how long question but if it goes well we will know a lot in about 1 week. If it doesn't go so well we may have to modify the PCB design and that is about a 1 week slip. That's the joy of engineering.

[DiabloD3]

I couldn't easily find this, but what are the dimensions of the board?

Since I know I wasn't the only one curious about it's dimensions.
Yohan confirmed these details via email when I ordered.

"Board is 126.492mm x 188mm give or take manufacturing tolerance. Main
mounting holes are 5mm (x) and 5mm (y) in from the corners."

Huh, thats actually bigger than miniITX which is 170x170. ATX is 305x244, which could probably fit 12 FPGAs with room to spare.

Maybe one of the manufs will start making 12 FPGA boards with the four outer mount holes fit for ATX cases and power itself off a P4 (16a 12v) or EPS12v (28a 12v) plug?

[Glasswalker]

The board width is almost perfect for a 3U rackmount case though, so with a motherboard/backplane to carry/power them, and the daisy chained comms, you could do the motherboard to EATX standards, and probably fit 16 miners (4 FPGAs each) so total 64x FPGAs to a 3U rackmount case, which would draw in total around 800W so with a decent highend PSU that should work nicely for density. Allowing push/pull airflow past them, decent power density, and easy modular maintenance.

Also leaves the rest of the case (drive bays and so on) to mount a small host system, LCD status displays, or whatever else is needed. (or they could build the host onto the motherboard which would be really ideal). With that kind of setup you could grab a decent rackmount case for $100, a PSU for another $200 so at MOST (including tax/shipping) you're talking $500 per 3U 18board cluster in supporting bits. (I guess that doesn't include the cost of the (so far fictional) motherboard though).

Anyway that's the direction I'm thinking anyway.

[yohan]

Some mechanical details now available on the temporary webpage http://www.enterpoint.co.uk/cairnsmore/cairnsmore1.html.

[DiabloD3]

The board width is almost perfect for a 3U rackmount case though, so with a motherboard/backplane to carry/power them, and the daisy chained comms, you could do the motherboard to EATX standards, and probably fit 16 miners (4 FPGAs each) so total 64x FPGAs to a 3U rackmount case, which would draw in total around 800W so with a decent highend PSU that should work nicely for density. Allowing push/pull airflow past them, decent power density, and easy modular maintenance.

Also leaves the rest of the case (drive bays and so on) to mount a small host system, LCD status displays, or whatever else is needed. (or they could build the host onto the motherboard which would be really ideal). With that kind of setup you could grab a decent rackmount case for $100, a PSU for another $200 so at MOST (including tax/shipping) you're talking $500 per 3U 18board cluster in supporting bits. (I guess that doesn't include the cost of the (so far fictional) motherboard though).

Anyway that's the direction I'm thinking anyway.

Quite a few rackmount cases will fit EATX (330x305), thats 4.18 times more area, so that could theoretically fit 16 FPGAs (assuming ATX fit 12), and it'll draw about 200 watts (assuming 4x Spartan 6 uses 50 watts with the new ~250 mhash bitstreams). You can't just fit 4x existing boards in manually because you can't fit 376x252 inside of 330x305 (and EATX is pretty cramped inside cases as it is), so they'll have to place stuff differently on a new board.

[Glasswalker]

Yes you can.

If you use a motherboard design (with sockets to vertically mount the cards). The new 1.1 rev cards have a high current power connector mounted vertically, and the ribbon connectors for data up/down link from each card, so you can mount the cards vertically plugged into a power distribution backplane, and daisy chain several together using short ribbon jumpers (like crossfire bridges).

Even without the motherboard, you can do this with brackets to hold the cards vertically.

You loose the on-card fan and use push/pull through the case as I suggested.

Vertically the cards are just right to fit on a board in a 3U case. and this way they can be laid side by side to fill an EATX board (2 rows).

And yeah I know not all rackmount support EATX but many do, I've already priced out several with plenty of room. (yeah I know EATX might not have room for 8 cards by 2 rows deep might only be able to do 12 cards, but the cases I'm looking at relocate the PSU to the front of the case, allowing the full 17" wide back half for motherboard. That will easily fit 8 cards, 2" each for plenty of airflow. 2 rows deep.

[spiccioli]

Glasswalker,

do you mind sharing some link?

thanks.

spiccioli

[DiabloD3]

Yes you can.

If you use a motherboard design (with sockets to vertically mount the cards). The new 1.1 rev cards have a high current power connector mounted vertically, and the ribbon connectors for data up/down link from each card, so you can mount the cards vertically plugged into a power distribution backplane, and daisy chain several together using short ribbon jumpers (like crossfire bridges).

Even without the motherboard, you can do this with brackets to hold the cards vertically.

You loose the on-card fan and use push/pull through the case as I suggested.

Vertically the cards are just right to fit on a board in a 3U case. and this way they can be laid side by side to fill an EATX board (2 rows).

And yeah I know not all rackmount support EATX but many do, I've already priced out several with plenty of room. (yeah I know EATX might not have room for 8 cards by 2 rows deep might only be able to do 12 cards, but the cases I'm looking at relocate the PSU to the front of the case, allowing the full 17" wide back half for motherboard. That will easily fit 8 cards, 2" each for plenty of airflow. 2 rows deep.

You're trying to go for a modminer "blade server" kind of design, which isn't entirely appropriate either. Lets try this instead.

Lets say you use a 4U case (3U won't fit full height cards) and used full sized PCI-shaped cards and no backplane at all (power and USB/GPIO (say, using a SATA plug instead like BFL is doing) is on the end of the card like GPUs), and you put 4 FPGAs on the card. The card will have to be 107mm by 312mm (or less: many cases, even rackmount, won't fit a full length card; 6990s and 7970s are full length), so you're fitting the FPGAs in a straight line instead of a quad configuration.

Now, because of the heatsink requirements, you're going to have to fit these in double thick configurations (using, I assume, pure copper 2U northbridge heatsinks and forcing airflow through them from the front of the case), so you're putting around 16 FPGAs in a case (7 case slots, 1 overlaps with the case so it needs a 1 slot bracket)

Now, if you pack these in even harder using 1U northbridge heatsinks, that'd be 4*7, powered using 7 PCI-E 6/8 plugs, or 28 FPGAs in 4U.

Now, my 16 FPGA in a 1U or 2U case? In 2U, thats 32 FPGA per 4U, in 1U, thats 64 FPGA per 4U. I think my idea wins.

[Glasswalker]

I'm not arguing that a super high density board would be the ultimate in high density mining. Point is there is a sweet spot for a distributor between low volume sales and profit. With this board design they can make a high volume of boards, and reduce their per-unit overhead. Meaning better profit margins. A super high density board would allow better density, but would likely have much lower sales volumes (requiring thicker markup to make back their R&D overhead and such). Also manufacturing yield is easier to control on smaller boards. Lastly, customer hardware failures are mitigated more with many small boards versus one large board. (if you blow a couple FPGAs on your 64x FPGA board what do you do about it?)

My ideas are purely within the confines of the cairnsmore1 product. How to pack as many into a rack as possible. Not a hypothetical new board. (enterpoint has said they may consider additional boards later depending on the success of the cairnsmore1, for now this is what we have to work with).

That said, you have a very good point about card height. Looking again at the mechanical drawings, once you consider motherboard thickness, connector height, and raised MB plate thickness the board likely won't fit in a 3U. (it would be damn close though, 3U is 133MM but internal space will be less, this board is 126.4 so likely won't fit in the end).

But I don't see why you think a "blade server" type approach is inappropriate? It offers high density (at least close to that of a super high-density board) and it offers modularity, easy maintenance (swap boards out), lower risk in the event of failure, and most importantly to many, smooth scalability (easy to keep buying small volumes of boards and expanding rather than having to drop $30K at a time).

Now considering 4U cases, an ideal option would be something like http://www.newegg.ca/Product/Product.aspx?Item=N82E16811165475
Remove the motherboard mounting plate, and mount the power supply where the 3x 5.25" bays are in the front. That would allow a full case width and a flat mounting surface in the back for the cards. You can easily fit 8 cards wide with plenty of room for heatsinks, and should be able to fit 2 cards deep in that config. This gives the same yield of FPGAs/U that your idea has, but is overall cheaper (1U rackmount cases are more expensive generally. or custom rackmount enclosures are also expensive, meaning per 1U you would be looking at probably $200-$300 per case if you include a power supply, making for $800-$1200 per 4U) in my setup that's $300 for a 4U case, $200 for PSU, so $500 total in case cost per 4U, same total number of FPGA, and I still have room to grow in the front half of the case (could always remove the hotswap drive cage and mount more cards up there, or have room for an added controller for standalone mining or whatever).

I think considering these cards specifically, that type of solution is the best way to go. Too bad it's so close to the 3U spec. If it would fit in a 3U that makes it more dense, which is even better.