BitFury Design, Licensing, Mass production

[bitfury]

Dear BitCoiners, BitCoin talk users!

First of all I would like to thank to other developers of Spartan-6 based works, especially those who contacted me soon - shalab.si, fpgamining.com team, Greg and all those who understand how hard that work is. Also I would like to thank all those who supported me during development of these mining racks and invested money, so I haven't went into "preorder"
fall like offering more Mh/s power than hardware actually could deliver.

I am actually in love with bitcoins, because this is exactly the thing I lacked in 2006 year, and tried hard to invent... Unfortunately I were not smart enough to find out proof-of-work based blockchain. And this solution actually useful even in wider range of applications than money transfer themselves, as it can limit or remove human factor in distributed database consistency. It is interesting how this will evolve.

Second, I was surprised by reaction about comparision with ButterFly Labs with 'Estimated price'. People, that's $90k is with 20% VAT, which is not paid in US for example and with having in mind that it would have single hot air outlet and that's all about it... but to get mini-rig we would pay +20% on customs VAT, so it would be not $15'295 but $18'354 + shipment costs, so it is $0,728 per Mh/s. We've spent on boards production about $0,50 per Mh/s and on installation about $0,12 per Mh/s, totalling $0,62 per Mh/s. Count bitstream offer (it will be later in this topic) - it would be $0,69 per Mh/s. SO IT IS $0,03 LESS THAN THAN BFL Mini-Rig and it really works for almost 2 month, so we are sure now that it does not fry chips etc. We have not even managed to fry chip, when fed it with 1.5 V VCCINT core voltage. But, price would be definately higher (near estimated production cost), if counted other issues, like climate control, getting specialized area for them, getting power (as it consumes 20 kW together with chiller). All of these costs were already paid, because it is installed in basement of household with already existing heat pump.

Third, 300 Mh/s is not limit of this bitstream, it can give even more rounds, if you count that almost all DSPs are not used in left part of design, and some free space in topmost part. So it could definitely give 8% at least better performance, it would then cost to us $0,57 per Mh/s, which is even less than BFL Mini-Rig.

I've been asked by email and skype about smaller editions. And I would say that in my opinion best solution would be standard 4-U chassis, 0.5 meter long, with 14-15 boards with 6 chips on board installed (that is 84 - 90 chips), like with shalab.si original ideas, but a bit different layout to prevent overheating chips. Withing this chassis single Intel Atom D525 motherboard is installed. Boards with 6 spartans can be even without microcontrollers and flash, everything could be programmed right via LPT-port. Bandwith required to communicate with every chip is quite low - about 300 bps. So with all chips it would be about 27 kbps. Bitstream loading over LPT port however will be slow. For smaller scale RS-485 is overkill. Why to bother about it and not implement using USB - simply because flashing chip or flashing controller
adds up cost of controller and also costs of programming and testing them, also when something should be updated, and you have to reprogram every controller - that rises service cost. I would like to say, that current design of BitFury rack, where controller only translates RS-485 to SPI bus with Spartan and back requires almost zero maintenance.

Cost of such chassis with power supply and Intel ATOM motherboard could vary in $400 - $600 range. Cost of Spartan6 chips when purchased in bulk quantities (WITHOUT VAT) would vary in $70 - $95 range, depending on shipment location and quantity of chips ordered. Cost of other components (using numbers from our current design):

motherboard 4-layer $111 ($24,5 PCB and $8,53 soldering), daugthercard 6-layer $18.39 ($2,2 PCB and $5,5 soldering). these are costs with components with VAT and connectors. If connectors will be removed and VAT will be removed, then cost would be like $77,7 for motherboard for 6 chips and daugthercard $12,83 . Totalling $154,68 per board .

So server with 14-15 boards could cost in range $8'445 to $11'470. More likely that actual manufacturing cost plus work will be somewhere about $10'000 to $10'500. Then, say it would have 90 chips. I would point out how important Mh/s  are:

BFL Mining rig quite comparative product promised to be sold at $15'295 for 25,2 Gh/s - that is $0.607 per Mh/s

200 Mh/s bitstream - would produce 18 Gh/s - it would be compared to 71% of Mini-Rig and so product price could be $10'859.
250 Mh/s bitstream - would produce 22.5 Gh/s - it would be compared to 89,2% of Mini-Rig and so product price could be $13'643.
300 MH/s bitstream - would produce 27 Gh/s - it would be compared to 107% of Mini-Rig and so product price could be $16'365.
325 MH/s bitstream - would produce 29.2 Gh/s - it would be compared to 116% of Mini-Rig and so product price could be $17'722.

So at 200 Mh/s - there's almost no difference between bulk order and product price. At 250 Mh/s there's $3'143 income.
At 300 Mh/s there's $5'865 income. And at 325 Mh/s there's $7'222 income. Additional 50 Mh/s per chip gives 86%  income increase if prices are set at BFL levels of $0.607 per Mh/s. Additional 75 Mh/s per chips gives +129% income.

Calculating these costs and also costs handling sales, manufacturing etc. lead me to following licensing targets for about 1000 chips per month installed:

$20 - $25 per chip (depending on chip price and costs incurred by AES-key programming) for current bitstream and
$5 - $7.5 for future upgrade (separately), which can be opted when such update actually done.

From our side most important point is bitstream protection. This incurs costs of AES-key programming by moving trusted person to assembly plant from time-to time, powering on boards and fusing chips with AES-key. For simplicity it would be great to have 6 spartans on single board tied on JTAG line via buffers. Then encrypted bitstream could be available without any additional protection to it. Of course if quantities will be small and location distant, it would be difficult to execute programming. We are already planning to program chips with AES key in Hong-Kong, and we have good access to EU, because we are located in Ukraine. Existing boards could be upgraded as well.

Also - why I am insisting that 4U design one of the best sizes:
1) it would consume 1.3 - 1.5 kW per 4U, and that fits into envelope 350W per 1U heat production, which is not difficult
to implement in either datacenter or homebrew setup;
2) Chassis itself could be sold with minimal margin, so people could build mining power within nice chassis step by step;
3) When local price to electricity would be unaffordable, people can send what they built to Iceland for $0.04 per kw*h
special setup, so compatibility with datacenters is nice feature;
4) It gives $1'800-$1'900 "entry-level" price and board-by-board upgrade possibility;
5) "special single board" can be available as well - with USB, and it would be nice option that it can be put into server,
basically I expect to lower costs to have boards with pads for USB-related components, just they will not be soldered for
items that will be put into servers;
6) also it would be nice to have some DRAM on board for other purposes, but again - this would be not soldered at all, but later, if boards will be re-used for other tasks such DRAM capability and at least 1G ethernet external connectivity would make difference for long-term product life. At stage of PCB design it adds only NRE costs of designing PCB itself, and no cost at manufacturing.

Thinking where equipment will be placed - in small quantities - probably in homes, but when it will get larger - I have already found location and discussed this issue with Andreas Fink (Skype: andreasfink ) from datacell.com

[5/22/2012 10:24:36 AM] Andreas Fink: Our datacenter is designed for 20kW cooling per rack
[5/22/2012 10:24:37 AM] bitfury.org: but if air inlet drops 10 degrees Celsius, performance can be increased. and optimal temperature is about 0 degrees Celsius if we use industrial chips. that gives 20% performance increase.
[5/22/2012 10:24:44 AM] Andreas Fink: we use hot aisle concept
[5/22/2012 10:25:17 AM] bitfury.org: buyers of these devices actually concentrate on pure performance and cost of electricity, cooling etc. many of them installing such installations right at home, because it is cheaper.
[5/22/2012 10:25:27 AM] bitfury.org: you can google for "bitcoin mining"
[5/22/2012 10:25:40 AM] Andreas Fink: Well we operate our datacenter in iceland. energy is cheap there and coolign is easy.
[5/22/2012 10:27:05 AM] Andreas Fink: We could build a custom datacenter just for that if there's enough demand.
[5/22/2012 10:28:05 AM] Andreas Fink: what is the potential of your installation if it produces bitcoins?
[5/22/2012 10:29:38 AM] bitfury.org: very difficult to say, because we have just started to announce our solution.
[5/22/2012 10:29:54 AM] Andreas Fink: a rough estimate to get an idea.
[5/22/2012 10:29:54 AM] bitfury.org: current bitcoin network hashpower is 10'000 Gh/s, single bitfury produces 110 Gh/s
[5/22/2012 10:30:03 AM] bitfury.org: so estimation is about 30-40 racks
[5/22/2012 10:30:28 AM] bitfury.org: that's potential for FPGA bitcoin mining.
...
[5/22/2012 11:37:42 AM] Andreas Fink: the electricity company charges like 3-4 eurocents per kWh
...

It depends - if we manage to start with 4U working with them and without using diesel/ups-backed power which is  expensive and climate control equipment - it would be cheapest collocation for mining available in world.

[5/22/2012 11:02:04 AM] Andreas Fink: http://www.sgi.com/products/data_center/ice_cube_air/
[5/22/2012 11:02:17 AM] Andreas Fink: put them in such a shelter, we provide power and internet and put it into the green field

I suppose that shelter should be designed as well as SGI could be expensive, but generally you've got the point. And that is why I am sticking with rackmount design. It will be big pain to move devices that are non-compliant with datacenters, putting Mini-Rig there for example. I suppose this is the moment, when buyers of BFL should start thinking twice, what they would do with their rigs, when income will be about the same as consumed power. For myself that problem is solved - mining equipment is integral part of house heating now. For cold countries this can be as well solution - integrating house / DHW heating with mining. Today you will probably say that it is absolutely crazy, but 5 years in the future things would change, and those who make useful use of excessive heat will save on electricity bills, while those who will pay additionally for air conditioning would loose, as they could not compete.

So finally, I would welcome everyone interested to join efforts. First - design chassis and PCB with more-or-less interexchangable parts. Getting for example with-USB and without-USB version, with-flash and without-flash version, etc. so actual board can be customized by needs of our partner who provide product, but in general they all are near the same, and could be fitted inside 4U chassis. This way we can save on designing this thing, build product in which our customers will be confident, that even if one supplier stops selling boards, he can possibly with higher costs order elsewhere, but still finish his 4U box. More important is different metal pieces like holders of boards, etc. Which would depend on supplier of these chassis solution.

I am going to provide soon draft of 4U box, as I see it in 3D with airflow calculations. For those who asked about current interface, I am attaching right into this message .vhd source code of bitstream side, current .ucf file (but please, it can change - we used 1 mm FGG484C, and it could be possibly better to use CSG 0.8 mm steps), only important point that communication enters BOTTOM of chip. And communication part in dsPIC33F firmware. As you probably would see - same communication can be performed over LPT port without bothering with production of additional conversion boards.

Here are links to download:
http://www.bitfury.org/bfdetails/sha_top.ucf
http://www.bitfury.org/bfdetails/sha_top.vhd
http://www.bitfury.org/bfdetails/shaspi.vhd
http://www.bitfury.org/bfdetails/jobs.c

Meanwhile - we used FGG484 because we though to put more bypass capacitors below chip for overclocking purpose, as with CSG there
will be difficult to place such numbers of capacitors. However we've aimed for 320 Mhz that makes too much errors, with 240 Mhz
it would be fine to place less capacitance, and also get industrial CSG chip, as this would remove any possible overheating problems.
CSG chips have about 1.5 times (!) lower junction-case thermal resistance according to datasheet. However this would be great to be
confirmed by someone who used CSG chips - I saw ZTEX used them.

So - how we shall proceed ? I suppose best way will be to:
1) Write to this topic, if you are in (possibly after getting in touch in skype: bitfury.org );
2) Writing estimated quantities per month, assembly plant location (place where chips can be turned on);
3) Writing your final destination where you are going to ship these boards for further packaging;
4) Writing number of chips you already use - because if there's many already - then possibly re-work on IR station could cost
   significantly less than buying new chips;
5) Writing your requirements to board - specifying way of Spartan programming, currently used interfaces (i.e. USB with USB chip name),
   so we can get common specification for board interfaces and chips that shall be on board. Probably for overclockers it would be nice
   to have ability of voltage setting.

Then next iteration will be to approve specification and doing PCB schematic design, and then doing layout, and
doing similar way with chassis. Hopefully there's people who like working 24 hours + night

Kind regards,
Mr. V // Skype: BitFury.org

PS. Special note for those who know me personally. I dislike indexing of personal information via google, etc, so please, do not write here - "oh I know this guy"... Write it in skype in personal communication (it will be visible at least for
special services and not to general johny hacker who mines information and would like to get some cash), not on public forums  or public web pages. And I discourage everyone from doing so with their information. Imagine that it is possible to enter your name to google not only to your friends and partners, but also to your foes, those who you even do not know...

PPS. I've also uploaded stats file http://www.bitfury.org/bfdetails/bitfury_status.txt with current number of working cores, you may see that some cores are not working, error rates.... There were no special handling with them - we've just put cores "as is" into motherboards, and there are no visible defects. We've not bothered replacing these cores. Just to those who say that it is not real thing

EDITED - also forgot part of conversation with Andreas Fink about Iceland VAT there is 25.5% but for collocated servers it is returned. So it is like having 0% VAT compared to 18-20% VAT in EU countries.

[1715268581]

1715268581

[1715268581]

1715268581

[1715268581]

1715268581

[1715268581]

1715268581

[1715268581]

1715268581

[rjk]

Wow! This is an amazing development.

[MrTeal]

Very interesting. Do you have any plans to license out just the bitstream?

[MoonShadow]

I'm interested, but I'm not understanding something.  What is the "bitstream" in this context, and what is it for?

[rjk]

I'm interested, but I'm not understanding something.  What is the "bitstream" in this context, and what is it for?

Also known as firmware. It is quite literally a stream of compiled bits that the fpga can execute.

[Inspector 2211]

My 2 cents:
The pricing is pretty much OK, but the AES key programming could turn into a logistical nightmare.
In a few days, EldenTyrell will reveal his offer and if it is competitively priced, but does not involve sending boards back and
forth for AES key implantation, it'll probably be the offer I'll pursue.

[MoonShadow]

I'm interested, but I'm not understanding something.  What is the "bitstream" in this context, and what is it for?

Also known as firmware. It is quite literally a stream of compiled bits that the fpga can execute.

Ah, I see.  I know what firmware is, but I've never heard it called a bitstream before.

[bitfury]

I'm interested, but I'm not understanding something.  What is the "bitstream" in this context, and what is it for?

This topic is more intended for developers of FPGA boards. I suppose they would join soon. For better understanding I will explain whole cycle of these devices:

Xilinx (www.xilinx.com) produces programmable logic chips - XC6SLX150 - the key is that they are "programmable" bitstream is configuration that configures wires and logic cells inside of chip, so it is not like usual program of GPU miner etc. There is no such thing like CPU in FPGA by default, although FPGA designer can implement many CPUs. Depending on effort - there could be many different implementations of same algorithm within FPGA. Most difficult part is LAYOUT within chip to get high performances, as unlike programming there you have to deal with wire and logic latency, design synchronization and etc. So - to get nice product you should definitely have best bitstream available for FPGA. - This point is solved - BITSTREAM IS QUITE COMPETETIVE .

Then comes in play PCB (printed circuit board) that provides connectivity and power for FPGA chips, it has many cheaper components. But their sole role to give jobs to FPGA, download results and make FPGA and its user happy. PCB itself is manufactured in multiple specialized factories. Also there's multiple assembly plants around globe who would gladly take PCB without components and solder them. At this point you'll have working PCB.

Then comes assembly of PCBs into devices - basically putting boards into chassis, adding PSU, configuring.

Then it comes to delivery to end-user. With also servicing it within warranty period, if required.

So with the rest of points - making PCB, assembling them - I am looking for partners that would deliver these devices best to their customers.

So if you are in - and would like to buy such devices - you can (a) ask someone nearby you who already have Spartan-6 solution - that you are aware and would like to buy NNN {put your numbers here } devices from them; (b) if there's nobody nearby, you could contact shalab.si for example soon and import modules from Slovenia etc.

Luckily, with help of this topic, I suppose to find those who are interesting in making and selling this stuff. So you could go locally and just order it. To make things more interesting for developers, I would like to ask you tell numbers - how many Mh/s you want. Target price I think is $0.60 for Mh/s without VAT (I suppose that everyone knows well what is Mh/s and possibly only problems with these fancy bitstream Mhz numbers).

[bitfury]

My 2 cents:
The pricing is pretty much OK, but the AES key programming could turn into a logistical nightmare.
In a few days, EldenTyrell will reveal his offer and if it is competitively priced, but does not involve sending boards back and
forth for AES key implantation, it'll probably be the offer I'll pursue.

Well - sending boards back and forth is bad case, and it will not work that way. Better case to program it in place where they are assembled. But that requires high level of trust to one who is doing such programming. Also it is cheaper to assemble in higher volumes. So if there will be 1, 2 or 3 points of PCB assembly with nice flows of chips - it is not problem to get trusted person, who will turn board on, program AES and done. I am ready to solve it, so it would be without any delay. But not for 20 points of assembly (soldering of PCB, not building complete product) with 10 chips per month each.
Anyway - to get good prices you have to make volumes, this is why we started with racks, and not with small units - it was much cheaper that way.

EDITED: Or - how else could it be protected ? Because selling it for $20 is like uploading it to ftp. Say you will not pirate it, but one of 100 users would certainly do, and they'll even think that they did it right way.

EDITED: Or another possibility was offered - that if chips are supplied to different board vendors using same channel, then we could program chips at some point during their initial delivery. Probably this could be best option for smaller purchases, but it depends.

[Inspector 2211]

My 2 cents:
The pricing is pretty much OK, but the AES key programming could turn into a logistical nightmare.
In a few days, EldenTyrell will reveal his offer and if it is competitively priced, but does not involve sending boards back and
forth for AES key implantation, it'll probably be the offer I'll pursue.

Well - sending boards back and forth is bad case, and it will not work that way. Better case to program it in place where they are assembled. But that requires high level of trust to one who is doing such programming. Also it is cheaper to assemble in higher volumes. So if there will be 1, 2 or 3 points of PCB assembly with nice flows of chips - it is not problem to get trusted person, who will turn board on, program AES and done. I am ready to solve it, so it would be without any delay. But not for 20 points of assembly (soldering of PCB, not building complete product) with 10 chips per month each.
Anyway - to get good prices you have to make volumes, this is why we started with racks, and not with small units - it was much cheaper that way.

EDITED: Or - how else could it be protected ? Because selling it for $20 is like uploading it to ftp. Say you will not pirate it, but one of 100 users would certainly do, and they'll even think that they did it right way.

EDITED: Or another possibility was offered - that if chips are supplied to different board vendors using same channel, then we could program chips at some point during their initial delivery. Probably this could be best option for smaller purchases, but it depends.

I am prototyping my own design as we speak, and thinking of having 100 boards (or 127 boards, because the assembly quote was for 127 - don't ask) assembled in China, at a relatively unknown assembly house, and then shipped to me.

I do not know whether this unknown assembly house would have the technical know-how to burn in an AES key into the FPGAs, in fact right now I have no plans to have a JTAG connector populated. And I do not know whether this unknown-to-you assembly house in the middle of China (literally) would have your "trust".

But, since I'm in NorCal and you seem to be in Los Angeles, maybe I can just drive down to you one weekend, you program in your AES key into all 100 (or 127) boards, and I hand over a check for 2 grand (or 2.5 grand). Or cash, if desired. Maybe we can work something out.

Right now, the current revision of my board has a relatively weak DC-DC converter chip on it (which I actually bought from someone on this forum, but I don't blame him - who could have foreseen bitstreams so fast) and I need to redesign the DC-DC converter to a 12 amp one. While I redesign it, could you be so kind and disclose the PINOUT that you have, i.e. the .ucf file. In other words, I'm redesigning the board ANYWAY, might as well redesign it to conform to your exact pinout...

[ice_chill]

Explain your marketing:
BFL couldn't sell enough units at $30k so they came up with one at $15k, and you jump in straight with $90k?

[rjk]

Explain your marketing:
BFL couldn't sell enough units at $30k so they came up with one at $15k, and you jump in straight with $90k?

Looks like it is possible to split into smaller rigs.

[Inspector 2211]

Explain your marketing:
BFL couldn't sell enough units at $30k so they came up with one at $15k, and you jump in straight with $90k?

As explained on their website, they were expecting even higher hash rates. The ISE tool even led them to believe higher hash rates are achievable, but in practice the error rate was too high and they had to downclock to 240 MHz.

Wow, I can't believe I just said "downclock to 240 MHz".

[Gomeler]

Would it be possible to have some sort of client/server setup for the bitstreams? Basically have some way to pay for a license and run a properly designed miner that pulls down the bitstream from the server in a secure fashion. I guess in the end, once the user has the bitstream loaded on to the fpga they could simply unload it and store it locally? I'd pay money monthly for a bitstream that dramatically increased my hashing power if it meant keeping that bitstream secure. Would it be easier to have a kickstarter fund started up where miners and fpga manufacturers could pay to opensource the bitstream?

[Cablez]

Would it be possible to have some sort of client/server setup for the bitstreams? Basically have some way to pay for a license and run a properly designed miner that pulls down the bitstream from the server in a secure fashion. I guess in the end, once the user has the bitstream loaded on to the fpga they could simply unload it and store it locally? I'd pay money monthly for a bitstream that dramatically increased my hashing power if it meant keeping that bitstream secure. Would it be easier to have a kickstarter fund started up where miners and fpga manufacturers could pay to opensource the bitstream?

Wasn't this what largecoin was attempting?

[Gomeler]

Would it be possible to have some sort of client/server setup for the bitstreams? Basically have some way to pay for a license and run a properly designed miner that pulls down the bitstream from the server in a secure fashion. I guess in the end, once the user has the bitstream loaded on to the fpga they could simply unload it and store it locally? I'd pay money monthly for a bitstream that dramatically increased my hashing power if it meant keeping that bitstream secure. Would it be easier to have a kickstarter fund started up where miners and fpga manufacturers could pay to opensource the bitstream?

Wasn't this what largecoin was attempting?

No clue, never paid attention to Largecoin. Weren't they the ASIC company that has gone nowhere?

[rjk]

Would it be possible to have some sort of client/server setup for the bitstreams? Basically have some way to pay for a license and run a properly designed miner that pulls down the bitstream from the server in a secure fashion. I guess in the end, once the user has the bitstream loaded on to the fpga they could simply unload it and store it locally? I'd pay money monthly for a bitstream that dramatically increased my hashing power if it meant keeping that bitstream secure. Would it be easier to have a kickstarter fund started up where miners and fpga manufacturers could pay to opensource the bitstream?

Wasn't this what largecoin was attempting?

No clue, never paid attention to Largecoin. Weren't they the ASIC company that has gone nowhere?

Apparently they had a structured ASIC in the works, but no one has seen one yet, and they've been pretty quiet.

[bitfury]

Would it be possible to have some sort of client/server setup for the bitstreams? Basically have some way to pay for a license and run a properly designed miner that pulls down the bitstream from the server in a secure fashion. I guess in the end, once the user has the bitstream loaded on to the fpga they could simply unload it and store it locally? I'd pay money monthly for a bitstream that dramatically increased my hashing power if it meant keeping that bitstream secure. Would it be easier to have a kickstarter fund started up where miners and fpga manufacturers could pay to opensource the bitstream?

Unloading bitstream is impossible....

About remote activation - it is pretty possible thing. There's actually several possibilities - one possibility that bitstream reads Device DNA code (it's serial number), encrypts and transmit it to server, server sends activation code, bitstream matches it - and if Device is in white list proceeds with execution. But this is possibly moderate protection, as bitstream could be reverse-engineered and wires going to Device DNA could be replaced with connection to logic fabrics.

More complex protection would require 2-stage loading - first-stage bitstream that is auto-generated, and mainly generates "special" kinds of public-private key pairs with embedded proof of work (like calculating it for 4 seconds in true Spartan). It would not work in simulators, and it would be difficult to reverse-engineer as code for calculation is generated. Then comes DPR (dynamic partial reconfiguration) to load first protected bitstream.

Also this bitstream could be "rented" - say working with nTime less than specified.

But pity moment about this I wrote you is that it requires quite high effort to be done, compared to fusing AES key.

It is bad that chip manufacturer implemented AES only, because if they would implement in silicon some public-private key infrastructure with Xilinx certificate - it would be much simpler.

I would just take public key + xilinx signature for public key that you supplied, check against xilinx certificate and done, I can deliver bitstream. This would work like charm, but there's no such feature.

Even with e-fuse it is less protection compared to SRAM + battery for AES key.

Just compare attack cost... You pay $20, and get thing that costs say $50k+ you could pay additionally then $20k just to break it. I understand that you are interested, believe me that when there will be no heat about Spartan - I will disclose bitstream + disclose ways how it was achieved, because it can be valuable for learning.

EDIT: https://bitcointalk.org/index.php?topic=49971.msg918095#msg918095 - claims that bitstream format is available under NDA... So - as long as you can reverse-engineer bitstream and patch it - protection won't work. So it would be just lost time. Protection should be assisted with hardware.

[MoonShadow]

I understand that developing such a product requires huge investments and risks, but this community is rabidly pro open source.  You might succeed at this for a time, but someone is going to replicate it and release it to the wild eventually.  I'm sure that you understand that, so the goal is to get as much profit from it as you can before the open source developers catch up.  May I offer an alternative to all this?  If you have an idea of how much revenue you expect to earn, let us know.  You can proceed with your licensing method however you see fit, but if the community can raise the funds, would you accept a collective donation in order to release the bitstream open source?

[2112]

EDIT: https://bitcointalk.org/index.php?topic=49971.msg918095#msg918095 - claims that bitstream format is available under NDA...

Since this refers to my message that wasn't clear: Xilinx bitstream format WAS at one time available to Xilinx EDA software partners, possibly under NDA, possibly under some different sort of legal and financial protection.

I no longer consult for that EDA vendor and I have no relevant knowledge regarding current products. I only remain friends with people who continue to maintain and upgrade the EDA software applications that I co-authored years ago.

[Inspector 2211]

EDIT: https://bitcointalk.org/index.php?topic=49971.msg918095#msg918095 - claims that bitstream format is available under NDA...

Since this refers to my message that wasn't clear: Xilinx bitstream format WAS at one time available to Xilinx EDA software partners, possibly under NDA, possibly under some different sort of legal and financial protection.

I no longer consult for that EDA vendor and I have no relevant knowledge regarding current products. I only remain friends with people who continue to maintain and upgrade the EDA software applications that I co-authored years ago.

It seems plausible that companies like Synopsys get access to the bitstream format, possible for a fee, but less plausible that any Tom, Dick and Harry could get access to it.

[DiabloD3]

Second, I was surprised by reaction about comparision with ButterFly Labs with 'Estimated price'. People, that's $90k is with 20% VAT, which is not paid in US for example and with having in mind that it would have single hot air outlet and that's all about it... but to get mini-rig we would pay +20% on customs VAT, so it would be not $15'295 but $18'354 + shipment costs,

Remember, US customers don't pay VAT, so you've probably quoted a EU price that is useless for US customers. If you can get BFL Minirig prices on Spartan 6 hardware, BFL is screwed.

Third, 300 Mh/s is not limit of this bitstream, it can give even more rounds, if you count that almost all DSPs are not used in left part of design, and some free space in topmost part. So it could definitely give 8% at least better performance, it would then cost to us $0,57 per Mh/s, which is even less than BFL Mini-Rig.

Someone said in another thread that the sea of hashers design will have three types of cores, ones near DSPs, one near BRAM, ones near neither.

I've been asked by email and skype about smaller editions. And I would say that in my opinion best solution would be standard 4-U chassis, 0.5 meter long, with 14-15 boards with 6 chips on board installed (that is 84 - 90 chips), like with shalab.si original ideas, but a bit different layout to prevent overheating chips.

That gives people mostly what they want.

Withing this chassis single Intel Atom D525 motherboard is installed. Boards with 6 spartans can be even without microcontrollers and flash, everything could be programmed right via LPT-port. Bandwith required to communicate with every chip is quite low - about 300 bps. So with all chips it would be about 27 kbps. Bitstream loading over LPT port however will be slow. For smaller scale RS-485 is overkill. Why to bother about it and not implement using USB - simply because flashing chip or flashing controller
adds up cost of controller and also costs of programming and testing them, also when something should be updated, and you have to reprogram every controller - that rises service cost. I would like to say, that current design of BitFury rack, where controller only translates RS-485 to SPI bus with Spartan and back requires almost zero maintenance.

Use SATA plugs for the actual connector, but normal serial over it. SATA has 7 pins and is enterprise ready. The cost of serial is the complex plug, not the actual design. You could use a very tiny FPGA for the controller on each board to interface with the serial using GPIO pins or something.

Cost of such chassis with power supply and Intel ATOM motherboard could vary in $400 - $600 range. Cost of Spartan6 chips when purchased in bulk quantities (WITHOUT VAT) would vary in $70 - $95 range, depending on shipment location and quantity of chips ordered. Cost of other components (using numbers from our current design):

No. Use something smaller and lower power to run this, not some shitty Atom board. Use a Pi or that new $50 Via x86 board http://www.geek.com/articles/chips/via-launch-a-49-android-pc-20120522/

200 Mh/s bitstream - would produce 18 Gh/s - it would be compared to 71% of Mini-Rig and so product price could be $10'859.
250 Mh/s bitstream - would produce 22.5 Gh/s - it would be compared to 89,2% of Mini-Rig and so product price could be $13'643.
300 MH/s bitstream - would produce 27 Gh/s - it would be compared to 107% of Mini-Rig and so product price could be $16'365.
325 MH/s bitstream - would produce 29.2 Gh/s - it would be compared to 116% of Mini-Rig and so product price could be $17'722.

Remember, if you're overclocking these, provide some way to underclock+undervolt these to extend the life of these once diff goes too high in 3-4 years. This is a big feature that a lot of people are asking for.

Also, another thing, make sure the total power usage for the box fits so an integer number of these fits on a 120v 20a  (the most common circuit in DCs, you often get two of these per rack). ~1000 watts each would be fine if you intend on putting two on a circuit.

[bitfury]

Second, I was surprised by reaction about comparision with ButterFly Labs with 'Estimated price'. People, that's $90k is with 20% VAT, which is not paid in US for example and with having in mind that it would have single hot air outlet and that's all about it... but to get mini-rig we would pay +20% on customs VAT, so it would be not $15'295 but $18'354 + shipment costs,

Remember, US customers don't pay VAT, so you've probably quoted a EU price that is useless for US customers. If you can get BFL Minirig prices on Spartan 6 hardware, BFL is screwed.

Well, if we count on bulk purchase by miners directly via assembly house without any intermediate - then BFL is blow
away definitely. As BitFury-110 was built for less money than BFL asked, if consider that we pay VAT. And it is not possible to bring such quantity of BFLs through customs and declare them as personal items when nobody cares.

Third, 300 Mh/s is not limit of this bitstream, it can give even more rounds, if you count that almost all DSPs are not used in left part of design, and some free space in topmost part. So it could definitely give 8% at least better performance, it would then cost to us $0,57 per Mh/s, which is even less than BFL Mini-Rig.

Someone said in another thread that the sea of hashers design will have three types of cores, ones near DSPs, one near BRAM, ones near neither.

There's already three types, more or less common however. One difficult piece is 28-bit adder in the middle as there exists cuts for PLL/DCM modules of slices. It was tricky to implement, but done.

I've been asked by email and skype about smaller editions. And I would say that in my opinion best solution would be standard 4-U chassis, 0.5 meter long, with 14-15 boards with 6 chips on board installed (that is 84 - 90 chips), like with shalab.si original ideas, but a bit different layout to prevent overheating chips.

That gives people mostly what they want.

Yep, and my initiative is to make things more or less compatible. Why to invent multiple fancy form-factors with fancy fans ? When it would be nightmare to support it for miner ? Mining is not that profitable - to play with the equipment long days and nights. Unfortunately instead of getting here exact numbers / requests / discussion about future design, we have more common discussion. Every designer still wants to invent own kind of wheel. I understand this of course, even for myself - because every developer thinks that he understands details better, which is actually true... But when it comes to maintenance - it is better to have simpler things as MOST RELIABLE THING - IS THE THING THAT DOES NOT EXISTS.... So there's tradeoff between creativity and will to produce maintainable and reliable devices.

Withing this chassis single Intel Atom D525 motherboard is installed. Boards with 6 spartans can be even without microcontrollers and flash, everything could be programmed right via LPT-port. Bandwith required to communicate with every chip is quite low - about 300 bps. So with all chips it would be about 27 kbps. Bitstream loading over LPT port however will be slow. For smaller scale RS-485 is overkill. Why to bother about it and not implement using USB - simply because flashing chip or flashing controller
adds up cost of controller and also costs of programming and testing them, also when something should be updated, and you have to reprogram every controller - that rises service cost. I would like to say, that current design of BitFury rack, where controller only translates RS-485 to SPI bus with Spartan and back requires almost zero maintenance.

Use SATA plugs for the actual connector, but normal serial over it. SATA has 7 pins and is enterprise ready. The cost of serial is the complex plug, not the actual design. You could use a very tiny FPGA for the controller on each board to interface with the serial using GPIO pins or something.

SATA plugs! thanks, nice idea! 4-in cables are there.... This as simple as just installing proper size of SATA PCB layout on PCB! $1.67 x 16 = $26.7 for all of the connectivity. Just connecting each board to another one. Simply installing them and connecting. 1.6m long wire.

http://www.satacables.com/

About controller - I've mentioned only 500 bps per chip, so for 90 chips it would be max. 45k bps in and out...
Information about protocol is avail in initial post. So pinout can be following:

PIN1 - SCK
PIN2 - MOSI
PIN3 - MISO
PIN4 - GROUND
PIN5 - RESET
PIN6 - PROGDATA
PIN7 - PROGSCK

So it would be possible (a) upload bitstream (b) reset all chips (c) send/receive work.

OR - ANOTHER POSSIBILITY IS TO USE JTAG (but I don't know actually how well it work via such long chains like 90 chips, even when TCK, TMS will be transmitted using buffers.

PIN1 - TMS
PIN2 - TCK
PIN3 - TDI
PIN4 - GROUND
PIN5 - UNUSED
PIN6 - RST
PIN7 - TDO

Please not that key is not important, as improper insertion of cabling won't fry things. However it would be
perfectly possible to transmit jobs over JTAG. JTAG also seems to be nicer, because key can be programmed into board right using this slot.

So single SATA and single power molex. Are there cons in JTAG vs 2 SPIs ?

Cost of such chassis with power supply and Intel ATOM motherboard could vary in $400 - $600 range. Cost of Spartan6 chips when purchased in bulk quantities (WITHOUT VAT) would vary in $70 - $95 range, depending on shipment location and quantity of chips ordered. Cost of other components (using numbers from our current design):

No. Use something smaller and lower power to run this, not some shitty Atom board. Use a Pi or that new $50 Via x86 board http://www.geek.com/articles/chips/via-launch-a-49-android-pc-20120522/

Nice board, is there any other boards that support GPIO for example or SPI or JTAG output without any additional converters ? So I could launch software on linux ? That would be excellent for this project. Possibly with 2 cores, so one core could drive GPIO in realtime manner, while other core is communicating with world. But not necessary, as SPI tolerate lags!

200 Mh/s bitstream - would produce 18 Gh/s - it would be compared to 71% of Mini-Rig and so product price could be $10'859.
250 Mh/s bitstream - would produce 22.5 Gh/s - it would be compared to 89,2% of Mini-Rig and so product price could be $13'643.
300 MH/s bitstream - would produce 27 Gh/s - it would be compared to 107% of Mini-Rig and so product price could be $16'365.
325 MH/s bitstream - would produce 29.2 Gh/s - it would be compared to 116% of Mini-Rig and so product price could be $17'722.

Remember, if you're overclocking these, provide some way to underclock+undervolt these to extend the life of these once diff goes too high in 3-4 years. This is a big feature that a lot of people are asking for.

These are not overclocked, overclocking could give +20 - 25%, but will consume more power and possibly damage chips permanently.

Also, another thing, make sure the total power usage for the box fits so an integer number of these fits on a 120v 20a  (the most common circuit in DCs, you often get two of these per rack). ~1000 watts each would be fine if you intend on putting two on a circuit.

Well one box 4U would consume about 1.3 - 1.5 kW power that's 10,8 - 12,5 Amps @ 120 V seems to be bad.
With 11-12 boards it would be not more than 10 Amps @ 120 V however.

[DiabloD3]

Withing this chassis single Intel Atom D525 motherboard is installed. Boards with 6 spartans can be even without microcontrollers and flash, everything could be programmed right via LPT-port. Bandwith required to communicate with every chip is quite low - about 300 bps. So with all chips it would be about 27 kbps. Bitstream loading over LPT port however will be slow. For smaller scale RS-485 is overkill. Why to bother about it and not implement using USB - simply because flashing chip or flashing controller
adds up cost of controller and also costs of programming and testing them, also when something should be updated, and you have to reprogram every controller - that rises service cost. I would like to say, that current design of BitFury rack, where controller only translates RS-485 to SPI bus with Spartan and back requires almost zero maintenance.

Use SATA plugs for the actual connector, but normal serial over it. SATA has 7 pins and is enterprise ready. The cost of serial is the complex plug, not the actual design. You could use a very tiny FPGA for the controller on each board to interface with the serial using GPIO pins or something.

SATA plugs! thanks, nice idea! 4-in cables are there.... This as simple as just installing proper size of SATA PCB layout on PCB! $1.67 x 16 = $26.7 for all of the connectivity. Just connecting each board to another one. Simply installing them and connecting. 1.6m long wire.

http://www.satacables.com/

About controller - I've mentioned only 500 bps per chip, so for 90 chips it would be max. 45k bps in and out...
Information about protocol is avail in initial post. So pinout can be following:

PIN1 - SCK
PIN2 - MOSI
PIN3 - MISO
PIN4 - GROUND
PIN5 - RESET
PIN6 - PROGDATA
PIN7 - PROGSCK

So it would be possible (a) upload bitstream (b) reset all chips (c) send/receive work.

OR - ANOTHER POSSIBILITY IS TO USE JTAG (but I don't know actually how well it work via such long chains like 90 chips, even when TCK, TMS will be transmitted using buffers.

PIN1 - TMS
PIN2 - TCK
PIN3 - TDI
PIN4 - GROUND
PIN5 - UNUSED
PIN6 - RST
PIN7 - TDO

Please not that key is not important, as improper insertion of cabling won't fry things. However it would be
perfectly possible to transmit jobs over JTAG. JTAG also seems to be nicer, because key can be programmed into board right using this slot.

So single SATA and single power molex. Are there cons in JTAG vs 2 SPIs ?

JTAG might work but I don't think you're supposed to use JTAG that way. If you can make it work and not damage the hardware, go ahead, its your product, all of us will just treat it like a black box. BTW, BFL minirigs are using SATA connectors for their serial communication between boards, and Ive seen other designs repurpose connectors SATA internally or Infiniband connectors externally.

BTW re power Molex: Remember, Molex chain has a maximum of 5 amps/60 watts of 12v, and 5 amps/25 watts of 5v. That is not per plug, and a lot of chains have 3 or 4 plugs on it. PCI-E 6 is 2 amps/75 watts of 12v, PCI-E 8 is 4 amps/150w of 12v, P4 is 16 amps/192 watts of 12v, EPS12v is 32 amps/384w of 12v.

If you're putting 6 or 8 FPGAs on a board inside the rig, you're going to want to be using PCI-E plugs not Molex.

Cost of such chassis with power supply and Intel ATOM motherboard could vary in $400 - $600 range. Cost of Spartan6 chips when purchased in bulk quantities (WITHOUT VAT) would vary in $70 - $95 range, depending on shipment location and quantity of chips ordered. Cost of other components (using numbers from our current design):

No. Use something smaller and lower power to run this, not some shitty Atom board. Use a Pi or that new $50 Via x86 board http://www.geek.com/articles/chips/via-launch-a-49-android-pc-20120522/

Nice board, is there any other boards that support GPIO for example or SPI or JTAG output without any additional converters ? So I could launch software on linux ? That would be excellent for this project. Possibly with 2 cores, so one core could drive GPIO in realtime manner, while other core is communicating with world. But not necessary, as SPI tolerate lags!

You're probably only going to see GPIO on ARM boards meant for embedded usage, otherwise you're looking at a custom build of a standard USB->serial controller. Raspberry Pis are like $25 and have enough power to run cgminer, so if you can get them in bulk it seems they'll do what you want.

Problem is you might run out of GPIO pins because you'd need one to every controller (one per 4?).

Just whatever you do, don't put USB cables in the case. They easily get disconnected during transit and sometimes just unplug themselves due to case vibration or high pressure air cooling.

200 Mh/s bitstream - would produce 18 Gh/s - it would be compared to 71% of Mini-Rig and so product price could be $10'859.
250 Mh/s bitstream - would produce 22.5 Gh/s - it would be compared to 89,2% of Mini-Rig and so product price could be $13'643.
300 MH/s bitstream - would produce 27 Gh/s - it would be compared to 107% of Mini-Rig and so product price could be $16'365.
325 MH/s bitstream - would produce 29.2 Gh/s - it would be compared to 116% of Mini-Rig and so product price could be $17'722.

Remember, if you're overclocking these, provide some way to underclock+undervolt these to extend the life of these once diff goes too high in 3-4 years. This is a big feature that a lot of people are asking for.

These are not overclocked, overclocking could give +20 - 25%, but will consume more power and possibly damage chips permanently.

Wait, you're doing 325 mh per Spartan 6 SLX150? Holy crap man. Is that just back of the napkin math, or do you have a working bitstream that can do that?

Still, consider some method to undervolt them. At some point within the next 2-3 years for most users, the cost of electricity will exceed the value of the coins, but undervolting will catch them up for at least another year of usage.

Also, another thing, make sure the total power usage for the box fits so an integer number of these fits on a 120v 20a  (the most common circuit in DCs, you often get two of these per rack). ~1000 watts each would be fine if you intend on putting two on a circuit.

Well one box 4U would consume about 1.3 - 1.5 kW power that's 10,8 - 12,5 Amps @ 120 V seems to be bad.
With 11-12 boards it would be not more than 10 Amps @ 120 V however.

Well, on typical hardware, you shouldn't exceed 10 amps on 120v. That gives you a 1200 watt continuous PSU. Two of those will just barely fit on an enterprise 120v 20a line in a DC and hopefully not trip it, or one will fit on a household 120v 15a line (note: household lines _suck_, never drive them at >12a 24/7).

There ARE servers that require 208/240v service in DCs, usually some nearline data warehousing server that you shove 40 drives in, but a lot of DCs don't offer this or require a special order for it, and no house in America would typically have that service outside of an electric oven range or dryer socket.

[bitfury]

JTAG might work but I don't think you're supposed to use JTAG that way. If you can make it work and not damage the hardware, go ahead, its your product, all of us will just treat it like a black box. BTW, BFL minirigs are using SATA connectors for their serial communication between boards, and Ive seen other designs repurpose connectors SATA internally or Infiniband connectors externally.

Well. Yep. JTAG could devastate things.... So better to leave there only bitstream programming and communication
with chips.

BTW re power Molex: Remember, Molex chain has a maximum of 5 amps/60 watts of 12v, and 5 amps/25 watts of 5v. That is not per plug, and a lot of chains have 3 or 4 plugs on it. PCI-E 6 is 2 amps/75 watts of 12v, PCI-E 8 is 4 amps/150w of 12v, P4 is 16 amps/192 watts of 12v, EPS12v is 32 amps/384w of 12v.

If you're putting 6 or 8 FPGAs on a board inside the rig, you're going to want to be using PCI-E plugs not Molex.

6 Amps per pin limit for molex from datasheet, if I remember it correctly and about 100 times plugging before tin
degrades guaranteed. So this should not make big problems. 4 pins used, 2 pins for +, 2 pins for -, 12V powered.
The only problem is how to make efficient down conversion from 12 V to 1.2 V. I used TPS40090-based one,
and its COP and size is not very nice, however works without any problems.

You're probably only going to see GPIO on ARM boards meant for embedded usage, otherwise you're looking at a custom build of a standard USB->serial controller. Raspberry Pis are like $25 and have enough power to run cgminer, so if you can get them in bulk it seems they'll do what you want.

Problem is you might run out of GPIO pins because you'd need one to every controller (one per 4?).

Just whatever you do, don't put USB cables in the case. They easily get disconnected during transit and sometimes just unplug themselves due to case vibration or high pressure air cooling.

That's it! I felt that USB is good only when you connect one board device to your home PC :-)

200 Mh/s bitstream - would produce 18 Gh/s - it would be compared to 71% of Mini-Rig and so product price could be $10'859.
250 Mh/s bitstream - would produce 22.5 Gh/s - it would be compared to 89,2% of Mini-Rig and so product price could be $13'643.
300 MH/s bitstream - would produce 27 Gh/s - it would be compared to 107% of Mini-Rig and so product price could be $16'365.
325 MH/s bitstream - would produce 29.2 Gh/s - it would be compared to 116% of Mini-Rig and so product price could be $17'722.

Remember, if you're overclocking these, provide some way to underclock+undervolt these to extend the life of these once diff goes too high in 3-4 years. This is a big feature that a lot of people are asking for.

These are not overclocked, overclocking could give +20 - 25%, but will consume more power and possibly damage chips permanently.

Wait, you're doing 325 mh per Spartan 6 SLX150? Holy crap man. Is that just back of the napkin math, or do you have a working bitstream that can do that?

not working yet... bitstream requires approx. 300 hours of additional complex work to make that working. Just if you look carefully, there's unused approx. 1200 slices in the left part of the design + approx. 40 DSP48 + 80 BRAMs my estimations that I can fit there 6 rounds definitely. 8 sha256 cores would be quite difficult however. which would work at same clock rates like rest of design. And also there's about 550 slices left in top area - again - enough to fit 2 sha256 cores. Also hole in center - but I cannot fit there round exactly as below - lacking 2 slices and it is difficult to make workaround for that, but maybe I can put there special smaller round that would work at 2/3 of main clock. So it would be certainly 90/65 x clock speed compared to current 82/65 x clock. But if there will be request to implement it (as I stated in license - I would jump in @ 5000 chips approx at $5 per chip level) - I will start doing that work. 90/82 speedup is minimal estimation, actually it can become 92.6 / 82 - but I don't know if I really could get it... it depends whether BRAMs will be used or not actually if BRAMs will be used - then 8 additional cores seems feasible, or at least 7 cores... if however they will not work even at smaller clocks or will need additional set of registers after them, then I'll have to get back to concept without BRAMs and it will give 6 rounds. I've estimated this already - it is very quick... then more or less difficult is write correct code - I believe that it would take about 60 hours for two different DSP-BRAM and DSP-nobram models and about 40 hours for top rounds modifications... And then most interesting game starts - fitting it into a chip :-) Which would require lots of design modification, manual separation of control wires, looking how routing of these new cores would intersect with routings of big design.... This will take all of the rest time... Maybe if I'll be lucky it would be done in another 100 hours, but as from previous experience - it is not the way - as you only think that you are "near", while in reality you have not finished half road.... That's like estimating distance to oasis in desert :-)

Still, consider some method to undervolt them. At some point within the next 2-3 years for most users, the cost of electricity will exceed the value of the coins, but undervolting will catch them up for at least another year of usage.

I actually think that programmable voltage in range 1.15 V .. 1.45 V would be good. If someone would decide to risk with Spartans but get higher returns - then that's OK. Their devices would give +20%. If one would decide to go with less electricity consumption - then switch to 1.15 V and lower clocks. It just adds challenge with PSU - it should be still efficient in all ranges of power consumption.

Also, another thing, make sure the total power usage for the box fits so an integer number of these fits on a 120v 20a  (the most common circuit in DCs, you often get two of these per rack). ~1000 watts each would be fine if you intend on putting two on a circuit.

Well one box 4U would consume about 1.3 - 1.5 kW power that's 10,8 - 12,5 Amps @ 120 V seems to be bad.
With 11-12 boards it would be not more than 10 Amps @ 120 V however.

Well, on typical hardware, you shouldn't exceed 10 amps on 120v. That gives you a 1200 watt continuous PSU. Two of those will just barely fit on an enterprise 120v 20a line in a DC and hopefully not trip it, or one will fit on a household 120v 15a line (note: household lines _suck_, never drive them at >12a 24/7).

There ARE servers that require 208/240v service in DCs, usually some nearline data warehousing server that you shove 40 drives in, but a lot of DCs don't offer this or require a special order for it, and no house in America would typically have that service outside of an electric oven range or dryer socket.

Well - I am using now 3-phase 0.4 kV L-L @ 33 amps when chiller is running and @ 22 amps when chiller is not running. I think that if someone would like to install many of these things @ home - he should invest in cable connected directly to transformer to not disrupt power distribution for other houses. How much would it cost to me, if I would ask in America utility company for say 50 kW power to my household ?

I have installed PSUs with power factor correction (PFC). and it is important, as otherwise power losses would be significant.

About DCs I thought that way - 350 W per 1 U x 4 = 1.4 kW.... Otherwise how they would setup 4 x 1 U @ 350 W ?

[2112]

Also hole in center - but I cannot fit there round exactly as below - lacking 2 slices and it is difficult to make workaround for that,

I have one more question/suggestion. I understand that you're using LUT-RAM to store the the FIPS-180-x constants. I also guess that each hashing cell in your design has a dedicated private copy of that ROM. Have you tried pairing (or quad-ing) the neighboring cells to share a single ROM storage? If the cell clocks are in sync the neighbors could share those without paying to much cost in switch/route overhead.

I'm just asking, I'm not trying to question your skills or anything like that. I'm not involved in Bitcoin mining and I read this board for intellectual stimulation only.

[DiabloD3]

BTW re power Molex: Remember, Molex chain has a maximum of 5 amps/60 watts of 12v, and 5 amps/25 watts of 5v. That is not per plug, and a lot of chains have 3 or 4 plugs on it. PCI-E 6 is 2 amps/75 watts of 12v, PCI-E 8 is 4 amps/150w of 12v, P4 is 16 amps/192 watts of 12v, EPS12v is 32 amps/384w of 12v.

If you're putting 6 or 8 FPGAs on a board inside the rig, you're going to want to be using PCI-E plugs not Molex.

6 Amps per pin limit for molex from datasheet, if I remember it correctly and about 100 times plugging before tin
degrades guaranteed. So this should not make big problems. 4 pins used, 2 pins for +, 2 pins for -, 12V powered.
The only problem is how to make efficient down conversion from 12 V to 1.2 V. I used TPS40090-based one,
and its COP and size is not very nice, however works without any problems.

Molex's peripheral cable uses AMP 1-480424-0, AMP 60619-1, AMP 1-480426-0, and AMP 60620-1 for the socket housing, socket, pin housing, and pins respectively. This is rated for 13 amps. However, these are commonly wired with 18 AWG wiring. Putting 13 amps through this will melt the wiring, let alone 13 amps on every single plug in the chain. And although it has 4 pins, only one set are for 12v, the other set is for 5v.

Also, I hope you're using enterprise rated PSUs. You don't need redundant, but the largest non-redundant PSUs you can typically buy are in the 850w to 900w range. Not quite the 1100w you're aiming for. They do make 600-700w 2U non-redundant PSUs though, so you can always double up.

Still, consider some method to undervolt them. At some point within the next 2-3 years for most users, the cost of electricity will exceed the value of the coins, but undervolting will catch them up for at least another year of usage.

I actually think that programmable voltage in range 1.15 V .. 1.45 V would be good. If someone would decide to risk with Spartans but get higher returns - then that's OK. Their devices would give +20%. If one would decide to go with less electricity consumption - then switch to 1.15 V and lower clocks. It just adds challenge with PSU - it should be still efficient in all ranges of power consumption.

If you can get them still mining at 1.10v that would be far enough for most people, since most Spartan 6s are ran at 1.20 to 1.25v now.

Well - I am using now 3-phase 0.4 kV L-L @ 33 amps when chiller is running and @ 22 amps when chiller is not running. I think that if someone would like to install many of these things @ home - he should invest in cable connected directly to transformer to not disrupt power distribution for other houses. How much would it cost to me, if I would ask in America utility company for say 50 kW power to my household ?

I have installed PSUs with power factor correction (PFC). and it is important, as otherwise power losses would be significant.

About DCs I thought that way - 350 W per 1 U x 4 = 1.4 kW.... Otherwise how they would setup 4 x 1 U @ 350 W ?

In the US we use 240v single phase for electric ovens and driers and electric water heaters and larger air conditioners. Its provided as two hots and a neutral, with the two hots 120v lines that have their phases 180 degrees out of phase (so peak + on one is peak - on the other). Seems that Europe uses 3 phase 400v for the same reasons.

If we're limited to 4 120v 20a circuits per rack (which is what I've been told is a common limit, and you have to pay extra for the other two), and each 4U uses 9a (= 1100 watts) and we place an empty U between each (for airflow/cooling and easy maintenance reasons), thats 40u used in a 42u rack, or 8800w.

You shouldn't include your chiller, because a DC provides its own cooling. So, 22 amps of 400v is 8800w.

So, either your math is wrong or you misstated something, because we're coming out to the same power usage.

[bitfury]

I understand that developing such a product requires huge investments and risks, but this community is rabidly pro open source.  You might succeed at this for a time, but someone is going to replicate it and release it to the wild eventually.  I'm sure that you understand that, so the goal is to get as much profit from it as you can before the open source developers catch up.  May I offer an alternative to all this?  If you have an idea of how much revenue you expect to earn, let us know.  You can proceed with your licensing method however you see fit, but if the community can raise the funds, would you accept a collective donation in order to release the bitstream open source?

Well. Here's the plan - finish before 20th-30th June design of smaller device (4U or 2U sized - we have debates with shalab.si about that). To accomplish it, I actually need at 6th-15th June spec ready and circuit in approved state. So if till 1st June there will be no constructive discussion here about device, then I'll go finishing it alone. As far as I understood for many people that's just nice hobby... But I am kind of person who dislikes things half-done and half-working... So I've already invested in it half year full time + many nights into it and all non-spent cash as well, and I expect others who would like to deliver hashing power to have more serious approach as well. About open source - if someone would do this "as hobby" investing like 10 hours per week or so, he would need approx. 40 weeks to use all clues I gave already to make design. By that time I would glad to know that person, because his skill is great, I would be even more excited if someone would make it 350-370 Mh/s without rising VCCINT voltage.  But I consider case that someone replicate within next 33-50 days AND PUBLISH OPEN-SOURCE as low chance. Then at that time _WE_ will have everything ready for deployment. And yes - I love competition as it enforces you to do best things and be one step ahead.

Meanwhile still doing design negotiations who will make assembly of boards and where, who will do assembly of these devices and delivery and who will sell and service them worldwide. Then we'll start production - I aim at 2-3 month lead time, but during first period I plan to reinvest all of income into devices production to lower lead times and make interesting offer - basically you pay higher if you get device quick. So if someone would like to install 10 devices, he could start testing first one tomorrow.

During production period I expect to ramp of from 500 chips per month level to 1000-1200 chips per month delivered. within 6 month frame. That is about 4500 - 5000 chips delivered. But it could be better, if additional financing would come in play, that would allow to use other equities to finance bitcoin farms productions instead of just paying in cash. Actually I would like to push spartans hard and make target higher at 10k chips, but that would be difficult to get.

So expectations are for licensing - $100k - $200k for chips licensing and $25k-$50k for bitstream hashing power improvement within current offer.

Then, when spartan's will not be hot anymore - all of existing solution will go opensource, and main production line would switch to Artix, so price per Mh/s would drop another time and profit margin for manufacturing will be about difference between DIY spartan6 and artix7 solution.

So what is problem with opensource - I am pro-opensource, but I really doubt that with donations it is possible to get that amounts for such non-popular project like spartan6-miner. If it would be mass-product with say $1 - $10 donations it would work however. It is the problem I see - amounts are high compared to % of users interested in it within current bitcoin community size. Manufacturers of boards unlikely would jump into it. Miners possibly, but that depends % how much they would invest into disclosure and how many chips they would get then. And I believe many of miners would just hate this initiative, because they would like better that things won't change, they get still 50 BTC coinbase, etc. etc.

But if I am wrong - then I expect you to prove it. Let's develop procedure on how fundraising will be performed, how much efforts needed to advertise it, who will oversee funds (say what will happen if only 30% of target amount raised, as funds should be returned back). Right today I am not bound with any agreements about this bitstream usage, but I think it will be fair and necessary step of mine to use raised money and give buy offer to all current co-owners of current equipment, because this open-sourced bitstream could give sharp difficulty rise. Also this should be done relatively quickly - within 4-week time frame. Because when designing will be finished, I would definitely enter into bindings with different parties, and would not be able to decide alone whether this should go open source or not, and it could be difficult to get consensus. But anyway - this will be interesting case - if it works, then definitely for many chips there will be best bitstreams available, nice competition among designers would rise.

[MoonShadow]

I can't disprove you, and didn't intend to.  I was just asking what your price point was, should the community prove both willing and able to hit that price point.  I consider compatition a good thing, but it's still replication of work and if it can reasonablely be avoided, benefits all concerned.  Replication of work can't always be reasonablely be avoided.  I'm personally not interested in a fpga solution, even though I would consider it technically superior.  I'm personally interested in an asic solution, because I can use the waste heat to warm my home in winter, so I believe that basement mining operations will always be a factor, because professional mining operations not only have to deal with the overhead of dumping heat, they also have to outperform the basement miners to such a degree to overcome the economic advantages of co-generation.

[bitfury]

BTW re power Molex: Remember, Molex chain has a maximum of 5 amps/60 watts of 12v, and 5 amps/25 watts of 5v. That is not per plug, and a lot of chains have 3 or 4 plugs on it. PCI-E 6 is 2 amps/75 watts of 12v, PCI-E 8 is 4 amps/150w of 12v, P4 is 16 amps/192 watts of 12v, EPS12v is 32 amps/384w of 12v.

If you're putting 6 or 8 FPGAs on a board inside the rig, you're going to want to be using PCI-E plugs not Molex.

6 Amps per pin limit for molex from datasheet, if I remember it correctly and about 100 times plugging before tin
degrades guaranteed. So this should not make big problems. 4 pins used, 2 pins for +, 2 pins for -, 12V powered.
The only problem is how to make efficient down conversion from 12 V to 1.2 V. I used TPS40090-based one,
and its COP and size is not very nice, however works without any problems.

Molex's peripheral cable uses AMP 1-480424-0, AMP 60619-1, AMP 1-480426-0, and AMP 60620-1 for the socket housing, socket, pin housing, and pins respectively. This is rated for 13 amps. However, these are commonly wired with 18 AWG wiring. Putting 13 amps through this will melt the wiring, let alone 13 amps on every single plug in the chain. And although it has 4 pins, only one set are for 12v, the other set is for 5v.

1.5 sq. mm wires. ( AWG 15 ).

Also, I hope you're using enterprise rated PSUs. You don't need redundant, but the largest non-redundant PSUs you can typically buy are in the 850w to 900w range. Not quite the 1100w you're aiming for. They do make 600-700w 2U non-redundant PSUs though, so you can always double up.

For current big bf-110 we used RSP-3000-12 - 200 Amps @ 12 V . Boards however consume only ~160 Amps. So there's reserve for overclocking or for grid under-voltage condition. I expect to put something nice and reliable installed into box as well. They're not cheap, but it is crazy to put cheap PSU to power such expensive equipment.

Well - I am using now 3-phase 0.4 kV L-L @ 33 amps when chiller is running and @ 22 amps when chiller is not running. I think that if someone would like to install many of these things @ home - he should invest in cable connected directly to transformer to not disrupt power distribution for other houses. How much would it cost to me, if I would ask in America utility company for say 50 kW power to my household ?

I have installed PSUs with power factor correction (PFC). and it is important, as otherwise power losses would be significant.

About DCs I thought that way - 350 W per 1 U x 4 = 1.4 kW.... Otherwise how they would setup 4 x 1 U @ 350 W ?

In the US we use 240v single phase for electric ovens and driers and electric water heaters and larger air conditioners. Its provided as two hots and a neutral, with the two hots 120v lines that have their phases 180 degrees out of phase (so peak + on one is peak - on the other). Seems that Europe uses 3 phase 400v for the same reasons.

If we're limited to 4 120v 20a circuits per rack (which is what I've been told is a common limit, and you have to pay extra for the other two), and each 4U uses 9a (= 1100 watts) and we place an empty U between each (for airflow/cooling and easy maintenance reasons), thats 40u used in a 42u rack, or 8800w.

You shouldn't include your chiller, because a DC provides its own cooling. So, 22 amps of 400v is 8800w.

So, either your math is wrong or you misstated something, because we're coming out to the same power usage.

I have not misstated. 0.4 kV L-L - that's about 230 Volts phase - to - neutral fed from transformer. When it gets to me, due to losses in cables - it gets to standard 215 - 220 Volts phase - to - neutral voltage. And then
22 x 218 x 3 = 14.4 kW approx.

I am not using 4U design right now - 4U design is during planning... look @ photos on www.bitfury.org - that are the bf-110 used now for mining!

[bitfury]

I can't disprove you, and didn't intend to.  I was just asking what your price point was, should the community prove both willing and able to hit that price point.  I consider compatition a good thing, but it's still replication of work and if it can reasonablely be avoided, benefits all concerned.  Replication of work can't always be reasonablely be avoided.  I'm personally not interested in a fpga solution, even though I would consider it technically superior.  I'm personally interested in an asic solution, because I can use the waste heat to warm my home in winter, so I believe that basement mining operations will always be a factor, because professional mining operations not only have to deal with the overhead of dumping heat, they also have to outperform the basement miners to such a degree to overcome the economic advantages of co-generation.

Well. ASIC is much more expensive. If you would get simple ASIC that would just work, but you get it with 100% success probability at affordable level - that would be rather slow 90nm or 130 nm chip, it would be like what 28-nm chips could give with their best performance and special price from fpga manufacturers. Say if altera or xil would decide to sell their chips less but more q-tys for mining - then ASIC project would end up with failure. And it would took long time to develop it as well.

If you would like to make really tricky and quick ASIC, then there could be epic failures and risks are very high. I would like to design such ASIC, but only if backed with investments about $1 - $1.2 mio and having possibility to order several times multiwafer which would be thrown away as "bad" result.... But when I am talking about that conditions people say that I am completely crazy. But that is true - without doing actual experiments, you would not get TOP results, as going proven way - you will get proven results of course... Or even simpler things like e-asic or altera hardcopy.

However really quick ASIC will outperform, but lot of money could be just thrown away with it, so there will be gradually improving and competing ASIC solutions. And this will be risky hop... Look how difficult hop from GPU to FPGA is, however FPGA are superior... Still most miners use GPU, and I still see much space to promote GPU mining, when these cards are installed @ home. And with ASIC it will be even more difficult. So for ASICs to succeed difficulty AND exchange rate should rise to level, so whole mining market would increase in side - currently it is like $1 mio per month mined, then if exchange rate will not rise - it will be about $0.5 mio per month... For low-end ASIC it would be tough play, also when you consider that optimized ASIC or discounted 28-nm chips would blow it out from game even without returning NRE costs.

Also ASIC must be available to interested parties, because myself is interested in condition, that there will be no single party in bitcoin who owns network... And if that happens - I'll really work hard to prevent this condition. Also I see that FPGAs can be competetive to ASIC while GPU would have much bigger gap in performance, so ASIC over GPU takeover is much more likely than ASIC vs FPGA.

+++ for mining co-generation - that's really nice. I can now implement pool and heat in in summer :-) without mining I would not waste heat for that :-) Now I have more than enough useful heat. However many still thinks that it is crazy way.

[DiabloD3]

Hrm, I was wrong. Athena Power sells a 1100w non-redundant: http://www.newegg.com/Product/Product.aspx?Item=N82E16817338021

[bitfury]

Also hole in center - but I cannot fit there round exactly as below - lacking 2 slices and it is difficult to make workaround for that,

I have one more question/suggestion. I understand that you're using LUT-RAM to store the the FIPS-180-x constants. I also guess that each hashing cell in your design has a dedicated private copy of that ROM. Have you tried pairing (or quad-ing) the neighboring cells to share a single ROM storage? If the cell clocks are in sync the neighbors could share those without paying to much cost in switch/route overhead.

I'm just asking, I'm not trying to question your skills or anything like that. I'm not involved in Bitcoin mining and I read this board for intellectual stimulation only.

Why not to wait before full disclosure ? It would be interesting anyway... I think that you do not even imagine scale of these optimizations, as you are asking questions that I started to ask when started this journey :-) I would even describe storyline how different decisions were make, and if would have enough time for that - arrange in a way that you could try to guess what was done next, not disclosing it before you read next page. Just be around to remind me to do this.

[DiabloD3]

1.5 sq. mm wires. ( AWG 15 ).

For current big bf-110 we used RSP-3000-12 - 200 Amps @ 12 V . Boards however consume only ~160 Amps. So there's reserve for overclocking or for grid under-voltage condition. I expect to put something nice and reliable installed into box as well. They're not cheap, but it is crazy to put cheap PSU to power such expensive equipment.

Wait, you're already using redundant modules for the PSU? Thats totally okay then. Theres 2 or 3 1400w or more models out there, although they all use a different incompatible interface.

I have not misstated. 0.4 kV L-L - that's about 230 Volts phase - to - neutral fed from transformer. When it gets to me, due to losses in cables - it gets to standard 215 - 220 Volts phase - to - neutral voltage. And then
22 x 218 x 3 = 14.4 kW approx.

I am not using 4U design right now - 4U design is during planning... look @ photos on www.bitfury.org - that are the bf-110 used now for mining!

So you're going to sell 2U boxes initially? 500w in a 2U could be easily done.

[rjk]

I dunno about redundant, but they are Mean Well PSUs, which are some of the best in the industry. You can see the MW logo in the pics.

[DiabloD3]

I dunno about redundant, but they are Mean Well PSUs, which are some of the best in the industry. You can see the MW logo in the pics.

Those kind of modules (lots of manufs make them) are typically used in either redundant PSU housings hold 2 to 4 modules, or people use them with custom wiring harnesses if they have specific wiring requirements.

[rjk]

I dunno about redundant, but they are Mean Well PSUs, which are some of the best in the industry. You can see the MW logo in the pics.

Those kind of modules (lots of manufs make them) are typically used in either redundant PSU housings hold 2 to 4 modules, or people use them with custom wiring harnesses if they have specific wiring requirements.

I was totally going to use them in my rig, but they are damn expensive. http://www.meanwell.com/search/rsp-3000/rsp-3000-spec.pdf
Nice lugs for attaching bus bars to.

[DiabloD3]

I dunno about redundant, but they are Mean Well PSUs, which are some of the best in the industry. You can see the MW logo in the pics.

Those kind of modules (lots of manufs make them) are typically used in either redundant PSU housings hold 2 to 4 modules, or people use them with custom wiring harnesses if they have specific wiring requirements.

I was totally going to use them in my rig, but they are damn expensive. http://www.meanwell.com/search/rsp-3000/rsp-3000-spec.pdf
Nice lugs for attaching bus bars to.

Yeah, its a 2400w PSU module. Something like that really should cost $550-600.

[RoadStress]

https://bitcointalk.org/index.php?topic=76351.msg923533#msg923533

Any bets on performance?

[DiabloD3]

https://bitcointalk.org/index.php?topic=76351.msg923533#msg923533

Any bets on performance?

[bitfury]

The proposed board. Mechanical drawing. Board size is 190 (width) x 160 height. Total height is about 32 mm (counting board width, chip width, and heatsink height of 28 mm).

1. Thanks to guys who showed me http://www.raspberrypi.org/ controller
    That controller has nice GPIO outputs, that could drive FPGA legs. And that controller can drive multiple boards at once.
    So no microcontroller per-board required.

    Just some GPIO pins would be used for SATA.

    It costs only $25 per item.... Which is nice, so when connected to multiple boards, cost of controller would be little.

2. Thanks to DiabloD3 for SATA cables idea, so when decided over JTAG vs non-JTAG I decided to put them all!
    SATA1 is input SATA2 is output to next board of following signals:

PIN1 - SCK
PIN2 - MOSI
PIN3 - MISO
PIN4 - GROUND
PIN5 - RESET
PIN6 - PROGDATA
PIN7 - PROGSCK

    SPI interface to talk with bitstream and another SPI to program bitstream plus generic RESET.

    And another SATA interface, again SATA3 input, SATA4 output:

PIN1 - TMS
PIN2 - TCK
PIN3 - TDI
PIN4 - GROUND
PIN5 - UNUSED
PIN6 - RST
PIN7 - TDO

    On board design will contain buffers and resistors connected in a way, that connecting these cables in wrong direction won't cause problems.

3. PSU is TPS40090 based 100 Amp power supply, but used at lower amps to get performance better, in overclocked
    condition performance at about 90% expected, in normal conditions about 92-94%. Amps delivered are quire high.
   Also PSU input would be in 10 - 15 V range, so it can be connected to automotive power supplies / lead acid backup /
   wind energy storage batteries directly if someone would like to do it.

4. Connectivity of 12 V is via MOLEX socket (4-pin) and also can be connected via bolt connection to copper bus / bar,
    which could be used to mount boards as well. (This 12 V connection is specifically for Greg). However in our chassis
    we may put separate cables with molex connectors and add fuses on them. If will be requested - we may add fuse on
    board by adding special PCB layout, that will burn if there will go about 20-30 amps current suddenly.

5. Board has connectors on TOP to not distract airflow, and also they would sit tightly there, when pressed by chassis top.

6. There will be 3 jumpers around PSU that would select voltage level - 1.15 V to 1.5 V in 0.05 V steps. We will set it up @ 1.3 V which is safe by datasheet. That way performance could be improved up to 280-290 Mhz clock depending on chip when voltage rised to 1.5 V. TSP40090 actually useful for that.

7. Every board would have 6 jumpers to select board identifier, that can be later used by software.

8. Communicating with board will be done using 32-bit SPI. Basically you first supply 32-bit command and then either read or write 32-bit word of data. 32-bit command would contain address (6-bit board number 3-bit chip number), command type (read or write 1-bit) and register number (5-bit) totalling 15 bits AND some framesync/error check information for quick bus resetting etc.

This communication can be made practically with every GPIO software, and even with LPT-port via resistors, at any clock rates, which will not produce significant error rates. So board will be as cheap as possible then. WITHOUT any RS-485 or USB interfaces. And without any FLASH chips on it.



In next message I'll write about temperature simulations... For maximum overclocked mode - 18W per chip on 1.5 V / 18W on power supply dissipation and 1 m/s airflow...

Then I'll go drawing draft for 4-U device.

[nedbert9]

In light of BFL's SC model announcement can we discuss realistically what the chances are of protocol change that would obsolete such ASIC solutions?  If such an ASIC solution comes to pass what percentage loss would a miner have to bear, compared to ASIC, to remain with FPGA-based solutions to mitigate protocol change risk.

[bitfury]

Ok. I put this board into single air channel virtual with flow setup - 293.2 K input (that's 20 degrees Celsius).

And here's thermal images I got:

Chips (Temperature of first chip in airflow - 309 K / 36 Celsius second chip - 315 K / 42 Celsius
and power supply - 326 K / 53 Celsius - however this is rude to model PSU that way... there will be much hotter parts like MOSFETs and inductors and much colder like drivers):



Board (about 308K / 35 C beneath second chip and 299K / 26 C  beneath first one ):



Air flow (cuts middle of heatsinks - air at 1 m/s overheats at about 20 C - average outlet temp.):



Now let's do some calculations to know temps inside of chip based on Spartan6 datasheets (spartan6-pack-ug385.pdf)
FG(G)484 23x23 says - 3.7 C/W junction-case and 6.3 C/W junction-board.

Let's assume Tcase = 42 Celsius Tboard = 35 Celsius and calculate Tj. Having P = 18 W.

Solve[{Tj - Pjc*3.7 == 42, Tj - Pjb*6.7 == 35, Pjb+Pjc == 18}, { Tj, Pjb, Pjc}]

Tj = 82.5 ; Pbj = 7 ; Pjc = 11

However that are not exactly true, as in model I've modeled device as piece of copper. So real temperature will be in range of about Tj = 86 degrees.

For INDUSTRIAL chip it will fit perfectly WITH 20 degrees C air input.

With 34 degrees inlet air Tj will be at absolute maximum.

So this rises questions whether we should put two such boards in serial way or not.... As IF COLD AIR SUPPLIED, we definitely can, if not - then we're screwed.

Also if we not overclock it - temperature raise would be proportionally less - to about 65 degrees C from 20 degrees in a chip. So if chips are not overclocked, then 2 such boards can be put in serial way leading to same inlet air requirements.

Also this shows that it is better to use industrial chips, although they are $4 more expensive.

Also we can increase air speed twice, and we would get only about 10 degrees temperature rise and install two boards.

So... it seems to better go to sleep and then think about this concept....

Within single 0.2 m space of 4U device one can fit 12 of such boards consuming max 1.95 kW of power @ max. clock. / max. voltage. In normal condition 1.3 kW only. But only 72 chips there.

In denser design it could be 4 kW / 2.6 kW per 4-U chassis, which seems to be OVERKILL. How to cool them down then....

[rjk]

Could you have the chips offset from one another so that the hot air from the first ones isn't warming the second ones?

Even if they weren't perfectly offset, you could use strips of plastic to duct the air around them.

[bitfury]

Could you have the chips offset from one another so that the hot air from the first ones isn't warming the second ones?

Even if they weren't perfectly offset, you could use strips of plastic to duct the air around them.

Yes, this can be done... But let me save how air temp. looks at output....



You mean shifting second row of boards a bit, so inlet air would be on these darker areas, right ?

11 boards in one row and 12 boards in second one ? This would help a bit I believe, but mostly volume would be hotter.

But still, isn't 2.6 kW for 4U too much ?

About plastic duct - that's bad idea, as this would add assembling difficulties = costs.

[bitfury]

In light of BFL's SC model announcement can we discuss realistically what the chances are of protocol change that would obsolete such ASIC solutions?  If such an ASIC solution comes to pass what percentage loss would a miner have to bear, compared to ASIC, to remain with FPGA-based solutions to mitigate protocol change risk.

Easily. BitCoin is small network actually. And it is small, because it was started as e-currency. "Normal" people wouldn't even understand what these hashes are about. This makes it actually interesting, because here on forum, people are smarter than average. But payment is actually application... Value is in things that you can buy with that e-currency. So if you start with thing, and then get nice e-currency for that thing - then all of your audience will use said e-currency. So when bitcoin matures and ASIC control will be in one hands - then people who would start integrating it with bittorrent world would make choice of another currency, with protected protocol depending on their need. And yes - their audience impact CAN BE MUCH BIGGER :-)

Current our position with BitCoin - we're trying it, digging into deeps What actually scares a bit - that exchanges, and people who do business with BitCoins should BUY HASHING POWER FOREMOST.... But it seems not to happen....

[DiabloD3]

2. Thanks to DiabloD3 for SATA cables idea, so when decided over JTAG vs non-JTAG I decided to put them all!
    SATA1 is input SATA2 is output to next board of following signals:

    SPI interface to talk with bitstream and another SPI to program bitstream plus generic RESET.

    And another SATA interface, again SATA3 input, SATA4 output:

If you're going to use multiple different connections all using SATA connectors, remember that you can buy SATA cables in different colors so people don't accidentally plug the wrong ones in when doing maintenance.

[bitfury]

2. Thanks to DiabloD3 for SATA cables idea, so when decided over JTAG vs non-JTAG I decided to put them all!
    SATA1 is input SATA2 is output to next board of following signals:

    SPI interface to talk with bitstream and another SPI to program bitstream plus generic RESET.

    And another SATA interface, again SATA3 input, SATA4 output:

If you're going to use multiple different connections all using SATA connectors, remember that you can buy SATA cables in different colors so people don't accidentally plug the wrong ones in when doing maintenance.

I'll first do precaution that nothing bad happen if they do...
And second point - there will be only one cable from one board to another board...
So one cable per board, short one like 4-inch SATA. When I saw them I loved them.

But - JTAG chain is nice to have for other applications for example.

Anyways thanks for JTAG idea!

[rjk]

Forgive me for hacking up your cool picture, but this is what I meant when I said offset:


Is it worth it?

[bitfury]

Forgive me for hacking up your cool picture, but this is what I meant when I said offset:

// pic eaten by wolves ///

Is it worth it?

We did it in hardware already. Yes it worth it, but not much, and in this design even less, than when we could offset whole daughter boards so they get colder air on their bottom side. It would be non-uniform then, and central chip would be hottest. You may see what we did in following pic... But I really want to remove that chip plug-and-pray technology:

[rjk]

Beautiful. I like the pluggable design.

[bitfury]

Beautiful. I like the pluggable design.

Well, me too. BUT there's big cons... Soldering these plugs is done by hand... and error-prone when a lot of them should be soldered... within some of them were balls of solder inside - so it was not possible to put there daughter card without pressing it too much... Then when daughter card is so small, you can only glue heatsink there and it is very RESPONSIBLE work... We did it in two person for about 3 days, and it can't be given to plant, because they can screw things... As if you put too much glue, Spartans could fry.... And we're not cooks :-)))) Fried spartan is expensive and not tasty dish )))
Then if you get it bigger it becomes 60x60 mm daughter card to put desirable-sized heatsink on it... And it wastes too much space :-( Those small heatsinks that you saw on other boards are not suitable well for such power output from Spartan.

[AzN1337c0d3r]

However, these are commonly wired with 18 AWG wiring. Putting 13 amps through this will melt the wiring, let alone 13 amps on every single plug in the chain.

18 AWG wire is allowed to carry 14 amps according to NEC:

http://en.wikipedia.org/wiki/American_wire_gauge

In fact, I'm using them in an application that carries 13A (4x Delta AFB1212GHE-CF00, which draw 3.24A each according to the label).

The wires themselves barely get warm, although if I try to pull 13A off one connector, it gets pretty hot. Probably exceeding some kind of molex pin specification there. I moved each fan onto its own connector after that.

[MrTeal]

However, these are commonly wired with 18 AWG wiring. Putting 13 amps through this will melt the wiring, let alone 13 amps on every single plug in the chain.

18 AWG wire is allowed to carry 14 amps according to NEC:

http://en.wikipedia.org/wiki/American_wire_gauge

In fact, I'm using them in an application that carries 13A (4x Delta AFB1212GHE-CF00, which draw 3.24A each according to the label).

The wires themselves barely get warm, although if I try to pull 13A off one connector, it gets pretty hot. Probably exceeding some kind of molex pin specification there. I moved each fan onto its own connector after that.

You can do that, but you really should upgrade the wire for 13A. I doubt the fans are actually pulling that much, but even outside the heating the extra inefficiency is unneeded. At 13A you'll be dissipating a little over 1W/ft just in the wire.

[bitfury]

However, these are commonly wired with 18 AWG wiring. Putting 13 amps through this will melt the wiring, let alone 13 amps on every single plug in the chain.

18 AWG wire is allowed to carry 14 amps according to NEC:

http://en.wikipedia.org/wiki/American_wire_gauge

In fact, I'm using them in an application that carries 13A (4x Delta AFB1212GHE-CF00, which draw 3.24A each according to the label).

The wires themselves barely get warm, although if I try to pull 13A off one connector, it gets pretty hot. Probably exceeding some kind of molex pin specification there. I moved each fan onto its own connector after that.

Well if you put 2 of them - then no problem... I use 1.5 sq. mm (about 14 - 15 AWG), and wired are at about 50 C compared to 25-30 C surrounding temperature on 8 Amps. If you consider that losses are square of current - i.e.
when you get say 14 amps - you'll get 14*14/8/8 = 3 times more power dissipation in wires.... You'll get into troubles.
Because device is labelled something does not means they consume all of that current!!! Use test equipment to measure.
Say - there's 3 x 3 kW power supplies on BF-110 - but that does not mean that it consumes 9 kW (rack alone). Or you can see 2.5 kW on wash machine, but it would actually consume on average more like 0.5 kW, because heater element is on for short periods of time. What is labelled usually peak power consumption.

[AzN1337c0d3r]

Use test equipment to measure.

Fans plugged in = 593W
Fans pulled out = 453W

Wattage of 4x Delta = 140W at the wall. Power supply is 89% efficient at that load, so 125W DC. 

About 10.5A on that 12V connector. Like I said the Wire barely gets warm and I'm sure I could put another Delta on it if I'd like without it melting.

[TheSeven]

2. Thanks to DiabloD3 for SATA cables idea, so when decided over JTAG vs non-JTAG I decided to put them all!
    SATA1 is input SATA2 is output to next board of following signals:

PIN1 - SCK
PIN2 - MOSI
PIN3 - MISO
PIN4 - GROUND
PIN5 - RESET
PIN6 - PROGDATA
PIN7 - PROGSCK

Did you look at the internal layout of a SATA cable? IIUC most of them look like this:

...which doesn't seem to be compatible with your pinout.
You'll have to deal with pins 1, 4 and 7 being connected internally and being laid out fundamentally different (as a shield/foil) than the data pins.

[bitfury]

2. Thanks to DiabloD3 for SATA cables idea, so when decided over JTAG vs non-JTAG I decided to put them all!
    SATA1 is input SATA2 is output to next board of following signals:

PIN1 - SCK
PIN2 - MOSI
PIN3 - MISO
PIN4 - GROUND
PIN5 - RESET
PIN6 - PROGDATA
PIN7 - PROGSCK

Did you look at the internal layout of a SATA cable? IIUC most of them look like this:
 // CUT IMAGE //
...which doesn't seem to be compatible with your pinout.
You'll have to deal with pins 1, 4 and 7 being connected internally and being laid out fundamentally different (as a shield/foil) than the data pins.

No I didn't ... Big thanks... Saved my time... As otherwise I would deal with it during PCB schematic / layout stage... Now have to guess how to live with that... Interesting to know, whether all grounds connected or no...

With JTAG it seems not be problem to reorder connections, but with 2 SPIs it seems to be.