BitFury Design, Licensing, Mass production

[Inspector 2211]

EDIT: https://bitcointalk.org/index.php?topic=49971.msg918095#msg918095 - claims that bitstream format is available under NDA...

Since this refers to my message that wasn't clear: Xilinx bitstream format WAS at one time available to Xilinx EDA software partners, possibly under NDA, possibly under some different sort of legal and financial protection.

I no longer consult for that EDA vendor and I have no relevant knowledge regarding current products. I only remain friends with people who continue to maintain and upgrade the EDA software applications that I co-authored years ago.

It seems plausible that companies like Synopsys get access to the bitstream format, possible for a fee, but less plausible that any Tom, Dick and Harry could get access to it.

[1714782174]

1714782174

[1714782174]

1714782174

[1714782174]

1714782174

[DiabloD3]

Second, I was surprised by reaction about comparision with ButterFly Labs with 'Estimated price'. People, that's $90k is with 20% VAT, which is not paid in US for example and with having in mind that it would have single hot air outlet and that's all about it... but to get mini-rig we would pay +20% on customs VAT, so it would be not $15'295 but $18'354 + shipment costs,

Remember, US customers don't pay VAT, so you've probably quoted a EU price that is useless for US customers. If you can get BFL Minirig prices on Spartan 6 hardware, BFL is screwed.

Third, 300 Mh/s is not limit of this bitstream, it can give even more rounds, if you count that almost all DSPs are not used in left part of design, and some free space in topmost part. So it could definitely give 8% at least better performance, it would then cost to us $0,57 per Mh/s, which is even less than BFL Mini-Rig.

Someone said in another thread that the sea of hashers design will have three types of cores, ones near DSPs, one near BRAM, ones near neither.

I've been asked by email and skype about smaller editions. And I would say that in my opinion best solution would be standard 4-U chassis, 0.5 meter long, with 14-15 boards with 6 chips on board installed (that is 84 - 90 chips), like with shalab.si original ideas, but a bit different layout to prevent overheating chips.

That gives people mostly what they want.

Withing this chassis single Intel Atom D525 motherboard is installed. Boards with 6 spartans can be even without microcontrollers and flash, everything could be programmed right via LPT-port. Bandwith required to communicate with every chip is quite low - about 300 bps. So with all chips it would be about 27 kbps. Bitstream loading over LPT port however will be slow. For smaller scale RS-485 is overkill. Why to bother about it and not implement using USB - simply because flashing chip or flashing controller
adds up cost of controller and also costs of programming and testing them, also when something should be updated, and you have to reprogram every controller - that rises service cost. I would like to say, that current design of BitFury rack, where controller only translates RS-485 to SPI bus with Spartan and back requires almost zero maintenance.

Use SATA plugs for the actual connector, but normal serial over it. SATA has 7 pins and is enterprise ready. The cost of serial is the complex plug, not the actual design. You could use a very tiny FPGA for the controller on each board to interface with the serial using GPIO pins or something.

Cost of such chassis with power supply and Intel ATOM motherboard could vary in $400 - $600 range. Cost of Spartan6 chips when purchased in bulk quantities (WITHOUT VAT) would vary in $70 - $95 range, depending on shipment location and quantity of chips ordered. Cost of other components (using numbers from our current design):

No. Use something smaller and lower power to run this, not some shitty Atom board. Use a Pi or that new $50 Via x86 board http://www.geek.com/articles/chips/via-launch-a-49-android-pc-20120522/

200 Mh/s bitstream - would produce 18 Gh/s - it would be compared to 71% of Mini-Rig and so product price could be $10'859.
250 Mh/s bitstream - would produce 22.5 Gh/s - it would be compared to 89,2% of Mini-Rig and so product price could be $13'643.
300 MH/s bitstream - would produce 27 Gh/s - it would be compared to 107% of Mini-Rig and so product price could be $16'365.
325 MH/s bitstream - would produce 29.2 Gh/s - it would be compared to 116% of Mini-Rig and so product price could be $17'722.

Remember, if you're overclocking these, provide some way to underclock+undervolt these to extend the life of these once diff goes too high in 3-4 years. This is a big feature that a lot of people are asking for.

Also, another thing, make sure the total power usage for the box fits so an integer number of these fits on a 120v 20a  (the most common circuit in DCs, you often get two of these per rack). ~1000 watts each would be fine if you intend on putting two on a circuit.

[bitfury]

Second, I was surprised by reaction about comparision with ButterFly Labs with 'Estimated price'. People, that's $90k is with 20% VAT, which is not paid in US for example and with having in mind that it would have single hot air outlet and that's all about it... but to get mini-rig we would pay +20% on customs VAT, so it would be not $15'295 but $18'354 + shipment costs,

Remember, US customers don't pay VAT, so you've probably quoted a EU price that is useless for US customers. If you can get BFL Minirig prices on Spartan 6 hardware, BFL is screwed.

Well, if we count on bulk purchase by miners directly via assembly house without any intermediate - then BFL is blow
away definitely. As BitFury-110 was built for less money than BFL asked, if consider that we pay VAT. And it is not possible to bring such quantity of BFLs through customs and declare them as personal items when nobody cares.

Third, 300 Mh/s is not limit of this bitstream, it can give even more rounds, if you count that almost all DSPs are not used in left part of design, and some free space in topmost part. So it could definitely give 8% at least better performance, it would then cost to us $0,57 per Mh/s, which is even less than BFL Mini-Rig.

Someone said in another thread that the sea of hashers design will have three types of cores, ones near DSPs, one near BRAM, ones near neither.

There's already three types, more or less common however. One difficult piece is 28-bit adder in the middle as there exists cuts for PLL/DCM modules of slices. It was tricky to implement, but done.

I've been asked by email and skype about smaller editions. And I would say that in my opinion best solution would be standard 4-U chassis, 0.5 meter long, with 14-15 boards with 6 chips on board installed (that is 84 - 90 chips), like with shalab.si original ideas, but a bit different layout to prevent overheating chips.

That gives people mostly what they want.

Yep, and my initiative is to make things more or less compatible. Why to invent multiple fancy form-factors with fancy fans ? When it would be nightmare to support it for miner ? Mining is not that profitable - to play with the equipment long days and nights. Unfortunately instead of getting here exact numbers / requests / discussion about future design, we have more common discussion. Every designer still wants to invent own kind of wheel. I understand this of course, even for myself - because every developer thinks that he understands details better, which is actually true... But when it comes to maintenance - it is better to have simpler things as MOST RELIABLE THING - IS THE THING THAT DOES NOT EXISTS.... So there's tradeoff between creativity and will to produce maintainable and reliable devices.

Withing this chassis single Intel Atom D525 motherboard is installed. Boards with 6 spartans can be even without microcontrollers and flash, everything could be programmed right via LPT-port. Bandwith required to communicate with every chip is quite low - about 300 bps. So with all chips it would be about 27 kbps. Bitstream loading over LPT port however will be slow. For smaller scale RS-485 is overkill. Why to bother about it and not implement using USB - simply because flashing chip or flashing controller
adds up cost of controller and also costs of programming and testing them, also when something should be updated, and you have to reprogram every controller - that rises service cost. I would like to say, that current design of BitFury rack, where controller only translates RS-485 to SPI bus with Spartan and back requires almost zero maintenance.

Use SATA plugs for the actual connector, but normal serial over it. SATA has 7 pins and is enterprise ready. The cost of serial is the complex plug, not the actual design. You could use a very tiny FPGA for the controller on each board to interface with the serial using GPIO pins or something.

SATA plugs! thanks, nice idea! 4-in cables are there.... This as simple as just installing proper size of SATA PCB layout on PCB! $1.67 x 16 = $26.7 for all of the connectivity. Just connecting each board to another one. Simply installing them and connecting. 1.6m long wire.

http://www.satacables.com/

About controller - I've mentioned only 500 bps per chip, so for 90 chips it would be max. 45k bps in and out...
Information about protocol is avail in initial post. So pinout can be following:

PIN1 - SCK
PIN2 - MOSI
PIN3 - MISO
PIN4 - GROUND
PIN5 - RESET
PIN6 - PROGDATA
PIN7 - PROGSCK

So it would be possible (a) upload bitstream (b) reset all chips (c) send/receive work.

OR - ANOTHER POSSIBILITY IS TO USE JTAG (but I don't know actually how well it work via such long chains like 90 chips, even when TCK, TMS will be transmitted using buffers.

PIN1 - TMS
PIN2 - TCK
PIN3 - TDI
PIN4 - GROUND
PIN5 - UNUSED
PIN6 - RST
PIN7 - TDO

Please not that key is not important, as improper insertion of cabling won't fry things. However it would be
perfectly possible to transmit jobs over JTAG. JTAG also seems to be nicer, because key can be programmed into board right using this slot.

So single SATA and single power molex. Are there cons in JTAG vs 2 SPIs ?

Cost of such chassis with power supply and Intel ATOM motherboard could vary in $400 - $600 range. Cost of Spartan6 chips when purchased in bulk quantities (WITHOUT VAT) would vary in $70 - $95 range, depending on shipment location and quantity of chips ordered. Cost of other components (using numbers from our current design):

No. Use something smaller and lower power to run this, not some shitty Atom board. Use a Pi or that new $50 Via x86 board http://www.geek.com/articles/chips/via-launch-a-49-android-pc-20120522/

Nice board, is there any other boards that support GPIO for example or SPI or JTAG output without any additional converters ? So I could launch software on linux ? That would be excellent for this project. Possibly with 2 cores, so one core could drive GPIO in realtime manner, while other core is communicating with world. But not necessary, as SPI tolerate lags!

200 Mh/s bitstream - would produce 18 Gh/s - it would be compared to 71% of Mini-Rig and so product price could be $10'859.
250 Mh/s bitstream - would produce 22.5 Gh/s - it would be compared to 89,2% of Mini-Rig and so product price could be $13'643.
300 MH/s bitstream - would produce 27 Gh/s - it would be compared to 107% of Mini-Rig and so product price could be $16'365.
325 MH/s bitstream - would produce 29.2 Gh/s - it would be compared to 116% of Mini-Rig and so product price could be $17'722.

Remember, if you're overclocking these, provide some way to underclock+undervolt these to extend the life of these once diff goes too high in 3-4 years. This is a big feature that a lot of people are asking for.

These are not overclocked, overclocking could give +20 - 25%, but will consume more power and possibly damage chips permanently.

Also, another thing, make sure the total power usage for the box fits so an integer number of these fits on a 120v 20a  (the most common circuit in DCs, you often get two of these per rack). ~1000 watts each would be fine if you intend on putting two on a circuit.

Well one box 4U would consume about 1.3 - 1.5 kW power that's 10,8 - 12,5 Amps @ 120 V seems to be bad.
With 11-12 boards it would be not more than 10 Amps @ 120 V however.

[DiabloD3]

Withing this chassis single Intel Atom D525 motherboard is installed. Boards with 6 spartans can be even without microcontrollers and flash, everything could be programmed right via LPT-port. Bandwith required to communicate with every chip is quite low - about 300 bps. So with all chips it would be about 27 kbps. Bitstream loading over LPT port however will be slow. For smaller scale RS-485 is overkill. Why to bother about it and not implement using USB - simply because flashing chip or flashing controller
adds up cost of controller and also costs of programming and testing them, also when something should be updated, and you have to reprogram every controller - that rises service cost. I would like to say, that current design of BitFury rack, where controller only translates RS-485 to SPI bus with Spartan and back requires almost zero maintenance.

Use SATA plugs for the actual connector, but normal serial over it. SATA has 7 pins and is enterprise ready. The cost of serial is the complex plug, not the actual design. You could use a very tiny FPGA for the controller on each board to interface with the serial using GPIO pins or something.

SATA plugs! thanks, nice idea! 4-in cables are there.... This as simple as just installing proper size of SATA PCB layout on PCB! $1.67 x 16 = $26.7 for all of the connectivity. Just connecting each board to another one. Simply installing them and connecting. 1.6m long wire.

http://www.satacables.com/

About controller - I've mentioned only 500 bps per chip, so for 90 chips it would be max. 45k bps in and out...
Information about protocol is avail in initial post. So pinout can be following:

PIN1 - SCK
PIN2 - MOSI
PIN3 - MISO
PIN4 - GROUND
PIN5 - RESET
PIN6 - PROGDATA
PIN7 - PROGSCK

So it would be possible (a) upload bitstream (b) reset all chips (c) send/receive work.

OR - ANOTHER POSSIBILITY IS TO USE JTAG (but I don't know actually how well it work via such long chains like 90 chips, even when TCK, TMS will be transmitted using buffers.

PIN1 - TMS
PIN2 - TCK
PIN3 - TDI
PIN4 - GROUND
PIN5 - UNUSED
PIN6 - RST
PIN7 - TDO

Please not that key is not important, as improper insertion of cabling won't fry things. However it would be
perfectly possible to transmit jobs over JTAG. JTAG also seems to be nicer, because key can be programmed into board right using this slot.

So single SATA and single power molex. Are there cons in JTAG vs 2 SPIs ?

JTAG might work but I don't think you're supposed to use JTAG that way. If you can make it work and not damage the hardware, go ahead, its your product, all of us will just treat it like a black box. BTW, BFL minirigs are using SATA connectors for their serial communication between boards, and Ive seen other designs repurpose connectors SATA internally or Infiniband connectors externally.

BTW re power Molex: Remember, Molex chain has a maximum of 5 amps/60 watts of 12v, and 5 amps/25 watts of 5v. That is not per plug, and a lot of chains have 3 or 4 plugs on it. PCI-E 6 is 2 amps/75 watts of 12v, PCI-E 8 is 4 amps/150w of 12v, P4 is 16 amps/192 watts of 12v, EPS12v is 32 amps/384w of 12v.

If you're putting 6 or 8 FPGAs on a board inside the rig, you're going to want to be using PCI-E plugs not Molex.

Cost of such chassis with power supply and Intel ATOM motherboard could vary in $400 - $600 range. Cost of Spartan6 chips when purchased in bulk quantities (WITHOUT VAT) would vary in $70 - $95 range, depending on shipment location and quantity of chips ordered. Cost of other components (using numbers from our current design):

No. Use something smaller and lower power to run this, not some shitty Atom board. Use a Pi or that new $50 Via x86 board http://www.geek.com/articles/chips/via-launch-a-49-android-pc-20120522/

Nice board, is there any other boards that support GPIO for example or SPI or JTAG output without any additional converters ? So I could launch software on linux ? That would be excellent for this project. Possibly with 2 cores, so one core could drive GPIO in realtime manner, while other core is communicating with world. But not necessary, as SPI tolerate lags!

You're probably only going to see GPIO on ARM boards meant for embedded usage, otherwise you're looking at a custom build of a standard USB->serial controller. Raspberry Pis are like $25 and have enough power to run cgminer, so if you can get them in bulk it seems they'll do what you want.

Problem is you might run out of GPIO pins because you'd need one to every controller (one per 4?).

Just whatever you do, don't put USB cables in the case. They easily get disconnected during transit and sometimes just unplug themselves due to case vibration or high pressure air cooling.

200 Mh/s bitstream - would produce 18 Gh/s - it would be compared to 71% of Mini-Rig and so product price could be $10'859.
250 Mh/s bitstream - would produce 22.5 Gh/s - it would be compared to 89,2% of Mini-Rig and so product price could be $13'643.
300 MH/s bitstream - would produce 27 Gh/s - it would be compared to 107% of Mini-Rig and so product price could be $16'365.
325 MH/s bitstream - would produce 29.2 Gh/s - it would be compared to 116% of Mini-Rig and so product price could be $17'722.

Remember, if you're overclocking these, provide some way to underclock+undervolt these to extend the life of these once diff goes too high in 3-4 years. This is a big feature that a lot of people are asking for.

These are not overclocked, overclocking could give +20 - 25%, but will consume more power and possibly damage chips permanently.

Wait, you're doing 325 mh per Spartan 6 SLX150? Holy crap man. Is that just back of the napkin math, or do you have a working bitstream that can do that?

Still, consider some method to undervolt them. At some point within the next 2-3 years for most users, the cost of electricity will exceed the value of the coins, but undervolting will catch them up for at least another year of usage.

Also, another thing, make sure the total power usage for the box fits so an integer number of these fits on a 120v 20a  (the most common circuit in DCs, you often get two of these per rack). ~1000 watts each would be fine if you intend on putting two on a circuit.

Well one box 4U would consume about 1.3 - 1.5 kW power that's 10,8 - 12,5 Amps @ 120 V seems to be bad.
With 11-12 boards it would be not more than 10 Amps @ 120 V however.

Well, on typical hardware, you shouldn't exceed 10 amps on 120v. That gives you a 1200 watt continuous PSU. Two of those will just barely fit on an enterprise 120v 20a line in a DC and hopefully not trip it, or one will fit on a household 120v 15a line (note: household lines _suck_, never drive them at >12a 24/7).

There ARE servers that require 208/240v service in DCs, usually some nearline data warehousing server that you shove 40 drives in, but a lot of DCs don't offer this or require a special order for it, and no house in America would typically have that service outside of an electric oven range or dryer socket.

[bitfury]

JTAG might work but I don't think you're supposed to use JTAG that way. If you can make it work and not damage the hardware, go ahead, its your product, all of us will just treat it like a black box. BTW, BFL minirigs are using SATA connectors for their serial communication between boards, and Ive seen other designs repurpose connectors SATA internally or Infiniband connectors externally.

Well. Yep. JTAG could devastate things.... So better to leave there only bitstream programming and communication
with chips.

BTW re power Molex: Remember, Molex chain has a maximum of 5 amps/60 watts of 12v, and 5 amps/25 watts of 5v. That is not per plug, and a lot of chains have 3 or 4 plugs on it. PCI-E 6 is 2 amps/75 watts of 12v, PCI-E 8 is 4 amps/150w of 12v, P4 is 16 amps/192 watts of 12v, EPS12v is 32 amps/384w of 12v.

If you're putting 6 or 8 FPGAs on a board inside the rig, you're going to want to be using PCI-E plugs not Molex.

6 Amps per pin limit for molex from datasheet, if I remember it correctly and about 100 times plugging before tin
degrades guaranteed. So this should not make big problems. 4 pins used, 2 pins for +, 2 pins for -, 12V powered.
The only problem is how to make efficient down conversion from 12 V to 1.2 V. I used TPS40090-based one,
and its COP and size is not very nice, however works without any problems.

You're probably only going to see GPIO on ARM boards meant for embedded usage, otherwise you're looking at a custom build of a standard USB->serial controller. Raspberry Pis are like $25 and have enough power to run cgminer, so if you can get them in bulk it seems they'll do what you want.

Problem is you might run out of GPIO pins because you'd need one to every controller (one per 4?).

Just whatever you do, don't put USB cables in the case. They easily get disconnected during transit and sometimes just unplug themselves due to case vibration or high pressure air cooling.

That's it! I felt that USB is good only when you connect one board device to your home PC :-)

200 Mh/s bitstream - would produce 18 Gh/s - it would be compared to 71% of Mini-Rig and so product price could be $10'859.
250 Mh/s bitstream - would produce 22.5 Gh/s - it would be compared to 89,2% of Mini-Rig and so product price could be $13'643.
300 MH/s bitstream - would produce 27 Gh/s - it would be compared to 107% of Mini-Rig and so product price could be $16'365.
325 MH/s bitstream - would produce 29.2 Gh/s - it would be compared to 116% of Mini-Rig and so product price could be $17'722.

Remember, if you're overclocking these, provide some way to underclock+undervolt these to extend the life of these once diff goes too high in 3-4 years. This is a big feature that a lot of people are asking for.

These are not overclocked, overclocking could give +20 - 25%, but will consume more power and possibly damage chips permanently.

Wait, you're doing 325 mh per Spartan 6 SLX150? Holy crap man. Is that just back of the napkin math, or do you have a working bitstream that can do that?

not working yet... bitstream requires approx. 300 hours of additional complex work to make that working. Just if you look carefully, there's unused approx. 1200 slices in the left part of the design + approx. 40 DSP48 + 80 BRAMs my estimations that I can fit there 6 rounds definitely. 8 sha256 cores would be quite difficult however. which would work at same clock rates like rest of design. And also there's about 550 slices left in top area - again - enough to fit 2 sha256 cores. Also hole in center - but I cannot fit there round exactly as below - lacking 2 slices and it is difficult to make workaround for that, but maybe I can put there special smaller round that would work at 2/3 of main clock. So it would be certainly 90/65 x clock speed compared to current 82/65 x clock. But if there will be request to implement it (as I stated in license - I would jump in @ 5000 chips approx at $5 per chip level) - I will start doing that work. 90/82 speedup is minimal estimation, actually it can become 92.6 / 82 - but I don't know if I really could get it... it depends whether BRAMs will be used or not actually if BRAMs will be used - then 8 additional cores seems feasible, or at least 7 cores... if however they will not work even at smaller clocks or will need additional set of registers after them, then I'll have to get back to concept without BRAMs and it will give 6 rounds. I've estimated this already - it is very quick... then more or less difficult is write correct code - I believe that it would take about 60 hours for two different DSP-BRAM and DSP-nobram models and about 40 hours for top rounds modifications... And then most interesting game starts - fitting it into a chip :-) Which would require lots of design modification, manual separation of control wires, looking how routing of these new cores would intersect with routings of big design.... This will take all of the rest time... Maybe if I'll be lucky it would be done in another 100 hours, but as from previous experience - it is not the way - as you only think that you are "near", while in reality you have not finished half road.... That's like estimating distance to oasis in desert :-)

Still, consider some method to undervolt them. At some point within the next 2-3 years for most users, the cost of electricity will exceed the value of the coins, but undervolting will catch them up for at least another year of usage.

I actually think that programmable voltage in range 1.15 V .. 1.45 V would be good. If someone would decide to risk with Spartans but get higher returns - then that's OK. Their devices would give +20%. If one would decide to go with less electricity consumption - then switch to 1.15 V and lower clocks. It just adds challenge with PSU - it should be still efficient in all ranges of power consumption.

Also, another thing, make sure the total power usage for the box fits so an integer number of these fits on a 120v 20a  (the most common circuit in DCs, you often get two of these per rack). ~1000 watts each would be fine if you intend on putting two on a circuit.

Well one box 4U would consume about 1.3 - 1.5 kW power that's 10,8 - 12,5 Amps @ 120 V seems to be bad.
With 11-12 boards it would be not more than 10 Amps @ 120 V however.

Well, on typical hardware, you shouldn't exceed 10 amps on 120v. That gives you a 1200 watt continuous PSU. Two of those will just barely fit on an enterprise 120v 20a line in a DC and hopefully not trip it, or one will fit on a household 120v 15a line (note: household lines _suck_, never drive them at >12a 24/7).

There ARE servers that require 208/240v service in DCs, usually some nearline data warehousing server that you shove 40 drives in, but a lot of DCs don't offer this or require a special order for it, and no house in America would typically have that service outside of an electric oven range or dryer socket.

Well - I am using now 3-phase 0.4 kV L-L @ 33 amps when chiller is running and @ 22 amps when chiller is not running. I think that if someone would like to install many of these things @ home - he should invest in cable connected directly to transformer to not disrupt power distribution for other houses. How much would it cost to me, if I would ask in America utility company for say 50 kW power to my household ?

I have installed PSUs with power factor correction (PFC). and it is important, as otherwise power losses would be significant.

About DCs I thought that way - 350 W per 1 U x 4 = 1.4 kW.... Otherwise how they would setup 4 x 1 U @ 350 W ?

[2112]

Also hole in center - but I cannot fit there round exactly as below - lacking 2 slices and it is difficult to make workaround for that,

I have one more question/suggestion. I understand that you're using LUT-RAM to store the the FIPS-180-x constants. I also guess that each hashing cell in your design has a dedicated private copy of that ROM. Have you tried pairing (or quad-ing) the neighboring cells to share a single ROM storage? If the cell clocks are in sync the neighbors could share those without paying to much cost in switch/route overhead.

I'm just asking, I'm not trying to question your skills or anything like that. I'm not involved in Bitcoin mining and I read this board for intellectual stimulation only.

[DiabloD3]

BTW re power Molex: Remember, Molex chain has a maximum of 5 amps/60 watts of 12v, and 5 amps/25 watts of 5v. That is not per plug, and a lot of chains have 3 or 4 plugs on it. PCI-E 6 is 2 amps/75 watts of 12v, PCI-E 8 is 4 amps/150w of 12v, P4 is 16 amps/192 watts of 12v, EPS12v is 32 amps/384w of 12v.

If you're putting 6 or 8 FPGAs on a board inside the rig, you're going to want to be using PCI-E plugs not Molex.

6 Amps per pin limit for molex from datasheet, if I remember it correctly and about 100 times plugging before tin
degrades guaranteed. So this should not make big problems. 4 pins used, 2 pins for +, 2 pins for -, 12V powered.
The only problem is how to make efficient down conversion from 12 V to 1.2 V. I used TPS40090-based one,
and its COP and size is not very nice, however works without any problems.

Molex's peripheral cable uses AMP 1-480424-0, AMP 60619-1, AMP 1-480426-0, and AMP 60620-1 for the socket housing, socket, pin housing, and pins respectively. This is rated for 13 amps. However, these are commonly wired with 18 AWG wiring. Putting 13 amps through this will melt the wiring, let alone 13 amps on every single plug in the chain. And although it has 4 pins, only one set are for 12v, the other set is for 5v.

Also, I hope you're using enterprise rated PSUs. You don't need redundant, but the largest non-redundant PSUs you can typically buy are in the 850w to 900w range. Not quite the 1100w you're aiming for. They do make 600-700w 2U non-redundant PSUs though, so you can always double up.

Still, consider some method to undervolt them. At some point within the next 2-3 years for most users, the cost of electricity will exceed the value of the coins, but undervolting will catch them up for at least another year of usage.

I actually think that programmable voltage in range 1.15 V .. 1.45 V would be good. If someone would decide to risk with Spartans but get higher returns - then that's OK. Their devices would give +20%. If one would decide to go with less electricity consumption - then switch to 1.15 V and lower clocks. It just adds challenge with PSU - it should be still efficient in all ranges of power consumption.

If you can get them still mining at 1.10v that would be far enough for most people, since most Spartan 6s are ran at 1.20 to 1.25v now.

Well - I am using now 3-phase 0.4 kV L-L @ 33 amps when chiller is running and @ 22 amps when chiller is not running. I think that if someone would like to install many of these things @ home - he should invest in cable connected directly to transformer to not disrupt power distribution for other houses. How much would it cost to me, if I would ask in America utility company for say 50 kW power to my household ?

I have installed PSUs with power factor correction (PFC). and it is important, as otherwise power losses would be significant.

About DCs I thought that way - 350 W per 1 U x 4 = 1.4 kW.... Otherwise how they would setup 4 x 1 U @ 350 W ?

In the US we use 240v single phase for electric ovens and driers and electric water heaters and larger air conditioners. Its provided as two hots and a neutral, with the two hots 120v lines that have their phases 180 degrees out of phase (so peak + on one is peak - on the other). Seems that Europe uses 3 phase 400v for the same reasons.

If we're limited to 4 120v 20a circuits per rack (which is what I've been told is a common limit, and you have to pay extra for the other two), and each 4U uses 9a (= 1100 watts) and we place an empty U between each (for airflow/cooling and easy maintenance reasons), thats 40u used in a 42u rack, or 8800w.

You shouldn't include your chiller, because a DC provides its own cooling. So, 22 amps of 400v is 8800w.

So, either your math is wrong or you misstated something, because we're coming out to the same power usage.

[bitfury]

I understand that developing such a product requires huge investments and risks, but this community is rabidly pro open source.  You might succeed at this for a time, but someone is going to replicate it and release it to the wild eventually.  I'm sure that you understand that, so the goal is to get as much profit from it as you can before the open source developers catch up.  May I offer an alternative to all this?  If you have an idea of how much revenue you expect to earn, let us know.  You can proceed with your licensing method however you see fit, but if the community can raise the funds, would you accept a collective donation in order to release the bitstream open source?

Well. Here's the plan - finish before 20th-30th June design of smaller device (4U or 2U sized - we have debates with shalab.si about that). To accomplish it, I actually need at 6th-15th June spec ready and circuit in approved state. So if till 1st June there will be no constructive discussion here about device, then I'll go finishing it alone. As far as I understood for many people that's just nice hobby... But I am kind of person who dislikes things half-done and half-working... So I've already invested in it half year full time + many nights into it and all non-spent cash as well, and I expect others who would like to deliver hashing power to have more serious approach as well. About open source - if someone would do this "as hobby" investing like 10 hours per week or so, he would need approx. 40 weeks to use all clues I gave already to make design. By that time I would glad to know that person, because his skill is great, I would be even more excited if someone would make it 350-370 Mh/s without rising VCCINT voltage.  But I consider case that someone replicate within next 33-50 days AND PUBLISH OPEN-SOURCE as low chance. Then at that time _WE_ will have everything ready for deployment. And yes - I love competition as it enforces you to do best things and be one step ahead.

Meanwhile still doing design negotiations who will make assembly of boards and where, who will do assembly of these devices and delivery and who will sell and service them worldwide. Then we'll start production - I aim at 2-3 month lead time, but during first period I plan to reinvest all of income into devices production to lower lead times and make interesting offer - basically you pay higher if you get device quick. So if someone would like to install 10 devices, he could start testing first one tomorrow.

During production period I expect to ramp of from 500 chips per month level to 1000-1200 chips per month delivered. within 6 month frame. That is about 4500 - 5000 chips delivered. But it could be better, if additional financing would come in play, that would allow to use other equities to finance bitcoin farms productions instead of just paying in cash. Actually I would like to push spartans hard and make target higher at 10k chips, but that would be difficult to get.

So expectations are for licensing - $100k - $200k for chips licensing and $25k-$50k for bitstream hashing power improvement within current offer.

Then, when spartan's will not be hot anymore - all of existing solution will go opensource, and main production line would switch to Artix, so price per Mh/s would drop another time and profit margin for manufacturing will be about difference between DIY spartan6 and artix7 solution.

So what is problem with opensource - I am pro-opensource, but I really doubt that with donations it is possible to get that amounts for such non-popular project like spartan6-miner. If it would be mass-product with say $1 - $10 donations it would work however. It is the problem I see - amounts are high compared to % of users interested in it within current bitcoin community size. Manufacturers of boards unlikely would jump into it. Miners possibly, but that depends % how much they would invest into disclosure and how many chips they would get then. And I believe many of miners would just hate this initiative, because they would like better that things won't change, they get still 50 BTC coinbase, etc. etc.

But if I am wrong - then I expect you to prove it. Let's develop procedure on how fundraising will be performed, how much efforts needed to advertise it, who will oversee funds (say what will happen if only 30% of target amount raised, as funds should be returned back). Right today I am not bound with any agreements about this bitstream usage, but I think it will be fair and necessary step of mine to use raised money and give buy offer to all current co-owners of current equipment, because this open-sourced bitstream could give sharp difficulty rise. Also this should be done relatively quickly - within 4-week time frame. Because when designing will be finished, I would definitely enter into bindings with different parties, and would not be able to decide alone whether this should go open source or not, and it could be difficult to get consensus. But anyway - this will be interesting case - if it works, then definitely for many chips there will be best bitstreams available, nice competition among designers would rise.

[MoonShadow]

I can't disprove you, and didn't intend to.  I was just asking what your price point was, should the community prove both willing and able to hit that price point.  I consider compatition a good thing, but it's still replication of work and if it can reasonablely be avoided, benefits all concerned.  Replication of work can't always be reasonablely be avoided.  I'm personally not interested in a fpga solution, even though I would consider it technically superior.  I'm personally interested in an asic solution, because I can use the waste heat to warm my home in winter, so I believe that basement mining operations will always be a factor, because professional mining operations not only have to deal with the overhead of dumping heat, they also have to outperform the basement miners to such a degree to overcome the economic advantages of co-generation.

[bitfury]

BTW re power Molex: Remember, Molex chain has a maximum of 5 amps/60 watts of 12v, and 5 amps/25 watts of 5v. That is not per plug, and a lot of chains have 3 or 4 plugs on it. PCI-E 6 is 2 amps/75 watts of 12v, PCI-E 8 is 4 amps/150w of 12v, P4 is 16 amps/192 watts of 12v, EPS12v is 32 amps/384w of 12v.

If you're putting 6 or 8 FPGAs on a board inside the rig, you're going to want to be using PCI-E plugs not Molex.

6 Amps per pin limit for molex from datasheet, if I remember it correctly and about 100 times plugging before tin
degrades guaranteed. So this should not make big problems. 4 pins used, 2 pins for +, 2 pins for -, 12V powered.
The only problem is how to make efficient down conversion from 12 V to 1.2 V. I used TPS40090-based one,
and its COP and size is not very nice, however works without any problems.

Molex's peripheral cable uses AMP 1-480424-0, AMP 60619-1, AMP 1-480426-0, and AMP 60620-1 for the socket housing, socket, pin housing, and pins respectively. This is rated for 13 amps. However, these are commonly wired with 18 AWG wiring. Putting 13 amps through this will melt the wiring, let alone 13 amps on every single plug in the chain. And although it has 4 pins, only one set are for 12v, the other set is for 5v.

1.5 sq. mm wires. ( AWG 15 ).

Also, I hope you're using enterprise rated PSUs. You don't need redundant, but the largest non-redundant PSUs you can typically buy are in the 850w to 900w range. Not quite the 1100w you're aiming for. They do make 600-700w 2U non-redundant PSUs though, so you can always double up.

For current big bf-110 we used RSP-3000-12 - 200 Amps @ 12 V . Boards however consume only ~160 Amps. So there's reserve for overclocking or for grid under-voltage condition. I expect to put something nice and reliable installed into box as well. They're not cheap, but it is crazy to put cheap PSU to power such expensive equipment.

Well - I am using now 3-phase 0.4 kV L-L @ 33 amps when chiller is running and @ 22 amps when chiller is not running. I think that if someone would like to install many of these things @ home - he should invest in cable connected directly to transformer to not disrupt power distribution for other houses. How much would it cost to me, if I would ask in America utility company for say 50 kW power to my household ?

I have installed PSUs with power factor correction (PFC). and it is important, as otherwise power losses would be significant.

About DCs I thought that way - 350 W per 1 U x 4 = 1.4 kW.... Otherwise how they would setup 4 x 1 U @ 350 W ?

In the US we use 240v single phase for electric ovens and driers and electric water heaters and larger air conditioners. Its provided as two hots and a neutral, with the two hots 120v lines that have their phases 180 degrees out of phase (so peak + on one is peak - on the other). Seems that Europe uses 3 phase 400v for the same reasons.

If we're limited to 4 120v 20a circuits per rack (which is what I've been told is a common limit, and you have to pay extra for the other two), and each 4U uses 9a (= 1100 watts) and we place an empty U between each (for airflow/cooling and easy maintenance reasons), thats 40u used in a 42u rack, or 8800w.

You shouldn't include your chiller, because a DC provides its own cooling. So, 22 amps of 400v is 8800w.

So, either your math is wrong or you misstated something, because we're coming out to the same power usage.

I have not misstated. 0.4 kV L-L - that's about 230 Volts phase - to - neutral fed from transformer. When it gets to me, due to losses in cables - it gets to standard 215 - 220 Volts phase - to - neutral voltage. And then
22 x 218 x 3 = 14.4 kW approx.

I am not using 4U design right now - 4U design is during planning... look @ photos on www.bitfury.org - that are the bf-110 used now for mining!

[bitfury]

I can't disprove you, and didn't intend to.  I was just asking what your price point was, should the community prove both willing and able to hit that price point.  I consider compatition a good thing, but it's still replication of work and if it can reasonablely be avoided, benefits all concerned.  Replication of work can't always be reasonablely be avoided.  I'm personally not interested in a fpga solution, even though I would consider it technically superior.  I'm personally interested in an asic solution, because I can use the waste heat to warm my home in winter, so I believe that basement mining operations will always be a factor, because professional mining operations not only have to deal with the overhead of dumping heat, they also have to outperform the basement miners to such a degree to overcome the economic advantages of co-generation.

Well. ASIC is much more expensive. If you would get simple ASIC that would just work, but you get it with 100% success probability at affordable level - that would be rather slow 90nm or 130 nm chip, it would be like what 28-nm chips could give with their best performance and special price from fpga manufacturers. Say if altera or xil would decide to sell their chips less but more q-tys for mining - then ASIC project would end up with failure. And it would took long time to develop it as well.

If you would like to make really tricky and quick ASIC, then there could be epic failures and risks are very high. I would like to design such ASIC, but only if backed with investments about $1 - $1.2 mio and having possibility to order several times multiwafer which would be thrown away as "bad" result.... But when I am talking about that conditions people say that I am completely crazy. But that is true - without doing actual experiments, you would not get TOP results, as going proven way - you will get proven results of course... Or even simpler things like e-asic or altera hardcopy.

However really quick ASIC will outperform, but lot of money could be just thrown away with it, so there will be gradually improving and competing ASIC solutions. And this will be risky hop... Look how difficult hop from GPU to FPGA is, however FPGA are superior... Still most miners use GPU, and I still see much space to promote GPU mining, when these cards are installed @ home. And with ASIC it will be even more difficult. So for ASICs to succeed difficulty AND exchange rate should rise to level, so whole mining market would increase in side - currently it is like $1 mio per month mined, then if exchange rate will not rise - it will be about $0.5 mio per month... For low-end ASIC it would be tough play, also when you consider that optimized ASIC or discounted 28-nm chips would blow it out from game even without returning NRE costs.

Also ASIC must be available to interested parties, because myself is interested in condition, that there will be no single party in bitcoin who owns network... And if that happens - I'll really work hard to prevent this condition. Also I see that FPGAs can be competetive to ASIC while GPU would have much bigger gap in performance, so ASIC over GPU takeover is much more likely than ASIC vs FPGA.

+++ for mining co-generation - that's really nice. I can now implement pool and heat in in summer :-) without mining I would not waste heat for that :-) Now I have more than enough useful heat. However many still thinks that it is crazy way.

[DiabloD3]

Hrm, I was wrong. Athena Power sells a 1100w non-redundant: http://www.newegg.com/Product/Product.aspx?Item=N82E16817338021

[bitfury]

Also hole in center - but I cannot fit there round exactly as below - lacking 2 slices and it is difficult to make workaround for that,

I have one more question/suggestion. I understand that you're using LUT-RAM to store the the FIPS-180-x constants. I also guess that each hashing cell in your design has a dedicated private copy of that ROM. Have you tried pairing (or quad-ing) the neighboring cells to share a single ROM storage? If the cell clocks are in sync the neighbors could share those without paying to much cost in switch/route overhead.

I'm just asking, I'm not trying to question your skills or anything like that. I'm not involved in Bitcoin mining and I read this board for intellectual stimulation only.

Why not to wait before full disclosure ? It would be interesting anyway... I think that you do not even imagine scale of these optimizations, as you are asking questions that I started to ask when started this journey :-) I would even describe storyline how different decisions were make, and if would have enough time for that - arrange in a way that you could try to guess what was done next, not disclosing it before you read next page. Just be around to remind me to do this.

[DiabloD3]

1.5 sq. mm wires. ( AWG 15 ).

For current big bf-110 we used RSP-3000-12 - 200 Amps @ 12 V . Boards however consume only ~160 Amps. So there's reserve for overclocking or for grid under-voltage condition. I expect to put something nice and reliable installed into box as well. They're not cheap, but it is crazy to put cheap PSU to power such expensive equipment.

Wait, you're already using redundant modules for the PSU? Thats totally okay then. Theres 2 or 3 1400w or more models out there, although they all use a different incompatible interface.

I have not misstated. 0.4 kV L-L - that's about 230 Volts phase - to - neutral fed from transformer. When it gets to me, due to losses in cables - it gets to standard 215 - 220 Volts phase - to - neutral voltage. And then
22 x 218 x 3 = 14.4 kW approx.

I am not using 4U design right now - 4U design is during planning... look @ photos on www.bitfury.org - that are the bf-110 used now for mining!

So you're going to sell 2U boxes initially? 500w in a 2U could be easily done.

[rjk]

I dunno about redundant, but they are Mean Well PSUs, which are some of the best in the industry. You can see the MW logo in the pics.

[DiabloD3]

I dunno about redundant, but they are Mean Well PSUs, which are some of the best in the industry. You can see the MW logo in the pics.

Those kind of modules (lots of manufs make them) are typically used in either redundant PSU housings hold 2 to 4 modules, or people use them with custom wiring harnesses if they have specific wiring requirements.

[rjk]

I dunno about redundant, but they are Mean Well PSUs, which are some of the best in the industry. You can see the MW logo in the pics.

Those kind of modules (lots of manufs make them) are typically used in either redundant PSU housings hold 2 to 4 modules, or people use them with custom wiring harnesses if they have specific wiring requirements.

I was totally going to use them in my rig, but they are damn expensive. http://www.meanwell.com/search/rsp-3000/rsp-3000-spec.pdf
Nice lugs for attaching bus bars to.

[DiabloD3]

I dunno about redundant, but they are Mean Well PSUs, which are some of the best in the industry. You can see the MW logo in the pics.

Those kind of modules (lots of manufs make them) are typically used in either redundant PSU housings hold 2 to 4 modules, or people use them with custom wiring harnesses if they have specific wiring requirements.

I was totally going to use them in my rig, but they are damn expensive. http://www.meanwell.com/search/rsp-3000/rsp-3000-spec.pdf
Nice lugs for attaching bus bars to.

Yeah, its a 2400w PSU module. Something like that really should cost $550-600.

[RoadStress]

https://bitcointalk.org/index.php?topic=76351.msg923533#msg923533

Any bets on performance?

[DiabloD3]

https://bitcointalk.org/index.php?topic=76351.msg923533#msg923533

Any bets on performance?