Algorithmically placed FPGA miner: 255MH/s/chip, supports all known boards

[eldentyrell]

I've been working on an unconventional design and doing 100%
algorithmic placement. This design contains three rings, each of
which is 64 stages of SHA-256.  Therefore, on each clock cycle each
ring gets 0.5 hashes (a nonce has to go through the ring twice in
order to be fully hashed).

Update, 20-Jul-2012: TML-1.0 has been released.  We are getting an average of 255MH/s/chip across a large farm of speed grade 2 chips (not the more-expensive grade 3 chips).  These are nexus6 boards, which have a power supply which is much more powerful than the one on the ztex boards.

All known boards (ztex, modminer, icarus, x6500, enterpoint, the-nameless-board-of-rph, nexus6) are supported.  You might need a JTAG cable.  If you would prefer to use your board's proprietary USB connector, contact your board vendor for assistance.

Additional announcement: during the last week, tricone mining has secured funding for its next (i.e. post-TML) bitcoin project (no, not the joke press release).  Unfortunately this means that between now and the end of August, further development of the TML will be at a reduced priority.  We will continue to maintain the signcryption servers, fix bugs, and produce new bitstreams (we have a farm of servers that sweeps through combinations of the various Xilinx tool command line parameters), so you can expect some minor improvements in performance and a steadily-growing supply of bitstreams to choose from.  However, new features will have to wait until 07-Sep-2012.  In order to compensate our users for the unfortunate timing of this priority shift, the commission rate is set to 0% until at least 07-Sep-2012.

Please do not send me forum PMs; I find the PM system very cumbersome to use.  My email address is in my profile.

______________________________________________________________________________

Update, 12-May-2012: Preliminary power numbers posted.

Update, 9-May-2012: New numbers posted.  This is the first design resulting from a substantial re-architecting begun about a month ago (and, I expect, the last one).  All the designs up to 145mhz were incremental improvements; the 170mhz design was a major change.

Update, 19-Mar-2012: sorry for the long delay.  Had two major
problems: one was a hardware/simulator mismatch that could
only be debugged with git bisect (thank you Linus!), but each
build takes ~24hours.  The other problem was voltage sag on my crappy
PCBs (I'm not a PCB designer), which was why the actual speeds were
below what the Xilinx tools said they should be.  I can now run at the
rated speed.  Results of overclocking tests will be posted shortly.  I
also have another build about to finish that should produce another
small bump in the design clock rate this weekend.

Update, 9-Mar-2012: new build finished last night, 140mhz
design rate, 135mhz actual clock rate, 202.5MH/s actual hashrate.
You'll notice that my actual clock rates are slower than my
design clock rates, meaning that I have to very slightly underclock
the design to get optimal error rates.  I don't know why this is,
although I only have a single SG-3 chip and it's on one of the first
PCB's I made -- before I got my reflow oven tuned up.

Update, 8-Mar-2012: The results are finally out of the
totally-embarrassing region, so I think I'll start posting them.  I
want to emphasize that these are very preliminary.  I've only
been doing performance optimization on the full design for three weeks
now.  As you can see, I'm getting these hashrates at 128mhz and others
have demonstrated that the SHA-256 critical path on a Spartan-6 can
run well above 200mhz.

All power consumption numbers are measured at the 12V rail of an
ATX power supply (SATA connector).  So they include any
inefficiencies introduced by the 12V->1.2V stepdown.  Power
consumption measured at the wall will be higher; how much
depends on the power supply's efficiency.

I use adaptive clocking (DCM_CLKGEN) at 1mhz granularity (yes, I
figured out how to fix the jitter issues) and a separate clock
adjustment for each ring.  This means that a performance defect in
one part of the chip will force the slowdown of only one of the three
rings.  This is another reason why I designed for three separate
half-pipelines instead of one full pipeline and one half-pipeline.

The "actual" power and frequency numbers produced by running the
design on an SG-3 chip at 1200mV, adapting the three clocks to
maximize (non-erroneous) hashrate, then averaging the three
frequencies.  Most of my cluster is SG-2, but I quote the SG-3 numbers
since that's what everybody else quotes numbers for.  I have digital
control of the voltage supply on my boards, so I'll try adaptive
overvolting at some point; the chips are rated up to 1320mV.

See the end of the thread for map output, or if you are
interested in licensing.

[1714159879]

1714159879

[1714159879]

1714159879

[1714159879]

1714159879

[1714159879]

1714159879

[ElectricMucus]

wow, this is incredible 

Are you going to share it? Under which conditions?
Very, very interesting

[tinman951]

Just wondering, what OS, programs, and other stuff are you running.  Let's say I have the hardware (which I obviously don't).  How would you set this up from a flash drive?

[pusle]

@big-chip-small-board:  So you get 100Mhash/s from this design at what size fpga?

[eldentyrell]

@big-chip-small-board:  So you get 100Mhash/s

As you can see, there is room for at least one more copy of the pipeline on the chip.

from this design at what size fpga?

You're looking at an LX150.

[Dexter770221]

And 3 months ago someone said that theres no way to achieve 200MH/s

[eldentyrell]

And 3 months ago someone said that theres no way to achieve 200MH/s

In fairness, I still do not have the "corner turn" routes running at that speed.  But I have verified that the circuit you see works (i.e. mines actual shares)... though at 45mhz since the corner turn routes are a ridiculous 22ns.

So, no celebrations just yet...

[ElectricMucus]

Excuse the nagging, but are you planning to share your work? 

[eldentyrell]

Excuse the nagging, but are you planning to share your work? 

Undecided, for now.  But honestly, at 45MH/sec it doesn't matter just yet.

[eldentyrell]

But I have verified that the circuit you see works (i.e. mines actual shares)... though at 45mhz since the corner turn routes are a ridiculous 22ns.

Oh, and, to Xilinx: whatever you did to columns 66 and 67 drives me berzerk.  I can't use the entire pair of columns because slices are randomly MISSING for no apparent reason.

[rph]

Heh, it was a matter of time until somebody hand-placed it. Hardcore.

-rph

[eldentyrell]

Heh, it was a matter of time until somebody hand-placed it. Hardcore.

Emphasis on "time"

FWIW, I don't think it's feasible to do this directly in Verilog/VHDL.  I had to write a Java library to generate the (totally illegible) Verilog.

[gmaxwell]

Emphasis on "time"
FWIW, I don't think it's feasible to do this directly in Verilog/VHDL.  I had to write a Java library to generate the (totally illegible) Verilog.

I am reloading this thread like a crazed weasel.  I'm quite eager to see what kind of results you get from the manual placement.

[BTCurious]

Oh wow. I know next to nothing about FPGAs*, but this looks very awesome nonetheless.

*I know that they're field-programmable gate arrays, and they're basically logic components which can be reconnected in different ways, making different chips depending on what the designer wants.

[rph]

ztex reached 200MHz on -3 with ISE 13.2 - it builds in about 40 minutes total (map + par) with -xt 5 -t 19.
I bet he used a lot of CPU time to find those settings..  

TBH I don't think hand placement will improve Fmax very much, although it could reduce the build time.

-rph

[eldentyrell]

Emphasis on "time"
FWIW, I don't think it's feasible to do this directly in Verilog/VHDL.  I had to write a Java library to generate the (totally illegible) Verilog.

I am reloading this thread like a crazed weasel.  I'm quite eager to see what kind of results you get from the manual placement.

Corner turns now running at 80mhz.

I know this sounds completely nuts, but there is a very, very, very remote chance of being able to cram three full pipelines onto the chip.  If that doesn't work I can put a "half-pipeline" in the empty space, although this takes a bit more than 50% of the area of a full pipeline (since I can't hardwire the K-values into the LUT equations anymore).  So I think the clock:hashes ratio will be at least 2clocks:2.5hashes.

All of my results are on the less-expensive -2 chip (not the more-expensive -3).

The next week will probably be pretty quiet, I have a major non-bitcoin deadline I have to deal with.  Progress should resume after 9-Nov.

[eldentyrell]

TBH I don't think hand placement will improve Fmax very much, although it could reduce the build time.

If you leave half the chip empty, then yes -- automatic placement will get pretty much the same frequency as hand placement.

If the chip is nearly full, automatic placement won't come even close to hand-placement -- if it can finish at all.

It's been three weeks since PAR has been able to finish my design with the RLOCs turned off at any frequency -- I even tried at 1mhz!  If the placement is crap there simply aren't enough wires to get from point A to point B.

It also helps to do your layout keeping in mind the wiring structure of the Spartan switchboxes: fast-path between slices in the same CLB, the 1x 2x 4x routing lines, and anything longer than that is too slow to be worth thinking about (unless it's a pure register-to-register path with no combinational logic).

[sadpandatech]

TBH I don't think hand placement will improve Fmax very much, although it could reduce the build time.

If you leave half the chip empty, then yes -- automatic placement will get pretty much the same frequency as hand placement.

If the chip is nearly full, automatic placement won't come even close to hand-placement -- if it can finish at all.

It's been three weeks since PAR has been able to finish my design with the RLOCs turned off at any frequency -- I even tried at 1mhz!  If the placement is crap there simply aren't enough wires to get from point A to point B.

It also helps to do your layout keeping in mind the wiring structure of the Spartan switchboxes: fast-path between slices in the same CLB, the 1x 2x 4x routing lines, and anything longer than that is too slow to be worth thinking about (unless it's a pure register-to-register path with no combinational logic).

  MMM, talk nerdy to me! Subbed just to follow. Though I am curious to know what you did to improve the corner turns and do you already have solutions in mind to improve them further?

[pusle]

Cudos to you for doing this manual routing! it's one hell of a job 

a couple of ideas:

Perhaps  you could use the dedicated busses between the DSP blocks to help out with the speed of the "corner turns".
Or put RAM blocks between there with a few cycles pipe delay to separate the "trees"

It might also be worth investigating smaller devices to get a better "fit" since the cost pr LUT is fairly constant.
They are also easier to cool and might run faster and/or have more headroom for overclocking

[BTCurious]

I'm trying to derive the theory from reading the FPGA threads, and some rudimentary knowledge. I'm curious if I'm close, can anyone let me know if I'm wrong?

FPGAs (Field programmable gate arrays) are collections of logic gates on a chip. The gate types themselves can be changed, and the wiring between them can be changed, so you can make your own high speed chip layout.

Every timing tick, all gates "do their process", and update their outputs, based on what they just had as inputs.

Corner turns: Is this like, all processing is flowing to the right, and then at the end of the chip you need to do some wiring or tricks to continue processing to the left?