X6500 Custom FPGA Miner

[sadpandatech]

That is a ridiculously awesome system. Although, it seems like an awe fully expensive way to get to that much power.

In the long run, its extremely cheap!

Nobody wants to think about it but ... what happens when / if BTC fails "in that long run"

Loving to see you try sell those FPGAs

For most of the guys with the funds to buy these kinds of setups it's a non issue. Atleast unless there was some reason to believe BTC would fall. The alternaitves are to send another couple grand to ur broker, but for what? None of them are performing for crap right now and are just as likely to lose your money too. Atleast with this, you hold it and control it.

[O_Shovah]

News from the waterfront...


As promised i created a watercoolingblock for the x6500.

Some pixels

                           

[99Percent]

News from the waterfront...


As promised i created a watercoolingblock for the x6500.

Some pixels

I am interested!

[Pipesnake]

News from the waterfront...


As promised i created a watercoolingblock for the x6500.

Some pixels

I am interested!

+1

[fizzisist]

News from the waterfront...

As promised i created a watercoolingblock for the x6500.

That looks really nice, O_Shovah! I like how you wrapped around to keep the regulators cool, too. Very clever!

[fizzisist]

OK, I'm not using X6500 boards, and my array is half the size of yours (25x Ztex 1.15x boards, plus my initial 1.15d dev board - forgetting the 9 Ghash of GPUs, costing me a fortune and to be switched off soon...) - but I'm having aggro with my systems recognising all the USB devices on the bus.

Theoretically, I should be able to connect 127 of the Ztex single boards to one computer, on one USB bus.

However, this just isn't happening... and that is fucked up, fucked up..

(sorry, listening to Thom Yorke)

There's definitely nothing wrong with the Ztex boards - they're performing brilliantly in idiotically insane environments - ambient air 38˚C, no typo, cheap £3 4-port daisy-chained Amazon USB2 hubs, and MUCH WORSE than I imagined... 5-to-1 2.1/5.5mm barrel plug daisy-chain cable rated at 60W but gets very hot to touch under load, and on top of this - unsupported miner platform... I run Mac OS X and recompiled Stefan's entire SDK and bitminer-src for Mac OS X Snow Leopard on both 32-bit (hackintosh - Dell Mini v10 - double unsupported), and 64-bit (proper full-blown Mac Pro workstation loaded to the gills)...

It appears that you have a similarly sized USB hub / chain problem that I do. I need to connect 25 USB cables to one netbook running Mac OS X.

Stefan explained why my cables were hot - and I now have a solution for that problem.

But the issue remains - are these cheapo Amazon 4-port USB2 hubs simply a disaster waiting to happen? What have you used for all the USB connections you need? Or have you used multiple logic boards? One option for me would be to simply plug the USB cables into my modular GPU miners, which have one logic board (with 6 USB sockets, normally) and 4 GPUs. This should work, but I don't want to be running 5 high-power ATX PSUs when I'm intent on selling off the 6950 / 5850 / 5830 / 5770 GPUs (no use to me as a Mac user, other than the 5770s, which I'll keep to make a 4-GPU Hackintosh tower with 16 GB ram and the i5-2500 CPU clocked to the moon, running Snow Leopard of course).

It's overkill to use separate logic boards and CPUs (wasted power). But high quality 10x USB2 hubs cost a lot of £££ and, if they require their own power bricks, then it defeats the object of consolidating points of failure. A single cheap USB2 hub could easily short out (I've taken photos of the internals of these things, and it's embarrasing - I am awful at soldering, but I could do a better job than these chaps).

Also, part of the project goals include the ability to make the entire structure portable, with a single power input and a single Ethernet output. It also has to look good in Jonathan Ive fashion (i.e. the downside of being an Apple hacker and concomitant need for good industrial design...)

Stefan (Ztex) has given me a good wiring diagram to power the FPGAs. But I'm still left with all 25 of the boards needing USB connections. Since USB2 spec allows for 4-port and 7-port hubs (IIRC), any other number (such as the 10-port hubs) is actually a 4-port and 7-port chained together internally.

This is where quality comes in. I've got cheap hubs... but are the crappiest of hubs 'adequate and functional' if the power supply to the FPGAs is clean, low resistance and properly soldered (i.e. no hot spots)?

After all, I'm looking at 12VDC at 5A - and 5 amps is quite a current for a cheap USB2 hub with ethernet-thin internal cabling...

Any top tips? If simply sorting the PSU to present a low resistance to the FPGAs will allow the cheapest of crap USB hubs, then I'll stick with what I've got... but if quality USB hubs are required... I'd appreciate any recommendations for the USB daisy-chain stage. After all, if I ever get to 127 FPGAs on one bus, the USB hubs need to work. I've had problems with even only connecting *5* to a single port...

I'm a little confused by what's going on here. Maybe you can help me understand before I get too far debugging this. What it sounds like is that the power supply ground is not good enough, so some significant current is returning over the USB cables, making them hot. Is this your understanding?

If so, I don't think the USB hubs can really be to blame (although powering them could potentially help, since it might help to keep their grounds from floating relative to the host). The best thing to do is to beef up the grounding to the power supply. This could be done by adding another wire in parallel, increasing the wire guage, shortening the wires, or making sure they are "star connected," meaning they all have their own personal path back to a common ground point, instead of the grounds all running in series.

Let me know if I'm on the right track.

[Garr255]

As awesome as that waterblock is, we need to keep in mind that the fpga itself has a plastic casing, making this considerably less efficient.

What temps do you get at 200mhz?

[O_Shovah]

As awesome as that waterblock is, we need to keep in mind that the fpga itself has a plastic casing, making this considerably less efficient.

What temps do you get at 200mhz?

I have an early revision, so there is no thermal sensor on the PCB.
But according to my thermal images the voltage regulator temps dropped slightly (~2°C)
and the bottom PCB temp dropped by 3-4 °C ( The two FPGA's differ a little).

It's running on ~ 194 Mhz and haven't applied the resistor swap yet.

I only cool the FPGA housings so the plastic is certainly a problem.

Another revision will be made afther i have finalised the cooler for the Bitforcebox  Rev 03.


Is there anybody seriously interested in gettin such a block ? And wich price range would be interesting ?

[tgmarks]

As awesome as that waterblock is, we need to keep in mind that the fpga itself has a plastic casing, making this considerably less efficient.

What temps do you get at 200mhz?

I have an early revision, so there is no thermal sensor on the PCB.
But according to my thermal images the voltage regulator temps dropped slightly (~2°C)
and the bottom PCB temp dropped by 3-4 °C ( The two FPGA's differ a little).

It's running on ~ 194 Mhz and haven't applied the resistor swap yet.

I only cool the FPGA housings so the plastic is certainly a problem.

Another revision will be made afther i have finalised the cooler for the Bitforcebox  Rev 03.


Is there anybody seriously interested in gettin such a block ? And wich price range would be interesting ?

I think a block like this could be very appealing once some overclocker bitstream comes out.  For the time being though, I can go as fast as I want on the current bitstreams and air is all I need.  Not worth the hassle of water yet.

[hashking]

Here's my 10GH/s rig I just set up today using 25 x6500 FPGA boards:

Are you using just one power supply for all your boards?

[Garr255]

Here's my 10GH/s rig I just set up today using 25 x6500 FPGA boards:

Are you using just one power supply for all your boards?

That's what it looks like. 750W is more than sufficient.

[bonks]

In this picture it was using one psu and I had everything connected to two molex cables and some splitters... which I learned was very bad.  So I ordered a few PCIE to barrel adapter splitters from Cablez which can handle the load and in the meantime I'm actually running each stack on its own PSU.

Here's my 10GH/s rig I just set up today using 25 x6500 FPGA boards:

Are you using just one power supply for all your boards?

That's what it looks like. 750W is more than sufficient.

[Garr255]

In this picture it was using one psu and I had everything connected to two molex cables and some splitters... which I learned was very bad.  So I ordered a few PCIE to barrel adapter splitters from Cablez which can handle the load and in the meantime I'm actually running each stack on its own PSU.

Here's my 10GH/s rig I just set up today using 25 x6500 FPGA boards:

Are you using just one power supply for all your boards?

That's what it looks like. 750W is more than sufficient.

Ohh good call. I assumed you had already done that.

[punningclan]

That is a ridiculously awesome system. Although, it seems like an awe fully expensive way to get to that much power.

In the long run, its extremely cheap!

Nobody wants to think about it but ... what happens when / if BTC fails "in that long run"

Loving to see you try sell those FPGAs

Unlike ASICs FPGA's can easily be repurposed, most of these devices come with JTAG or some other way of upgrading the firmware, luckily.

[O_Shovah]

I have a question regarding the exchange of  the R2 and R8 resistors.

Wich voltage cuircut is affected by the resistors ? I guess the 1,1 V core voltage.

From wich to wich voltage this change of the resistors would be effectual ?

And wich other voltages would be interresting.


On funy fact is:

I use the water cooling i postet.

While using the autoscaling bitstream i get 190 Mhz/~2% stale on FPGA 1 and 172/6%stale Mhz on FPGA 0 with a 2 °C temperature
increase to the second one.

I cant really spot any difference in the cooling power on those two sides of the PCB.

But while i use the old 200 Mhz bitstream i get 200Mhz on both an ~ 0,1 % stales on both.

So  nanisore ?

[fizzisist]

I have a question regarding the exchange of  the R2 and R8 resistors.

Wich voltage cuircut is affected by the resistors ? I guess the 1,1 V core voltage.

From wich to wich voltage this change of the resistors would be effectual ?

And wich other voltages would be interresting.


On funy fact is:

I use the water cooling i postet.

While using the autoscaling bitstream i get 190 Mhz/~2% stale on FPGA 1 and 172/6%stale Mhz on FPGA 0 with a 2 °C temperature
increase to the second one.

I cant really spot any difference in the cooling power on those two sides of the PCB.

But while i use the old 200 Mhz bitstream i get 200Mhz on both an ~ 0,1 % stales on both.

So  nanisore ?

To change VCCINT for each FPGA from 1.2V to 1.23V, you can replace the 1.91k resistors (R2 and R8) with 1.82k resistors. You can use any resistor values in between these extremes to get different voltages, if you were curious about experimenting.

You'll see variations in the acceptable clock using the overclocker firmware because of variations in the silicon, and it looks like this variation is more important than temperature, at least within some reasonable range of temp. With sufficient cooling and 1.23 V core voltage, it looks like typical clock speeds using the x6500-overclocker-0402 bitstream are between 192 and 208 MHz, with the average about 198 MHz with 1% invalids. Using ztexmerge_200mhz and 1.23V VCCINT, you should see more like 0.01% invalids.

[99Percent]

I am proud to report that according to BTC Guild one of my X6500 found a block

[Garr255]

I am proud to report that according to BTC Guild one of my X6500 found a block

Cool :}
Now switch to P2Pool

[nbtcminer]

@Anyone:

Questions about power for the X6500 Rev 2.0:

1.) I know that you can get 5v power from the on-board molex connector on the Rev 2.0; it is possible to power it the 12v though the board?

1.)a.) If not, would something like this work? One of the Male would goto the board and the other would be cut / re-wired to directly supply power to the fans with 12v.
 http://www.ebay.ca/itm/260953850104?ssPageName=STRK:MEWNX:IT&_trksid=p3984.m1497.l2649

2.) Is a 12v 2a wallwort enough to do the rev 2.0 and two Silenx fans (1.56w each) or would that be too much? If my calculations are right, the rev 2.0 + two silenx fans should use up only 1.67a of the 2a on 12v; I just need some help confirming this.


Rev 3.0:

2.) Does the rev 3.0 supply 12v power through the molex connector or only 5v?

Thanks in advance for all your help!

*Note: I'm only asking this because I want to get more CFM out of the fans I have on the unit now and I find that even though 5v is very silent, I would like to make the unit a little cooler.


Cheers,
nbtcminer

[Garr255]

Just switch the 5v(red) and 12v(yellow) cables to the molex connecter you're powering the board with.