Free 4xSpartan 6 DIY design and schematics!!!

[dropt]

Just wondering..... Why would I want to hook up fan headers to my converter thats running at 1.2v and put more load oon it when there is perfectly good 12v and 5v lines coming from a PC power supply?

This.  All that is needed is a 2pin male fan connector and a trace to the 12V input.  Nothing more complicated than that is necessary. 

You're both completely right, brain fart as I wrote the reply in a rush.

I'm partial to using PSpice, but that's just because it's what I was taught.

[BTC-engineer]

Hi stcupp thanks for sharing your DIY design with the community.

I'm a hw-engineer, and also quite experienced with EAGLE.  I did not yet look longer time on your design. Maybe I will find some time this weekend.

But a few thinks directly came into my mind.

You mentioned to choose the FT232, because it will already be used in other FPGA miner designs, so you expect to be compatible with the other bitstreams.
Unfortunately I also haven't looked how it will be done in the X6500 design, so maybe my worries are stupid, but did you check that the wiring between the FT232 and the FPGA's is exactly the same?
From your signal names it looks like you would like to use the JTAG pins of the FPGA's for communication. Is this how the X6500 design is doing the communication? Are you sure it doesn't matter for the existing mining-software/bitstream if there are 1,2 or 4 or even more FPGA's in the chain? I don't know how you wire the daughter boards with the FPGA on board, but from your backplane I can see that the TDO (output) of one daughter board is not going to the TDI (input) of the next board and so on (JTAG chain). All TDO's/TDI's are on the same pinnumber.  I don't understand this kind of wiring.   

[TheSeven]

You mentioned to choose the FT232,

Not really a good choice...

because it will already be used in other FPGA miner designs, so you expect to be compatible with the other bitstreams.
Unfortunately I also haven't looked how it will be done in the X6500 design, so maybe my worries are stupid, but did you check that the wiring between the FT232 and the FPGA's is exactly the same?

It isn't. And the X6500 will move away from using the FT232 soon, because that's really awful from a software point of view.

From your signal names it looks like you would like to use the JTAG pins of the FPGA's for communication. Is this how the X6500 design is doing the communication?

Yes.

Are you sure it doesn't matter for the existing mining-software/bitstream if there are 1,2 or 4 or even more FPGA's in the chain?

The current software for the X6500 only supports one setup: 2 FPGAs, hooked up to one set of JTAG pins each. Multiple devices in a chain aren't supported, and there is no reason to add support for that because that whole interface will be redesigned.

I don't know how you wire the daughter boards with the FPGA on board, but from your backplane I can see that the TDO (output) of one daughter board is not going to the TDI (input) of the next board and so on (JTAG chain). All TDO's/TDI's are on the same pinnumber.  I don't understand this kind of wiring.   

That looks just broken.

[BTC-engineer]

You mentioned to choose the FT232,

Not really a good choice...

because it will already be used in other FPGA miner designs, so you expect to be compatible with the other bitstreams.
Unfortunately I also haven't looked how it will be done in the X6500 design, so maybe my worries are stupid, but did you check that the wiring between the FT232 and the FPGA's is exactly the same?

It isn't. And the X6500 will move away from using the FT232 soon, because that's really awful from a software point of view.

From your signal names it looks like you would like to use the JTAG pins of the FPGA's for communication. Is this how the X6500 design is doing the communication?

Yes.

Are you sure it doesn't matter for the existing mining-software/bitstream if there are 1,2 or 4 or even more FPGA's in the chain?

The current software for the X6500 only supports one setup: 2 FPGAs, hooked up to one set of JTAG pins each. Multiple devices in a chain aren't supported, and there is no reason to add support for that because that whole interface will be redesigned.

I don't know how you wire the daughter boards with the FPGA on board, but from your backplane I can see that the TDO (output) of one daughter board is not going to the TDI (input) of the next board and so on (JTAG chain). All TDO's/TDI's are on the same pinnumber.  I don't understand this kind of wiring.  

That looks just broken.

Ok, so it looks that stcupp's current design will not work with existing bitstreams of other designs and has to be changed. This feedback should help him.
It looks like stcupp has to decide if he would go one with a design >2FPGA's and if, he has to change his wiring (maybe use multiple FT232's in his design, 1*FT232 per 2FPGA's), then it should look for the existing software simply like multiple FPGA mining boards.
Or he is thinking about another solution then 1*FT232 per 2FPGA's. In this case he has additional HW design effort (maybe a uC), the mining sofware has to be changed and maybe also the FPGA bitstreams. This is all together a lot of work, especially if you have to do it alone.
I just took a quick look on the newest X6500 rev3 design (picture of the board). Even on rev3 is still a FT232.

TheSeven, do you know what is planned as a replacement for the FT232 solution in newer designs of the X6500?
Or what to you recommend stcupp to use between his FPGA's and the Host-CPU?

ZTEX is using a Cypress uC, but his design only handles one FPGA.

[TheSeven]

Ok, so it looks that stcupp's current design will not work with existing bitstreams of other designs and has to be changed. This feedback should help him.
It looks like stcupp has to decide if he would go one with a design >2FPGA's and if, he has to change his wiring (maybe use multiple FT232's in his design, 1*FT232 per 2FPGA's), then it should look for the existing software simply like multiple FPGA mining boards.
Or he is thinking about another solution then 1*FT232 per 2FPGA's. In this case he has additional HW design effort (maybe a uC), the mining sofware has to be changed and maybe also the FPGA bitstreams. This is all together a lot of work, especially if you have to do it alone.
I just took a quick look on the newest X6500 rev3 design (picture of the board). Even on rev3 is still a FT232.

TheSeven, do you know what is planned as a replacement for the FT232 solution in newer designs of the X6500?
Or what to you recommend stcupp to use between his FPGA's and the Host-CPU?

ZTEX is using a Cypress uC, but his design only handles one FPGA.

Yes, the rev3 is basically a rev2 + temperature sensors + fan headers + heatsink mounting holes, there haven't been any interface changes.
The next generation will have a µC, most likely some STM32 part. Those are very well capable of handling 4 FPGAs, or even more if desired.

I just don't think you'll be happy with an FT232-based solution.

We really need a Spartan6 mining core hard macro, so that you can resynthesize just the interface part without affecting the mining core itself. That would make adapting the FPGA bitstream to different PC interfaces a lot easier.

[jake262144]

I love it but honestly I would like to see a "bigger one".

PCIe 6/8 pin connector is good for 150W.  I would love to see a backplane which gets power from a PCIe 8 pin "socket" (and can work with either 6 pin or 8 pin connector, usb controller for connection to a host, and a 120W 12VDC to 1.2VDC supply and 12 (yes 12) connectors.  

Each board would have only a heatsink no fan and you could put 2x 140mm fans along the "long side of the board to blow cool air across all the heatsinsk.  The board could have L brackets (or just screw holes) to mount the fans and 2x 3pin fan headers to power the fans.

210 MH/s * 12 = ~2.5 GH/s fully loaded and user could add daughter cards incrementally.  An 900 to 1200W PSU w/ 6 PCIe power connectors could power 6 boards or 12.5 GH/s.  It could be a "poor mans" Rig Box.  Start small w/ one backplane and 4 or 6 FPGA and build as you go.

Ummm... DAT, the ATX spec regulates the GPU power consumption, not the connector itself.
The fact that a GPU is lmited to 75/150W does not imply the connector itself can't deliver (much) more power.

The PEG-6 connector is actually the Molex Mini-Fit Jr connector configured with HCS contacts. These are rated for 13A each.
With three 12V wires/contacts, you're looking at 3*13*12 = 468W max theoretical load.
To be on the safe side, take 80% of that value and you are looking at 375W with a single PEG-6 connector.
The PEG-8 connector is pretty much the same as PEG-6 as the number of 12V wires/contacts is unchanged.

[stcupp]

Before I made this design I looked into the FT232R a bit and how it works with the software....

The current software x6500 wouldn't work because the jtag connections on the spartan 6's arn't daisy chained together so yes there would have to be a little bit of software modification.....

The x6500 actually uses 2 different jtag chains 1 for each spartan 6.

And the reason the FT232R isnt really a good chip choice is because it doesnt support jtag! they're using bitbang mode on the chip to use jtag

http://vak.ru/doku.php/proj/bitbang/bitbang-jtag

Also the chaining on my board isnt broken lol thats what the jumpers are for the middle pin goes to the tdo of a spartan 6 and you can jump it to either go back to the tdo of the ft232r or jump it to go to the tdi of the next spartan 6.

This makes it so you can just add spartan 6 boards  and wont need all 4 to complete the daisy chain

say you only got the money for one spartan 6 and youd jump the tdo back to the ft232r

but then you add another spartan6 so youd jump the first jumper to the tdi of the second spartan6 and the second jumper back the the ft232r.

[DeathAndTaxes]

Ummm... DAT, the ATX spec regulates the GPU power consumption, not the connector itself.
The fact that a GPU is lmited to 75/150W does not imply the connector itself can't deliver (much) more power.

But think about that a little harder.  What is on the other end of the wire.... the PSU.

The spec doesn't just regulate the GPU it also regulates the power the PSU must deliver.

What assumption was the PSU built for?  375W or 150W?

Assuming you can pull 375W from a connector just because the connector is rated for that is not going to make a very useful product.

Plug in, power on, instantly PSU powers off  or plug in, power on, PSU smokes.  

150W is plenty.  It can power 14 FPGAs and would work with 99.9% of PSU as simply plug and play.  More "might" work or might be very frustrating to end users, or might destroy the PSU or in the case of junk PSU might cause an electrical fire.

Doesn't seem much value in going beyond the ATX power spec given how power efficient FPGAs are.

[jake262144]

But think about that a little harder.  What is on the other end of the wire.... the PSU.
The spec doesn't just regulate the GPU it also regulates the power the PSU must deliver.
What assumption was the PSU built for?  375W or 150W?

Sorry, you're just plain wrong.

For the ease of explanation and without loss of generality let's assume our PSU is a single rail, non-modular device.
All the wires are soldered at the same place of the PCB.
Ever seen an open PSU, DAT? A whole bunch of 12V wires are soldered in one place, usually using very generous amounts of solder to improve the wire-to-wire electrical interface.
The same goes for ground wires - they all end up at the same spot.
The PSU circuitry doesn't know or care whether it delivers 375W across two PEG connectors and a bunch of molex 4 pin peripheral connectors or just one connector.
The PSU was built for one assumption only: to deliver 100% of the rated power as cleanly as possible - this is a prerequisite for receiving the 80 PLUS certification.

While it is true that the wires themselves have their rated amperage it is much higher than the rating of any commonly used connector and could only pose a problem if the manufacturer used substandard cabling. The spec recommends 18 AWG as the minimum wire gauge and only crappy low-end, low-power units are equipped with 20 AWG wiring when the manufacturer needs to cut corners.

The situation gets slightly more complicated when a multi-rail device is used: you need to know the shutoff threshold for the virtual rail you're drawing power from.

A modular PSU includes an additional potential point of failure - the modular board and connectors.
These are most often the same Mini-Fit Jr, HCS variety as the PEG-6 connector so no problem here.
Should the manufacturer go with some non-standard design, it's up to the user to parse the relevant specs and verify how much power the connectors can deliver.

[jake262144]

Speaking of wires, this is never a good sign:

Each PEG connector should come with its own wire to prevent overloading the PEG connector closest to the PSU.
This is another sign of a manufacturer going cheap.

If I had a PSU like this one and had to push the PEG connectors hard, I'd disassemble the PEG connector and apply solder to the place where both wires are crimped to the contact in order to minimize resistance.

[DeathAndTaxes]

Not true at all.  All PSU have current limitations for safety.  They wouldn't be UL compliant if they didn't.

So while the PSU could deliver 1200W on a single giant cable it won't.  When current on one connector exceeds the current limit it will trip.  This has been verified multiple times by many different sources when testing PSU.

Here is just one example:
http://www.jonnyguru.com/modules.php?name=NDReviews&op=Story3&reid=273

A single 1200W rail but it trips if excessive current occurs on a single connector.

Call it "Virtual rail" or "safety current limit" but you try to pull more than the PSU was designed to push on 1 wire and it will trip.  

Once again given that 150W is more than enough to power 16 FPGA (>3 GH/s) on a giant 2 foot board w/ 14 backplane slots and $2000 in FPGA (and more than enough for anything less) why make it complicated?  
Why would you need more current?  
Why take risk that it will be incompatible w/ PSU "current protection" limitations?
If you honestly needed to make a backplane which powered 30 FPGAs (>6 GH/s) would adding second PCIe connector by that much a "hassle"?


Could you get away with exceeding the spec?  Probably.  Would it work with 100% of the PSU?  No it wouldn't work in this very highly rated PSU in the link above.  As you pointed out in your PSU putting 2 PCIe connectors on one set of wires leads to more potential confusion to the end user.  If board is pulling 300W from a single connector and the user plugs two boards into 2 connectors on the same set of wires the PSU will trip unless its current protection allows 50A+ on a set of wires.

More confusion, more complications.  Simply put one PCIe 6 pin connector for each 150W you need and it is universally compatible with 100% of PSU on the market without hacking or "gotchas".  Given a 1200W PSU could power 6 boards each >3GH/s it is hardly a "limitation" to follow the spec.

[jake262144]

DAT, please... take a deep breath, re-read the spec, and ponder on it for a while.

Current version of the ATX spec abolished the 20 amps per rail limitation.
The 20A per wire limitation still stands but it's so much higher than the 13A Mini-Fit Jr per-contact limitation that it's pretty much moot - we can't violate this limitation without burning the connector first.
This 20A per wire limit is not enforced on hardware level.

A single-rail PSU does not measure the load at any individual DC output, only total OCP (over current protection) is hopefully active (BTW, I never advocated crossing the global OCP).
That's precisely the difference between single rail and multi rail devices: a multi-rail PSU measures load levels at each configured virtual rail.
I said that in the case of a multi rail PSU you need to take the per-rail OCP limit levels into consideration.

I don't like repeating myself but let's look at what I disagreed with you about:

You said that a PEG-6/8 connector is only good for 75/150 watts (6.25/12.5 amps).
Using ATX spec and manufacturer data I proved that neither the wires themselves nor the connectors are limited to such low amperage.
With three 12V wires, the PEG-6 connector is in ATX spec until 60A wire-wise.
The connector contacts max out at 13A thus limit the connector to 39A (468W). I suggested a safer value of 375W as the one to go with.
A multi-rail PSU might impose a current limit on the rail, e.g. 25A (300W) but a single-rail device cannot.
Here are a few examples of how single rail PSUs are wired up internally - take a glance at the PCB pictures, see how the wires are soldered in together.  If you ever took a class in electronics and circuits or electrical engineering the PCB picture should tell you a lot.

I know you're an expert in software and not in hardware but that's really all there is to single rail PSUs. Not exactly rocket science.
A single-rail unit cannot shut off because one of the connectors is overloaded as long as it's not overloaded high enough to trip the global OCP limit.

The ATX spec limits the PCIe devices, not connectors. Re-read it if you don't believe me.
That's the only thing I disagreed with you about.

TL;DR
There's no hacking involved nor "gotchas".
When using a multi-rail unit, you need to be aware of the individual per-rail OCP limits.

In a single-rail device the only limit you need to worry about is global load - a sum of all DC outputs.
A connector consists of wires (the spec mandates 20A per wire limitation but no PSU actually checks that) and contacts (the Mini-Fit Jr HCS are rated at 13A each).
The ATX spec limitation of 75W/150W applies to devices, not to connectors.
Electrically, the PEG-6 connector is capable of 468W. I suggested 80% of that value as a safer load.

[jake262144]

Here is just one example:
http://www.jonnyguru.com/modules.php?name=NDReviews&op=Story3&reid=273

Damn it, it is a QUAD rail PSU you're linking to.
Four 12V rails with their individual OCP limits to worry about instead of one!

Look at the load table:
[  3.3V  |  5V  |  12V1  |  12V2  |  12V3  |  12V4  ]
[  25A   |  25A |  30A   |  30A    |  45A   |   45A   ]

Try actually reading the site you're linking to:

With a single +12V rail power supply, I usually don't pay too much mind to what plugs in where. I typically just plug in as many connectors as possible into the front of my load tester and fire it up. But for this power supply, I couldn't do a full load and was wondering if the power supply was really a 1250W or if it was tripping off because of the high temperatures. I even ran my tests at room temperature and the power supply still shut down.

XFX ensured me that the power supply should do 1250W without issue and sent me a second unit. I plugged the second unit into the load tester and experienced the same problem. At high loads, the power supply kept shutting down.

So what's the deal? Well, it turns out that EasyRail isn't EasyRail after all. This power supply actually has four +12V rails. The way I hooked up the cables to the power supply, and the way the +12V is distributed was causing the OCP to trip. I was plugging the 4+4-pin into my load tester and the 4+4-pin shares a +12V rail with the 24-pin connector; so putting a load on both of these connectors was putting the total load on one +12V rail.

The "Easy Rail" is just meaningless marketroid blabbering with no real life implications.
XFX used to use this keyword to indicate single-rail devices but apparently every PSU they make is now advertized as Easy Rail, no matter the number of rails.

[jake262144]

Once again given that 150W is more than enough to power 16 FPGA (>3 GH/s) on a giant 2 foot board w/ 14 backplane slots and $2000 in FPGA (and more than enough for anything less) why make it complicated?  

You imply that delivering more than 75 watts through the PEG-6 connector is in violation of the spec.
For the n-th time, the spec regulates the GPU, not the connectors.
For reasons already stated, the PEG-6 connector is electrically capable of delivering over 400W and it will do so unless the rail it's feeding from has a lower OCP limit.

As to the merit of the whole discussion, let's assume we're using BFL FPGA devices each drawing 90W.
You imply that a single PEG-6 connector could at most power one such device and would be overloaded doing so (75W vs 90W).

I have shown that the PEG-6 connector can safely run three BFL FPGAs, each of them feeding off its own 12V wire (there are three of those).
All that without "hacking" or "gotchas" you accused me of, all in-spec.

With some additional work (merging the three 12V wires' output together and feeding the FPGAs from that) the same PEG-6 connector can safely power four BFL FPGAs without getting close to manufacturers' ratings for the wires or contacts.

[DeathAndTaxes]

For reasons already stated, the PEG-6 connector is electrically capable of delivering over 400W and it will do so unless the rail it's feeding from has a lower OCP limit.

EXACTLY.  THAT WAS MY ENTIRE DAMN POINT.  If you draw only 150W from a PCIE connector then you are GUARANTEED UNIVERSAL COMPATIBILITY with all existing PSU.  If you draw more well it may or may not work.

As to the merit of the whole discussion, let's assume we're using BFL FPGA devices each drawing 90W.
You claim that a single PEG-6 connector could at most power one such device and would be overloaded doing so (75W vs 90W).

Well we aren't talking about BFL.  We are talking about Spartan however if you are going to derail threads and debate how about you STOP PUTTING WORDS IN MY MOUTH.  I never said 75W I said 150W.  150W DC can drive 2x BFL FPGA.

Can you draw more?  Sure and it may not work with all PSU.  You may cause OCP trips, or you may overload the PSU.   Given the # of PCIe connectors directly scales to the # wattage of the device I see it as a non-issue.

150W = universal compatibility.
A unit w/ 4 PCIe connectors lets you draw 600W
A unit w/ 6 PCIe connectors lets you draw 900W
A unit w/ 8 PCIe connectors lets you draw 1200W

You will notice almost all 1200W PSU have .... drumroll ... 8 PCIe connectors and 900W units have ... 6 PCIe connectors.

Pushing more through a connector provides absolutely no benefit and may not work with all PSU.

If you pull 150W from a PCIe connector it is going to work w/ every single PSU.  If you pull more it is going to trip OCP on some PSU.  It will work for some and not work for others.   A lot will depend on the OCP limits, "virtual rail configuration", and how many connectors per wire (as you pointed out sometimes 2 are put on a single wire).

Given 150W is enough to power 15 Spartan-6 FPGA trying to do some >150W seems stupid.  I get you disagree.   If hypothetically someone made a board with 30 Spartan-6 I would recommend they use 2x PCIe connectors.  Wow a whole $2.30 in cost to guarantee universal compatiblity with all ATX PSU.  Seems good insurance to me.  You would say use only one even if the board uses 400W.  My recommendation would work on every single power supply on the planet, yours wouldn't.  Seems foolish to take risks going >150W when PSU MAY not be able to supply it without issue.

I get you disagree.  For highest compatitbility with ALL existing ATX PSU it simply doesn't make sense to go over 150W IMHO

I won't say anything further as we have been disrespectful and totally derailed this topic.   I won't see any responses.

[Lis]

read

[testconpastas2]

Nice and interesting thread.

[jake262144]

My last post in this topic, I swear.

EXACTLY.  THAT WAS MY ENTIRE DAMN POINT.  If you draw only 150W from a PCIE connector then you are GUARANTEED UNIVERSAL COMPATIBILITY with all existing PSU.  

150W is only 12.5A - no contemporary PSU has OCP limit that set low.
While it's true that 150W will always work I'd like to point out that more power will always be available.
The information how much more load the rail can handle is right there on the side of every PSU in the load table: read the amperage, divide it by 12 and voila! - you know how much power you can pull without tripping the OCP.


Please note how counterproductive sticking to 150W per PEG connector is on high-end single rail PSUs miners have heavily invested into: you'd be using less than 40% of the power available through the PEG connector.
Maximizing the number of FPGAs per PSU is very important for anyone building an FPGA farm and following my advice allows to do that cleanly, without using unnecessary cables or gadgets like molex adapters - that was my entire point.


Let me draw an analogy - 640x480 is a standard resolution any monitor in the world should be able to display.
You're saying "Stick to 640x480 and you're guaranteed universal compatibilty".
My advice is "Any contemporary device can do better than that. 640X480 is counterproductive as heck so read the manufacturer's paperwork to find the maximum safe resolution (load per a 12V rail powering the PEG connector) if you need more".

[triplehelix]

i'm not at a level to understand all the tech details.  where is this from a dev standpoint?  could someone give the schematics to a manufacturer and get a reliable product?  whats the max hashrate?

[triplehelix]

btw, my interest is for personal rigs, not commercial interests.  just wanted to be clear on that in case there were any concerns, or if commercial use of the design would violate even the spirit of the project.