Cairnsmore2 - What would you like?

[DiabloD3]

We are looking to make this rack stand alone and yes it is up against Butterfly's larger products. We would like to make this run free of a host PC. This is very much a big system concept. I hope we do a better product that the competitors but time will tell on that point. It's very unlikely we will do a PCIe backplane for this although we are looking at these for our general HPC products. The bought in industrial backplanes are usually very expensive so it's unlikely we would but that in. However we can use one of our standard ones that we have designed or even a derivative of one of them. The cost is quite reasonable doing it this way.

I would forget any secondary value on any sort of mining kit. If you are banking on that your equations will be wrong. FPGA families have replacements on average every 2 years and the old family will have limited value even still as chips never mind in a system that has either to be reused or silicon recovered. GPUs are even worse for this. Try selling a 2-3 year old GPU. It might have cost £500 but in 2/3 years you can buy a brand new board of equivalent performance usually for less than £100. Second hand maybe it goes for £30. I for one would not want to buy a ex-mining GPU given the stress put on them but of course most people don't mention that on Ebay.

The only thing you need to focus on is total cost of ownership per MH without lowering the product quality like BFL has. A 4U rig with boards that can be installed after purchase would get a lot of people on board: however, 4U high density implies 2000w redundant 208/240v-only PSUs, and a lot of people in the US and a lot of DCs in the US simply do not have access to that.

So, maybe a 2U rig with half-height boards in a custom case that does 1200w max? As long as the density is as high as possible, you will have customers.

Plus, if you make this a generic rig and sell different sorts of boards for this, you essentially have an FPGA blade server. You could sell mining boards along with non-mining boards that your non-mining customers buy, and non-mining customers could very well buy mining oriented boards to combine them with non-mining boards to handle whatever they need.

Mining FPGAs, as far as I can tell, are largely just FPGA designs with high amp VRMs and no external memory chips and no high speed IO wired in. If you just need computation power without local memory or mass IO, mining boards really would be useful to you.

Centralizing all your FPGA products into a single FPGA-blade design would lower the cost for all customers theoretically.

[mrb]

My advice to you, yohan:

• Definitely use Ethernet. Not USB: the maximum cable length is only 5 meters, and large scale miners have racks spaced out my much more than 5 meters. Not cable PCIe: it is overkill, too expensive, and its extra bandwidth unnecessary.
  
• If you put an embedded PC in the unit, use USB to link it to the internal FPGA boards. You don't necessarily have to use USB cables, instead you may want to design a backplane populated with SATA data and power connectors repurposed to carry the USB signal (over the SATA data connector) and power to the FPGA boards (over the SATA power connector). The boards would be plugged into the backplane, much like a SATA drive is plugged into the SATA backplane of an x86 server. Put a USB hub controler IC on the backplane to make it a USB switch (1 upstream link to the embedded PC, multiple downstream links to the FPGA boards). A SATA power connector is only rated 4.5A for the 3.3V line, 4.5A @ 5V, and 4.5A @ 12V, therefore I would suggest to repurpose the useless 3.3V and 5V rails to 12V, giving you a total of 13.5A for 12V, or 162W per connector, which should be sufficient for a board with up to 16 LX150's. BFL seems to be following a similar idea by carrying USB signals over SATA cables (not backplane connectors though) in their mini rig.
• Use 19" rackable chassis. And make them at least 2U. Rationale: easier to cool, and bigger more efficient fans can be installed in them, compared to the 41 mm fans in 1U chassis (this is why Facebook uses 1.5U chassis in their Prineville datacenter instead of 1U: http://opencompute.org/wp/wp-content/uploads/2011/07/Server-Chassis-Specifications.pdf )
  
• Use commodity ATX power supply units. And allow users to purchase your chassis without PSUs. A lot of miners like myself have invested in efficient PSUs for their GPU rigs, and would like to re-use them. The 20/24-pin connector can power the embedded PC, while the 6- or 8-pin PCIe power connectors can power the backplane, which powers the FPGA boards.
  
• Keep it simple stupid. Fewer components means less chance of hardware failure, reduced costs, and reduced time-to-market. Particularly: (1) no LCD, and (2) no Wifi. Rationale: (1) I want to remotely configure and monitor the FPGA unit over a web interface, I don't want to deal with an LCD display and buttons that require me to be physically present in front of the unit, and (2) large miners are likely to already have cat5 deployed in their datacenters and Wifi is unreliable in some of these environments.
  
• Temperature probes for each FPGA.
  
• Fan speed monitoring (however don't necessarily make them PWM controllable)
  
• Easily replaceable fans (like some 1U chassis where fans are not screwed in, but can be slided in and out of a plastic frame, with rubber to absorb vibrations).
  
• Size each unit so that its fully-loaded configuration is about 960W. This will allow users to put exactly 2 of these units per 120V-20A circuit, or 4 per 240V-20A circuit, or 9 per 208V-30A 3-phase circuit (while not exceeding 80% of the circuit's maximum current rating). Rationale: in datacenter environments, users pay a fixed monthly price per circuit. A circuit not used at its maximum capacity (or a config not fully-loaded to not over-consume) is wasted money. BFL failed to follow this advice of mine when designing their 1250W mini rig.
  

[DiabloD3]

My advice to you, yohan:

Size each unit so that its fully-loaded configuration is about 960W. This will allow users to put exactly 2 of these units per 120V-20A circuit, or 4 per 240V-20A circuit, or 9 per 208V-30A 3-phase circuit (while not exceeded 80% of its maximum current rating). Rationale: in datacenter environments, users pay a fixed monthly price per circuit. A circuit not used at its maximum capacity (or a config not fully loaded to not over-consume) is wasted money. BFL failed to follow this advice of mine when designing their 1250W mini rig.

This so much. Ye typical DC gives you two 120v 20a circuits per rack and you have to pay for more at prices that are insane (but still cheaper than residential and small business electrical prices), and from what I've seen they max out at four 120v circuits at a rack. So, if all we have is 80a of 120v, we have to make the most of it as much as possible.

Given 1 units per 10a, each unit is, say, 4u big, a 42u rack will hold 8 of these units if we put 1u space between each one (which I've been told is normal for high density units so they don't cook each other) (= 5u), 8 * 10 = 80a, so we should be fine in an average DC, plus we have 2U to spare.

[DiabloD3]

BTW, re programmable VRMs. You're right, Yohan, they're expensive.

So, what about just having a jumper that we can change to swap out some of the power circuitry (resistors, etc) so that it'll lower (Spartan 6 example voltages) from 1.2 to, say, 1.0 and cut the clock rate just as much (assuming FPGAs are stable at low voltages, I have no experience with this on FPGAs, only on CPUs and GPUs).

[Lethos]

I agree to the sentiments that a blade and/or modular like design should work out quiet well.
I did see some examples on this forum of someone who did something like that.
Not sure if it went it to big production, but it certainly looked good and impressed many.

Upgrades and replacements could become a more viable thing, allowing a low starting cost to upgrade later.

[bulanula]

LOL at the guys talking about 120V

For the US you have BFL and their crappy delays and excuses

For the EU we have yohan and this real company that delivers

Too bad we cannot import / export due to outrageous costs / VAT / duty and other gubbment crap.

I'd like to see someone in the UK import a BFL rig and somebody in the US import one of yohan's rigs.

[mtminer]

1) Cost mega hash per $
2) Back plane
3) Ethernet or USB box doesn't need to be self sustaining, controlling computer can live somewhere else.

I am not sold on the idea of it needing to fit in a computer rack. Allot of people use shelving units, floors, and benches to hold their computers. An open air cage would be fine in my opinion.

[jjshabadoo]

I agree with a modular design that can be sold and then upgraded or added to. The $15,000 BFL entry point is just focking stupid and actually it's bullshit because it kills the democratic/decentralized component of mining. You are now completely pushing out small scale miners and any form of ingenuity. This $15,000 bullshit is just a money box and will be worth .50 if bitcoin goes to shit. It's the nuclear weapon of bitcoin, once it exists you just up the ante too much and screw everyone who can't afford one.

If you guys can produce a 4-5 Gh/s base product which could be upgraded to 20-25 with additional modules added to the unit over time that would be ideal.

Please beat out BFL so we can piss on their poor business practices and customer service.  Mining is speculative enough without being forced to put out your money for months in advance and then still get delayed.

[DiabloD3]

I agree with a modular design that can be sold and then upgraded or added to. The $15,000 BFL entry point is just focking stupid and actually it's bullshit because it kills the democratic/decentralized component of mining. You are now completely pushing out small scale miners and any form of ingenuity. This $15,000 bullshit is just a money box and will be worth .50 if bitcoin goes to shit. It's the nuclear weapon of bitcoin, once it exists you just up the ante too much and screw everyone who can't afford one.

If you guys can produce a 4-5 Gh/s base product which could be upgraded to 20-25 with additional modules added to the unit over time that would be ideal.

Please beat out BFL so we can piss on their poor business practices and customer service.  Mining is speculative enough without being forced to put out your money for months in advance and then still get delayed.

OTOH, that $15k entry point means nothing if they're getting that high of a mh/$. That said, for a large scale farm, a $5k box or a $15k box, there is little inbetween. Especially since they've already sold like 25 of them and no one even has one yet.

[Keninishna]

LOL at the guys talking about 120V

For the US you have BFL and their crappy delays and excuses

For the EU we have yohan and this real company that delivers

Too bad we cannot import / export due to outrageous costs / VAT / duty and other gubbment crap.

I'd like to see someone in the UK import a BFL rig and somebody in the US import one of yohan's rigs.

I'm buying from enterpoint and I'm in the US? How much are export costs from UK to US?

[mrb]

LOL at the guys talking about 120V

My remark about sizing for 960W also applies to 240V circuits.

I agree with a modular design that can be sold and then upgraded or added to. The $15,000 BFL entry point is just focking stupid

Actually a $10-15k pricing point is fine... The way I see it, is that as long as there are mining solutions at the following price points (orders of magnitude), anybody should be able to find something for their budget:
- around $100: entry-level/mid-level video cards.
- around $1k: quad Spartan6 boards, BFL single, etc.
- around $10k: BFL mini rig, Cairnsmore2 (hopefully).

yohan: for Cairnsmore2, please arrange the fans like rackable servers: placed at the front or back of the chassis (ie. not attached to heatsinks or FPGA boards).

[yohan]

Export costs vary widely by country where it is being imported and the type of equipment it is. The key to all of this is the harmonised code which is a system that identifies the type of equipment and then the customs processor knows what duty level is to be applied. Duty is different from local sales taxes. Most countries have a mechanism for applying local/federal sales taxes on imports so you don't avoid those.

For sales between supplier and customers in different EU countries duties have been totally eliminated but sales tax (Value Added Tax - VAT) still exists and we have a system that is not totally different from intra-state system in the US. If we are selling to a domestic customer in the EU we will always apply VAT to the price. If the EU customer is a business, with a valid VAT number, we can ship without charging VAT ourselves but it is still due to the authorities local to the customer. This is backed up with a reporting system so that fraud doesn't occur and we report every VAT free shipment to a EU country.

As far as I understand it, and this is complex, US duties vary with sourcing country of the item being imported and the harmonised code used. They also used to have an associate tax called MFP which is very small, 0.02% from memory, as well. Generally as a country UK items don't get hit hard by duty as we are a friendly country. VAT does not get applied to non-EU countries. The hardminised code that we use for most of our products is a fairly low duty, or none, but Cairnsmore1 may be different especially if we ship it with bitstreams loaded i.e. a known function that changes the harmonised code. We are looking at what harmonised code we can use for Cairnsmore1. Once that is established we will publish that. Most of the couriers have duty calculators so cost can be checked.

South America countries have very high duty levels from what we have heard.

[Glasswalker]

Here are my thoughts on this (just noticed this thread, had a busy weekend lol):

• Low initial chassis cost. Chassis includes PSU and backplane, and controller. Then you can add boards at a reasonable price per board. Allows a smooth upgrade curve. The challenge is finding the sweet spot in MHash/$ but while maintaining a low "per board" cost. You want optimal MHash/$, but you don't want your boards to be much over $2K each.
• Controller should use Ethernet. Make it standalone.
• Consider power density. We want as high a density as possible, but also don't want to exceed common power density limits. I would keep the unit's power consumption under 200W/U. So if you do a 3U chassis keep it under 600W. At current FPGA tech/bitstreams (though I'm hoping that with your skilled team you can come up with a better bitstream that milks more performance per chip and per watt). that means about 15GHash/s per 3U Case. With this setup you don't need to space out the gear in the rack, and can fit 14x rigs in a rack. which puts you at about 8.4KW of draw per rack, which works with most datacenters limits (but on the high end without paying a lot more) and 210GHash/s per rack. You can go higher density than this, but at that point you're starting to fight power density at the datacenter, and thermal density issues as well.
• Monitoring! You need monitoring of as many metrics as possible. Fans, Heat, Power, everything that doesn't blow up the price too much, you should be monitoring, and reporting over either webservice, or SNMP. So that existing infrastructure monitoring tools can manage the health of the cluster
• Depending on the cost of the chassis, adding an accessory later, allowing slotting of 1 or more cairnsmore1 boards into a "blade" for this new system would provide an upgrade path for those already investing in your Gen1 product
• Bitstream upgrade abilities over ethernet. Allowing remote upgrade of individual blades (without bringing down the entire cluster)

I'm sure I have more than that, but that's the main stuff I can think of right now.

[yohan]

As you all keep asking this is what what we are doing as regards Bitcoin mining. It is intended only as a test board for Goliath technology and yes it does look strangely familiar but you never know we might spin it as a product. It doesn't have the full Goliath functionality which is why we might sell it but it does have enough to be a competative Bitcoin solution. Please don't ask on the emails for more info or pricing as we are very busy currently. None will be forecoming until we are ready to talk more about what we are doing. Meanwhile enjoy!

Yohan

[Clearfly]

Looking sweet Yohan, no need to re-invent the wheel  .

Don't leave it too long though, some of us have their fingers hovering over the order button.

I would much rather send my money your way than BFL's but time is money as they say.

[Beaflag VonRathburg]

Looking sweet Yohan, no need to re-invent the wheel  .

Don't leave it too long though, some of us have their fingers hovering over the order button.

I would much rather send my money your way than BFL's but time is money as they say.

The nice thing is that they've proven they can go from concept to working product relatively quickly.

[Slipbye]

Gets the thumbs up from me 

[spiccioli]

Looking sweet Yohan, no need to re-invent the wheel  .

Don't leave it too long though, some of us have their fingers hovering over the order button.

I would much rather send my money your way than BFL's but time is money as they say.

The nice thing is that they've proven they can go from concept to working product relatively quickly.

True,

but now they have to rush it, since difficulty is already rising fast.

And they have to have chips which cost less than 25 USD/GHs (Avalon price of 1500 USD/60 GHs) and/or use a lot less power than Avalons


spiccioli

[spiccioli]

Last but not least,

add a power cycle button, on CM1s you need to remove pcie connector and usb connetor to power cycle them.

Remove power from usb connector, makes life easier finding a usb hub and, maybe, a different usb chip which does not lockup when powering boards on like it happens on a couple of my CM1s.

spiccioli

[yohan]

Last but not least,

add a power cycle button, on CM1s you need to remove pcie connector and usb connetor to power cycle them.

Remove power from usb connector, makes life easier finding a usb hub and, maybe, a different usb chip which does not lockup when powering boards on like it happens on a couple of my CM1s.

spiccioli

At the moment this is a test board but if we are going to make it a board for sale we will probably do a tidy up. The performance we get out of the technology is more of interest to us than anything. We can simulate and do all sorts of calculations to gauge the new chip operation and performance but nothing to beat having a chip on the bench working. We knew how good the CM1 cooling solution was so we just reused that for this test board and the main reason for using the CM1 format. It won't be long until we have the performance gauge and we will decide then our development path for CM2 and the Goliath systems.