I made a giant, overpriced mining rig

[TCollar]

and to top it off I made two of them.

I decided to go with GPU rather than with the non existent ASIC.  I see it as deciding to buy a v-8 sports car for $100k rather than wait for the $1k atomic powered flying sports car to be finished (like with BFL).

I work with both clusters and co-location, so I got a good deal on a massive amount of hardware and I got a good deal on coloco (air kept at 66F all day plus a staff).

My rigs are a cluster of 4 nodes each and one embedded *BSD licensed controller of my own design.  Each one has 8 8 core cpus, 512GB of RAM and 12 GPU cards with logging and storage done in an encrypted VIA Mini ITX controlling a storage array (so an error won't crash the system by filling up system disks).  The other rig is exactly the same, located in a different location with the same colo company and has hot failover capability with the other machine using something Linux HA heartbeat.  So if one rig fails it takes over the tasks of the other rig as its virtual instances switch tasks (why I needed so much RAM).

I really think I spent too much for this ($60k for both.  if I had not got the deal I did I would have paid about $90k) and tonight or tomorrow I should be able to start mining once it is set up at my colo.

The amount I spent is making me have doubts about bitcoin to be honest but I am here and I am obviously jumping all in.  So I figured I would introduce myself.

Hi!

[papamoi]

hi
how much hashing speed it s having?

[ionux]

Hey man, that is an amazing setup.  I'm trying to get into mining myself but don't have the funds at the scale you do!  Can we talk offline, perhaps?  I'd love to pick your brain about GPU mining setups!

[TCollar]

Hey man, that is an amazing setup.  I'm trying to get into mining myself but don't have the funds at the scale you do!  Can we talk offline, perhaps?  I'd love to pick your brain about GPU mining setups!

For the first commenter:  I'll let you know tomorrow.

For Ionux.  Maybe we should see how I do first before you start getting information from me.  I might suck

But seriously, Go with ATI / Radeon.  Its gotta be as open as possible.   What I am doing now is this:

Trying to tweak my system on a theoretical computing level.  That is why I have the BSD licensed controller, so I can protect my work.  The USA is lawsuit happy and if this works like I want it to work I want to make sure that someone does not file a suit for violating the GPL or something.

Before GPU mining I would advise you make a cluster.  If you can find some old VIA mini ITX boards you can have a low power 3 node cluster than can fit in an old case.  Those boards usually have one PCI slot to experiment with an old GPU.  You wont get a mining rig that will give you plenty o' bitcoin but you will learn how to make a cluster, learn Linux and learn how to tweak a GPU.  By doing this you will learn something that could even become a profession just in case the whole bitcoin thing turns out to be a fraud someday.

What I have to do with my clusters now is see if I can arrange for cluster leasing or something in case the whole bitcoin thing is nonsense and I end up getting stuck with $60k worth of future landfill.

[gyverlb]

and to top it off I made two of them.

I decided to go with GPU rather than with the non existent ASIC.  I see it as deciding to buy a v-8 sports car for $100k rather than wait for the $1k atomic powered flying sports car to be finished (like with BFL).

You know that all the new bitcoind roads are designed for the atomic powered flying sports car and that your $100k v-8 sports car will have to remain parked in your garage?

I work with both clusters and co-location, so I got a good deal on a massive amount of hardware and I got a good deal on coloco (air kept at 66F all day plus a staff).

My rigs are a cluster

Why a cluster? Mining bitcoins is a parallel task: you don't need any failover or load balancing tech.

of 4 nodes each and one embedded *BSD licensed controller of my own design.  Each one has 8 8 core cpus, 512GB of RAM

High-end CPUs and lots of RAM? For mining? Seriously if you want to throw money away you might as well give it to me: I could invest it in hardware that would produce some income and not generate electric bills like there's no tomorrow...

and 12 GPU cards with logging and storage done in an encrypted VIA Mini ITX controlling a storage array (so an error won't crash the system by filling up system disks).

I use USB keys for each of my "nodes". On some of the first USB keys filesystems were remounted read-only on medium errors: the mining process was undisturbed...

  The other rig is exactly the same, located in a different location with the same colo company and has hot failover capability with the other machine using something Linux HA heartbeat.  So if one rig fails it takes over the tasks of the other rig as its virtual instances switch tasks (why I needed so much RAM).

I really think I spent too much for this ($60k for both.  if I had not got the deal I did I would have paid about $90k) and tonight or tomorrow I should be able to start mining once it is set up at my colo.

You think so? You have a grand total of 12*4*2 = 96 GPU. If these are 7970 you may have a total of 96 * 700MH/s (overclocked) = 67.2GH/s for $60k.

The fact that you have 12 GPUs/node will certainly be a problem (there's a limitation of 8 GPUs/server in the AMD drivers).

So you will probably be restricted to 64 GPUs -> 44.8GH/s (still for $60k). Being in a datacenter, power won't come cheap: even if you manage to put these 44.8GH/s online you may not even make a dime mining Bitcoins (and unless the BTC price rise to absurd levels you certainly will begin to lose money in one or two month).

For a point of reference I spent ~$7k for 12GH/s one year ago: your hardware is 3x less dollar efficient than mine one year later. Mining is about optimizing the cost of hardware and the power usage, your setup is simply over-engineered.

[stumpper98]

If you can post some pictures..those are some big money machines.

[dree12]

This is a gigantic waste of money. An Avalon mining rig would use less power and cost 10% of your rig, for the same amount of hashpower. Your setup is depreciating so fast it is more efficient to avoid the sunk cost fallacy and sell the entire thing now. All it will accomplish if it runs is contributing to global warming.

[BlackBison]

First: Ouch!

Second: I suggest you compare mining litecoins when the btc difficulty starts to skyrocket soon.

Very bad timing on the purchase but best of luck..hopefully you can at least break even on it

[TCollar]

Just to clarify I have 12 GPU cards.    I do things for failover and redundancy because that is how I set up everything that handles potential income.  It might be excessive but it is just a habit.

As far as ASIC goes,  How many people do you know personally with ASIC rigs? 

As far as my huge amount of memory, I will need that to run numerous instances.  One thing I am just checking out is the possibility of adding nodes even if I could virtualize an ASIC chip.  They have such low power useage and would take up low CPU and perhaps GPU cycles that if I can virtualize and ASIC instance , then maybe this rig would not be a waste of money.

My one test was a dismal failure at about 26600 Mh but I was testing on cracking phpBB3 hashes when I got that speed, so I will see for sure in a day or two.

Most likely this will be a bad investment though.  I can face that but I am going to look for ways to "hedge" a potential loss.

No pictures yet.  On the off chance this does work, I really don't want copycats with my rig chugging away at what could be my bitcoins.  If it works I might be open to "leasing" as part of my hedge plan.  Being that if it works, I lease instances as a hedge against BTC for dollars.

[Tuckie]

Very interesting choice on BSD for the OS as opposed to win/linux.  How's the driver support?

[jwzguy]

No pictures yet.  On the off chance this does work, I really don't want copycats with my rig chugging away at what could be my bitcoins.  If it works I might be open to "leasing" as part of my hedge plan.  Being that if it works, I lease instances as a hedge against BTC for dollars.

I'm guessing copycats are the absolute least of your worries. I hope you have a use for this hardware beyond mining.

[TCollar]

Very interesting choice on BSD for the OS as opposed to win/linux.  How's the driver support?

The controller is *BSD.  The rest is CentOS.

[TCollar]

No pictures yet.  On the off chance this does work, I really don't want copycats with my rig chugging away at what could be my bitcoins.  If it works I might be open to "leasing" as part of my hedge plan.  Being that if it works, I lease instances as a hedge against BTC for dollars.

I'm guessing copycats are the absolute least of your worries. I hope you have a use for this hardware beyond mining.

Heh. You are probably correct.  But it is unique on the inside but really, really boring on the outside (quad 6U cases being driven to two racks somewhere now). 

[ionux]

My one test was a dismal failure at about 26600 Mh but I was testing on cracking phpBB3 hashes when I got that speed, so I will see for sure in a day or two.

Most likely this will be a bad investment though.  I can face that but I am going to look for ways to "hedge" a potential loss.

Well, according to the Bitcoinx mining profitability calculator, that hashrate would gross $4403.21/month.  Granted, you did spend a lot for your rig, but I wouldn't call that a dismal failure!

[paraipan]

Pics or it didn't happen.

[TCollar]

Just to clarify I have 12 GPU cards.    I do things for failover and redundancy because that is how I set up everything that handles potential income.  It might be excessive but it is just a habit.

As far as ASIC goes,  How many people do you know personally with ASIC rigs? 

As far as my huge amount of memory, I will need that to run numerous instances.  One thing I am just checking out is the possibility of adding nodes even if I could virtualize an ASIC chip.  They have such low power useage and would take up low CPU and perhaps GPU cycles that if I can virtualize and ASIC instance , then maybe this rig would not be a waste of money.

My one test was a dismal failure at about 26600 Mh but I was testing on cracking phpBB3 hashes when I got that speed, so I will see for sure in a day or two.

Most likely this will be a bad investment though.  I can face that but I am going to look for ways to "hedge" a potential loss.

No pictures yet.  On the off chance this does work, I really don't want copycats with my rig chugging away at what could be my bitcoins.  If it works I might be open to "leasing" as part of my hedge plan.  Being that if it works, I lease instances as a hedge against BTC for dollars.

Failover? Redundancy? can you elaborate further on those topics? because I don't think they apply to the bitcoin world unless we are talking about a pool that needs to be up 24x7 or a market exchange, but from the miner's perspective?

Failover means that when , lets say, The State of Missouri is destroyed in an act of clumsiness typical of that state and my rig is destroyed with it, my other rig either surrenders half of its tasks and takes on the tasks of the dead Missouri rig or it takes over in full by creating more instances to make up for those lost.  This is why I need huge logging space because the failover rig will have to collect data from the failed rig.  The colos provide 24/7 AC and backup power so I figured I might as well take advantage of them. 

Redundant means they are the same and have the ability to failover.

[TCollar]

Pics or it didn't happen.

It'd be nice if lack of pics could get me a refund.

But here we are.  Things happened.

[cedivad]

Pics or it didn't happen.

+1

We want pics! Anyway, what an ugly choose! 512GB of ram, you are trolling us, correct?

Failover means that when , lets say, The State of Missouri is destroyed in an act of clumsiness typical of that state and my rig is destroyed with it, my other rig either surrenders half of its tasks and takes on the tasks of the dead Missouri rig or it takes over in full by creating more instances to make up for those lost.  This is why I need huge logging space because the failover rig will have to collect data from the failed rig.  The colos provide 24/7 AC and backup power so I figured I might as well take advantage of them.  

Redundant means they are the same and have the ability to failover.

Yes... you must be really trolling.

GG

[TCollar]

Pics or it didn't happen.

+1

We want pics! Anyway, what an ugly choose! 512GB of ram, you are trolling us, correct?

Failover means that when , lets say, The State of Missouri is destroyed in an act of clumsiness typical of that state and my rig is destroyed with it, my other rig either surrenders half of its tasks and takes on the tasks of the dead Missouri rig or it takes over in full by creating more instances to make up for those lost.  This is why I need huge logging space because the failover rig will have to collect data from the failed rig.  The colos provide 24/7 AC and backup power so I figured I might as well take advantage of them. 

Redundant means they are the same and have the ability to failover.

Yes... you must be really trolling.

GG

Do you think I could use my cluster for trolling and make some trollcoin with it?

[mokahless]

Failover? Redundancy? can you elaborate further on those topics? because I don't think they apply to the bitcoin world unless we are talking about a pool that needs to be up 24x7 or a market exchange, but from the miner's perspective?

Failover means that when , lets say, The State of Missouri is destroyed in an act of clumsiness typical of that state and my rig is destroyed with it, my other rig either surrenders half of its tasks and takes on the tasks of the dead Missouri rig or it takes over in full by creating more instances to make up for those lost.  This is why I need huge logging space because the failover rig will have to collect data from the failed rig.  The colos provide 24/7 AC and backup power so I figured I might as well take advantage of them. 

Redundant means they are the same and have the ability to failover.

I think what he was asking is how you are making this work for bitcoin.
Failing over the running applications does not help you because there is no "progress" in bitcoin mining.