I have....

[Darrel]

looked through much of the forum sections and have so many questions and also have answers to other peoples questions.
 
I have been a Linux Nut for nearly 10 Years and a serious hardware guru for even longer.  My background is in Network Engineering, Data Center and Telco Cable-Plant and Facilities Design.     

I can't wait to get a mining system put together with Linux and I already have 4 6990's on order along with a motherboard, PSU, water cooling system, etc. and will be building a custom solution for mounting all the hardware.

I will have access to 600 Amp service soon and will be plugging this thing in.  If it all works the way I think, I will be building many more.  I see a real future in bitcoin and I WILL be a part of it in a big way.

I see many people saying that Bitcoin Mining is no longer profitable and I am calling Bull$hit on that.  Maybe it's not as profitable as you want it but I've $100K I'm going to throw at something to make some serious dough and right now it's Bitcoin!

So, my questions for the group:
 
1. Does anyone know how to get past 8 GPU's or is there a solution to the 8 GPU limitation?  This is apparently a limit of the Operating Systems and not the Hardware as most motherboards today can enumerate as many as 128 PCIe bus devices and a few can do 256 PCIe Bus Lanes via PCIe Bridging.

I ask this mainly due to the fact that there are some really great PCIe expansion systems, for industrial use, out there and have as many as 18 PCIe slots.  With Riser Cards/Cables and custom mounting we could get 126 GPU Cards onto one motherboard using 7 expansion boards.  The Hardware for doing this in fact out there and a power solution exists, not out of the box mind you but there is a way to physically make it work.  This system would of course have to be water cooled simply due to it's sheer size, and I have just the solution for this type of setup.

2. Since Linux is so extremely flexible and the 8 GPU limit is probably in the Kernel is there anyone here up to the challenge of doing some Custom Kernel Hacking?

Anyone, Anyone, Anyone???  For the right person there could be a steady paycheck involved.  Let's Talk!

Cheers,

Darrel

[1713975005]

1713975005

[1713975005]

1713975005

[1713975005]

1713975005

[Stephen Gornick]

Mining is a form of speculation that is only slightly less risky than buying bitcoins themselves outright.

It is all relative though.  The miner with a dozen rigs enjoys economies of scale with regard to creating a configuration and replicating that to each rig.  That miner though is at a disadvantage to those who are mining their single rig in their business or home where the cost of electricity is included in the rent, for instance.

With the mining profitability remaining near its recent low, the miners bailing are generally those who are more affected by the noise, heat and space constraints that mining imposes.  Additionally mining requires a commitment of time for keeping current with mining methods and for administering a rig.  Though mining might still return a profit the levels are increasingly insufficient to keep the operator sufficiently motivated in  continuing operations.



(from http://bitcoinx.com/charts/ )

Another risk today's miner faces is that a replacement technology is right around the corner.  There could be an ASIC design being developed that could make difficulty rise so quickly that the GPU miner gets displaced just like how GPU mining displaced CPU mining less than a year ago.

[ElectricMucus]

Well the first fpga miners were being sold yesterday, but it's gonna take a while till the initial costs go down.
Also what some people understand as asic differs from person to person. Chips configured with block roms in them and with standard layout are not asics, even if they can surpass fpgas.

[MiningBuddy]

If you have $100k to throw around I would seriously reconsider spending it on mining rigs, you WILL have more coins buying directly now then those rigs will ever produce.
You need to sit down and work over all the numbers, this is a good start http://bitcointalk.org/?topic=7531.0

[sd]

1. Does anyone know how to get past 8 GPU's or is there a solution to the 8 GPU limitation?  This is apparently a limit of the Operating Systems and not the Hardware as most motherboards today can enumerate as many as 128 PCIe bus devices and a few can do 256 PCIe Bus Lanes via PCIe Bridging.

Chances are no-one has ever had a need for more than 8 GPUs in a Linux system before. Few graphical applications would need more than 8 GPUs.

Ask on the Linux Kernel Mail List. You might get a useful response. Those people HATE limits in their kernel and will likely fix it if fixing it is at all possible.

If might be worth asking ATI technical support. You can bet you will have to talk to 2 or 3 layers of reboot-monkey script readers before you get though to someone who knows what they are talking about. If you get though to a real techie you might get a sensible response.



However mainboards are cheap, are you sure you need more than 8 GPUs per board? And are you really sure about water cooling? It's expensive.

[jh1523]

I very much doubt an 8-gpu limit exists in the kernel. If you're looking for a culprit, look at fglrx.

[Darrel]

Thank you to everyone who has replied to my post.

QUOTE:
Mining is a form of speculation that is only slightly less risky than buying bitcoins themselves outright.
END QUOTE:

Right you are!  But lets just say hypothetically I dumped my $100K into the BTC system, That's nearly 9000 BTC and would increase the total Trading Volume 7.8% and it would shoot the price of BTC way up, by my initial calculations the history of BTC it should nearly double in price to around 21. By infusing that much at one time into such a small market of 7.1 Million would have adverse effects, I can assure you.  I feel that if the market shot up that much in one day too many would check out and would drive the price down possibly even further than it is today in the 11 range.  This is probably not what you want.  Mass sell-offs in such small markets causes panic among the smaller players which I think dominate the market and could likely kill any remaining trust left in the Bitcoin system.

I played the Forex game for 6 Years only to profit about 8% in total with the final year at a loss.  Most Forex traders do better at it than I did but Forex was not something I did Day-to-Day.  Much of the loss was due to the failing USD and not paying enough attention to the daily changes around the world.  It seemed that every time Obama opened his mouth a foreign currency would drop an entire percent. 

bottom line is that I'm not here to just write someone a paycheck while they bail out of Bitcoin.

Let me address the issue of Miners who are getting free electricity.

How long do you think a landlord of a home or office space is going to let their tenant take advantage of them with FREE electricity?  Even mom and dad are going to start questioning their kids about the power consumption, heat, noise, etc.  Especially when the power bill was 100-200 monthly and suddenly spikes to 400 then 500 then 600 as they add more machines.  So even the kids that are mining will probably be shut down in due time.  Typically the only type of office space that comes with electricity included are Class A locations aka "Executive Office Suites" most like Regus or OfficeHQ and especially high-rise suites have abuse clauses in their terms that allows them kick them out despite say a one year term agreement.

Bottom line is that power is not unlimited nor is it free for the taking.  Those NOT paying will have their day.  As the US economy tanks its just a matter of time. 

On the Topic of FPGA's and ASIC's   

Bring em on!  I have already been talking with a couple EE's I know from Cisco Systems already on the topic, and both are confident that these will be the future of any computational platform.  Stanford University is already working from FPGA's and are working on an ASIC platform to update their Private Protein Folding system.

FPGA's and ASIC's are not to be feared.  It will be no different than going from CPU mining to GPU Mining.  FPGA's are just the "Next Step" in the LifeCycle of Bitcoin.
Additionally, since systems can be sold off at nearly what you paid for them then your GPU exit strategy should be pretty well cut out for most.

I can't wait to start seeing some of the performance metrics of FPGA's and the advancements of ASIC's and other emerging technologies.
 
Who is selling the FPGA's???

QUOTE from "sd"
And are you really sure about water cooling? It's expensive.
END QUOTE

it's expensive when you independently water cool each machine to room air.  When dealing with larger scale there are extremely efficient ways to use water cooling.
This is how Data Centers are able to survive.  Bulk Heat Dumps are absolutely necessary when you are direct cooling numerous machines.  Once the Bulk of the heat is removed then you can chill the water using a system similar to an Air Conditioner except instead of cooling air you chill the water to somewhere near 4 degrees Celsius or 40 Degrees Fahrenheit.  Its called a water based Heat Exchanger system.  it is about 10 times cheaper to run than a Air Cooling system, it scales easily, and virtually eliminates the Second Watt Problem.  That is 1 watt to heat and 1 watt to cool back down.  In the water system you get 1 watt of heat and it takes about .5 to .6 to cool. 

Now I live in the desert so dumping bulk heat is difficult unless your water is 150F or higher.  So I don't plan to build here.  I am looking at the Pacific Northwest where it's quite cool year round and power is as low as 6 cents per KWH through a co-op power company.

8 GPU limitation:  Thanks for the suggestion!  I was already going down that path on IRC with the Kernel group.  I was just hoping that someone had solved that issue already.

Thanks JH1523 for the FGLRX item.  It seems that it is the one thing causing Segmentation Faults which still leads back to the Kernel.  I will investigate FGLRX further to see what causes the seg faults.


I encourage and invite any questions anyone has.  I am full of information on this stuff. 

Cheers Everyone!


Darrel

[antares]

1. Does anyone know how to get past 8 GPU's or is there a solution to the 8 GPU limitation?  This is apparently a limit of the Operating Systems and not the Hardware as most motherboards today can enumerate as many as 128 PCIe bus devices and a few can do 256 PCIe Bus Lanes via PCIe Bridging.

Chances are no-one has ever had a need for more than 8 GPUs in a Linux system before. Few graphical applications would need more than 8 GPUs.

Ask on the Linux Kernel Mail List. You might get a useful response. Those people HATE limits in their kernel and will likely fix it if fixing it is at all possible.

If might be worth asking ATI technical support. You can bet you will have to talk to 2 or 3 layers of reboot-monkey script readers before you get though to someone who knows what they are talking about. If you get though to a real techie you might get a sensible response.



However mainboards are cheap, are you sure you need more than 8 GPUs per board? And are you really sure about water cooling? It's expensive.

About that 8 GPU-Limit - It's 99.9% within fglrx, which is a precompiled binary shipped with the fglrx-driver by amd. During the driver installation, the fglrx.a gets statically linked into the "kernel module", and by it's size alone this is where the magic happens. So, unfortunately, unless you (know someone who) speak assembler very well there's no chance of getting this fixed. I for my part can't understand how AMD sells a "professional GPGPU" solution that is limited to 8 GPUs per system, comes with a closed source driver that is almost impossible to install on custom kernels and requires X11 to work. A shame that nvidia cards are flawed when it comes to integer calculations...

[Stephen Gornick]

Bottom line is that power is not unlimited nor is it free for the taking.  Those NOT paying will have their day.

Because there are so few was to monetize "free" electricity, landlords generally are unaware that a tenant or two are exceeding by a good margin the average electric consumption for a unit.

In other words, landlords can stay irrational longer than miners can stay solvent.  :-)   (with my apologies to Keynes)

[1905]

lol nice get an apt w free electricity and punish it. lololol. ill look into that. gotta read the fine print on the lease agreement. 30 days to evict where im at.

[silverchair]

You have...

[indio007]

I thought the 8 GPU limit was a result of the opencl driver not the OS driver. Atleast that's what I read on AMD dev forums.