Bitcoin Forum

I originally posted this as a reply to a topic of a guy wondering why his 1500-watt PSU was overloading (rather, "why the computer just shuts off") with his quad-GPU rig. I figure it could be very useful to anyone running a serious mining operation, so I'm making this into a proper post...

---

For one thing, sucking up 1500 watts is about the equivalent of running a hair dryer constantly... or one of those box room heaters... or something obscenely power-hungry like that. I mean, we're talking like, "dude, you'd better check your power cord to the wall outlet and make sure it's a couple gauges heavier than normal" kind of crap. I'll bet you it gets warm, if not hot. So in light of that topic of conversation, you really need to make sure you've run the ELECTRIC BILL analysis on this kind of completely ludicrous rig... it's going to cost you almost triple-digits on your power bill, so you'd better be CERTAIN you're going to make that back in Bitcoin. Not to mention, if you use it indoors and you need to run air conditioning in that area, it's going to cost about 1.5x that amount in electricity to remove the heat produced by the PC. It's no joking matter. It's summer here and I run my mining rig literally, outside on the porch (2nd floor = great anti-theft), to offset the cooling problem. And I'm still barely breaking even on the power bill, if that much.

That said, your GPUs are stupidly over-powered. Knock them back to pure stock and you should be fine. You also need to make sure:

1) Your CPU usage is ZERO PERCENT while running miners. Any indication of CPU load is typically a rat's nest of API version, driver version, miner settings, etc., that results in any amount of CPU usage caused by a GPU miner. This was a long struggle for me to find the best way to solve. Basically it boils down to one thing: start with low aggression settings, and test each setting for 10-20 seconds to see which one starts causing CPU usage while you're seeing a "Mhash/sec" reading (actually running, that is). If it causes CPU usage, knock the aggression down to the last one you saw 0 at. And drivers, evidently there are some issues with Catalyst 11.7 and 11.8 (the latest), so you ought to move to either 11.6 (browse AMD's archive list), or find the 11.9 beta at guru3d.

2) Always keep your eyes on each GPU's temperature. With that many GPUs, god knows how you fit them in the case. Fans don't move air when they're in a vacuum, so be sure you either have a handful of fans COORDINATED to push air from the front of the case through the back (pulling air in is always easier than pushing it out, it seems), or just leave your case side off and direct a blower fan at it or something. But I can tell you one thing: those GPUs were NEVER designed to operate at full-capacity, right next to each other, for an extended period of time. You've got to make sure you stay on top of their "health" to make sure they stay working properly.

3) Clocks. First of all, don't "just" overclock. Mining is a very unusual situation to put a GPU in, so it doesn't need the same uber-high clock speeds that 3D rendering usually requires. For example, on my 6770, setting the memory clock (default at 1200, I think) to 300MHz actually puts the GPU in some sort of timing mode that essentially super-charges its mining speed, but drops about 10 C off its running temperature. Maybe it shuts down some unused clock modules or something. Whatever it does, it lets me run the thing at 950MHz/300Mhz (core/mem) when its default is more like 850/1200, and bangs out an extra 30 Mhash/sec while keeping the GPU core cooler. Find the right settings, search around, and don't just "move the slider up". And FFS, put the voltage back to stock, that's a freakin' sweet way to nuke your cards in no time...

4) Clients and settings. There's a HUUUUGGEEE difference that can be made between one client and another, and between one parameter and another. For example, my GPU goes from 211Mhash/sec to 160Mhash/sec when I just remove "worksize=128"; the kernel (currently using a modified phatk posted here somewhere, but very similar to the phatk2 provided with Phoenix) defaults to a worksize of 256 and my GPU only has a 128-bit memory bus (it doesn't know that). Performance goes through the floor. You may mess your pants when you see some 300-odd Mhash/sec, but what you don't know is that it could be straining itself with some bad settings and you could be seeing 400 if you played around a bit. "BFI_INT" is pretty much a necessity. Play with "VECTORS"/"VECTORS2"/"VECTORS4" in combination with different "WORKSIZE=" (64, 128, 256) values - the "vectors" values mesh with the given "worksize" to produce each sort of work "chunk", and finding the right combination is key to a smooth-running system.

5) You don't need a new power supply, just quit sucking power out of the wall like it's going out of style! Run the numbers - consider that 1500 watts continuous is what you're drawing (since we already know we're blowing past that as it is), run that as kilowatt-hours (kWh), which is approximately 1.5kW (long form, "1.5 kWh per hour"), and run that out to running 24/7 for a month. 732 hours in an average month, so we're talking... 1,098 kWh for your rig each month. And we plug price into it at about my average power rate, which is $0.15/kWh, and... *click click*... yeah, that'll run you $164.70/month in power to the box alone. Remember what I said about air-conditioning; when you factor in the $247.05 for cooling the house from the heat generated by the rig, that's $411.75 a month. 'Ya sweating yet?

I dunno, maybe I tldr'd here, spent 20 minutes writing this crap for one guy's rig, but dude, seriously, 2000 bucks? I was sweating spending 100 on my 6770 and had to split it with a friend ;) If it helps you any, there's an address in the sig that could help me buy a compliment to my single 6770!

edit: Oh, and if you want to know how much your miner is REALLY costing you, down to the cent... it's called a "KILL A WATT". Get one!! http://amzn.com/B000RGF29Q (no, I don't make anything off this link, just convenience - go Google instead if you want.)

---

Any inaccuracies, explain them and I'd be more than happy to edit this. But I hope this brings a few common problems to light; I spent a huge chunk of time trying to squeeze extra performance out of my rig while watching the Kill-A-Watt meter, running the numbers, etc. Now I hope I can pass this on to someone else that can use it :)

You get 400 MH/s out of your 6770?
/me looks at his 5830 churning out 300

OUT!

ok, all kidding aside, looks like what I need to do is def download the 11.6 drivers, then look at miners other than CGMiner

wat no, lol.
"For example, my GPU goes from 211Mhash/sec to 160Mhash/sec..."
"You may mess your pants when you see some 300-odd Mhash/sec, but what you don't know is that it could be straining itself with some bad settings and you could be seeing 400 if you played around a bit."

No, my 6770 churns out 213Mhash/sec at the currently best combo I've found, 11.9 beta, 950 core/300 mem, with:
C:\Users\Falcon\Desktop\phoenix-orig\phoenix.exe -u http://***:***@**.**:8332/ -v -q 2 -k phatkmod VECTORS2 DEVICE=0 AGGRESSION=9 WORKSIZE=128 BFI_INT=true FASTLOOP=false

(where phatkmod is: https://bitcointalk.org/index.php?topic=25860.0 )

when done right, voltage modification can help.
only a few of my cards are running with stock volts.

Voltage modification has to be done with caution but as long as you don't crank it all the way to the top first try you shouldnt be cooking anything before the safeties try and stop you

Very true. I'm considering playing with a voltage tweak on my 6770 during the winter (when it's not relying on being cooled by summer 95-100F outside air), but mine's also a single-card setup. Multi-card setups... well... unless you've got a *really* powerful power supply, custom cooling, and excellent airflow (or may I recommend... water cooling?), it's really just begging for a toasting. Especially the +0.1V boost this guy was talking about - far as I gather, these chips only operate in the 1.0-1.5V range (absolute minimum to "wtf? nuke!" there), so a 0.1v boost is like a 20% over-volt! I'd expect playing around with 0.05 (10%) at the most, since that usually overcomes the 10-15% power consumption boost these things can usually handle with their cooling.

I'm really curious for input on the CPU usage/"aggression" correlation, though. Has anyone else tested this? In all the threads I've browsed around here, I haven't seen anyone besides myself mentioning this relationship - just "unexplained" high CPU usage with people on a constant wild-goose chase of drivers, APIs, miners, etc... I couldn't really be stumbling on something unique here, could I?

if you have proper cooling, 0.1v overvolt will be okay.
your power consumption, on the other hand, will nearly double.

It's generally not thermodynamically possible to cost more in cooling than it is to cost in mining. Essentially the cost is related to the power used. The power used from the cards represents a certain amount of Watt-hours. The cooling system would generally have less Watt-hours than this depending on the efficiency of your cooling system. A typical heat pump central air system should be able to give in the 80-90% range and a standard window unit would be around 80% as well. As an example, I have a window unit to supplement my central air that cools 10000 btus using 450 watts. Using the factor 3.41 btus per Wh, we see that to cool 10000/3.41 = 2932 watts of energy takes 450 watts, so only 15.4% (http://www.google.com/search?sclient=psy&hl=en&client=firefox-a&rls=org.mozilla%3Aen-US%3Aofficial&source=hp&q=450Wh%2F10000+btu&btnG=Search) of the mining cost can go to cooling. This also makes a very basic assumption that all the energy used on the mining rig is converted to heat, which is clearly not true but I have no real way to measure it.

So given this and using a conservative 20% for your cooling factor, the cooling on $164.70 should be no more than $32.94.

What is it converted to?  Ignoring LED's and fans of course.

Bitcoins, silly!

Except for tiny amounts escaping as radiation through windows or as sound and vibrations, energy put into LEDs and fans ends up as heat as well.

I just read that whole paragraph and didn't understand a shit.

I did run out the numbers that's why I'm fucking buying GOLD PSU to save money 3 year from now.
Is that what you meant ?

And here I was trying to proactively head off someone being pedantic ;)

pay close attention to how your PSU distributes its wattage.  I blew up a 850watt psu cos I thought 800watts on the molexes thats cool.... when in fact only about 600 watts was on the molex connectors for the GPUs to use, the rest was motherboard and 5v.

not worth it at $5 per bitcoin imo

My take on AC cooling gpu is that if you manage to direct the very cold air in the fan intake without much lost you can save energy by having the fan spin much slower for the same cooling.

AC cooling just one end of the room, does little to fix the problem of having ~30 fan consuming ~200W at stirring their own shit. (30*0.5mA)

undervolt not worth it?

lol, whatever.

I love you... and unlike the usual forum troll that mucks these boards, I must say: "I now know this". Yep, internet history here, a guy will admit he's wrong. God, I love real numbers and facts. You're awesome.

OK, so... since I'm a little too stoned to dial in those numbers and recrunch the figures (which would still be damn high), could you be bothered to run 'em and see if they put a happy face on your calculator?

Since you guys are discussing this I will let you in on how I was able to have four GPU's tight in together in a case without using pcie risers or special cables and I get very good temps, the highest temp is 64 degrees Celsius.

http://btcnetwork.com/images/screenshot.jpg

There are 11 GPU's mining in my small office. A 12.5k BTU window air conditioner takes care of the cooling. Because winter is coming I modified the Air Conditioner (some air conditioners like mine will shut off when its cold out site) by taking the sensor and putting in on a long wire and having the A/C sensor actually sit inside one of my GPU's that way it always thinks its hot. Plus I made a thick plastic cover that partially covers the air conditioner to trap in some of the heat on the outside . I use this on cold days because the air conditioner will freeze itself and stop working if it's too cold out side. The plastic cover keeps some of the air around the outside of the A/C and keeps it from freezing.

Now as you can see in this picture I have a desk immediately in front of the a/c you can only see one of my rigs on the desk but there are two there, both have a fan that blows the cool a/c air a crossed the gpus and they both have a powerful sucker/blower fan attached to the back where the gpu video connections are on the back of the PC case. These fans suck the air from the gpu's and blows it away, this keeps cool air constantly flowing a crossed the gpu's and hot air constantly exhausting.

http://btcnetwork.com/images/cooling.jpg