Bitcoin Forum

So the original thread is here (but the plan changed a lot and the thread got sidetracked at points w/ flames & trolls)
https://bitcointalk.org/index.php?topic=66606.0

The Purpose (also know as "are you fraking crazy FPGA are going to take over the world ... ")
I ALREADY have a farm of six 3x5970 rigs plus another 6x 5970s cards looking for a home.  The hardware has long since been paid off and I have some measure of pride knowing the rigs I built are just about as efficient as is possible with GPU tech (although some optimizes 7970 rigs might edge them out).  The goal is to extend that investment as long as possible.  Kinda a GPU "last stand" against the encroaching FPGA hoards  

My goal was to find a way to:
a) survive the summer without the 1/3rd increase in electrical cost
b) avoid paying ~$3K to the upgrade AC system (farm was expanded in the fall so current AC won't be able to handle it)
c) have fun building a big insane system.
d) see if it would be possible to capture some of that heat for preheating hot water heater and/or heating the house.

So I had this crazy idea.  Can you put 24+ HD 5970s in a standard server rack and using a really powerful pump move all that heat (18,000 BTUs) outside?

Well the farm is 50% converted.

The Rack
From top to bottom is APC PDU, management server, gigabit switch, second APC PDU, and the first three water cooled rigs.
http://i.minus.com/ibcPqjLo6NhLFA.png

Closeup of the rigs
No I am not cooling it with milk.  I guess the flash made it look like that.  Just using plain distilled water.
http://i.minus.com/iP5LbeEyZpCtf.png

Connectivity & Power
I need some better cable management but power on left, connectivity on the right is a good start.  I hope to eventually replace all the flash drives with PXE.
http://i.minus.com/ibtPC83iYjiyd8.png

Manifold
First three rigs are hooked to the three left most ports.  Isolation valves on top, flow control valves on bottom.
The connected loop on the right ensures even if I absent mindedly disconnect all 3 rigs the pumps will still get about 1gpm flow (important to keep the pump cool).  The photo is kinda hard to tell but this is mounted on the right hand side of the rack when looking from the POV of the first photo.
http://i.minus.com/iBLM12pskhT2g.png

The pump
A real monster but quiet and has a nearly silent fan in the back which reduces the amount of heat which ends up in the loop.  A pair of unions simplify hooking it up to the loop. The pump and fan are the only component which can't run on 240 which necessitates running 120V extension cord.
http://i.minus.com/iFdeXoZyWc9Tb.png

The radiator
Has about 16x the pipe surface area as your "normal" 3x120mm radiator and about 6x the fin surface area.  :)   I am still using the poly tubing but will be replacing it with PEX when I get the time.  Fan is just a "normal" floor fan.  Probably 1800 cfm.  I may replace it with a car radiator fan or maybe just a outdoor rated workspace fan.
http://i.minus.com/iAys7Sg9WIxA9.png


The rack contains:
3x quad 5970s rigs
2x APC 9571 PDU (240V, 30A, 80% derate) for 11.5 KW usable power
1x Cisco 24 port gigabit switch
1x 1U watchdog server (runs bitcoind, p2pool, remote power relay card, and monitoring software)

Current config:
12 5970s (24 GPUs) ~ 9.2 GH/s

When complete:  
24 5970s (48 GPUs) ~ 18-20 GH/s

Theoretical max:
40 5970s (80 GPUs) ~ 30-32 GH/s

That thing looks so sexy. I want to see it full of shinny electronics.

Hm, hoses out the front, didn't really think of that. That would make routing them in the case easier I would think. Cool.

Yeah the bottom one is done the way I want it. I took the 3.5" plate and drilled 2 holes.  One half of the quick disconnect has a panel mount adapter so it mount directly to the plate.  The plate is then monted in the drive cage.  This makes the internal tubing loop fixed.  Less chance of moving around, stressing the line, or causing a leak.  Right now if you yanked on the tubing for the bottom rig the worst you could do is cause a leak outside the rig.

However the steel is pretty tough. It wore out my cheapo bit after the first plate.   Need to get a stronger bit to drill out the rest.

Looking good so far. Do you have any pics of the other end of your watercooling setup?

Yeah should be up this evening.

You know "this evening" is pretty useless without specifying a timezone. For me(gmt+2) "evening" was technically several hours ago  :P

How about wait for evening EST?  :P

That would be now... ;)

Sorry guys.  By midnight EST.  Guess I should have been more specific.

I wonder why haven't these php forum coders thought about this multi timezone operation. Everyone probably do set the timezone on their profile so it wouldn't be too difficult to show it in posts with the user info on the left. Most regional forums do just fine without it but Bitcointalk is hardly regional.

Edit: okay my bad. It does show if you check users profiles but that's a chore.

Updated first post w/ more pics & details.

Very nice. I just have been re-pasting my cards (because I had to remove the heatsinks in order to take the plastic bits off), and I can only imagine what hell it is to do so many. I've been keeping the same old thermal tape and pads, but even then I can only do 2 at a time before getting frustrated and having to take a break. Have done a dozen so far.

Maybe if/when you switch to 7990s I will be able to buy your 5970s with the blocks already applied to stuff in my rig. ;D Wonder how much they will be worth then.

Heh that beast of yours do deserve watercooling. It would be the most epic watercooling rig ever. I'd say if you play your cards right you might be able to get somebody sponsor those blocks for you rjk  8) I bet it could get a lot of coverage on IT-media.
I just wonder what kind of scores would both of yours (dt & rjk) rigs do folding. Top 10?

Car radiator fan is probably not designed for continuous use. I'd say go for the workspace fan.

Given the can of motor oil on the shelf, I assume this is in your garage?

It's looking great.  Any plans for a backup pump or will you just do software throttling?

Aha, I doubt it. Folding uses CPUs and Nvidia hardware, which are no good for mining. Mining hardware is AMD, and I doubt that it would fold vary well, although I don't know a whole lot about the folding scene.

When you exchange the heat into you warm water for your house you can make your rigs more efficient. This is what i am planning to do with the heat of my serverroom.... in germany its called luft-wärme-pumpe... in english it could be air-heat-exchanger (or something similar). your efficiency is even higher if you can exchange the heat of water to water....

Yeah.


No plans for backup pump at this point.  I have the ability to cut power to the rigs (the reason among others for the watchdog server).  I intend to install both a current transformer and flow meter.  In a pump failure situation the relay board in the watchdog server will trip power on all 6 rigs.

cgminer is already set to do software throttling (reduce clock @ 60C, idle thread @ 70C) but the "power kill" provides a further layer of protection.  It likely isn't necessary but it does buy some piece of mind when you go $4K worth of GPU relying of coolant.

This weekend I intend to waterblock some/all of the last 12 GPUs.  God I am not looking forward to that. :(

Yeah that is the goal once this is done and stable.  A water-water heat exchanger to preheat the cold water line of hot water heater. 

Let's say you preheat your water. You got 6kw of constant heat coming. Your water boiler doesn't heat the water constantly but just when it's temperature drops below a certain limit(when you use water). What happens to your excess heat? You don't plan on leaving your tap water running do you? :P

I guess it needs some kind of fancy temperature controlled bypass valve.

gpu loop ----> water 2 water heat exchanger ----> radiator ---> back to gpu loop

whatever heat isn't dumped into cold line of hot water heater is dumped into radiator.  Dumping the energy into hot water heater isn't necessary for cooling, I sized the radiator to handle 6 KW of heat even in 100F ambient temps.  However it is "free" energy.  Given the choice between heating the outside air or heating my hot water (and saving $300 per year)  will take the later. :)

Why wouldn't it work well?  I have already tested it. 

Cold side has 60F flowing @ 2 gpm.  Hot side has 140F water flowing @ 10 gpm.  More than enough Delta T.

Remember this is a hot water PRE-heater  The cold side has cold inlet water.  The hotside transfers heat to the water BEFORE it goes into hot water heater tank.  IF water isn't flowing there is no heat transfer.

Not sure what by pass valve and only using radiator over night would accomplish.

 

I have a high pressure pump on my LC rig, and I found that when a tiny leak appeared, I lost the card. I shifted the pump to after the cards, thus creating a negative pressure where the cards were. Now if a leak were to occur, I'd simply be suckin in air, rather than forcing out coolant.

That may be a good idea.  Cetrifugal pumps tend to have poor suction so if a small leak did develop the flow would drop way down once enough air is sucked into the system and that would trip the flow sensor shutting everything down.  I might need to think about moving the pump.

A shutdown sure beats a fried card.

Need and water blocks for 5870's?

P.s. Snag some pix of the innards.

Once you have a leak it isn't a closed loop. :)

Will do.  Don't one to pull the existing rigs out of the rack but I will be building/converting rig #4 this weekend.  Will snap some shots.

I get what you are saying now but remember heat only flows from hotter side to cooler side.

The cold water line will never be hotter than the hot loop (loop carrying GPU waste heat).  At worst it will be the same temp (Delta T = 0) and there will be no heat flow. 

So water is flowing on cold water inlet to hot water heater heat will be transfered from hot loop to cold water raising its temp (and saving me energy).  When hot water is shut off, the water in line will reach equilibrium with hot loop and heat transfer stops.

I have a strong feel to drop that radiator into a lake during summer  ;D

Yeah the radiator will be outside but my goal is to maximize the amount of energy "harvested" for heating water so the loop is

GPU (heat source)   ---> water2water exchanger (preheater) -----> radiator (outside) ----> coolant back to GPUs.


I will be using flat plate heat exchanger.
You can reduce pressure loss by using more plates.

http://www.brazetek.com/brazed-plate-heat-exchangers

these plate heat exchangers sound like a good plan in theory. A couple of questions rises though. Are they capable of handling the water pressure in your cold water line and do they have enough flow? ...okay you said you could put a few in but still. You only run hot water like 0.1% of a day, double or triple that if you got a wife. So during that time you could save money. My point being that you would actually have to ditch that water boiler all together and build a custom water boiler that constantly circulates the boiler water through your heat exchanger to have any kind of hope to save money. Your water boiler is still going to keep that water heated during that 99.9% of the day ;)

House heating during winter times would yield a lot better ROI IMHO.

Oh and another thing I thought about. What is the temp that your pump can take(if you are considering moving it to the other side)? What will it do to its life span? I'd say the water coming from your 24 GPU rig will be burning hot. Your tubes will melt ;)

For high efficiency water heater standing losses are less than 20%.  So if you use x BTUs of energy per day/week/month/year 0.8x is heating up cold water and only 0.2x (or less) is keeping that water hot.

I use about ~$300 per year in hot water so lets assume that is ~$240 toward heating up water, and $60 towards keeping it hot.  Heat exchanger should be able to preheat 60F water to >120F for "free" using the waste heat of the GPU loop.

Sure having a custom built hot water heater with an internal water to water heat exchanger and bypass valve and backup heat source would provide even better efficiency but it also would cost a LOT more, and provide only a marginal increase in savings.

So:
spending $80 on heat exchanger (+ parts &  labor) = recovery 80%-90% of energy costs
vs
spending $1500 on new custom hot water heater (+ parts & a lot more labor = recovery 90% - 100% of energy cost.

Simply put using heat exchanger as a pre-heater is "good enough" and provides a much higher ROI%.  Another way to look at it is every BTU of energy dumped into the cold water line is a BTU the hot water heater doesn't need to expend.  "Free" BTUs vs paid BTUs.


Regarding pressure & flow.  Flat plate exchangers they can handle much greater pressure than home water systems.  They are often used in industrial cooling applications.  Those brazetek models have burst rating of 400 psi.  They make them in all sizes able to handle 10gpm up to 100gpm+ at various pressure losses. 

Hi,
I have a similar setup, with 12 cards (9 5970 and 4 5870) working flowlessly for about 1 year.  The diferrence is that I ran them in series. I'm using a Koolance heat exchanger connected to the cold water line. Will soon add 5 more cards.

Awesome.  First I have heard of anyone else doing this.

Yeah, the heat was driving me insane last summer.

Connecting the loops in series would allow higher temps to be dumped into the heat exchager right? The higher delta T the better. But it also means that your rigs run hotter or at least some of them do. Which might create more energy consumption due to the worse switching efficiency of higher temps on GPU.

Yes, you right. First card is at 20C last at 40C. To lower the temperatures I can increase the cold water debit, but I want to keep the noise down.

Besides, restrict the water flow in your closed loop (such as running it through a water block compared to 1 inch hose), and you can bet your ass you'll have negative and positive pressure on either side of the restriction.

AH, yours is a sealed system? MY system has the overflow tank, thus there is a positive and negative pressure. Thus if air enters from the suction, it's released in the overflow tank.

Please enlighten me on this simple theory.  Why does the resistance not cause more pressure on one side when the fluid is being pumped through a restriction?  Surely you can explain such a simple theory to someone with a degree in math from an engineering school.

https://i.imgur.com/YaLr0.jpg

Hoping you can post a video showing a walk around of the rack.
Love your hose connectors on the front.

I agree with this post. Simple fluid dynamics.  Fluid flows from high pressure to low pressure.  Or it wouldn't flow at all :)

The cards are in series. I have only one heat exchanger.
http://koolance.com/index.php?route=product/product&path=54_119&product_id=944

"The term "negative pressure" is used in physics and engineering to refer to a situation in which an enclosed area has lower pressure than the area around it. Any compromise in the divide between the area of negative pressure and the more highly pressurized area around it would cause substances to flow into the area of negative pressure."

Please, humor me.

It is not, however, you cannot simplify a liquid cooling system into a closed loop.  The reservoir contains a less dense fluid, "air" which compresses with less energy than the water.  This allows for a vibration model to be made due to the pauses in rotation by the induction pump. (4 per 2 radians?)  This creates a dampened oscillation, reinforced by the motor.  In turn there is a negative pressure in the reservoir with respect to the pump.

This can be solved by a long and complex differential equation, which I really don't feel like doing.

For more information on this, look up Bernoulli's principle

I use no radiator.

Amazing temp ! Now i'm tempted to do mine. With multiple exchangers, i cant wait to see how it works out.

Just to clarify Ciuciu, is your exchanger right b4 the water heater?  (preheating) or you just use cold line to cool the loop?

Ah ok.  I thought the debate was about negative pressure in the whole system o.o

It was, i assumed the system without reservoir. However, mr degree flasher came in with a silly talk.

If you want to create a negative sealed system, you have to heat up the liquid (say 50c) prior sealing. The inline pump inside the loop cant pressurize the loop itself.

or use an overflow valve after stress testing your cards :D

I told you I didn't know and ask you to teach me.  Instead you just act like an ass.  Good job.  I wasn't flashing my degree, just trying to show I have the ability to understand so you won't be wasting your time.  But whatever, now that I know positive pressure vs. negative pressure is not the same as high pressure vs. low pressure I can probably do my own research.  Way to turn a simple misunderstanding of terminology into a "You're such an idiot" post.  Ass.

Just cold water. Here is a pic:
www.mycpu.ca/Ebay/IMG_4327.JPG

You just run the tap water all the time?  :o

Yes, I do.

I guess you have free (or flat rate) water?

Yes, luckily I have free water.

 brilliant idea, really   ::)

It comes from the river, it goes to the river. I believe is better than to run the air conditioner non-stop.

How do you deal with the condensation?  Faucet water is much colder than ambient temps

I am not saying he is doing this but with a heat exchanger you can make the GPU loop any temp you want.  Keep GPU loop's flow fixed and adjust the cold side loop (the cold water -> heat exchanger -> drain) flow so that it pulls enough heat from the hot side (GPU loop) to cool it down but keep it above ambient.

I'm super jelly that he can just use fresh cold tap water to cool everything instead of us normal people that need to screw around with a radiator, a pump, and a reservoir. Must be soooooo nice having free water. Dont even need air conditioning! All the heat goes down the drain!

he still needs pump and reservoir, just not radiator as the heat exchanger is the "radiator"

I'm doing exactly what you said.
I lost a card a few months ago to condensation. I wasn't home and the Internet went out. The cards got very cold and condensation happened. The lost card was using an XSPC block, all EKs were OK. For this I try to no longer use XSPC, they sure have a condensation problem.

Okay, now that was good to know! This could easily happen to DT as well if you put your rad outside.. Although 24 GPUs would draw about 1060w while idle so there would still be a good amount of heat going in to the loop but if the fan blowing through the the rad keeps running full bore after the net is out temps would still drop rapidly and low.
A kill switch shouldn't be too difficult to do.

Ek blocks are half plastic but condensation would still happen on the steel plate some of the blocks have on top(5870V2 & 7970). Some condensation would also happen on the sides. I think the under side facing the card is getting some heat  and the air doesn't move there as much so it is probably ok.

Why don't you wrap the tubes and cover the blocks with cellophane?

I think that would make the rigs a nightmare to maintain. Besides as it isn't the purpose to do extreme cooling here simple temp controlled fan killswitch would do. And a mag valve would do for the tap water cooler dude.

DT: how many cards did you manage to convert during the weekend ?

Another 4.  Fourth rig is installed.  16 down, 8 to go.  God I hate installing waterblocks.  :)  If FPGA didn't exist and I planned to rig up an entire datacenter like this I would have to find someone to install blocks for me.

Will provide some updated pics this evening.

So I guess this was with the Swifttech links. Were they hard to press together vs phobya links?

Yes this is the first rig to use the Swiftechs. 

The link has some resistance when adjusting the width of the link but not much. 

I found it easiest to install it this way.
1) Unscrew the "collar" (gives you more room to get wrench on the connector).
2) Attach connector to the waterblock (side closest to the collar) *
3) Tighten w/ wrench according to spec.
4) Shorten the link as far as possible
4) Insert all cards into the rig (each is connected on only one side)
5) Attach other side of link **
6) Tighten w/ wrench according to spec

* My water blocks (Danger Den 5970) have a plastic top so I attached the links to that side of each card first (outside of rig)
** This is actually the hardest part.  Kinda hard to get it started (small space, resistance from extending the link and getting threads started).

So far I am really impressed with these links.   They are very very similar to the Koolance links ($15 ea) but are fitted to allow tightening with standard wrenches.  I honestly can't believe they are only $8. 

In your case when there was nothing connected to the motherboard wouldn't it have been easier to just tighten the links outside of the rig -> attach all the cards together -> place the whole stack on the mobo ?

I have never had luck with that.  If you got 2 card you can do that pretty easy.  But with 4 card trying to get all 4 to line up right is tough.  You kinda need 5 hands.

Probably the cleverest way to preheat hot water would be to get an old second water heater and put it before your main water heater - you can probably find a craigslist one with a burned out element or such. Instead of a heater element inside it, you would put a "radiator" inside of it for your water cooling. Since you can't open up the tank to put in a normal radiator, perhaps a dozen feet of pex or copper tubing inside the thing circulating your coolant would do the job. The best kind to modify would be an electric with two heater elements, as pictured below, so one element can be your coolant inlet, and one can be your outlet. You remove the elements, fish tubing from the top to the bottom, and shove as much extra pex as will coil around inside the water heater. Then the trick will be finding fittings that let the piping pass into the water tank and still seal the inlets where the tubing passes so they will hold water pressure.

Note that the city water pressure will want to collapse the low-pressure coolant tubing. Although it should resist this crushing pressure, your coolant pump should be right before the water heater pumping coolant into the exchanger. Also note that if the tubing/radiator inside the water pre-heater ruptures, this will allow 60-80psi of water pressure into your coolant lines, so an emergency coolant pressure relief valve should be installed where it can let out water to prevent a hose blowout inside your racks.

With such a setup, you could preheat 15C water up to 40-50C, as warm as the coolant coming out of the systems.

http://www.naturalhandyman.com/iip/infwaterheater/i/elecwhexplo.gif

That's actually a really ingenious solution. For a radiator, is there any reason you couldn't take the top off and either insert a real rad or a nice coil of 1/2" copper? You're using it as a tank and insulator, so even if the tank is sealed it's nothing that a Sawzall won't fix. The copper tubing would have no problem surviving 30-40PSI of external pressure.

If you think about this water boiler business real hard you'll come to the conclusion that this second boiler is just stupid over  the multiple heat exchanger idea. This doesn't save any power over the plate exchanger it just takes a huge amount of space.

In theory this would allow for greater delta T because it has longer time to build up the heat. Adding enough plate exchangers would do roughly the same however.

Are you sure on that? With the extra water heater, you should be able to get the preheater to the same temperature as your water block exhaust temperature. With plate exchangers, you're still limited by the power of your loop. If you're running the hot water full out and pulling say 10lpm, you would need to dump ~700W of heat through the exchanger to raise the water 1 degree C. You would need a massive amount of plate exchangers and a lot of water in your cooling loop in order to raise the water temperature to close to the exhaust temperature. That sounds a lot more expensive than getting a free or very cheap water heater and some copper tubing, and much more likely to cause pressure issues on your hot water line.

Why?

Flat plate heat exchanger on the cold water line cost ~$50 and provides a massive Delta T to the cold water.  It recovers most of the energy needed and is both cheap and simple.  The payback period is measure in weeks.  While one can do more remember hot water isn't that expensive maybe $300 per year so complex projects start cutting into your ROI% pretty quickly.

While there are ways to recover more of the heat I just don't see them as viable.  Hacking up an old hot water heater of dubious quality, building your own radiator, running copper loop (not that cheap these days), jury rigging some water tight seals all to get maybe another 10% to 20% of annual energy budget?  It doesn't seem worth it.    I am going to keep it simple.

I you're wrong on the last point. Consider a simple closed loop system where you have a pump, a block and a radiator. Your exhaust temperature will be a function of flow rate and power. Whether you have 2 ft of tubing between the GPUs and radiator or 20 ft, as long as you're not dissipating much heat through through the tubing the water between the blocks and radiator would be the same. You have 10x as much water in the tubing in the second case though, and thus 10x as much stored energy.

I don't see why that idea would be expensive though. It seems like something that could be done for $20, and perform as well or better than a large plate exchanger. The biggest downside is the floor space and low SAF.

I doubt you are going to find everything for $20.  Copper loops large enough to have any significant heat transfer aren't going to be cheap.  Still even if it was free this is the water my family drinks.   Not sure I want to be hooking up some salvage hot water heater of unknown origin or quality.

I fully concede that a flat plate exchange on the cold water inlet doesn't recovery 100% of theoretical energy but is does recover a good portion, is cheap and simple and doesn't require potentially contaminating the water supply.

If I went the storage tank w/ heat exchanger route I would look for something like this:
https://www.signalmarine.com/p-51907-atwood-eh-20-electric-water-heater-wheat-exchanger-20gal-110v.aspx

If someone made something like this without the backup electrical heat it likely would be cheaper.

You could always do some kind of radiant floor heat in the winter...just a thought...

Your points are valid, it would be significantly more work to ensure it was built well and safe. As for the amount of water, to efficiently transfer the heat you need a large amount of surface area. To have a large surface area without excessive restriction, you'd need a large volume. That requires more water with more stored energy.

I'd actually be very interested in your calculations for the plate heat exchanger you looked at, and what the stats are for your system in terms of flow rate and head pressure, temperatures and such. With 6kW, you have a lot of leeway for heating water, but looking at my own supply even in the summer the incoming cold supply is around 10C. Getting it to 30C requires a lot of power.

Its  not about getting it to 30c. But it's sure as hell higher than 10c. Not much energy saved but free is better.

Beside, i'm doing it to help with the heat being dumped in the house during hot summer days.

Now, i'm not sure how well it work. I hoped high untill i saw ciuciu's pic. LOL  

Any responsible human being wouldnt waste water like that. Clean water isnt cheap. Its infact more expensive than gasoline where i am.

ok.. so I've been thinking...  flat plate on cold water inlet vs. 2nd tank with sidearm heat exchanger...

I see the trade off as being:  with the flat plate method, you are only getting benifit while you are taking a shower.  Lets say the entire household totals 1 hour per day.  you get maximum effenciency/minimum price with the flat plate, but only for (1/24) = 4.1% of the time.. the other 95.9 percent, you are getting nothing. 

So the question is. how much heat are you recovering during that ~4% of the day?  after doing some research (yer geeks, you know how to use google..) I found that 1 Kilowatt/hr will heat 409 gallons 1 degree.  Lets assume you need a delta increas of 40 degrees.. so 1 Kilowatt/hr will cover approx 10 gallons.  assuming 2 gallons per minute on the shower, the question is:  does your water loop provide 1 kwh of heat energy every 5 minutes or less?  If yes, then you are golden with the flat plate setup. 

If not, then your water loop cannot keep up with your shower, and you need to extend the time where you are extracting heat.  To do this, it seems to me the best way would be to install a 2nd tank (cheap/new or used.. ) with a sidarm heat exchanger.  Put this inline with your hot water heater on the cold side instead of the flat-plate.  While this is the more expensive option, in the long run it should pay off as you will be actually be pre-heating all the shower water instead of just some.

If like me, your needed temp increase is closer to 60 degrees, then the calculations lean even further in the direction of 2nd tank.

Alternatively, you could just install water saving shower heads...  Ugg, I hate those things.

Sigg