Building Cheap Miners : My "Secret"

[erikstarfish]

What is the vesting period of B21 tokens? It is not clearly stated.

[1714754488]

1714754488

[1714754488]

1714754488

[1714754488]

1714754488

[1714754488]

1714754488

[1714754488]

1714754488

[MinersRus]

I never had a problem w/ linux and XMR-Stak w/ the 8837's.  The default auto config for me worked right out the box.  I do think hwloc ends up putting the double memory ones up first though.

I've had a hell of a time with XMRig and that's because it doesn't support NUMA.

It turns out that you cannot use custom cpu settings on more than one machine even if the systems are exactly the same because Linux disables different cores for no apparent reason.

I had affinity errors on my second HP DL580 G7 running HiveOS because Linux did just that disabled different cores. The only way I could run without getting affinity errors was to let XMR-Stak auto-generate it's own on startup like you have.

For me not having control is not optimal but because Linux is stupid I have too.

I have updated my original post: https://bitcointalk.org/index.php?topic=1955358.msg33408190#msg33408190

[MinersRus]

Just got my third HP DL580 G7 Rig up and running on HiveOS. All three are currently doing 7.75 KH/s on Monero V7.

Currently I only have four GTX 750's in each Rig. I am waiting on the special 10-pin to dual 6-pin PCIe power cables that I ordered from China to arrive so that I can add four more GTX 750's to each Rig. That will add another 2.88 KH/s to the three miners more if I get overclocking working. HiveOS overclocking is broken (as in it does nothing) but I was able to do overclocking manually. I just need to get that automated when the Rig(s) boot. When fully loaded and OCed I expect to see 11 KH/s on the three.

[MinersRus]

I am still waiting on the HP DL580 G7 10-pin to dual 6-pin power cables. I got the 10-pin to dual 8-pin I ordered later and they are very well constructed:

In the mean time I wanted to find a cheap frame for mounting my GTX 750's externally for the three HP DL580 G7 systems I have.

I came across this: https://www.ebay.com/itm/Multipurpose-GPU-Mining-Rig-Rack-Case-Frame-Bitcoin-AMD-NVIDIA-GTX-RX-580-1070/253405188911?hash=item3b0020532f:g:EkEAAOSwKcVadm9J for $24.99

After looking at the picture and measurements I was able to determine that it was actually: Whitmor 4 Tier Floor Shoe Rack - 20 Pair - Storage Organizer
https://www.amazon.com/gp/product/B005KC6DY0/ref=od_aui_detailpages00?ie=UTF8&psc=1

The ends are sturdy plastic resin and the horizontal round poles are coated metal.

I ordered four of them at $7.09 each on April 11th. Total with tax was only $30.68.

Current price is now $10.21 with free shipping over $25
 

[MinersRus]

whoa what are you doing with these servers exactly? very curious!

I will be CPU mining ITNS on them at the moment.

Assuming ITNS has an even partial price recovery I estimate I can get them paid off in 4-6 months on CPUs only.

My fallback is Yescrypt mining which is also quite profitable on CPUs... but I am a big holder of ITNS & I've mined since August on a smaller scale. I believe the project has potential so I want to mine as many as I can.

------

Also... I will be experimenting with adding GPUs to them as they do have robust PSUs and several PCI-E slots.

Did you ever get GPUs working in the R815?

I picked up two and would like to also run GPUs in them.

[MinersRus]

I've been deploying 20a 240v circuits and I terminate them with 2 15a 120v (each outlet getting a separate hot from the 240v circuit) and one L6-20R 20a 240v receptacles.  I then plug in a 16a 240v PDU that then powers all the big iron at 240v and allows me to have misc 120v items powered through the same circuit but through the 120v receptacles.  

I am looking to go the 240v route for all the servers I have acquired (5x HP DL580 G7, 2x Dell R815, various other SuperMicro) and have some questions in doing so.

I am looking to pick up these PDUs: https://www.ebay.com/itm/Lot-of-4-Spectrum-Power-Distribution-Unit-PDU-8x-240V-16A-72A-116BUHEN-01/192482251025

What cables do I purchase to go from these PDUs to the server power supplies?

Also how did you connect the PDU to the L6-20R?

Thanks

[MinersRus]

After killing myself for the last 24 hours troubleshooting 2 r815's I picked up, I have learned a thing or two.

If the cover is not on the box, it will not produce full power.  Board will limit CPU power evident by power drawn at plug and performance in hash.  I chased my tail for quite a bit yesterday and today until I figured it out.  I got the servers Friday and only had enough memory on hand to get 2 cpu's running in one box.  I saw appropriate hash numbers on it.  Since I got 2 different speed CPUs for testings on the boxes, I tried the other set in dual mode and also saw the appropriate numbers.  I then noticed the 50 memory sticks I ordered had arrived yesterday afternoon and rushed to get the boxes fully up and running.  Since I had so many issues with memory population with the 1GB UDIMMS and even various 2GB RDIMMS I have, I had left the covers off and then could not figure out WhyTF in 4 cpu configs, I was down on power by 500H/s.  They are also very picky about the memory that goes in them.  Memory that works fine on other boxes comes up as failures on the r815s.

You must populate 16 DIMM slots.  If they aren't all the same size, you will have an annoying F1 key to push everything the box boots, I have not found a way around that yet.  The box will boot with 8 sticks but multiple threads will not have hwloc bind memory so performance will be garbage.

You will also get a slightly higher hash rate (100H/s or so) by disabling the gui and running from CLI only.

These things run great off 16GB usb flash drives.  

le sigh...  

Thanks for this as it has saved me some issues with the R815's I picked up.

By the way I read somewhere that these R815 servers only work with 2Rx8 ECC Registered memory. I picked up 25 sticks of 2GB 2Rx8 PCE-10600R memory and it worked without issues.

Question on running off 16GB usb flash drives. Do these wear out pretty fast and die? When I tried running off a flash drive for HiveOS the flash drives died in about a week because of the constant log writing that was going on.

[Storx]

After killing myself for the last 24 hours troubleshooting 2 r815's I picked up, I have learned a thing or two.

If the cover is not on the box, it will not produce full power.  Board will limit CPU power evident by power drawn at plug and performance in hash.  I chased my tail for quite a bit yesterday and today until I figured it out.  I got the servers Friday and only had enough memory on hand to get 2 cpu's running in one box.  I saw appropriate hash numbers on it.  Since I got 2 different speed CPUs for testings on the boxes, I tried the other set in dual mode and also saw the appropriate numbers.  I then noticed the 50 memory sticks I ordered had arrived yesterday afternoon and rushed to get the boxes fully up and running.  Since I had so many issues with memory population with the 1GB UDIMMS and even various 2GB RDIMMS I have, I had left the covers off and then could not figure out WhyTF in 4 cpu configs, I was down on power by 500H/s.  They are also very picky about the memory that goes in them.  Memory that works fine on other boxes comes up as failures on the r815s.

You must populate 16 DIMM slots.  If they aren't all the same size, you will have an annoying F1 key to push everything the box boots, I have not found a way around that yet.  The box will boot with 8 sticks but multiple threads will not have hwloc bind memory so performance will be garbage.

You will also get a slightly higher hash rate (100H/s or so) by disabling the gui and running from CLI only.

These things run great off 16GB usb flash drives.  

le sigh...  

Thanks for this as it has saved me some issues with the R815's I picked up.

By the way I read somewhere that these R815 servers only work with 2Rx8 ECC Registered memory. I picked up 25 sticks of 2GB 2Rx8 PCE-10600R memory and it worked without issues.

Question on running off 16GB usb flash drives. Do these wear out pretty fast and die? When I tried running off a flash drive for HiveOS the flash drives died in about a week because of the constant log writing that was going on.

Im not sure about HiveOS, but ive been using USB drives for SMOS for nearly a year on rigs, not a single failure yet. Not even a corrupt file error from a block going bad.
Im using Samsung USB3.0 16GB drives

[optii]

And as profitability goes up, the diff will too as everyone floods it.  It's exactly what happened to Haven.  For a moment in time there, 1KH/s was $20/day on Haven!

I expect a spike to $3-4 for 1KH/s per day over the next week and then slowly go down to settle around $2-$2.50 for 1KH/s per day. That is certainly an improvement from the 0.56 for 1KH/s per day it was just a couple of days ago.

Since ASIC's are now useless and a whole lot of botnet miners are orphaned I don't see the hash rate going alot higher from the current global rate of 145 MH/s:

Difficulty   114.616 G -7.87% in 24 hours
Hashrate   144.596 Mhash/s -81.33% in 24 hours
Monero Mining Profitability   0.6414 USD/Day for 1 KHash/s

https://bitinfocharts.com/monero

Even if the current hash rate doubled to 300 MH/s it would still be significantly less than the 1.1 GH/s it was before the fork. At 300 MH/s Profitability should be 2.05 USD/Day for 1 KHash/s or 366% better than pre-fork.

I can certainly live with that.

---------------

Edit 1: 17:50 CDT 04/07/2018 - Difficulty getting easier and Profitability increasing

Difficulty   93.383 G -27.17% in 24 hours
Hashrate   156.719 Mhash/s -72.85% in 24 hours
Monero Mining Profitability   0.7847 USD/Day for 1 KHash/s

Edit 2: 22:20 CDT 04/07/2018

Difficulty   76.866 G -42.22% in 24 hours
Hashrate   167.254 Mhash/s -55.67% in 24 hours
Monero Mining Profitability   0.9408 USD/Day for 1 KHash/s

Edit 3: 08:24 CDT 04/08/2018

Difficulty   49.093 G -60.68% in 24 hours
Hashrate   190.917 Mhash/s +29.52% in 24 hours
Monero Mining Profitability   1.4867 USD/Day for 1 KHash/s

Edit 4: 12:57 CDT 04/08/2018

Difficulty   38.306 G -64.46% in 24 hours
Hashrate   226.997 Mhash/s +51.65% in 24 hours
Monero Mining Profitability   1.9081 USD/Day for 1 KHash/s

Looks like it will settle around $2-$2.50 for 1KH/s per day without the higher earlier spike since many farms have now moved to mining Monero.

Edit 5: 20:00 CDT 04/08/2018

Difficulty   32.073 G -62.05% in 24 hours
Hashrate   272.847 Mhash/s +68.03% in 24 hours
Monero Mining Profitability   2.3169 USD/Day for 1 KHash/s

Hopefully the Hashrate increases will start to slow down soon

Edit 6: 15:53 CDT 04/09/2018

Difficulty   41.794 G +21.26% in 24 hours
Hashrate   424.229 Mhash/s +72.64% in 24 hours
Monero Mining Profitability   1.658 USD/Day for 1 KHash/s

Well it is beginning to look like you were right in that the flood of farms now mining Monero keep coming. I have yet to see the Hashrate acceleration even decline. I thought 300 MH/s would be the top now I hope 500 MH/s can hold.

Edit 7: 17:44 CDT 04/14/2018

Difficulty   53.898 G +0.54% in 24 hours
Hashrate   459.753 Mhash/s +4.81% in 24 hours
Monero Mining Profitability   1.5033 USD/Day for 1 KHash/s

Thank goodness the 500 MH/s did hold. There was a brief period where it was exceeded but it now has settled into the 450-475 MH/s range

ohh you were so wrong... network hashrate goes now from 500+ to 400+....

[PharmEcis]

I've been deploying 20a 240v circuits and I terminate them with 2 15a 120v (each outlet getting a separate hot from the 240v circuit) and one L6-20R 20a 240v receptacles.  I then plug in a 16a 240v PDU that then powers all the big iron at 240v and allows me to have misc 120v items powered through the same circuit but through the 120v receptacles.  

I am looking to go the 240v route for all the servers I have acquired (5x HP DL580 G7, 2x Dell R815, various other SuperMicro) and have some questions in doing so.

I am looking to pick up these PDUs: https://www.ebay.com/itm/Lot-of-4-Spectrum-Power-Distribution-Unit-PDU-8x-240V-16A-72A-116BUHEN-01/192482251025

What cables do I purchase to go from these PDUs to the server power supplies?

Also how did you connect the PDU to the L6-20R?

Thanks

http://www.cointainer.life/2018/02/22/density-electrical-101/

That should answer all your electrical questions.

[PharmEcis]

Question on running off 16GB usb flash drives. Do these wear out pretty fast and die? When I tried running off a flash drive for HiveOS the flash drives died in about a week because of the constant log writing that was going on.

I'm using Sandisk USB 3.0 16MB sticks with no problems so far.  I did have a few issues early on with some cheap no name sticks.

The R815's, while power hogs are work horses.  They just plug away like nothing is wrong and require minimal baby sitting.

I'm having an issue getting more than 4 cards working on the 580s under Windows.  Two separate boxes lock up with 5 Nvidia cards connected.  I will try hive OS next.

[PharmEcis]

Under Linux, I have 6 Nvidia cards working.  Not sure what is up with the Proliant G7 series.  I have tried a DL360, DL380 and the DL580 and none of these will boot with more than 4 GPUs installed.  Windows will lock up at loading screen.

Slap a USB stick with HiveOS on it and 6 get recognized easily.

I have a ML350P G8 that has taken 6 GPUs in windows easily.  

Edit* Make that 9 1060s in one 580 in HiveOS.

[MinersRus]

Under Linux, I have 6 Nvidia cards working.  Not sure what is up with the Proliant G7 series.  I have tried a DL360, DL380 and the DL580 and none of these will boot with more than 4 GPUs installed.  Windows will lock up at loading screen.

Slap a USB stick with HiveOS on it and 6 get recognized easily.

I have a ML350P G8 that has taken 6 GPUs in windows easily.  

Edit* Make that 9 1060s in one 580 in HiveOS.

Nice.

I am only going to be able to do 8 GTX 750's for now.

I only want to use the internal power and no breakout box/power supply. That means the three 10-pin to dual 6-pin cables could power six cards. The GTX 750's take up two card slots so each on them cover two PCIe slots. That does leave 7 slots left but with only 6 PCIe 6-pin power plugs adds to 8 cards. A similar solution and what I will do is put three GTX 750's in the internal PCIe slots that cover 6 PCIe slots leaving 5 PCIe slots free. I will use 5 risers for the external GTX 750's. That also adds up to eight.

If I want to do more than 8 then I will need to rewire the SAS power. I believe that is what you did. Can you provide details in how to do that?

[nsummy]

Under Linux, I have 6 Nvidia cards working.  Not sure what is up with the Proliant G7 series.  I have tried a DL360, DL380 and the DL580 and none of these will boot with more than 4 GPUs installed.  Windows will lock up at loading screen.

Slap a USB stick with HiveOS on it and 6 get recognized easily.

I have a ML350P G8 that has taken 6 GPUs in windows easily.  

Edit* Make that 9 1060s in one 580 in HiveOS.

What version of Windows are you using?

[PharmEcis]

If I want to do more than 8 then I will need to rewire the SAS power. I believe that is what you did. Can you provide details in how to do that?

It's pretty easy.  The SAS power connector has 10 pins.  5 commons / grounds (black).  3 12V (red) and 2 3V (yellow). 

Assuming your cards only need 1 power connector, you are talking a total of 13 power connectors if 3 cars are installed in the system and 5 are on risers.  If the GPU connectors have 2 each on 3 cables that's 6.  You'll get another two out of the SAS connector.  Leaves you short 5 power connectors unless you are going to use splitters.

I've been successful increasing density on my breakout boards by using molex powered risers that I can power three off one pcie power header.  The pcie power header has 3 + and 3 negative at 12V.  Each gets a 12v to 5v step down and then are passed to a molex plug.  In essence I've turned one of the breakout board power connectors into 3 functioning molex connectors that will drive 3 risers saving 2 of pcie power connectors.

What version of Windows are you using?

Since it's quad CPU the DL580s are running Server 2016.  However, the 4 GPU limitation I have experienced on both Server 2016 AND Win10.  The DL360s and 380s run 10 since they are only dual proc.

[Storx]

Question on running off 16GB usb flash drives. Do these wear out pretty fast and die? When I tried running off a flash drive for HiveOS the flash drives died in about a week because of the constant log writing that was going on.

I'm using Sandisk USB 3.0 16MB sticks with no problems so far.  I did have a few issues early on with some cheap no name sticks.

The R815's, while power hogs are work horses.  They just plug away like nothing is wrong and require minimal baby sitting.

I'm having an issue getting more than 4 cards working on the 580s under Windows.  Two separate boxes lock up with 5 Nvidia cards connected.  I will try hive OS next.

Hows the servers working for yall?

Just realized your a fellow container farmer like myself, but i have mine setup on an all watercooled setup to control temps... how have you managed to control temps in yours?

mine is located in Florida, i bought some property that had 2 old mobile home pads with separate power panels that old abandoned burned down mobile homes lived once a long time ago...... Picked up the property for little to nothing and settled with the county to offset all the fees tacked on the property from all the junk and 2 abandoned mobiles if i cleaned it all up and got rid of the junk... so i spent a few months burning junk from the 80's lol....

[jmigdlc99]

Question on running off 16GB usb flash drives. Do these wear out pretty fast and die? When I tried running off a flash drive for HiveOS the flash drives died in about a week because of the constant log writing that was going on.

I'm using Sandisk USB 3.0 16MB sticks with no problems so far.  I did have a few issues early on with some cheap no name sticks.

The R815's, while power hogs are work horses.  They just plug away like nothing is wrong and require minimal baby sitting.

I'm having an issue getting more than 4 cards working on the 580s under Windows.  Two separate boxes lock up with 5 Nvidia cards connected.  I will try hive OS next.

Hows the servers working for yall?

Just realized your a fellow container farmer like myself, but i have mine setup on an all watercooled setup to control temps... how have you managed to control temps in yours?

mine is located in Florida, i bought some property that had 2 old mobile home pads with separate power panels that old abandoned burned down mobile homes lived once a long time ago...... Picked up the property for little to nothing and settled with the county to offset all the fees tacked on the property from all the junk and 2 abandoned mobiles if i cleaned it all up and got rid of the junk... so i spent a few months burning junk from the 80's lol....

I'm new to using USBs to run OSs for mining. Is it possible to use or does anyone have experience running Windows as an OS on these USBs? or do i need a custom mining OS like Hive or EthOs? I'm familiar with linux/ubuntu as well but windows is just way user friendly for configuring and remote troubleshooting.

[spinx]

Under Linux, I have 6 Nvidia cards working.  Not sure what is up with the Proliant G7 series.  I have tried a DL360, DL380 and the DL580 and none of these will boot with more than 4 GPUs installed.  Windows will lock up at loading screen.

Slap a USB stick with HiveOS on it and 6 get recognized easily.

I have a ML350P G8 that has taken 6 GPUs in windows easily.  

Edit* Make that 9 1060s in one 580 in HiveOS.

I hit a wall at 9 cards on the 580 also, using SMOS. Spent countless hours getting 9 cards to work. In one way i love the HP servers, H/s on cryptonight is a bonus but as a gpu-mining platform they look good on paper but is in my experience not fun maintaining. Did anyone try to run Windows 10 on the 580 with success?

[PharmEcis]

Hows the servers working for yall?

Just realized your a fellow container farmer like myself, but i have mine setup on an all watercooled setup to control temps... how have you managed to control temps in yours?

mine is located in Florida, i bought some property that had 2 old mobile home pads with separate power panels that old abandoned burned down mobile homes lived once a long time ago...... Picked up the property for little to nothing and settled with the county to offset all the fees tacked on the property from all the junk and 2 abandoned mobiles if i cleaned it all up and got rid of the junk... so i spent a few months burning junk from the 80's lol....

Currrently I'm forcing tons of air through the unit.  Soon I will have to AC.  I've painted both containers white and am in the process of building a solar shade.  I have my work cut out for me!  Even though the servers have given me some fits, I'm glad I went that route.

I'm new to using USBs to run OSs for mining. Is it possible to use or does anyone have experience running Windows as an OS on these USBs? or do i need a custom mining OS like Hive or EthOs? I'm familiar with linux/ubuntu as well but windows is just way user friendly for configuring and remote troubleshooting.

USB sticks for Linux OK!  USB sticks for Windows, NOPE.  Sucks.  Might work, you won't be happy.  HIVEOS is the easiest of ALL once you figure out interface.

I hit a wall at 9 cards on the 580 also, using SMOS. Spent countless hours getting 9 cards to work. In one way i love the HP servers, H/s on cryptonight is a bonus but as a gpu-mining platform they look good on paper but is in my experience not fun maintaining. Did anyone try to run Windows 10 on the 580 with success?

I haven't tried more than 9.  I have one box that doesn't have the addin pcie board so I tried to use the 6th 8x PCIe Connector that is located on the SPI board that is supposed to be for a dual 10gb adapter so I could have 6 1080s going in it.  Well when I powered on I heard a pop and I'm currently down one server awaiting a new SPI board.         

You can run Win10 on the 580 if you only want 2 procs recognized.

In my experience, Win10 and Server 2016 act identical on these servers.

Sooo beat.  I was up until 3:30AM last night trying to finish my migration.  I'm currently back to 90% operational.

[Storx]

Hows the servers working for yall?

Just realized your a fellow container farmer like myself, but i have mine setup on an all watercooled setup to control temps... how have you managed to control temps in yours?

mine is located in Florida, i bought some property that had 2 old mobile home pads with separate power panels that old abandoned burned down mobile homes lived once a long time ago...... Picked up the property for little to nothing and settled with the county to offset all the fees tacked on the property from all the junk and 2 abandoned mobiles if i cleaned it all up and got rid of the junk... so i spent a few months burning junk from the 80's lol....

Currrently I'm forcing tons of air through the unit.  Soon I will have to AC.  I've painted both containers white and am in the process of building a solar shade.  I have my work cut out for me!  Even though the servers have given me some fits, I'm glad I went that route.

I'm new to using USBs to run OSs for mining. Is it possible to use or does anyone have experience running Windows as an OS on these USBs? or do i need a custom mining OS like Hive or EthOs? I'm familiar with linux/ubuntu as well but windows is just way user friendly for configuring and remote troubleshooting.

USB sticks for Linux OK!  USB sticks for Windows, NOPE.  Sucks.  Might work, you won't be happy.  HIVEOS is the easiest of ALL once you figure out interface.

I hit a wall at 9 cards on the 580 also, using SMOS. Spent countless hours getting 9 cards to work. In one way i love the HP servers, H/s on cryptonight is a bonus but as a gpu-mining platform they look good on paper but is in my experience not fun maintaining. Did anyone try to run Windows 10 on the 580 with success?

I haven't tried more than 9.  I have one box that doesn't have the addin pcie board so I tried to use the 6th 8x PCIe Connector that is located on the SPI board that is supposed to be for a dual 10gb adapter so I could have 6 1080s going in it.  Well when I powered on I heard a pop and I'm currently down one server awaiting a new SPI board.         

You can run Win10 on the 580 if you only want 2 procs recognized.

In my experience, Win10 and Server 2016 act identical on these servers.

Sooo beat.  I was up until 3:30AM last night trying to finish my migration.  I'm currently back to 90% operational.

I need to do some research on these servers that all of yall are running to see if they are still profitable to buy... GPU prices have dropped drastically, but profits have been marginal at best..

Ya i tried to use forced air at first, but the heat here in the summer just made it to impossible to maintain temps on the gear, so i had to build a script that would underclock the gear to 50%tdp during the hottest part of the days and back up to 65% during the night hours, so i searched into better means of cooling and invested in a large single waterloop that i run my entire farm off of now. I rented a ditch witch and dug a series of 2ft deep trenches on the property and i ground buried a few hundred feet of pex tube in the ground as my source of radiation of heat. My first attempt at running a test rig off the setup showed that running a typical watercooled pc pump was out of the picture, because the head pressure was to large on pushing fluid through 6GPU's on the first test rig. Then i moved to commercial inline pumps and over time they kept dieing on me also.. the head pressure combined with the heat just destroyed a commercial pump in a matter of a month and would start leaking from everywhere.. so i moved on to a spa/jacuzzi style pump thinking it was heat doing the issue and it turned out the heat was the issue all along killing the pumps, because it lastest the longest, so i started scaling up adding rigs to the waterloop and quickly i found myself pumpless again as the headpressure just was to much for the rigs. So a forum member mentioned an idea and its been rock stable for 6+ months now... I currently use a double pump system, but 1 is only running at a time, other is on pressure switch just as an emergency, but now i use a gravity fed system to cool all the gear. I have large 4inch pipes running across the top of the container wall that is pumped with water from the pump and on the end of the pipe about at the 90% level of the pipe, i have a large pipe dumping runoff back into the pump tank. Basically im using gravity to flow the water from the large 4inch pipe, into a diy manifold spreading the water to all 6 cards and flowing back out to the ground loop via a pipe at a level above the level of the pump tank. This has reduced the watts the pump runs at greatly and allowed it to stay cooled and reliable so far.. im letting gravity do all the work for me now... granted i went from a 40c loop to now a loop in the 50's, but it is the best way to go about it for me. I run universal copper CPU coolers i bought from alibaba in bulk, so my cost on them in bulk was around $8 each. after converting the system over to water, i dropped about 1500watts from the farm from removing all the GPU fans and just running a water pump and using mother earth to remove the heat. Now my entire farm runs at currently 58c max so far on the 90f+ days.. not sure yet what 100+ days will bring me yet, but if i have to im prepared to add additional cooling via a large radiator i purchased meant for industrial a/c system, but so far its been super nice running it this simple.