Bitcoin Forum

Hi Everyone,


I am setting up a 50TH S1 Farm

Those of you with experience on setting up larger S1 FARMS, are there anything I should be aware of?

I read that you can use server power supplies as they are cheaper, but it looks like there's quite a bit of work behind setting up the PSU for 278 machines.

What would be the most cost efficient PSU setup? What type of networking Issues should I be aware of? What is the most efficient set up? How would you deal with airflow and heat?

What is the best type of RACK to set up on?

I look forward to all your experiences, creative solutions, and help.


BTCBuyer

why are you doing this without knowing or hiring someone who does know?

At 2W/GH you realize how big your electricity bill will be right?

What is the cost kWh in the location you are settings up?

278 S1 Ants x 180GH x 2W = 100,080W

Standard home user rate of 15c kWh = 100kW x 24hrs x $0.15 = $360 per day or $10,800 per month.

Buying the 1W/GH S2 would probably pay for itself pretty quickly with half the power cost and it will be a lot easier to deal with half the heat.

This is really only a little hobby, and project for me. I have access to cheap power, 7.5c per kWh and a large warehouse. I figure since I have the space and capacity, the only thing left is cheaper PSU, any suggestions?

I have read about using the dell server PSU, but there seems to be a lot of work with getting them to work.

I know that you are interested in answers for your 50TH S1 farm, but I would advise you to buy SP10s units instead. My group buy has the lowest $/GH with best density. You only need 33 SP10 units and each unit is 1.25U in size :)

I am not sure you understand how electricity works.
So you are going to have 130 PSU that will need to be 1,000W minimum.   What are you plugging them into?
The PSUs are not your problem, the heat may not even be your problem if the warehouse is large enough and the air cool enough.
You should start with 30 and see how you doing just powering the PSUs up.    You need to be learning about transformers as you seem to think you could just get 130 "plugs" and this would work.

I find the 1300W EVGA SuperNOVA G2 offer the best price per watt, 92% efficient, and it's rock solid with a 10 year warranty.

It can power 3x S1 Ants easy, even overclocked.

I read the dell Server PSU are the best.

To clarify, I have 400Amps, There's more than enough juice to run 100,000watts.

Is it hard to convert the 750W Dell PSU to run S1s?

https://bitcointalk.org/index.php?topic=503423.0

You can buy just the breakout boards or the whole kit with everything. I have one of these and it works great.

Awesome, I will look into this.

in an old groupbuy thread i ran across a link to these racks- https://openrigs.com/

was going to look into dimensions and such to see if theyd be suitable for my S1's but then stopped at 6 S1's so i decided it was unneccesary. Seem fairly affordable and modification ready. Then again, if it were a warehouse id just get my old man to bring over some old cinderblocks and planks haha.

I hand soldered the main terminals on mine and made up some links and resistor packs with jumper cables for the control pins.

In quantity I could probably do them in 10 mins a PSU.

They are loud though.

Good Luck

 don't go with the dells for a giant setup. (have had the dells and I use the evga's now)

Gator-rex is steering you correctly.

if you are real you want the evga super nova 1300.

 runs 3 overclocked 10 year warranty and cost is 195   that is 66 dollars a s-1.

two of these are 388 usd that is 6  s-1's units.    

three  dells will not over clock 6 s-1's safely .   Shock Hazard  fire hazard   not much resale value.


tell you what get 2 evga 1300's and setup 6 s-1's  over clocked to 393 or 387 freq.   get 3 dell and try to do 6 s-1's over clocked.


here is a link for the evga's

http://www.rakuten.com/prod/evga-supernova-1300-g2-1300w-power-supply-110-v-ac-220-v-ac-input/249973555.html

next you need a lot of switches.  get one of these to start

http://www.rakuten.com/prod/tp-link-tl-sg1016-10-100-1000mbps-16-port-gigabit-19-inch/213928097.html

that lets you do 1 in 15 out or 15 s-1's


oh you need two internet lines not one if you go over 250 or 256.

a router taps out at  amax of 256 including the router     192.168.0.   1- 256     these 1-256 are the most you can do on 1 router.

with that many machine 278 you want a second fan on every one  here is a link to the delta

i used  http://www.ebay.com/itm/151276722479?ssPageName=STRK:MEWNX:IT&_trksid=p3984.m1439.l2649


mount this fan with 4 pipe cleaners and 4 nylon nuts to allow the fan blades to clear.
http://imagizer.imageshack.us/v2/1600x1200q90/843/8909.jpg

Consider going with a few High current DC power supplies. For each 25kw (~1500A at 12-14Vdc) your 3 phase ac load would be ~ 90A at 208Vac.
Do the calculation for the S1's and your existing warehouse load to see if your 400A service will be enough.
Then consider running this power thru combiners and recombiners to safetly get fused power to each S1 (picture a solar farm running backwards with the S1's being the solar modules).

Going to cost but you must have the BTC.

don't know enough about your idea to vote on it.
do you have any photos?

here is how I power the second fan

http://imagizer.imageshack.us/v2/1600x1200q90/841/sq97.jpg


a miner with 2 fans and a psu attached

http://imagizer.imageshack.us/v2/1600x1200q90/842/kyee.jpg

No I don't have any Pictures, I'll leave that to the OP to research if he understands the concept! if not he should hire a EE or a good electrician

 Thanks trends    I think your idea has merit if the op has the correct people set it up.


To the op these miners are better off side by side on a solid surface in long rows.  think lunch room tables.

when the miner is side by side to a second miner they make a wind tunnel so ten side by side have  only 2 blades not in a wind tunnel the left and right ends do not get a wind tunnel effect every other blade in the long row best the wind tunnel.  the solid surface  also helps that tunnel effect.

I will take a shot of a very small s-1 farm to show what I mean

the center blades  have a wind tunnel
http://imagizer.imageshack.us/v2/1600x1200q90/845/uiae.jpg

ten in a row with the psu out of the way would be ideal

http://imagizer.imageshack.us/v2/1600x1200q90/836/fd15.jpg

Well 50th is not a toy farm

I wish I had a space to run 100 or so.

 2  more  shots  four more  powered by an evga 1300 and a sea sonic 1000

http://imagizer.imageshack.us/v2/1600x1200q90/841/v417.jpg


these have the two fans  each.
One is  stock and the second is the delta running flat out.  you always want the pull fan running faster then the push fan.

  they run loud but cool in the garage.
 The  solid shelf with all four  stacked side by side with the double fan creates powerful wind tunnels. I did find there is a ton of heat above the row of s-1s a fan above would help to cool the gear down



http://imagizer.imageshack.us/v2/1600x1200q90/845/c7n4.jpg

400 amps is not enough to run 100 kW

Only 80% is useable, which means this is the max:
400A * .8 * 240V = 76.8 kW
400A * .8 * 208V = 66.5 kW

Maybe Industrial with 440V or 575V

kendog is correct that 400 amps does not  do  100k watts unless the power is 440 multi phase   but at 220 208 or 110 not enough power.

oh my last shot of  the six ants running

you want ant monitor  the monitor I am using  link

https://bitcointalk.org/index.php?topic=596178.msg6548589#msg6548589

but start small  maybe 15  or get 3 of the 16 port switches.

  do 1 into 2.  check for latency issues   since 1 switch can do 15 switches giving you 200 plus ants.

feel free to ask me in a pm about setting up but frankly I am not a fan of the dells



http://imagizer.imageshack.us/v2/1600x1200q90/835/jtkd.png

There's a new 1500 watt psu from corsair that 80 PLUS TITANIUM CERTIFIED

94% peak efficiency, 125 amps at 12 volts, single rail


Corsair ax1500i. It's an absolute monster

Yoshi at Bitmain is recommending Either Dell 750W Server PSU or the HP 1200W Server PSU, for the most economic solution. Your thoughts?

We used the Dell 750's on all the S1's for our hosting customers

theres a really weird mix of advice in this thread, a lot of it coming from people with less than 5 antminers.

I currently run 17 antminers (and have run/sold several more on top of that). Here is my advice.

1) go with 208V power. If you are industrial you likely have 600V mains, and 208V is the most common transformer output. Its also about 3-5% more efficient than 120V.
2) use L6-30 outlets and find some PDUs (power distribution units) that match it. This gives ~4.98kW of power per outlet (after the 20% safety threshold) and is a common style that is cheap to find and install
3) get a few 48-port network switches that are 10/100 outputs. Most of these are build with a gigabit trunk so they can all daisy chain while providing plenty of good connections.
4) forget the secondary fans. Its an extra cost, wastes time to install, and is totally unnecessary in the setup you are going for.

5) you NEED the ability to ventilate the heat directly outside. a closed building requires much more air conditioning and thus even more power usage.
5b) use a hot/cold aisle design. 2 rows of shelves, exhaust from both pinted inwards, with strong fans pushing all the heat out the aforementioned ventilation to outside.
5c) you will still need a lot of AC to keep things cool, probably about 15% of the ASIC power draw if you can vent to outdoors, 30% otherwise.

6) go with DPS-2000 power supplies. These are the most effective use of space and time for getting the massive 12V amperage you will need. Alternatively, the DPS-800 can put out 1000W on a 208V input. (gigampz.com has some nice, but slightly pricy, breakout boards to make using this PSU easy)
6b) soldering fat wires to a PSU is a huge pain. either follow a method that requires minimal soldering, or invest in a proper 120W+ soldering gun that is at least $40. anything cheaper will burn up with extended use.
6c) I sell custom 16awg PCIe cables with tinned ends. They are thick wires and can handle more than enough current (beware 18awg labelled stuff from china). see my sig if you are interested in getting some - they work well if you splice a few onto thicker 10 or 12 awg wires for use with the DPS-2000.
6ci) PCIe connectors will make life easy if something needs swapping. the screw terminals are impossible to access if you stack miners or have them side by side.

7) stack the miners on thier sides,with bottoms facing the bottoms of those in the neighbouring stack. (see picture).  This will allow you to pack miners in with good density and contain the airflow across the units nicely.

https://i.imgur.com/iPQxGKH.jpg



Yes, the SP10 is smaller and more efficient. But it costs almost twice the $/GH. If you are bullish on BTC prices rising the S1 is by far the best option. (if you feel that in 6 months from now we will still be <$500, then by all means go for the more efficient hardware). The S1 is a solid unit with nice 120mm fans, and if you have the space and power for them they are my #1 pick.    You can always undervolt them (takes about 5-10 min per machine to do) to get <1.3w/GH when the time comes  (about 2 months by my estimate if $/BTC doesn't jump up first)

+1

How do you plan on cooling the usable 400A of power.  Have to have power reserved for that.

I skimmed most of the post and only thing mentioned is fans.  Which with the amount of Antminers or hell any miner you have to have proper cooling for it.

Watts to BTU/hr = 3.412141633 x Watts
RT (Refrigerated Tons) from BTU/hr = BTU/hr / 12000

Or at that scale you would be buying an AC setup for a datacenter style where you would be looking at it as kW being used like with a Liebert, etc. unit.

Learn how to internets.

Use the 10.x.x.x and setup a Class A or B network. 

If it were me, I would certainly not hesitate to use a wireless connection, even if it will be difficult to manage

See my Sig for setup guide. 200 is the same as 1.

Never mine with wi-fi....

But to stay on topic

Just a small batch:

https://i.imgur.com/1g0QFUrl.jpg

And a part of...

https://i.imgur.com/FQgfeYLl.jpg

But trust me, better go for larger machines in smaller quantities...  ;)

The long solid shelf you use make for a nice wind tunnel effect along with side by side for 10 or 12 miners.   those look like mdf so they won't allow heat to rise much


that setup could be improved using simple 20 inch box fans on the pull side venting much of the heat out of the building.

   obviously I am trying to get the op to avoid/lower ac cost as much as possible.

you have close to 40 in that setup the op would need 5 to 7  racks like that.  his biggest problem with your setup is access to the gear. but with 200 plus s-'1 anything other then a line of lunch tables makes the same problem.

 btw  if he gets some 1th machines it is less work.

I have an 22000 air m3/h industrial fan in the wall at the back of the shelf to suck out the warm air. :)

I've made the height of the shelf small because I didn't had more space. The new shelf is higher, have more space at the top of the miners, but no space between them because the cooling is better this way.

I have an other shelf but it's not shown at the picture.

I'll try to make a paint but don't laugh hard.  ;D

https://i.imgur.com/BH42Wpsl.png

Had had to do this way because of the spec of the room.

If I'll able to make a new room, and If i were op i'd do this. (I'll sure do it when I arrive that point to make a new room.)

https://i.imgur.com/DSq4VQC.png

You still have enough room to check the miners.

But I'll definitely go for 1 th or bigger machines.

First you be careful with heat from ants. You can make a tower with the ants and create a wind tunnel effect to easily exhaust the heat from building you can put a big fan.

In the electric problem each ant eat 360w from the wall so if you put 278 miners you need a 100Kw line in 220v.. You needs many main switches for isolate electric problems. Dell server power supply make extreme noise. Really EXTREME. Consider hp 1000W server power supply have less noise and it's more easily for hack and make the wire connections...

For 278 ants you possibly have to pay a bill near to 5000USD / month.. nice

Additional you need control the humidity of the air and playing with this you can increase the refrigeration of the facility.

For the network you need a gigabit main router you need a gigabit enterprise router o business router. And many 48th lines 100mb switches with less than 1 gigabit line to connect with the main router. And can use a watchdog for reset automatically every ant when this stop mining for any reason.

Happy mining!

now if the op wants he could get

1th miners from bitmaintech   the price is now 2529 - 400 usd if you have a coupon

  at 2129  they are pretty tempting .  he can pull 1 board from each one  as it gives the stock psu overhead. 

 sell a few boards on ebay keep a few board for spares.

 much easier to manage. 45-50 machines then 250plus .

If you're going to do 50TH, forget the S1. Get 36 SP10s instead. Call up Spondoolies and I'm sure you get can a discount for a 36 unit order. Setting up 36 units would be a piece of cake.

if he could get  30 sp10s at the right price it is the best way to go.

 not sure if they go low for a 30 unit buy.

I think they say they give different price for orders >50 pcs.
 I am not quite sure, but give them a call.

Good suggestions in this thread.

I'm running 30 s1's each with an individual psu. After much consideration I chose the the rosewill capstone 550w 80+ gold psu from amazon($74 each w/ shipping). An overclocked s1 on 220v lands squarely in it's peak efficiency range. For secondary fans I'm running delta afb1212she(150cfm). The ants are arrayed horizontally on heavy duty plastic grate type shelving units with each corresponding psu located on the shelf beneath.

For ventilating the room the ants are in I utilize (6)14" industrial fans arrayed in pairs on the room's three windows. Two pairs of fans pull fresh air in and the pair nearest the ants exhausts the heat. In addition there are (3)20" industrial fans pushing air around in this long room.

CGRemote is used to control the chaos.

While it's loud and ambient is peaking around 90f during the day near the ants I haven't had a single failure(psu or ant) in the three months I've been running this way.

Im working on a similar setup up to 30 ants and intending on getting more machines, 1ths or others, but ventilation iis a BIG issue!
so what size fan is it that you use to get 22000m3/h? and what do you have for cold air coming in and how cool does it stay.

in the second what is the tube section supposed to be?

thanks

Regarding the Dell 750W PSU, I've had three load-balanced running five overclocked S1 for a couple months with zero issues. It's not recommended but some folks have had a single PSU pushing two overclocked S1 for a while as well; it takes the supply to 100% but they're built like tanks. If you're considering going that way let me know and we can work out a bulk rate on boards and PSU. One nice thing about using the server supply and swappable interface board is if the PSU craps out, it's about a $10 replacement that takes all of ten seconds to pull and plug, no worrying about rerunning cables and paperclips and stuff.

https://i.imgur.com/pGabcRm.jpg
https://i.imgur.com/wPMEai6.jpg
https://i.imgur.com/ACxKst7.jpg

I built these. (DPS-2000BB)  You are going to need more than 400amps.  You also need power for exhaust fans and/or cooling.

How tall is the ceiling?  Can you cut a hole in the roof for an exhaust fan?

Find an electrician off craigslist that works for cash so you don't have to pull permits.

It cost me around $4000 to put in 600amps of 220V.  I already had a 2000amp main breaker so it was easier to pull off the main lines.

Then, when the power company or the city or the land lord comes around, you can get screwed by not having the permits. Seriously, I dont know why you would go around the permits, they exist for a reason and going around them can get your ass lit up here in the US.

There are many reasons to circumvent bs government regulations(money grabs). In many cities here in the US you're not allowed to change your own broken water heater in your own home without a permit. Land of the free indeed.

And when the power company comes by, wonders why your pulling 3x the load your permits say you can and they cut your power, you can complain about government regulations.

For myself, I don't care what goobermint does, but this whole line of conversation is off topic. I simply said there are many reasons to circumvent tyranny regulations.

If the building burns down due to shoddy electrical work and no permits, I guarantee insurance will not cover it and smracer will be financially liable.

Also, going from drawing 200 Amps to drawing 600 Amps without checking with the power company first sounds pretty dangerous. I know when I looked into upgrading my service from 200 Amps to 400 Amps, the cost was very high because the power lines coming into my house could not support a 400 Amp load so the power company would need to run new wires.

Buy Spondoolies-Tech boards either the 1st gen or Rockerbox and let them hash away in immersion cooling. Their Rockerbox boards give around 3 TH/S per board. You can buy 60,000 worth of those boards and stuff them in the 2-Phase immersion cooling. Electricity won't be a problem for that. It's the most efficient way to cut down on your costs. (http://www.allied-control.com/immersion-cooling) This will be a good project if you go this way.

The work was done by a licensed electrician just not on the clock.  It is completely up to code.

I have a 2000 amp panel.  All he did was run 3 X 200 amp GE boxes.   They are not that hard to hook up.  I just wanted a professional to do it.

I own the building.  I am not worried about the warehouse burning up.

It used to be a metal shop and the electric bill averaged $8,000 a month when I bought the place.

https://i.imgur.com/uZn1EeX.jpg

I see 2 panels with 8 breakout wires each.

 so a 200 amp panel divide by 8 = 25amps a wire.


the white box says 230 volts…  So what do you have in  each of the gray boxes 230 volts at 200 amps?


if you do and each breakout is on a 25 amp breaker what gauge is that wire.  12 or 10?  I hope not 14. 

As those breakout wires look like 12 gauge to me.

Not to go off topic but is it possible to even ROI once you factor in the cost of the work, electrical costs, and cost of the miners or is this an expensive hobby for people with a lot of coins and money. 

The other panel is on the other side out of shot.   The 230V box was for a screw air compressor that is no longer in service.  I used the 2 X 120V lines to make 220V for one of the 200amp boxes.  The 208V stinger is not being used.  I had the whips made - They are 10 gauge.

 you are fine with 10 gauge.  no worries with overheating the breakout wires.


to psjw4450

 the wirework he did will last 20 years.  so roi becomes different.  Also remember tax laws vary and this is a longterm capital improvement.

his warehouse may use s-1's for a while and he may decide to be a hosting service.   The s-1's  would be a way to learn how to do a hosting service. So s-1 roi is meh  but gets written off a bit as an expense. 

If I recall he has 7.5 cents  a k-watt for power.  he may be a great hoster in 3 to 6 months time.  lets say oct 1st.  he can ramp up to many units and host them.

 I would love to have 5 or 6 sp10's running at 7.5 cents a k-watt. or better yet run 2 sp30's at 7.5 cents a kwatt.

^  +1

electrical work and long-term installations will outlast a single generation of bitcoin equipment.