Agree.  Left and right side switches meet at the firewall.  Everything is working off of DHCP... but that brings up a good point.  Possible some of my "Down" machines have simply been given new addresses!  Off to check...

Hi all.  I had over a PH of Kano hash go offline this morning.  Not a problem with the pool, just had ~150 antminers decide to not talk on my network.  Going nuts trying to figure out whats going on.  Any assistance would be appreciated.  Details at https://bitcointalk.org/index.php?topic=3444035.msg35897218#msg35897218

Around 9am CST6CDT the vast majority of my antminer farm went offline.  At the moment I have about 150 S9s and a few L3s offline.

The miners physically look fine, with flashing green lights, no red lights (mostly), network activity, etc.  However the majority can not be pinged, and a few that can be are not starting up.

Current network topology:   PFSense firewall -> right side Netgear GS316 -> <fans out to 7> Netgear GS316 -> Antminer S9s
                                                          ->  left side Netgear 10-100 switch on right side -> Netgear 7 port gigabit switch on left side -> <fans out to 3> Netgear GS316 -> Antminer S9s & L3s
                                                                                                                                                                                                 -> Avalon 821s & 841s

My normal topology is to have the right side Netgear GS316 also support the left side, but I split the network using a redundant feed I had back to the PFSense firewall with morning in an attempt to isolate the problem.

Of my (76) 14TH S9s, (24) are active, the rest can not be pinged.  Likewise (9) of my (25) 13.5TH S9s are reachable - mostly on the right side.

The left side has (11) of (13) 13TH S9s working, (2) T9s, and (2) of ( "problem children" S9s.  All of the Avalons are fine, but of course, they are clustered behind a few PIs, so have a lower network port count.  (22) of my (26) L3s cannot be pinged.

So both the left side and right side is having problems, and they are independent of each other network wise back to PFSense firewall box.  Occasionally I'm seeing Antminers go blinking Red, but a quick power cycle clears that. 

I'm at wits end without a clue.  I'd be fine if I lost a switch.  But my problem children appear to be spread across several switch, and in fact, several physical networks.  The LAN side IP addressing is shared at the firewall, but I can't see how that would be a problem.

Although growing (with the latest batch of 34 mixed Antminers being added early this week), the network has been otherwise stable until this morning.

Somebody please!  Give me some ideas of what I am overlooking...

Yes, mine are set to: 
W:172.16.4.0/22,W:127.0.0.1/32

Which allows me to control them from anywhere on my lan.  Should probably tighten that up someday.

Hmmm, your pool stats are showing 0 miners.  With 0 fee I suppose that would be an option if I wanted to solo mine, but you don't appear to do merge mining.  Honestly don't know if that only amounts to dust or not, so don't know if I should care about it.

I decided to diversify my SHA256 farm and start mining some Scrypt, mostly because of coupons from Bitmain and then $505 list prices of L3s, combined with some empty shelf space in my data center.

Spent some time on Google, but unlike BTC pool information, not finding as many pool comparisons for LTC and its friends or specific fee structures.
Update:  Finally found https://litecoin.info/index.php/Mining_pool_comparison

Recommendations I found so far:

Solo hash - Pro:  no fees.  Cons:  LTC only unless I was to setup merged mining and manage multiple wallets.  Doable.

Litecoinpool - Pro:  Claims 0 fee compliments of merged mining.  Cons:  Cost money to run a pool, guessing they shave some off the merged mining and keep (currently trivial) transaction fees.  Concerns:  PPS pools TEND to have higher fees to cover themselves.  Reported in the Pool Comparison to charge 4%

Give-Me-Coins - Pro:  Smaller pool, merged mining, payout directly to each wallet.  PPLNS - so SHOULD have lower fees. Cons: (Minor) Need to maintain multiple wallets (already setup).  Concerns:  Pool Comparison claims 0% fee, but a pool must charge something to survive.

ProHashing - Pro:  Profit Switching, rumored to pay out substantially more than merged mining sites.  Ability to pay out in any coin, including BTC.  Cons:  Somebody is paying for all the transaction fees associated with mining multiple coins and converting them to payout coins.  Concerns:  Unknown fee structure.  Also, if they are so great, and pay a lot more, why isn't everyone mining there?

FYI: Out of principle, I refuse to mine on any Chinese owned or controlled pool.  Also trend towards smaller pools for the health of the network - so long as blocks are found on a weekly basis I consider them big enough (which is why Litecoinpool is just a backup for me now).

Basically, this boils down to trying to maximize my return, which usually means minimizing expenses.

Can anyone please help educate me on this branch of mining?

Well, its something like 2150 before we hit that point, since the reward get smaller with each "halving", but yes.  In the end, just transaction fees.

So for us idiots in the USA, that is about 2000 sq ft.  Lets assume a 40 ft by 50 ft building.  40' is enough for (5) 8' aisle - (2) cold with equipment, (3) hot for exhaust air only.  So we have both sides of the cold aisle with racks of equipment - so (4) rows total.  With doors into the hot aisle on the inside of the building, we are going to have room for, oh, 40' of equipment more or less (rather conveniently enabling construction of the racks out of 10' unistrut.  So, 160 ft of equipment rows.  Say 1 miner per foot... 900 miners = 5 (800 miners) or 6 rows (960 miners).  5 rows is tall enough that you can access each row without a step-stool.  6 rows would require one.  In any case, the space checks out as doable.

If you are going to build this Yahoo chicken-coop style to minimize the need for exhaust fans, your going to need about 1 square meter of vent space for every 10 miners (about 1 square foot per miner).  Any smaller and you will need added cost fans that use added cost power to run.  Lets presume your going to get clever and use the entire area of both ends and check that math:  960 miners = 960 sq ft.   Building is 40 feet wide, times 2 (for both ends) = 80 feet.  So the ventilation end-walls would need to be 12 feet high.  Doable.

I plan on running 15 per Pi 3 controller - (3) strings of 5 with one open USB port and a spare AUC3.  That way if I need to take one miner out of a string, I can patch the remaining miners in that string to the 4th AUC3. 

I LOVE fork.lol - so much great information.  I also use https://bitcoinwisdom.com/bitcoin/difficulty when looking for a bit more history.

For what its worth, I've had batches of S9s drop as your described.   Worst example was about 80% of my miners, sometimes its half, sometimes only one or two.  These are spread across (2) transformers and (10) separate distribution panels, each with surge devices.

Give that I'm about 85% of the power draw on the lines serving my property, and only the miners are on those transformers, I'm guessing its not power related, although I clueless what it could be.  My wild ass guess is some blip on the Internet.

ps.  I use Awesome Miner and have coded a rule to deal with it.

Make sure you have the defaults set right.  Use the web interface to the miner to be sure.

If your going to use Awesome Miner to set defaults, change pools, etc. make sure you have enabled Privileged API access via the Awesome Miner Antminer tool.

Last but definitely NOT least, stop mining on Antpool and move over to Kano.is, Slushpool, or some other pool that pays transactions fees.  Antpool keeps all transaction fees for themselves.   The last block Kano mined had over 1 BTC in transaction fees on top of the 12.5 block reward.  They are not trivial.  Back in December, blocks were having 3-4 BTC of transaction fees on a fairly regular basis - all of that profit went to Antpool to "pay bonuses to the employees".  I can believe those bonuses were paid, after all, the owners likely consider themselves "employees" <smirk>.

Just me, or does that sound kind of like a bug?  Clearly the Web UI could make that change for us...

Wow.  I didn't know that.  Rather presumed using the Avalon Web Interface and selecting the pull down menu for voltage offset would do the trick.  That is rather surprising news.

FWIW - I switched (3) strings of Avalon 8s from +1 to -1 to help cool them down just a touch.  Before and after stats:

Before:  1a - 11464.6 - single 821
             3a - 66555.7 - string of (5) 841s
             3b- 64851.9 - string of (5) 841s
                   ----------
                   142872.2

After:    1a - 11538.1
            3a - 66540.3
            3b - 64928.6
                   ---------
                   143007.0

So virtually identical numbers.  Both test were over 16+ hours.  I should note the "after" group spent most of those hours in the evening and early morning with cooler intake air.

First the block need to have 100 confirm, then Kano post the reward transaction.  These days that kicks in by a blockchain block or two.  Worst case it gets paid by Kano's next block find.

Just remember the 5 block ramp-up... you will only see 1/5th of your reward (vaguely speaking, its a bit more complicated than that).  The rest if being buffered/delayed a bit.  You won't lose anything, but it will take 5 blocks before you reach full payment per block.

I'd temper that a bit with a "Presuming ASCI-Boost machines generate more Hash/Watt than other offerings".

Most Exchanges ask you NOT to mine directly too them.  Its too much overhead and indirect control I guess (you don't see a deposit, but you don't control the wallet it came from... makes it a finger pointing exercise for you - a headache for them)

Pretty sure I would NEVER do this since I would become a slave to the current price.  I normally sell BTC 3-4 times a month from my mining operation (to pay the bills!), I skipped almost all of March and most of April until BTC > $8500.  Just held them for a few weeks to wait for the dip to correct.  If I auto-sold, I would have been feeding that price decline - not good for my pocketbook, not good for the price of Bitcoin.

If one desire maximum hash rate and power is not an issue, should they be running at +1?