Discussion: Best Turnkey solution for 200 miners behind slow internet connection

[akagi82]

I need your help guys.

The scenario:
200 s9 miners.  1 internet connection (300KiloBytes down 100KiloBytes up).  One premium router that will be the gateway to the internet connected to cisco switches, connected to the miners.

The problem:
throughput is potentially a problem, latency is not.

What is my solution?

1) Use a proxy.  Which one should I use and is simple / turnkey as possible?
2) Just connect miners without a proxy
3) Other optimizations

[1714839489]

1714839489

[1714839489]

1714839489

[1714839489]

1714839489

[1714839489]

1714839489

[1714839489]

1714839489

[1714839489]

1714839489

[coffeefan]

4G connectivity is not really an option?
Purchase some powerful LTE gear like Mikrotik LHG LTE and you should have superb connection for mining.
https://mikrotik.com/product/lhg_lte_kit

[DaveF]

Can you get more connections or is that it for the area?

I had to deal with something like this for a construction site years ago. Only thing they could get was DSL that was slow to the post of useless.
5 DSL connections later each one running to a separate thing worked.

Still slow, just now 5x less slow....

If you can get more you just put batches of miners behind separate connections.

-Dave

[akagi82]

Can you get more connections or is that it for the area?

I had to deal with something like this for a construction site years ago. Only thing they could get was DSL that was slow to the post of useless.
5 DSL connections later each one running to a separate thing worked.

Still slow, just now 5x less slow....

If you can get more you just put batches of miners behind separate connections.

-Dave

Thanks for the replies guys, but adding internet connections is not feasible.  What did u calculate to be your per-rig bandwidth used?  Why did you not go with a proxy?

[Artemis3]

I need your help guys.

The scenario:
200 s9 miners.  1 internet connection (300KiloBytes down 100KiloBytes up).  One premium router that will be the gateway to the internet connected to cisco switches, connected to the miners.

The problem:
throughput is potentially a problem, latency is not.

What is my solution?

1) Use a proxy.  Which one should I use and is simple / turnkey as possible?
2) Just connect miners without a proxy
3) Other optimizations

Last time i measured, 1 S9 needed about 1 KB/s, so you are like half there... even with all optimizations, i think you will either end having a half hashrate or worse.

Anyway i would recommend:

No dhcp, manual configure all miners.
Local dns cache, your dns should be in your own LAN, and that DNS be a caching DNS. You could even just put the resolving addresses manually in hosts or whatever for the pool you use.

There was a mining proxy, but its abandoned. Don't bother with it. You need to get a better uplink asap, or just use 100 S9s, i'm not sure why you ended with such a slow bandwidth to begin with, even in my garbage country, its possible to do better with money.

Another thing could be to maximize efficiency which lowers THs but produces more per watt. So a typical S9 would consume like 810w for 10TH. This has already been done using the Braiins OS Free and Open Source firmware. But of course that's 10 vs 14, not exactly half...

Try to get another link and pass half your miners over it.

[Observate]

I heard it goes no higher than 500mb a month regardless of number of miners.

[PassThePopcorn]

Last time i measured, 1 S9 needed about 1 KB/s, so you are like half there... even with all optimizations, i think you will either end having a half hashrate or worse...

You're correct each S9 is anywhere from 0-2kbps, usually averages 1 when doing most things. The largest problems with S9's and bandwidth from my experience is their dns checks, so having a dns cache / forwarder on network is very useful especially for large operations. The S9 has a tendency to just keep checking.

More than likely if your bandwidth won't support the miners you'll have more problems than if you just ran less miners, think of the miners as trying to fight for the connection on small scale you might not have a problem but if your pushing the limits I've seen an entire DC network go down just because the traffic was so bad.

[Artemis3]

Oh if it was kbps (i might have mistaken my memory) then i doubt he will have any problems, as the link is 100KB/s not 100kbps, so 8 times better. If each S9 takes 2kbps he just needs 400kbps and still has 400kbps to spare...

Yeah i probably saw avg 1kbps instead of 1KB/s, my mistake. Still the recommendations to reduce bandwidth use apply, especially a local dns cache.

[mikeywith]

Maybe op meant 300 kbps (bits)and not bytes , I hope not , but really what are the chances of getting a 2.4mbps connection in a remote place where applying for an internet connection is too expensive compared to the income of 200S9s

... adding internet connections is not feasible.

Anyhow, below is the data for 10 miners.



now I will repeat myself and say

notice these values are the throughput not the bandwidth, now bandwidth is actually the maximum throughput that you can have, where throughout is the real-time packet/data rate , which means your bandwidth needs to be more than whatever the throughput shows here or else you will have a lot of troubles, latency and many other factors affect all these statistics, but to be on the safe side i would allocate at least 5kbps for each miner.

in other words, 0.6kB (killoBytes) for download , and 0.3kB (killoBytes) for upload per miner is needed , so now let's pretend op was accurate in the details he provided.

100KiloBytes / 0.3 = 333 miners
300KiloBtyes/  0.6 = 500 miners

Which means, 200 miners would work just fine.

if however op has 100Kilobits then maximum of 62 miners can fit in safely.

[Artemis3]

in other words, 0.6kB (killoBytes) for download , and 0.3kB (killoBytes) for upload per miner is needed , so now let's pretend op was accurate in the details he provided.

100KiloBytes / 0.3 = 333 miners
300KiloBtyes/  0.6 = 500 miners

Which means, 200 miners would work just fine.

if however op has 100Kilobits then maximum of 62 miners can fit in safely.

In this case downclocking for efficiency would help OP squeeze a few more (about 80 maybe), but 200 isn't going to work.

When aDSL service was inaugurated here, 20 years ago, the "normal" plan was 256/64 (kbps), and the "fastest" was something like 1024/128 (yes, it was already pitiful for 2001). I kept using that (Cisco) modem all this time until it finally died last year, then i bought a new one but the phone service died and with it dsl service which was intermittent for a few more months after that. Nominally my current plan is 4096/768. but it only worked a few days August last year, it quickly deteriorated afterwards. Ah the joys of state ownership and anti free market economy...

But yeah i know, it only sounds strange that in 2019 there would be still plans like that somewhere.

[mikeywith]

But yeah i know, it only sounds strange that in 2019 there would be still plans like that somewhere.

I don't know about the rest of the world, but where I live, in remote places , you can hardly get any internet connection , people there go for satellite internet, but if price and latency is an issue, we usually set up point-to-point connections for them.

In fact in many cases it's cheaper to buy cheap Ubiquitti airfiber equipment such as the af-2x which goes for about 500$ each , which can do up to 200km depending on the line of sight, and you can still get over 100mbsp in ranges above 50km easily, as long as you don't have to lower the frequency to penetrate any obstacles on the way.

in other words, let's say you live at location (A) or have any friend's or family whom you can borrow a 50 square cm on their roof (and of course some of their internet bandwidth) , and your farm is is at location (B) which is 100km away, if you have a bit of networking skills and can do the work alone, you end up feeding your farm with 50-100mbsp connection at very cheap cost, and in the course of a few months you would have ROIed the equipment investments compared to how much you have to pay for a shitty and expensive connection.

When I used to do this type of work, we went to the extent of renting people's roof's to be able to go as far as 500-1000km , but that won't be easy to do if you don't have the right skills and team, interference and frequency regulations might get in the way, but if the distance is not that huge, and you have a line of sight, it's really not a bad idea at all.

[akagi82]

Maybe op meant 300 kbps (bits)and not bytes , I hope not , but really what are the chances of getting a 2.4mbps connection in a remote place where applying for an internet connection is too expensive compared to the income of 200S9s

You're correct each S9 is anywhere from 0-2kbps, usually averages 1 when doing most things. The largest problems with S9's and bandwidth from my experience is their dns checks, so having a dns cache / forwarder on network is very useful especially for large operations. The S9 has a tendency to just keep checking.

Last time i measured, 1 S9 needed about 1 KB/s, so you are like half there... even with all optimizations, i think you will either end having a half hashrate or worse...

Wow... First off, Thank you all for your time and informative replies.  They will for sure help me hit the ground running with limited experience.  For Clariation My internet connection is indeed 300 KBps down 100KBps up (or 3.3Mbps down 0.8Mbps up)

In a nutshell here is what I have learned, and my gameplan going forward:

1) My internet connection is weak but will suffice with optmizations.
2) Use static IP addresses,
3) Have a local DNS cache
   -Router with DNS cache functionality
   -managed switch (i dont like this, as I am unfamailiar with Cisco Commands)
   -Other options???Such as typing in IP addresses in each antminer's configuration, does that mean they do not require DNS lookups?
4) Make sure my miner does not have infected firmware, update to optimal firmware later on.
4) Decide which Pool to join.
5) Learn and decide how to monitor mining performance, per miner and aggregate.

My remaining questions:
- Do you guys have a recommendation / solution for the easiest DNS cache?  I am hoping to find a premium router that can do this while being the gateway for the 200 devices that I need, all on one subnet... but it looks like this may not be realistic.
- Can I manually type in IP addresses in the S9 configuration screens to bypass the DNS issues?  If so, will that be a problem as their IP addresses change from time to time?

Thank you all once again.

[Artemis3]

I don't know about the rest of the world, but where I live, in remote places , you can hardly get any internet connection , people there go for satellite internet, but if price and latency is an issue, we usually set up point-to-point connections for them.

In fact in many cases it's cheaper to buy cheap Ubiquitti airfiber equipment such as the af-2x which goes for about 500$ each , which can do up to 200km depending on the line of sight, and you can still get over 100mbsp in ranges above 50km easily, as long as you don't have to lower the frequency to penetrate any obstacles on the way.

in other words, let's say you live at location (A) or have any friend's or family whom you can borrow a 50 square cm on their roof (and of course some of their internet bandwidth) , and your farm is is at location (B) which is 100km away, if you have a bit of networking skills and can do the work alone, you end up feeding your farm with 50-100mbsp connection at very cheap cost, and in the course of a few months you would have ROIed the equipment investments compared to how much you have to pay for a shitty and expensive connection.

When I used to do this type of work, we went to the extent of renting people's roof's to be able to go as far as 500-1000km , but that won't be easy to do if you don't have the right skills and team, interference and frequency regulations might get in the way, but if the distance is not that huge, and you have a line of sight, it's really not a bad idea at all.

Yes that is being done by several companies and individuals here, given the terrible service by the State owned ISP, but few people can afford it, only business and probably miners would have no problem.

Commonly you get no connection, shitty State connection, or mobile (state or private) and those links you mention. A few do have satellite, but that's a whole can of worms.

I wonder if that Ubiquity can penetrate a small hill that is blocking my line of sight to the only friend that could share internet with me... I don't need 100mbps, 1mbps would be good enough...

[akagi82]

I don't know about the rest of the world, but where I live, in remote places , you can hardly get any internet connection , people there go for satellite internet, but if price and latency is an issue, we usually set up point-to-point connections for them.

In fact in many cases it's cheaper to buy cheap Ubiquitti airfiber equipment such as the af-2x which goes for about 500$ each , which can do up to 200km depending on the line of sight, and you can still get over 100mbsp in ranges above 50km easily, as long as you don't have to lower the frequency to penetrate any obstacles on the way.

in other words, let's say you live at location (A) or have any friend's or family whom you can borrow a 50 square cm on their roof (and of course some of their internet bandwidth) , and your farm is is at location (B) which is 100km away, if you have a bit of networking skills and can do the work alone, you end up feeding your farm with 50-100mbsp connection at very cheap cost, and in the course of a few months you would have ROIed the equipment investments compared to how much you have to pay for a shitty and expensive connection.

When I used to do this type of work, we went to the extent of renting people's roof's to be able to go as far as 500-1000km , but that won't be easy to do if you don't have the right skills and team, interference and frequency regulations might get in the way, but if the distance is not that huge, and you have a line of sight, it's really not a bad idea at all.

This sounds very interesting to me.  For the performance (range/throughput) you are describing, it seems that seems like that is the way to go.  However, in the country where I am trying to operate, I would likely be violating some strict law governing radio emissions.  When you are describing 100km away, how is line of sight even a consideration?  The earths curvature would cause line of sight issues to start.  The internet service providers here have decent service in cities, but very little incentive to expand to rural areas, and usually at high cost.  I will need to look into this more.

[mikeywith]

I wonder if that Ubiquity can penetrate a small hill that is blocking my line of sight to the only friend that could share internet with me... I don't need 100mbps, 1mbps would be good enough...

Unless you go around or above that small hill there is no current technology that would allow penetration of a hill, you can search online for point-to-point simulator/calculator , not so accurate but they give you a clue.

If the hill is relatively high, then building a large tower or placing a AP on top of the tall building will be the only way, if that is not an option you will need multilink to go around  the hill.

When you are describing 100km away, how is line of sight even a consideration?  The earths curvature would cause line of sight issues to start.

But they said earth is flat!! Lol kidding,  curvature is nearly irrelevant in small scales, now this depends on the topographical relief , theoretically earth curvature is  10cm per 1km iirc,  so in 100km a curvature of 1000cm is to be expected but that assumes the land itself is completely flat, which is never the case, I am sure we did 80-100km with towers less than 100 meters high, so it really depends.

Anyhow if any of you guys want to go that route, send me the coordinates and I will try to help you set up a plan with the equipments needed and all, I have not been in this line of work for sometime now but I can still help.

[akagi82]

Thanks.  I will give you a shout if we decide to scale, probably post halvening.  This is an interesting topic for me.  The biggest thing to overcome would be topography, power, and permitting (we can't erect towers on the side of a mountain) without a lot of red tape to cut.

What kind of solution are you guys using to function as a DNS cache?

[mikeywith]

The biggest thing to overcome would be topography, power, and permitting (we can't erect towers on the side of a mountain) without a lot of red tape to cut.

You only have to worry about power if you need to have multi-link connection, where one point is in the middle of no where, but really these antennas require very little power, 2*100amh batteries and 2 solar panels will suffice in most cases, and those don't cost much, so power isn't really a concern, topography is,but as i mentioned in my previous post, if you are lucky,topography could make your life easier instead, this a good simulation application provided by Ubnt, you need to have much knowledge, just drag and drop https://link.ui.com/.

[akagi82]

The actual distances will be <10km, but lots of topographical challenges with these remote relays needing power and red tape of putting up at tower...., but as you have noted, power requirements are very low! thank you.

[Artemis3]

This sounds very interesting to me.  For the performance (range/throughput) you are describing, it seems that seems like that is the way to go.  However, in the country where I am trying to operate, I would likely be violating some strict law governing radio emissions.  When you are describing 100km away, how is line of sight even a consideration?  The earths curvature would cause line of sight issues to start.  The internet service providers here have decent service in cities, but very little incentive to expand to rural areas, and usually at high cost.  I will need to look into this more.

I just checked, those are wifi directional antennas. Wifi uses unregulated band worldwide, this is because the band used is considered garbage. It happens to be the same used in satellite tv among other things, like wireless phones, and microwave ovens. Yes, the infamous water resonance 2.4ghz band and all its harmonics (5.8, 11, etc). Also they are narrow beam, especially the higher frequencies. Pointing the things to each other sounds like fun (which reminds me of pointing satellite dishes, same thing only farther lol).

Well yeah, if i were to do it here i would need to rent a middle point to get internet here, meaning at least 4 antennas for 2k USD... OR get a real tall (few stories high) pole, which might be doable but pointing the thing would be... challenging. The hill is close and rather small, but enough to obstruct line of sight.

But who knows, maybe in the future i might end using one of these solutions. Hmm i think i could do the last leg far more cheaply, given how close the hill is... Unidirectional wifi isn't something new, after all... Oh i see they have a cheap 99$ antenna, that might be enough.

[bobzombie]

please correct me if i'm wrong ,
maybe by increasing the difficulty to a higher value by using (if pool support it , for example) "D=512000"
Then the miners will send out less data (less frequently)

[mikeywith]

please correct me if i'm wrong ,
maybe by increasing the difficulty to a higher value by using (if pool support it , for example) "D=512000"
Then the miners will send out less data (less frequently)

You are 100% correct, higher difficulty = less data transmission , setting a higher difficulty also means a lot of shares that you would have been rewarded for will go unpaid, but of course payout in the long run won't be any different because while you do submit less shares, those less shares worth just as much as those many, after all the hashrate is what matters.

However, to my understanding, most mining pools try to set high enough difficulty for your miner to avoid bandwidth congestion, after all the pool guys are more eager to keep the communications between you and them as low as possible than you are , so while your statement is true,  I doubt it does benefit the OP by any means.

[coffeefan]

I still don't understand why powerful 4G antenna + support construction wouldn't work in the location? 

[mikeywith]

I still don't understand why powerful 4G antenna + support construction wouldn't work in the location? 

There is no reason why this set-up wouldn't work, but come to think about it, 4G is relatively a new technology and many countries only started building 4G infrastructure recently, in fact wireless internet in general is very expensive to use, it is not expensive to set-up from the user stand-point, but it's too expensive to operate.

Where I live, using a 4G connection to run all my gears and despite the little bandwidth they use, I will probably have to pay a lot more for the ISP company than I can afford , 4G here is still a premium service, people use it only when they have no other option at all, I know this may sound strange for people who live in well developed countries , but to the rest of the world , 4G is never an option.

[Artemis3]

please correct me if i'm wrong ,
maybe by increasing the difficulty to a higher value by using (if pool support it , for example) "D=512000"
Then the miners will send out less data (less frequently)

There is no need for this, most pools will auto adjust the difficulty to what you can actually deliver, be it hashrate or bandwidth. Forcing it usually leads to a worse (suboptimal) situation, let the pool do it for you.

I still don't understand why powerful 4G antenna + support construction wouldn't work in the location?  

You can use 4g here (sometimes, lol) and when it works, it goes really fast. But they cap it to like 1g of data per month so you can empty it in like 1 day watching youtube. Then it stops entirely. A couple of the companies (there are just 3) charge you per byte after that. The third is State owned and is a weird mystery. It would be too complicated to explain, you can't even tell what data plan you have, and new customers are not offered data plans at all (everything is metered by the byte, and expensive).

Most likely there is no 4g coverage where that setup is. In many not so well developed countries, mobile data coverage is... random. Maybe no data but phone, maybe data but the slower one, maybe 4g but after an hour it switches to the slower one, or leaves you without data until you reset it...

And here sometimes you can even get 4g link but no internet, because they have problems upstream as well...

[akagi82]

I still don't understand why powerful 4G antenna + support construction wouldn't work in the location?  

A 4G Antenna / repeater is an option.  But would need to to coordinate with telecom to put it up.  They will gladly take your money to improve their infrastructure.  Time and cutting thru red tape to arrange such a thing is an issue (government involvement to put up a structure).  Plans in this country are also somwhat expensive for data, but somewhat coming down in price as of late.  Cheap electricity = remote = bad infrastructure for tech.

Thanks for your interest and posting though