Data limits becoming an issue......any thoughts?

[yochdog]

Hey all,

One of my three locations is using Verizon broadband for internet access, and it works great.  Unfortunately, I am now consistently going over the 10 GB data limit 2/3's through the billing cycle and am charged 10 bucks per GB overage. 

The facility hosts approximately 16-17 GH/s.

Is it normal to burn through 10 GB in data in only 20 days with this type of hashing power?  Is there any way to reduce the data usage?  Or am I just F-ed?

thanks! 

[1714860451]

1714860451

[1714860451]

1714860451

[davout]

10G isn't that much, your plan looks quite smallish

[jamesg]

I hard code pool ip addresses into the computer hosts files to cut down on DNS lookups. Not sure how much of your traffic they actually make up though and I've done this more to keep the router connections down than anything else.

[RandomQ]

I hard code pool ip addresses into the computer hosts files to cut down on DNS lookups. Not sure how much of your traffic they actually make up though and I've done this more to keep the router connections down than anything else.

DNS lookups are cached locally on each system by most OS, so this should produce very little data usage. My guess around data usage for DNS lookup would be under 100 megs per 10 gigs. your talking about asking to converting "www.greenbtc.com" to "124.0.0.1" very liittle data involved


A real way to save data is to increase your share difficulty, the higher your difficulty the less shares you will produce and the less data you will transfer.
Say you systems are working on difficulty 1 shares and you switch to difficulty 4 shares, this would cut down your shares produced ~ 75% and save data.
You would just get paid more for the higher difficulty Shares.

but your pool has to support this and your miner software has to support this

Are you using a 4G,3G,Wimax based broadband modem?

[yochdog]

I hard code pool ip addresses into the computer hosts files to cut down on DNS lookups. Not sure how much of your traffic they actually make up though and I've done this more to keep the router connections down than anything else.

DNS lookups are cached locally on each system by most OS, so this should produce very little data usage. My guess around data usage for DNS lookup would be under 100 megs per 10 gigs. your talking about asking to converting "www.greenbtc.com" to "124.0.0.1" very liittle data involved


A real way to save data is to increase your share difficulty, the higher your difficulty the less shares you will produce and the less data you will transfer.
Say you systems are working on difficulty 1 shares and you switch to difficulty 4 shares, this would cut down your shares produced ~ 75% and save data.
You would just get paid more for the higher difficulty Shares.

but your pool has to support this and your miner software has to support this

Are you using a 4G,3G,Wimax based broadband modem?

It is a router from Cradlepoint, with a USB 4G modem plugged into it. 

[Shadow383]

Where are you pointing all that hashrate? The new mining proxy on slush's pool would probably use less bandwidth, not sure if that's an option?

[DeathAndTaxes]

Some options:
a) Solo mining.
Enough said.  All the advantages and disadvantages which come with it.

b) Higher difficulty shares.
Using a pool which supports higher difficulty shares will reduce outgoing bandwidth.  You may also wan to encouraging other pools to adopt higher difficulty shares.  For example on a pool using difficulty 32 shares you will find (and thus transmit) 1/32 as many shares. Compensation is the same as each share is "worth" 32x as much.  Pools could support dynamic difficulty as optimal share difficulty varies by hashing power.  20GH/s at difficult 1 will find (and need to report) 12,069,941 shares every month. At ~ 0.5KB per share that is ~6GB per month.  Note that is just outbound data and is under optimal conditions.  Using a difficult of 100 would reduce that by 99%.  Inbound bandwidth remains the same at 1 (or more) getworks per physical processor (CPU core, GPU, FPGA chip, etc) per block.  

c) Getwork splitting.
This may require more work that it is worth.  Higher difficulty shares improve outbound bandwidth but even with longpolling and ntime rolling you need at least 1 getwork per physical processor per block.  So a farm with 50 GPU farm requests 50 block headers on every longpoll and one every time a share is found.  In theory a single getwork could be split across multiple GPU/FPGA (by giving each one of them part of the nonce range).  There is still an upper limit of roughly 1 getwork per 4GH/s per block.  That limit exists because more than 4GH/s will need more than 1 getwork per second and the timestamp has a precision limit of one second.

d) Local work.
Local work is likely the long term solution given the rise of TH/s ASICs.  It would operate similar to how p2pool works but still using a central "traditional" pool server.  Essentially solo mining with shared rewards/variance.   There is no real requirement that the server generate the blockheader for all miners and distribute them.  The only thing the server needs to provide (and verify on submitted hashes) is the coinbase address to ensure the pool (and all miners) are equitably paid.  It would require new mining software but a miner could construct the entire blockheader locally as needed.  In theory this would reduce inbound bandwidth to a negligible amount.   Outbound bandwidth per share is increased as miner needs to submit the merkle tree to pool for verification.  Still by using higher difficulty shares the overall outbound bandwidth can be reduced.  Say using 100 difficulty shares even if the share requires 20x as much bandwidth we are still talking an 80% reduction in overall outbound bandwidth and incoming bandwidth is reduced to a negligible (rounding error) amount.


c & d require real work however there is no reason pools couldn't support difficulty >1 today.

[Vandroiy]

Why are you paying mobile phone traffic prices, isn't there a decent connection available?

I once threw 1.2 TB over my home connection in one month, just because. And no, I did NOT pay 12k USD. Less than 100, actually. And I still live in the broadband-middle-ages. FTTH deployment is spreading... expect times when the network beats the RAID.

Anyway, that's a seriously bad connection for business stuff. Can't you do something about that rather than trying to cut down a few GB traffic? You'll be poor if someone uses YouTube from there!

[smracer]

I had the same problem at one location where I have 14Gh.  I upgraded to a DSL line for $29 a month.

If that is not an option can you add another 4G modem to your account with another 10GB and just split up the miners?

[Remember remember the 5th of November]

My monthly traffic is usually 2.5-3TB. 10GB in 20 days, Nowhere near enough for me.

[yochdog]

Are their any major pools accepting higher difficulty shares? 

I have to use the Verizon because the location is not served by normal broadband providers. 

[Shadow383]

My monthly traffic is usually 2.5-3TB. 10GB in 20 days, Nowhere near enough for me.

Have you ever thought about downloading porn that isn't 1080p?  

[Shadow383]

Are their any major pools accepting higher difficulty shares? 

I have to use the Verizon because the location is not served by normal broadband providers. 

Eclipse has a higher-difficulty pool, there's also HHTT, which isn't exactly major but it's nice and simple - 2% fee PPS with difficulty 32 shares:
https://bitcointalk.org/index.php?topic=95378.0

[crazyates]

This might seem pretty basic, but do both your pool and your miner support rollNtime? This drastically reduces the number of getworks you're requesting, especially for 17GH/s.

[yochdog]

This might seem pretty basic, but do both your pool and your miner support rollNtime? This drastically reduces the number of getworks you're requesting, especially for 17GH/s.

I use CGminer in conjunction with GPUmax to manage my farm.....

[crazyates]

This might seem pretty basic, but do both your pool and your miner support rollNtime? This drastically reduces the number of getworks you're requesting, especially for 17GH/s.

I use CGminer in conjunction with GPUmax to manage my farm.....

GPUmax just acts as a proxy. What pools do GPUmax point to?

[yochdog]

This might seem pretty basic, but do both your pool and your miner support rollNtime? This drastically reduces the number of getworks you're requesting, especially for 17GH/s.

I use CGminer in conjunction with GPUmax to manage my farm.....

GPUmax just acts as a proxy. What pools do GPUmax point to?

Mostly 50BTC and OZcoin

[Shadow383]

This might seem pretty basic, but do both your pool and your miner support rollNtime? This drastically reduces the number of getworks you're requesting, especially for 17GH/s.

I use CGminer in conjunction with GPUmax to manage my farm.....

GPUmax just acts as a proxy. What pools do GPUmax point to?

Mostly 50BTC and OZcoin

Are you actually getting enough GPUMax work for diff-1 shares and higher stale rates to be worthwhile?

[yochdog]

This might seem pretty basic, but do both your pool and your miner support rollNtime? This drastically reduces the number of getworks you're requesting, especially for 17GH/s.

I use CGminer in conjunction with GPUmax to manage my farm.....

GPUmax just acts as a proxy. What pools do GPUmax point to?

Mostly 50BTC and OZcoin

Are you actually getting enough GPUMax work for diff-1 shares and higher stale rates to be worthwhile?

works like a charm 99% of the time......

[sadpandatech]

This might seem pretty basic, but do both your pool and your miner support rollNtime? This drastically reduces the number of getworks you're requesting, especially for 17GH/s.

I use CGminer in conjunction with GPUmax to manage my farm.....

GPUmax just acts as a proxy. What pools do GPUmax point to?

Mostly 50BTC and OZcoin

Are you actually getting enough GPUMax work for diff-1 shares and higher stale rates to be worthwhile?

works like a charm 99% of the time......

Your rollNtime effectiveness is likely being dimineshed by the extra packet time bouncging thru gpuMAX.  Point them all direct for 7 days and check your bandwidth usage then.

[yochdog]

This might seem pretty basic, but do both your pool and your miner support rollNtime? This drastically reduces the number of getworks you're requesting, especially for 17GH/s.

I use CGminer in conjunction with GPUmax to manage my farm.....

GPUmax just acts as a proxy. What pools do GPUmax point to?

Mostly 50BTC and OZcoin

Are you actually getting enough GPUMax work for diff-1 shares and higher stale rates to be worthwhile?

works like a charm 99% of the time......

Your rollNtime effectiveness is likely being dimineshed by the extra packet time bouncging thru gpuMAX.  Point them all direct for 7 days and check your bandwidth usage then.

Sounds like a plan.....thanks

[eleuthria]

If data limits are a strong concern, the stratum protocol completely solves the issue.  Both slush and BTC Guild are running Stratum protocol beta pools.  On a large farm, you can see upstream bandwidth reduction in the 99%+ range with dynamic difficulty+stratum.  You'll also see inbound bandwidth reduced to ~3-5 KB per minute (~60 bytes per second average, though it happens in bursts of ~1 KB at a time).

[Ascholten]

Call Verizon.  I remember about a few months ago, them saying they were going to offer an unlimited plan, because sprint has one and was stealing too much of their business with it.  When I dumped the for sprint when I got my new smart phone, they asked why and I said unlimited data, they said oh but we got that now, I said, sorry, too little too late, plus they throttle.

look into sprint, you may find it cheaper overall if it is in your area, also call your current verizon, they might have the unlimited for you.

Aaron

[thebaron]

I'm not sure why pool operators would choose to use up so much bandwidth. You would think they could automatically detect your hashing power and adjust things accordingly.

[P_Shep]

2 things:

1. EMC now has variable difficulty shares on all their servers. It scales according to your Hashrate
2. There's a proxy for N-roll-time which could help distribute work to all your machines: https://bitcointalk.org/index.php?topic=103248.0

Untill GetBlockTemplate and Slush's Protocols become de-facto, these could well reduce your bandwith by some margin.

[Korbman]

After reading all this I'm still not quite sure I gathered how much bandwidth is needed for a typical rig. What sort of internet speeds (upload and download) are needed for typical mining pools of 500GH/s+? I know it won't translate the same to the ASIC devices (since they won't have thousands of simultaneous connections for mining), but I'm still curious.

[crazyates]

Cgminer now supports Stratum, and Slush's pool and BTCguild both support in on their side as well. Using a combination of these will drastically reduce your data sent/received.

[mdude77]

2 things:

1. EMC now has variable difficulty shares on all their servers. It scales according to your Hashrate
2. There's a proxy for N-roll-time which could help distribute work to all your machines: https://bitcointalk.org/index.php?topic=103248.0

Untill GetBlockTemplate and Slush's Protocols become de-facto, these could well reduce your bandwith by some margin.

I recommend EMC for now.  I'm thinking with 2% fee at ozcoin, but it has transaction fees included in payout, that they are roughly equal.  But that will change once the great halving occurs.  How, I don't know, but it'll change.

I tried the proxy mentioned above and it eventually started causing me problems.  I think it's a resource issue, but not sure.

As someone else mentioned, cgminer has stratum support now, but it's early release code, not necessarily stable yet.  However not many pools support stratum yet.  Using EMC (us1 server) should significantly increase your share size and hence decrease your bandwidth.

M

[Bogart]

It is a router from Cradlepoint, with a USB 4G modem plugged into it.  

On a side note, I use a USB 3G modem like this for my home internet (it's my only option where I live).

I had a Cradlepoint router like this, and it kind of worked, though sometimes it would hang and need to be reset.

Then I replaced it with an Asus RT-N13U and put dd-wrt on it, and what a difference!  My average data rate doubled, and stability is a lot better.

Somehow I suspect the Cradlepoint had trouble delivering enough power to the modem.  I never did try using a powered hub with it.

As for bandwidth, I solo mine.  I'm also on a grandfathered plan not subject to a data limit, but I think they're gonna try and take it away from me at the end of the year.   

[RyanRed]

+1 for Stratum. Supportability uses less bandwidth. I would think a lot of pools might move to it over time with the increased hash rate on the horizon.

[underminer]

Hey this is off the wall but I though it might be helpful:

I used to use these things that would connect internet through power lines (sometimes called poe I think) to connect customers to the internet who didn't have high demand for speeds.  I have linked up to 6 total before, as there is a sort of a "breadcrumbs" way of linking them together, and I have heard of people using them between houses.

Dlink makes a good one but I believe they are everywhere now.  Anyway, my thought is that you can feasibly run a durable extension cord between houses more easily than just a typical cord if you can negotiate with a neighbor somehow.  If it helps cool otherwise disregard lol!

[mdude77]

Hey this is off the wall but I though it might be helpful:

I used to use these things that would connect internet through power lines (sometimes called poe I think) to connect customers to the internet who didn't have high demand for speeds.  I have linked up to 6 total before, as there is a sort of a "breadcrumbs" way of linking them together, and I have heard of people using them between houses.

Dlink makes a good one but I believe they are everywhere now.  Anyway, my thought is that you can feasibly run a durable extension cord between houses more easily than just a typical cord if you can negotiate with a neighbor somehow.  If it helps cool otherwise disregard lol!

I used them here before I hard wired the house for cat5e.  They work well, although they are a bit slow.  Might work well for low bandwidth mining.  They can't get through transformers, so if you and your neighbor(s) are on the same transformer, you'll be able to access each other (within reason).  If that's not desirable, make sure you use one with encryption and make it a secure password!

M

[firefop]

Having dealt with your situation while playing admin for a med sized business with mobile (back of semi truck) workshops... I feel your pain.

By far the best solution is to just get another 4g plan identical to the one you have now. If your device supports it you can swap sim card every 15 days and stop worrying about it. If not you'd have to just swap each device out every 15 days. Not the most elegant solution but it works - is easy to manage and won't cost you an arm and a leg.

[BitcoinOxygen]

Cgminer now supports Stratum, and Slush's pool and BTCguild both support in on their side as well. Using a combination of these will drastically reduce your data sent/received.

+1 for Stratum

[yochdog]

Having dealt with your situation while playing admin for a med sized business with mobile (back of semi truck) workshops... I feel your pain.

By far the best solution is to just get another 4g plan identical to the one you have now. If your device supports it you can swap sim card every 15 days and stop worrying about it. If not you'd have to just swap each device out every 15 days. Not the most elegant solution but it works - is easy to manage and won't cost you an arm and a leg.

This is what I have been doing as a work-around......