[1500 TH] p2pool: Decentralized, DoS-resistant, Hop-Proof pool

[bongo1]

Thank you for your time and I appreciate your help.

I feel like my miners have been hijacked.I started mining using P2Pool in mid June. All seem fine until on or about 7/2/2014 10:14:08 PM and I notice that I am not getting pay for blocks found. My hash rate as shown on P2p stats page seem consistent about 80 Gh/s with what I should get but on the configuration page of my blades it is much lower. My hash rate on each of my blades use to be 10 Gh/s but now it fluctuates about 6-8Gh/s max on each blade.

I reset the blades to their original factory setting and reconfigured it again and every thing seem fine and now I see

Payout if a block were found NOW: 0.00154952 BTC to Xxxxxxxx was no longer 0. After a few days it went back to 0. I left it for a few days and nothing happened.

I reonfigured everything and it started working just fine for a few days except for the hash rate issues but the Payout amount also dropped by half. (I believe I read in this forum that it was because of the increase in the network hash rate).

I think I noticed a pattern (Well it could be a coincidence). When this happens there is always a rash of blocks are found and I miss out completely in the payments.

I also use CGwatcher to monitor my miners. I am concerned with the line entries in bold. What those it mean? I do not have a Pool 1. During this period I upgraded my CGwatcher and Bfgminer but going back to the earlier versions doesn't help either.

In addition my CGwatcher takes long to start up my mining configs than usual. It stops responding completely.

--- log entry when I started P2Pool. I had no Pool 1.

[6/26/2014 12:29:01 AM]  BFGMiner (5052): Process started using "P2Pool mining Bitcoin" profile on port 4028.
[6/26/2014 12:30:35 AM]  BFGMiner (5052): Pool 0 (@http://127.0.0.1:9332) status is ALIVE
[6/26/2014 12:30:35 AM]  BFGMiner (5052): Current pool is Pool 0 (1CpnaoUNINvaqvaIIINqmanTZ3m8@http://127.0.0.1:9332)
[6/26/2014 12:30:35 AM]  BFGMiner (5052): Network difficulty is now 13,462,785,540.
[6/26/2014 12:30:41 AM]  BFGMiner (5052): GPU0 (AMD Radeon HD 7800 Series) status is UNKNOWN
[6/26/2014 12:30:41 AM]  BFGMiner (5052): PXY0 status is ALIVE

[6/26/2014 8:24:33 PM]  BFGMiner (4028): Process started using "P2Pool mining Bitcoin" profile on port 4028.
[6/26/2014 8:25:35 PM]  BFGMiner (4028): Pool 0 (@http://127.0.0.1:9332) status is ALIVE
[6/26/2014 8:25:35 PM]  BFGMiner (4028): Current pool is Pool 0 (1CpnaoUNINvaqvaIIINqmanTZ3m8@http://127.0.0.1:9332)
[6/26/2014 8:25:35 PM]  BFGMiner (4028): Network difficulty is now 13,462,785,540.
[6/26/2014 8:25:35 PM]  BFGMiner (4028): GPU0 (AMD Radeon HD 7800 Series) status is UNKNOWN
[6/26/2014 8:25:35 PM]  BFGMiner (4028): PXY0 status is ALIVE
[6/26/2014 8:32:03 PM]  BFGMiner (4028): Miner restart requested...
[6/26/2014 8:32:06 PM]  BFGMiner (4028): Process closed.


--- When the Payout was 0 and miners started acting up.

[7/2/2014 10:14:08 PM]  BFGMiner (3508): Process started using "P2Pool mining Bitcoin" profile on port 4028.
[7/2/2014 10:14:09 PM]  BFGMiner (3508): Pool 0 (@http://127.0.0.1:9332) status is DEAD
[7/2/2014 10:14:09 PM]  BFGMiner (3508): Pool 1 (@http://localhost:8332/#getcbaddr#allblocks) status is FAILOVER
[7/2/2014 10:14:09 PM]  BFGMiner (3508): Current pool is Pool 0 (1CpnaoUNINvaqvaIIINqmanTZ3m8@http://127.0.0.1:9332)
[7/2/2014 10:14:17 PM]  BFGMiner (3508): PXY0 status is ALIVE


--- Most recent entry in CGwatcher log file.

[7/17/2014 12:53:31 PM]  BFGMiner (5564): Process started using "P2Pool mining Bitcoin" profile on port 4028.
[7/17/2014 12:54:46 PM]  BFGMiner (5564): Pool 0 (@http://127.0.0.1:9332) status is DEAD
[7/17/2014 12:54:46 PM]  BFGMiner (5564): Pool 1 (@http://localhost:8332/#getcbaddr#allblocks) status is FAILOVER
[7/17/2014 12:54:46 PM]  BFGMiner (5564): Current pool is Pool 0 (1CpnaoUNINvaqvaIIINqmanTZ3m8@http://127.0.0.1:9332)
[7/17/2014 1:10:42 PM]  BFGMiner (5564): Pool 0 (1CpnaoUNINvaqvaIIINqmanTZ3m8@http://127.0.0.1:9332) status is ALIVE
[7/17/2014 1:10:42 PM]  BFGMiner (5564): Network difficulty is now 17,336,581,514.
[7/17/2014 1:10:52 PM]  BFGMiner (5564): GPU0 (AMD Radeon HD 7800 Series) status is UNKNOWN
[7/17/2014 1:10:52 PM]  BFGMiner (5564): PXY0 status is ALIVE

--- Another log file

[7/17/2014 12:33:32 PM]  [d] Miner.StartMining.DeleteTemporaryBatchFile 'D:\bitminer\windows\scripts\asicminer\temp_P2PPOOL_bfgminer_asicminer_BTC.bat' deleted successfully.
[7/17/2014 12:33:32 PM]      Current pool is http://127.0.0.1:9332
[7/17/2014 12:33:32 PM]      Unauthorized pool found! Pool 1 (fdrpcuser @ http://localhost:8332/#getcbaddr#allblocks) was not added in CGWatcher, CGRemote, or the miner's config.
[7/17/2014 12:33:42 PM]  [d] Miner.SendCommand (devs) returned status of 'E' (STATUS=E,When=1405614822,Code=10,Msg=No PGAs,Description=bfgminer 4.3.0, )!
[7/17/2014 12:53:18 PM]      -- Begin CGWatcher v1.3.9.0 Process ---

[windpath]

When I put in the address you displayed it returns as invalid...

1CpnaoUNINvaqvaIIINqmanTZ3m8



http://minefast.coincadence.com/miner.php?id=1CpnaoUNINvaqvaIIINqmanTZ3m8

[-ck]

I'm wondering,  with high variance on p2pool or another somewhat smaller pool,  is it possible to actually lose money?  Like in the long run luck is irrelevant,  but what if you have really bad luck and then the difficulty goes up 30%?

In this situation do you lose money because of high variance?

Yes you can. Consequently you can also gain money. Contrary to what people will insist on, it does NOT even out in the end because difficulty keeps rising. However luck determines if you're ahead long term or behind long term in an irretrievable way. If diff did not continually rise and rose-and-fell or stayed the same it would even out.

Your argument is based on the statement that difficulty keeps rising.  While historically this is true, it is not a guarantee of future patterns.  Yes, we have seen some insane rises in difficulty, but that is due to a number of factors including wider acceptance, general knowledge of crypto-currencies, incredible hardware advances, etc.  I don't think we can state as a fact that difficulty will forever continuously rise.

I did not argue that it will rise indefinitely. However, it's realistic to predict it will continue to rise until it levels off. Once it levels off, it is never going to fluctuate as much as it did to pre-rise diff levels. Until it levels off (and this could be years away) this issue will hold true, and all the variance experienced till that point can never be caught up on.  As I said though it's not a guaranteed loss, it's almost as likely a guaranteed win. The fact that by definition bad luck means blocks take longer means the statistical bias is towards a loss, but that statistical bias is nowhere near as large as the variance effect itself is.

Put it this way, many diffs ago, on a lucky day in mining, you could earn 10 bitcoin in one day or 1 bitcoin on a bad day with 60GH.  In today's diffs, even if they fluctuate a little instead of just going up, there is no way in an eternity you could ever make up those 9 bitcoin with 60GH unless you used your avalon as a weapon to hit someone over the head and steal their money.

The next block halving will probably be the only time we'll ever see a significant drop again (unless BTC value crashes). What effect that will have on a tight margin network (unlike the GPU days) is unpredictable.

[nreal]

Thank you all for this great waste of time and electricity.

Even when im picking blueberies im thinking p2pool  

[abdullahadam]

Windpath, both your DNS entries (mining.coincadence.com and mining2.coincadence.com) are actually pointing to the same IP?

Pinging mining.coincadence.com [54.235.94.19]
Pinging mining2.coincadence.com [54.235.94.19]

[abdullahadam]

Put it this way, many diffs ago, on a lucky day in mining, you could earn 10 bitcoin in one day or 1 bitcoin on a bad day with 60GH.  In today's diffs, even if they fluctuate a little instead of just going up, there is no way in an eternity you could ever make up those 9 bitcoin with 60GH unless you used your avalon as a weapon to hit someone over the head and steal their money.

The next block halving will probably be the only time we'll ever see a significant drop again (unless BTC value crashes). What effect that will have on a tight margin network (unlike the GPU days) is unpredictable.

LOL.  Not only are you intelligent, you can also explain things very well.  Excellent work

[bongo1]

When I put in the address you displayed it returns as invalid...

1CpnaoUNINvaqvaIIINqmanTZ3m8

You are correct the address I gave is wrong. I edited out the correct address. Do you need the it?

Thanks.

[windpath]

Windpath, both your DNS entries (mining.coincadence.com and mining2.coincadence.com) are actually pointing to the same IP?

Pinging mining.coincadence.com [54.235.94.19]
Pinging mining2.coincadence.com [54.235.94.19]

Note: the subdomain is MINEFAST not mining

Yes, they are. Two reasons:

1. We occasionally start a new instance of p2pool (on the same node) and shut down the old process. This is a feature of p2pool and allows a restart of p2pool without having any downtime. When this happens, even with the second instance already running, miners will jump to one of the backup pools in your configuration, by using our node for pool 1 & pool 2, and having a separate p2pool node as pool 3 (as suggested in our configuration section) you minimize time a miner would be mining on another node/pool and increase the likelihood of them submitting a valid share.

2. We have big plans Currently we are running a single node in an eastern US data-center (a very well connected one). While our node has actually proven fast enough to handle miners that are geographically further away then I anticipated because of low latency, I do still plan to add more nodes in the future. I've been playing around with Latency Based Routing (LBR), and once I'm confident I have it working efficiently will use 54.235.94.19 as the gateway to a group of geographically distributed p2pool nodes that automatically route miner traffic to the lowest latency node. When this is operational minefast.coincadence.com will represent your closest node based on latency, minefast2.coincadence.com will represent the 2nd closest node based on latency... More on this later

[windpath]

When I put in the address you displayed it returns as invalid...

1CpnaoUNINvaqvaIIINqmanTZ3m8

You are correct the address I gave is wrong. I edited out the correct address. Do you need the it?

Thanks.

I don't necessarily need it, if I had it I'd be able to see your payout per block which would help verify what your talking about.

Also, if that was the actual (unedited) log output that obviously would be why you were not receiving payouts

[jonnybravo0311]

I'm wondering,  with high variance on p2pool or another somewhat smaller pool,  is it possible to actually lose money?  Like in the long run luck is irrelevant,  but what if you have really bad luck and then the difficulty goes up 30%?

In this situation do you lose money because of high variance?

Yes you can. Consequently you can also gain money. Contrary to what people will insist on, it does NOT even out in the end because difficulty keeps rising. However luck determines if you're ahead long term or behind long term in an irretrievable way. If diff did not continually rise and rose-and-fell or stayed the same it would even out.

Your argument is based on the statement that difficulty keeps rising.  While historically this is true, it is not a guarantee of future patterns.  Yes, we have seen some insane rises in difficulty, but that is due to a number of factors including wider acceptance, general knowledge of crypto-currencies, incredible hardware advances, etc.  I don't think we can state as a fact that difficulty will forever continuously rise.

I did not argue that it will rise indefinitely. However, it's realistic to predict it will continue to rise until it levels off. Once it levels off, it is never going to fluctuate as much as it did to pre-rise diff levels. Until it levels off (and this could be years away) this issue will hold true, and all the variance experienced till that point can never be caught up on.  As I said though it's not a guaranteed loss, it's almost as likely a guaranteed win. The fact that by definition bad luck means blocks take longer means the statistical bias is towards a loss, but that statistical bias is nowhere near as large as the variance effect itself is.

Put it this way, many diffs ago, on a lucky day in mining, you could earn 10 bitcoin in one day or 1 bitcoin on a bad day with 60GH.  In today's diffs, even if they fluctuate a little instead of just going up, there is no way in an eternity you could ever make up those 9 bitcoin with 60GH unless you used your avalon as a weapon to hit someone over the head and steal their money.

The next block halving will probably be the only time we'll ever see a significant drop again (unless BTC value crashes). What effect that will have on a tight margin network (unlike the GPU days) is unpredictable.

Con, the statement that I bolded is absolutely fantastic.  I laughed out loud at it.  I see what you're getting to here as well, that good luck in the past might have earned you 10BTC on a good luck day, but only 1BTC on a bad luck day.  The earnings are considerably less now, and I wonder if that same statement holds.  On a good day, you can earn 1BTC but on a bad day you might get 0.1BTC.  Can you make up that 0.9?  With the current state, would the variance be that large now - i.e. swinging by as much as 10x?  Personally, I've not noticed my payouts vary that wildly.  They have their up and down swings, but nothing that extreme.  It would be fun to see this kind of thing plotted out.

[jonnybravo0311]

So, I have now had my batch 1 Antminer S3s running for about 18 hours.  Throughout this time, I've been playing with them, tweaking settings, changing pools, etc.  Here are my results regarding p2pool to this point:

• Overclocking.  Everyone wants to know, "How well do they over clock?" The answer is, at least from my experience thus far, "Not well." You can over clock them up to a chip frequency of 250 using the asic-freq configuration file - which is the same file you used in the S1.  The stock clock is set to 218.75.  My recommendation is to keep them right there for the best stability and consistent hash rate on the pool.
• Hashing speed, hardware errors and ASIC failures.  I had different experiences with each of my S3s (I only have 2).  I left them both overclocked to 250 overnight to get a gauge on things.  On my first unit, the miner stats page reported about 450GH/s, and had an "x" or two sprinkled in the ASICs.  My second unit reported about 387GH/s average.  It had a "-" on one of the ASICs.  I noticed this behavioral pattern throughout my tests (set a clock speed, let it hash for an hour or so and look).  The hardware errors actually were exceptionally good no matter what clock speed I tried - I think 0.5% was about the highest I saw in one test.
• Hashing on a different pool.  I tried experiments on Eligius, BTCGuild and Ghash.io, hashing on each for about an hour or so at different clock speeds.  While I did see slightly higher reported rates in the S3 stats, it was not like what we saw with the S2 losing 10-15%.  Yes, at one point, one of my S3s was telling me it was hashing at 500GH/s, but that didn't last long and I got a bunch of "x" in the ASIC stats.
• Power.  I didn't try playing around with different cable configurations - like what happens if I try to set the clock to 250 but only put in 2 cables?  I might try something like that today.  I have all 4 PCI-e connectors powered on each device currently because I was playing around with over clocking.  I will be going down to 2 each, and will report back later on the effects running at stock clocks.  By the way, both S3s are driven on a single EVGA G2 1300.
• Draw at the wall.  Unfortunately I don't have a Kill-a-Watt, or any other tool to measure actual wattage, so hopefully somebody who does can chime in here.  All I can tell you is that even running full out at max over clocking the PSU fans were never spinning loudly.  I'm pretty sure it was barely even noticing the draw.
• Under clocking.  I have yet to try lower clock speeds and would be interested to see the hash rate and power draw results from somebody who does it.  If you could get a consistent 400-420GH/s for 250-300 watts, then running 4 on a single PSU would be a very worthwhile operation.
• Random weirdness.  I don't know how to describe this, or why it happens, but my S3s will randomly let out a beep.  It's the same beep as the "just coming online" or "you just rebooted" beep.  At first I thought my units were just constantly rebooting themselves, but that's not the case.  They just like to emit a random beep, I guess to let me know that they are still there.  There is a configuration in the Miner Configuration tab that allows you to switch off beeping - and it works for when you lose connection to the network - but it has no effect on this random beep.

At this point, I'm keeping my S3s running stock clocks and pointed to my local p2pool node.  Below are the screenshots of my 2 S3s at stock clocks hashing to my local node:

[bongo1]

When I put in the address you displayed it returns as invalid...

1CpnaoUNINvaqvaIIINqmanTZ3m8

You are correct the address I gave is wrong. I edited out the correct address. Do you need the it?

Thanks.

I don't necessarily need it, if I had it I'd be able to see your payout per block which would help verify what your talking about.

Also, if that was the actual (unedited) log output that obviously would be why you were not receiving payouts

@windpath: Here it is Payout address  1CpWdpMT3ygMUxJKrspF8ax4EDPmyTZ3m8

[windpath]

...

@windpath: Here it is Payout address  1CpWdpMT3ygMUxJKrspF8ax4EDPmyTZ3m8

I can see the gaps in your payouts now, whats your average expected time to share?

Have you tried taking a look at BFGminers pool management console while you think this may be occurring (press P to access pool management in the console)?

[windpath]

At this point, I'm keeping my S3s running stock clocks and pointed to my local p2pool node.  Below are the screenshots of my 2 S3s at stock clocks hashing to my local node:

This is good great news, looks like Bitmain came through!

My batch 1s are in transit, should be a Monday delivery.

Thanks for sharing

Edit: And look at that error rate:

Top miner: 0.00125648267%
Bottom miner: 0.00073813656%

WOW!

[Hunterbunter]

I'm a bit disappointed by my S3's on P2Pool. After 1 day mining:

The top one got a lot of hardware errors so I just run it at stock, the bottom one overclocked quite well to 250...started off well on P2Pool, but reported hashrate is pretty bad on both. Note: the node is not close to me, but it shouldn't show this much difference - my old S1's did fine with it.

EDIT: anyone know what the discarded shows? Mine seems pretty high.

[windpath]

I'm a bit disappointed by my S3's on P2Pool. After 1 day mining:

The top one got a lot of hardware errors so I just run it at stock, the bottom one overclocked quite well to 250...started off well on P2Pool, but reported hashrate is pretty bad on both. Note: the node is not close to me, but it shouldn't show this much difference - my old S1's did fine with it.

EDIT: anyone know what the discarded shows? Mine seems pretty high.

[IM]https://i.imgur.com/tCkQfJN.jpg[/img]

Your error rates look great, both well under 1%, have you tried another pool to see how they preform?

jonnybravo0311 seemed to get the best results running them at stock clocks....

Edit: Ant error rate calculator: http://www.coincadence.com/antminer-s1-hardware-error/

(was built for S1, but works the same for S3...)

[jonnybravo0311]

I'm a bit disappointed by my S3's on P2Pool. After 1 day mining:

The top one got a lot of hardware errors so I just run it at stock, the bottom one overclocked quite well to 250...started off well on P2Pool, but reported hashrate is pretty bad on both. Note: the node is not close to me, but it shouldn't show this much difference - my old S1's did fine with it.

EDIT: anyone know what the discarded shows? Mine seems pretty high.

[IM]https://i.imgur.com/tCkQfJN.jpg[/img]

Your error rates look great, both well under 1%, have you tried another pool to see how they preform?

jonnybravo0311 seemed to get the best results running them at stock clocks....

Edit: Ant error rate calculator: http://www.coincadence.com/antminer-s1-hardware-error/

(was built for S1, but works the same for S3...)

So far, yes, I've seen the best results on stock clocks.  One thing I have noticed is that one of my S3's performance slowly degrades.  It hashes at 440GH/s for a while... then slows down to 430 and 420.  Rebooting it sometimes helped, but a lot of times on reboot, I'd notice one of the ASIC would show a status of "-" instead of "o".

So, this morning I completely stripped the thing.  When I pulled off the heatsinks, I saw thermal paste everywhere.  It was on the chips, on the PCB, on everything.  So, I cleaned it all up and re-applied to the chips.  Everything is put back together and it's up and hashing again.  Hopefully, it will stay consistent now.  We'll see .

I'm probably going to do the same to my other one as well.  Then, I'll retry over clocking them both.  Some people have reported amazing success.  For example, Goxed has his happily hashing away at 504GH/s (250 clock).  Now that would be nice!

[windpath]

Yea, the S1s came littered with thermal paste as well, little easier to clean up without the extra heat sinks though....

[mdude77]

I'm a bit disappointed by my S3's on P2Pool. After 1 day mining:

The top one got a lot of hardware errors so I just run it at stock, the bottom one overclocked quite well to 250...started off well on P2Pool, but reported hashrate is pretty bad on both. Note: the node is not close to me, but it shouldn't show this much difference - my old S1's did fine with it.

EDIT: anyone know what the discarded shows? Mine seems pretty high.

[IM]https://i.imgur.com/tCkQfJN.jpg[/img]

Your error rates look great, both well under 1%, have you tried another pool to see how they preform?

jonnybravo0311 seemed to get the best results running them at stock clocks....

Edit: Ant error rate calculator: http://www.coincadence.com/antminer-s1-hardware-error/

(was built for S1, but works the same for S3...)

So far, yes, I've seen the best results on stock clocks.  One thing I have noticed is that one of my S3's performance slowly degrades.  It hashes at 440GH/s for a while... then slows down to 430 and 420.  Rebooting it sometimes helped, but a lot of times on reboot, I'd notice one of the ASIC would show a status of "-" instead of "o".

Sounds like S2 behavior.

M

[kgb2mining]

We just switched mining to p2pool from BTCGuild.  No general problems over there, but I have been reading about p2pool and think it may be a better choice in the long term for many reasons.

I'm considering building our own pool node since we have the computing resources to do it in a proper datacenter setting.  I have a couple main questions about resource needs, if anyone who's running pools currently could chime in with real-world experiences I'd appreciate it.

First off, can I run a "private" node, inside the firewall without opening any ports inbound?  Just something that is part of the network that we could point our miners to for our use for now.  Having the ability to do this would help the decision on where to run the node, being inside would help the latency between the miners and the node.  Eventually I could see us opening it up for others to use if there's enough need for a local node (Northern NJ, which seems to have good coverage for now I think).

Second, what are you seeing for bandwith needs, both spike and average consistent rate?  Pulling up to 10Mbit/s is doable without much notice or problem, but if we're pulling more than that on a consistent basis that wouldn't be too good.

The node would be based off of Ubuntu 14.04 virtual machines (4CPU, 8GB RAM to start).

Thanks.