Update:  

I just tried it again now and had the same problem.  In particular, I noticed errors about requests to bitcoind (http://127.0.0.1:8332) timing out (getwork requests and getting tx details, if I recall).

I was running this version of bitcoind: https://github.com/forrestv/bitcoin/tree/getrawtransaction because I thought it would be beneficial to have the rawtransaction support (side note: isn't our mining kind of useless to the network if we never include other people's transactions in our found blocks?  maybe I am misunderstanding the functionality added when getrawtransaction is present).

In any case, I tried switching to the officiall 0.3.24 release version of bitcoind and all of the problems went away.  The timeout messages are gone and my miner is now reliably mining and finding shares.

I tried this last night and didn't have much success.  Here is the sequence I saw a couple times before I gave up:

* Start local bitcoind
* Startup p2pool python code with the user and password to the local bitcoind and with the -a parameter to specify an address from my real wallet (the local bitcoind does not have my real wallet and I don't want it to hold funds since it is not secured)
* Start poclbm miner (up to date from git)
* poclbm starts mining ok initially
* At some point, in the p2pool kind of goes insane and there is a long flurry of back to back messages about "generating".
* At about the same point or slightly after, the miner starts reporting RPC problems.  'getwork' requests are timing out and not returning.  Restarting the miner does not help.
* The "generating" flurry stops and the output of p2pool goes back to normally looking debug spam, but the miner never seems to be able to 'getwork' again and is stuck.
* Restarting p2pool makes things start working again for another minute or two

I apologize that I don't have the actual output because I didn't save it.  If this is not a known problem and isn't happening to others, let me know, and I can try it again later and capture the verbose output of both the miner and of p2pool.

Odds are it is a bug in Bitmarket.  They are incorrectly attributing someone else's deposits to your account.

I am only seeing data for a 2.5 day time window.  Is it supposed to be a week's worth of data or did I misunderstand?

No it doesn't. 

Good idea!

https://github.com/bitcoin/bitcoin/pull/220

I can confirm now that at least some of the flurry of submissions were valid.  I can see on my pool stats page that my hashrate dropped while the internet connection was down, but then shot up to 50% higher than my actual hashrate as soon as it returned.  Since they calculate my hashrate based on accepted shares, I believe this means that I send a hugh backlog of valid shares right when the network returned and that made my 5-minute average hashrate look larger than it actually is.

I tested the latest git code (excluding the very recent kernel tweak) with a simulated internet conncetion failure:

1. I unplugged my internet for 10 minutes and cleared the DNS cache on my miner and on my proxy server.

2. Shortly after the internet connection went down, cgminer reported that the pool was slow and it started doing generating local work.

3. Shortly after that, there was a message about a submit timing out and being cached.

4. Then I waited 10 minutes.  The miner kept mining (at least hashrates on the display stayed high) the entire time.

5. I plugged the internet back in. 

6. There was a flurry of messages that went by very fast.  I assume these were all the cached share submissions.  I didn't see the ones at the start, but the last set were all rejected/stale (which is expected since there was definitly a new block on the network while I was disconnected).

7. After the flurry, cgminer returned to mining as normal without needing to be restarted (and reported it was resuming communications with the server).

So it appears to work better now than when I had my 4 hour internet outage.   At least as far as I can tell in a 10 minute test (and I can't bring myself to test longer than that).

Use a pool that doesn't tell you how many blocks you found? 

Yes, 2-3% reject rate is about right for slush as it doesn't do long polling.

While I like the idea, isn't there a timing issue here?  New blocks to not arrive at all nodes on the bitcoin network at the same time.  If a new block was detected on a miner before it arrived at a pool, the miner would just request new work and get back a work based on the old block.  This would still cause stale shares (or invalid blocks if a share looked like a valid block). 

Long polling has the advantage that it doesn't happen until pushpool for sure has new work to hand out based on the new block.  The long running HTTP requests are a problem, but perhaps the solution is to just set a max time on them and return after 5 minutes with or without a new block.  That would solve the problem with buggy NAT routers and not add too much overhead (a long poll returning early is basically just another getwork).  And because long poll requests are initiated from the universe of miners in a staggered way, the "early ending" long polls would be staggered as well.

This doesn't solve the problem of an actual long poll (caused by a new block arriving) requiring 5000 work units being generated for larger pools.  But I don't know that anything can solve that except possibly beefy CPUs + changes to bitcoind to allow it to calculate more getwork requests in parallel (and I think there may already be a patch for that floating around).

I don't know if the idle threads restart was causing problems or not.  The biggest problem was that after the network came back, it didn't start doing getworks again to restart mining.

If you set aggression (in phoenix) too high or -f (in poclbm) too low, my experience is that it interferes with multiple GPU mining.  The miner gets bogged down with the overhead of being too aggressive and can't keep the GPUs fed with work to do. 

I would try a higher -f until you find the point at which you can mine on all GPUs without the second GPU affecting the first GPU's hash rate.

I was giving this a test run today and coincidentally had a complete internet connectivity failure for 3-4 hours due to problems with my ISP.   Because the ISP problem was long, I would expect that there were not only network connectivity problems but that also at some point DNS requests started failing as well because the cache expired and it couldn't do new lookups.  Obviously, during the Internet downtime, the miner wasn't working, but after the ISP problems resolved themselves, cgminer did not recover.  I had to kill it and start it again for it to start working.  I didn't think to copy the then-current screen output, but I do remember that the screen was filled with messages about the miner being idle for more than 60 seconds (that was recently added to address OpenCL or the GPU breaking).   

In contrast, the one machine that I was still running poclbm on recovered as soon as the Internet connection was back up.

Fortunately, the deepbit.net LP spec doesn't call it JSON-RPC.  And it does happen to be perfectly compliant use of HTTP (make a GET request; get back a response).

Look at these two graphs:

Pool Hopper on a PPS pool (fixed price per share submitted) ends up with 2.5 BTC after 14 days: http://www.calindora.com/tmp/bitcoin/index.php?graph=pps_hopper_100

Pool Hopper on a Prop pool ends up with 4.2 BTC after 14 days: http://www.calindora.com/tmp/bitcoin/index.php?graph=prop_hopper_100 

You can see that there is significant gain from pool hopping on proportional pools.  By switching to the prop pool that has most recently started a new round, you can net much better than the expected reward per share (which should be (1/difficulty * 50 BTC) with no fees).

You might also want to compare what the 24/7 miner gets on that prop pool that has a lot of hoppers:  7.86 BTC instead of the 9 BTC they would have had in the absence of hoppers.

Except the fact that the person who invented it says it should be a GET in his documentation of how it should work.  But the intentions of the guy who defined the standard doesn't matter... 

And the argument that a JSON-RPC request has to be a POST is irrelevant because the LP request as defined by the Deepbit spec is not a JSON-RPC request just because its response looks like a JSON-RPC response.   Two different standards can have responses that look the same without being the same in all other ways.   

You've defined a new standard.  A "DiabloD3 Long Poll".  Cool.  But they people who are following the published guidelines for the "Deepbit Long Poll" are not idiots (as you appear to be implying).  They are just doing exactly what it says they should be doing.

It's a pointless argument.  Fortunately, it's easy enough for pools and proxies to support both by not caring if there is a POST body or not.

Bummer.  I'll stick with poclbm for now.  It appears to handle it by toggling RollNTime support on and off based on the response to every getwork.  And it uses the most recently indicated LP URL each time a LP request is started instead of caching the URL once at startup and using it over and over.  I don't think it does anything to abort an in progress LP and start a new one when a more recent getwork http header indicates that it should be different, but it could do that as well, I guess.

Anyway, cgminer looks well done and I'm sure I'll keep playing with it on and off just because it is cool.  Thanks for making it!

Ya, certainly doesn't make sense to do it by default.  But if a pool (Eligius, for example, which also has a very high CPU cost to return a getwork response) announces to the miner that it supports it via the X-Roll-NTime: Y http header in the response to the getwork, you could increment the ntime when you exhausted all possible nonces instead of switching to another getwork.

Another question for you.  Does cgminer adapt to changing pool behavior on the fly?  For example, a long poll URL changing to a new URL.  Or X-Roll-NTime support appearing and disappearing from getwork to getwork.  In other words, does cgminer work well when pointed at a proxy where any given getwork request may be handled by one of several real pools with different capabilities.

I had some problems with DiabloMiner with that and so I currently stick with a bunch of individually started poclbm miners, but I am very interested in the idea of one miner per rig instead of multiple miners per GPU.