These are the times/dates where the network has severed me from connection... I have no idea what the blocks are, because the wallet doesn't show that information. All times are (GMT -5)

(Last block found before being severed from 25+ conn - first block isolated on 1-2 conn, now is a ? block.)
Date: Just now connected again... after I woke-up...
Date: 8/18/2013 01:31 - 01:35
Date: 8/17/2013 16:31 - 16:34
Date: 8/17/2013 11:50 - 11:55
Date: 8/17/2013 00:53 - 00:57
Date: 8/15/2013 05:26 - ? (Stopped mining until 8/15/2013 12:51, when I could get back on any chain.)
Date: 8/15/2013 02:25 - ? (Same as above - 8/15/2013 04:55)
Date: 8/12/2013 23:41 - ? (Same as above - 8/14/2013 21:12)
Date: 8/12/2013 17:30 - ? (Same as above - 8/12/2013 20:32)

Prior to this date, I mined fine... on version 1.4.0

Note: Always eventually connecting to the chain that is listed on http://bottlecaps.kicks-ass.net/block_crawler.php using the peer-list from http://bottlecaps.kicks-ass.net/peers.php

Also note: Only version 1.4.1 connections after 8/17/2013 00:53, and POS has nothing to do with "all of us getting rejected and disconnected for misbehaving at the same time".

Yes, you will have to do this every few hours... because you will again be booted from the chain you are on, or they will leave. (They = the one largest pool that becomes the new tallest chain, every time.)

I am making a list of every time I was severed from "the tall chain", when my program was forced to mine with the others who were booted off the chain, which leads us into creating the many small isolated chains. In hopes that this will help to identify the exact issue... but without knowing why "we are all seen as misbehaving"... it will be of no real use.

If misbehaving = resubmitting a block that is not acceptable... due to being beat by another block... then that code has to be changed. CGminer is told to resubmit up to 12 times, which is NOT misbehaving, that is called persistence when the client does not respond with a valid reason for rejection, or does not respond with a rejection at all.

This is where a port-snooper comes in handy. You can actually see what it is sending before the program chews it up and spits out something that can't be translated.

You don't get it...

We are all being disconnected...

The chains are all going rogue after a certain period of time... All of them... any one we connect to...

Stopping POS has nothing to do with it... We are generating POS fine for hours...

Versions have nothing to do with it... The new versions are being booted too...

There is a problem with the clients. They are either f-ing up with something related to the block-heights, f-ing up with something related to "sending data" or "reading sent data", or f-ing up because someone figured-out an exploit and is exploiting the miners through the IRC rooms.

That, or the one we keep following is just the only one f-ed up and causing all the issues for all of us. Someones "special compiled version" that they screwed-up and are persistently convincing us to connect to.

Then there is that guy a few posts back who said something about branching his own version of CAP from some other client. Could his rogue-branch creation be f-ing up the network, feeding us invalid data because he has no f-ing clue what the hell he is doing with his "branch".

Connecting to the right chain does not solve anything. It keeps disconnecting us. Like the logs show... we are all acting-up after time. No... that client is not reading our information correctly, or he is right, because our clients are not sending information correctly. Then it isolates itself, and we isolate ourselves from everyone else, doing the same... refusing connections because they are acting-up too. Creating one fork for every client, until no-one new can connect to us, and our fork just dies and sits there not mining or doing anything.

P.S. This happens at about 1:00 AM (GMT -5) and at 11:00 AM (GMT -5) for me, every day... it is that predictable for those times, by observation of my wallet filling with (?) blocks, which all start around those times. (There are other times too... but those two times it absolutely disconnects, and goes into a fork.)

And the fact that the program doesn't even attempt to fix the issue by looking for taller chains, or joining taller chains when found or reported... is making the issue a game-stopper. Even when you find the right chains, you have to delete the whole chain, or manually delete and restore a point where the chain was below that point, or it never connects and keeps forcing you back onto the wrong chain, because it is not fixing itself with the correct longer chain. Without THAT fixed, or working... this will never have resolution.

How to do that is simple. If you need guidance on that.. don't hesitate for me to outline HOW. Not sure why it doesn't do that in the first place, or why it was removed as a function.

I am still using the list from the block-explorer... had to reload my DB from noon... but it jumped right on to the chain, and still running strong... for the last hour. Only 6 connections though... dropped from 8 to 6 of the 25 in the list.

Height @ 78887 now, diff 37K (0.5659)

[
{
"addr" : "92.5.104.42:7685",
"services" : "00000001",
"lastsend" : 1376797451,
"lastrecv" : 1376797450,
"conntime" : 1376794359,
"version" : 70000,
"subver" : "/Satoshi:0.7.2/",
"inbound" : false,
"releasetime" : 0,
"startingheight" : 78827,
"banscore" : 0
},
{
"addr" : "80.229.2.127:7685",
"services" : "00000001",
"lastsend" : 1376797450,
"lastrecv" : 1376797450,
"conntime" : 1376794371,
"version" : 70000,
"subver" : "/Satoshi:0.7.2/",
"inbound" : false,
"releasetime" : 0,
"startingheight" : 78827,
"banscore" : 0
},
{
"addr" : "106.187.100.78:7685",
"services" : "00000001",
"lastsend" : 1376797449,
"lastrecv" : 1376797449,
"conntime" : 1376794377,
"version" : 70000,
"subver" : "/Satoshi:0.7.2/",
"inbound" : false,
"releasetime" : 0,
"startingheight" : 78827,
"banscore" : 0
},
{
"addr" : "115.76.20.64:7685",
"services" : "00000001",
"lastsend" : 1376797449,
"lastrecv" : 1376797449,
"conntime" : 1376794397,
"version" : 70000,
"subver" : "/Satoshi:0.7.2/",
"inbound" : false,
"releasetime" : 0,
"startingheight" : 78828,
"banscore" : 0
},
{
"addr" : "106.186.115.58:7685",
"services" : "00000001",
"lastsend" : 1376797449,
"lastrecv" : 1376797450,
"conntime" : 1376794428,
"version" : 70000,
"subver" : "/Satoshi:0.7.2/",
"inbound" : false,
"releasetime" : 0,
"startingheight" : 78828,
"banscore" : 0
},
{
"addr" : "97.81.165.175:7685",
"services" : "00000001",
"lastsend" : 1376797443,
"lastrecv" : 1376797449,
"conntime" : 1376797285,
"version" : 70000,
"subver" : "/Satoshi:0.7.2/",
"inbound" : false,
"releasetime" : 0,
"startingheight" : 78876,
"banscore" : 0
}
]

My IP: 107.217.208.215

I guess we are still having to baby-sit this coin until 1.4.2 rolls-out... This is something that will not apparently fix itself.

Finally, I have more than one connection, actually 8 now out of 30, most with the correct height... (What the block-crawler indicates as the correct height. 78840 now.)

Not sure what you guys did, but it worked... all except the part where I was mining on the short-chain for five hours, with no connections.

There is still a major issue with our windows-wallet/clients not looking for new connections, once we get settled-in. Thus, never even knowing there is a taller chain... or, when we know, it still keeps mining on the shorter one. (With an inability to "fix/remove" the wrong blocks in our chain, to build the correct one at the larger height. Thus, leaving us stuck on a dead chain that no-one is mining, at a lower height.)

The list I am using is right from the block-chain explorer list of connections...
http://bottlecaps.kicks-ass.net/block_crawler.php
http://bottlecaps.kicks-ass.net/peers.php

The peers list...

YES... finally... fifth time's a charm! (Got it again. But It will be lost again, I assume. I have lost the chain five times so far.)

I still only have 1 to 2 connections... which is what is causing the splits, I am sure... we keep branching if we all only have one or two other connections to talk to, and we can't communicate to one another.

Are you sure someone didn't change an IRC relay room to chr-code UTF-8 or something, instead of an (ISO-8859-1) standard format? Or could it be buggin on the ipv6 formatted IP's, trying to relay those to computers that just can't connect to them.

I need some sleep before work... my head hurts.

That is the funny thing... this list is the list from the correct chain. But they are all giving me 78280 as the height, but I can see them at the height of 78415...

Like I said before, they are mining away and not telling anyone what they are building. They are isolating themselves from us, and feeding us the wrong block-heights. (I assume it is an array issue or something like that now. Are you using arrays that start at 0 but sending us 1-x not 0-x... or they are just sending corrupted data that can't be used?)

They tell us, "hey I am at block 78415", then we say, "great, send it!", then they say "Screw you, this is my block-list, get your own!", and kill our connection. Leaving us talking to the guy building his own chain which is shorter and 300 blocks behind. (Dramatic explanation. lol)

That was odd... I had three good connections... Then, I unlocked my wallet and they all disappeared and it took a connection with a starting height that was 2000 lower than the newest fork.

(The chain I was on, with that ipv6 connection forked and everyone started dropping-off and now I am seeing more "-1" start-heights on the version 70000 client 1.4.1.)

Funny thing is... the peers sending the wrong blocks are in the list of peers connected to the chain with the correct height...

I am getting -1 from about 12 on the 25 in the list, and everything from 75000 to 76000 to 77000 to 78000 from them. They are all version 1.4.1 (70000)

IP: 188.252.16.109

reports as -1 block-height, and it is the one that the wallet keeps connecting to, over and over.

Just got an odd ipv6 addr... (2001:0:5ef5:79fd:1cd3:64c:add1:6b8c)

[
{
"addr" : "92.5.104.42:7685",
"services" : "00000001",
"lastsend" : 1376751282,
"lastrecv" : 1376751280,
"conntime" : 1376744432,
"version" : 70000,
"subver" : "/Satoshi:0.7.2/",
"inbound" : false,
"releasetime" : 0,
"startingheight" : 78102,
"banscore" : 0
},
{
"addr" : "50.137.233.14:7685",
"services" : "00000001",
"lastsend" : 1376751281,
"lastrecv" : 1376751280,
"conntime" : 1376745877,
"version" : 70000,
"subver" : "/Satoshi:0.7.2/",
"inbound" : false,
"releasetime" : 0,
"startingheight" : 78125,
"banscore" : 0
},
{
"addr" : "115.76.11.1:7685",
"services" : "00000001",
"lastsend" : 1376751281,
"lastrecv" : 1376751280,
"conntime" : 1376746046,
"version" : 70000,
"subver" : "/Satoshi:0.7.2/",
"inbound" : false,
"releasetime" : 0,
"startingheight" : 78127,
"banscore" : 0
},
{
"addr" : "[2001:0:5ef5:79fd:1cd3:64c:add1:6b8c]:7685",
"services" : "00000001",
"lastsend" : 1376751281,
"lastrecv" : 1376751280,
"conntime" : 1376750607,
"version" : 70000,
"subver" : "/Satoshi:0.7.2/",
"inbound" : false,
"releasetime" : 0,
"startingheight" : 78205,
"banscore" : 0
}
]

Seems to be connecting fine to it... and it has the correct height, for the moment. Not all IPV4's can actually connect to IPV6 though. Lucky me, my ISP and router auto-convert. Since my ISP's modem doesn't actually have IPv6 ability.

You might just want to wait until after noon, when the old versions are kicked and the new versions are the only ones mining.

Then, sadly... redownload the whole DB again, deleting everything except your config file and your wallet.dat...

Once you hit block 72000, stop the client/wallet... wait for it to actually die... and save everything it created, except the config and wallet... into a folder called "backup". Then, if you do get on the wrong chain... (let it try and try... you may get the wrong one at first, but it should eventually hit the correct one... if it does not, then just delete what it downloaded, and paste your backup data back into your wallet folder, so you start again from 72000.)

Eventually one of these connections will lock and begin working for you. (I hope.)

Or mine on a pool that is working... (The easier option, until the connection issue is resolved.)

Did you restart the client... if it hangs, it will not continue until you shut down the client and restart again. Don't delete your blocks data if you are that far... it just hung on a bad send. (I had to do that multiple times, to get the last 200+ blocks)

The wallet never reported the final height once... until I hit a POS block... that refreshed the screen. Then it showed I was up to date... the whole time I was actually up to date, but the wallet just didn't display I was, when I actually was.

Unlock it, and try mining... if you start mining, then you are up to date, if you see one correct height in your list of peers.

78197 current height (CGminer reported diff 79-83K... if you see less, you are not on the right chain.)

That is all you need... the heights are the heights when they first connected... they will not update the start-heights, unless they disconnect and reconnect to you, after getting the missing blocks. Then you see the new heights.

What you don't want to see is this... (corrupted or NULL or invalid or cut/trimmed packet data)

 "addr" : "95.72.68.42:29670",
        "services" : "00000000",
        "lastsend" : 0,
        "lastrecv" : 0,
        "conntime" : 1376748399,
        "version" : 70000,
        "subver" : "",
        "inbound" : true,
        "releasetime" : 0,
        "startingheight" : -1,

or connections to older versions 60001, 60002, 60003

It has UPnP checked, but no option to uncheck it... It is locked that way.

I don't need port-forwarding, as all connections are unblocked in my router, except the most extreme known viral and attack detected packets themselves. (Nothing shows in logs as being blocked. Nor do I have any problem connecting to over 40 connections on other coins who have wallets that have had the 8-connection limit removed.) I have not altered anything except my RPC port, to keep people from reading my personal info, or attacking my miners.

NOTE: When I say my versions below... I am talking about the windows wallet, used as a daemon, for mining and as a wallet. Not the linux compiles or OSX compiles...

I see people connect, and drop-off, after a few minutes, so I assume they got a faster connection or disconnected, or the program itself is having connection issues.

Everything was fine, up until the last few days, with the old 1.4.0 program. Nothing has changed in the 1.4.1 program, except now I get POS blocks, where I didn't before, on the 1.4.0 program.

My connections to the older 1.3 programs was perfect, from 1.4.0... It was when the 1.3 programs disappeared, that the issues began. Thus, whatever was changed from 1.3 to 1.4 is the actual issue, related to connections. Because those 1.3 programs were talking finem and keeping us 1.4 users connected, but 1.4 to 1.4 connections have been the trouble for me... Though 1.3 was just not doing POS well. 1.4 didn't do POS well, and also had/has comm issues... 1.4.1 has POS fine, but comm issues still.

Even the block-explorer which has 100 possible connections, is only able to get 25... Though, I am sure there are more than 100 attempting to connect to that daemon.

How is the IPV4:IPV6 and IPV6:IPV4 connections? That is a major f-up, for most sockets. Could that be an issue here? Packets forward and have multiple forwards which can shift packet data out of the MTU range. Check that data fits within the MTU range for UPnP, and is also not being chewed-up by a IPV4->IPV6 or IPV6->IPV4 conversion that ISP's like to break, and programs don't like to digest.

EG, with 1 forward of IPV4:IPV6 and (data) the MTU may be fine... but if a packet hits another forward, or multiple 4->6 then 6->4 then 4->6... it pushes data out of the packet to make room for the growing header in the packet. Causing it to break the TCP send link. Change the data-size to a little less, and see if that helps. If it does, then that is the issue with the connections. Multiple failed TCP connections due to corruption from MTU failures will kill the TCP connection. It won't try to resend more than 6 times.

I am watching this like a hawk until 5:00 PM (GMT -5)...

Might as well, since I am here. I need my port-snooper to check-out the raw packet data. lol. Where did I put that program...

http://bottlecaps.kicks-ass.net/peers.php

Though, in that list... I still only get 2 connections...

I think everyone has connections set too low... (Due to the wallets not accepting more than 8 connections, it is like 8 are connecting to the same 8, and isolating themselves over and over. And the ones accepting more connections, are connecting, but not talking, or not sending data that the wallets are able to read/use.)

Got 4 connections now... lol... love the start-height variance...
09:30:11
[
{
"addr" : "92.5.104.42:7685",
"services" : "00000001",
"lastsend" : 1376746126,
"lastrecv" : 1376746182,
"conntime" : 1376744432,
"version" : 70000,
"subver" : "/Satoshi:0.7.2/",
"inbound" : false,
"releasetime" : 0,
"startingheight" : 78102,
"banscore" : 0
},
{
"addr" : "50.137.233.14:7685",
"services" : "00000001",
"lastsend" : 1376746182,
"lastrecv" : 1376746182,
"conntime" : 1376745877,
"version" : 70000,
"subver" : "/Satoshi:0.7.2/",
"inbound" : false,
"releasetime" : 0,
"startingheight" : 78125,
"banscore" : 0
},
{
"addr" : "115.76.11.1:7685",
"services" : "00000001",
"lastsend" : 1376746182,
"lastrecv" : 1376746130,
"conntime" : 1376746046,
"version" : 70000,
"subver" : "/Satoshi:0.7.2/",
"inbound" : false,
"releasetime" : 0,
"startingheight" : 78127,
"banscore" : 0
},
{
"addr" : "91.235.254.37:7685",
"services" : "00000001",
"lastsend" : 1376746188,
"lastrecv" : 1376746188,
"conntime" : 1376746159,
"version" : 70000,
"subver" : "/Satoshi:0.7.2/",
"inbound" : false,
"releasetime" : 0,
"startingheight" : 77921,
"banscore" : 0
}
]

Apparently they are catching-up... so I assume I am delivering blocks that they are reading... I hope.

My IP in the list of the 25 is 107.217.208.215

Do not use that as a connect, except for temporary connection... My IP changes hourly, I believe. But if you need a quick-connect... add it, then remove it once connected, so you don't restart with it next time you connect... a week later.

Shut-down your daemon... (Wait for it to completely die)... Then delete peers, and reconnect with the peers in that list. Wait a minute for it to run through the connections and find and link the right heights.

If it doesn't get the right height... do that again... until you do get the right height.

Also, save yourself some time... once you get the right height... save a backup of everything in your folder... so when/if this happens again, you can just restore that backup and continue from that point on... you connect faster that way too.

Apparently no-one wants to send the whole chain to everyone that keeps connecting with 0 blocks downloaded. Leaving the burden to one IP to deliver the whole chain over and over... and the chain it is sending is the wrong ending... you have to HOPE to get a taller connection before it begins sending you the wrong blocks. (Because if it does, you have to do it all over again.)

The block explorers just went down... but I am on height 78103 on IP: 50.137.233.14 which oddly reports the starting-height as 78026... That is the only connection I get from the list of connections.

Even blockcrawler is having connection issues, as we are... http://bottlecaps.kicks-ass.net/block_crawler.php

Just reported 0 connections, and 0 blocks, and now is back up with 15 connections and my height.

Now also having IP: 92.5.104.42 connect with the correct height as the starting height. Apparently the other IP isn't getting the missing blocks delivered, and is hung-up, but still relaying the top block.

Good to see that diff back up to normal... 100K (1.53) That comforts me a lot more than what it was last night. I don't mind making less and demanding more for my efforts, since the coins are majorly undervalued by those who are trading them. They will stop selling themselves short, once people stop hoarding them and actually start spending them.

Now that the number of places to spend is growing, and hopefully this next release will address many of the connection issues that is leading to the forking issues.

I still think the best solution is to have at-least one roaming connection, of the 8 that the wallets are limited to... used to traverse the rooms, looking for higher blocks, and 1 of the 8 to look for rooms with less blocks, to alert them and deliver the higher blocks.

However, that will not do much good if you HAVE to shut-down, delete the database, restart, hope to connect to a higher chain, and then wait for the whole chain to be delivered, and then unlock and restart the miners that were running, so they can mine again, on the unlocked taller chain.

Having an archive of the "block-0 to block-last-hard-coded", in addition to "new-blocks", and having the ability to "delete invalid block from database", up to the point of the branch... to rebuild it from that point forward... would make most of these issues go away. (Except the connection issues, where people keep getting connections killed and invalid packets are being sent, or packets are being improperly read/translated, once sent.)

I would also like to mention... I see "ping" in the debug info that was posted on the last few posts...

You do realize that 90% of the world's routers do not allow or respond to pings, even if a router does, ISP's do not allow them. Pings from the internet are "suspicious", as they are usually a sign of a hacker attempt to find an open port or get a response. Internally, in an intranet, routers will allow pings, because that is how home-devices let other devices know they are alive. But ISP's block pings, inbound and outbound, and most routers block any inbound pings, only allowing internal pings to be used.

If our connections are being killed because we are not responding to pings that we never hear... then you need to remove that part of the code.

That will FORCE segregation, as it will keep only intranet (private networks), who DO respond to pings, isolated as the only valid connections. EG, If I had 10 rigs, all connected to my wallet, and 20 connections, but 10 from the net didn't respond to pings, so I sever those connections, and my 10 rigs DO respond to pings... I would end-up isolating myself from the world, only talking to my 10 rigs with running wallets, building my own chains.

You can't ask people to "allow pings" and HOPE that works. (Not only because it exposes them to hacking detection, but also because many ISP's will drop most pings, and make it a useless fix.) They should just not be used, except to determine if that connection is possibly an internal connection on an intranet, or an external connection on an internet. Nothing more.

Spitting-out a UDP packet is the correct way to test for a connection, when a TCP connection is severed. Not a PING. Though, those too can be blocked, if they contain detectable data that is not encrypted. (Eg, if an ISP doesn't want mining, and they can detect mining packets, they will block them. If the packets are encrypted, they have no real detection, and usually allow them to pass. That is what TOR and torrents now do.)

did you use the command -rescan, after getting on the right block-height?

They should move back, in time... once enough blocks have gotten above them, beyond the point of (Confirmed invalid) eg, the point where they should be confirmed, but the chain indicates they were not the actual blocks that were confirmed as valid.

The only thing you have to deal with, is that those coins can not be staked, while they remain pending-stake, waiting for release back to your wallet.

grr... I went to sleep at 1AM (gmt -5), and just as I left, the programs all shut-down and apparently forked on a new chain.

That, or this program just hates me...

Only lost about 20 blocks, because the miner seemed to be connecting and rejecting randomly... The 16k diff on cgminer let me know something was up. lol.

Have you considered changing the POS diff to 1k (0.015), to stop massive burps of near-zero diff blocks from flooding the network. That is still an acceptable CPU diff, and would add more time between blocks.

I wasn't kidding when I said I was getting TONS of POS blocks. Just last night I believe I had found over 97 POS blocks and only 39 normal blocks. (Near the end, before I was severed from the net, by being rejected by the other miners... I had found 20 POS's in a row.)

Now, I am sitting at an uncomfortable 2 connections, out of the 25 listed... so that is a lot of connections not talking/communicating with anyone, on that list at http://bottlecaps.kicks-ass.net/block_crawler.php