Bitcoin core on testnet stops at block and won't sync

[73blazer]

I've got a core full node on mainnet and also on testnet running, both for years.
Currently on ubuntu with core 0.16.0

On saturday, my testnet node outbound connections dropped to 0 and it's stuck on block 1354779.
I've made no changes to my config, firewalls, network, it happened while it was running as I always leave it up, last reboot was weeks ago.
I see in the debug log it makes new outbounds but they quickly get kicked off within a few seconds to a few minutes, like my node is considered misbehaving by others or something. No blocks past the one mentioned ever get synced.
The mainnet node is running fine.

Any suggestions on what move try to remedy this? Any ideas why it happened?

Sample from log:
[

2018-07-21 16:13:30 GUI: Platform customization: "other"
2018-07-21 16:13:30 GUI: PaymentServer::LoadRootCAs: Loaded  146  root certificates
2018-07-21 16:13:31 New outbound peer connected: version: 70015, blocks=1354840, peer=0
2018-07-21 16:13:31 socket recv error Connection reset by peer (104)
2018-07-21 16:13:36 Imported mempool transactions from disk: 18657 succeeded, 0 failed, 0 expired, 0 already there
2018-07-21 16:13:37 connect() to [2001:0:9d38:90d7:433:113d:355b:3ed4]:18333 failed: Network is unreachable (101)
2018-07-21 16:13:38 connect() to [2001:0:9d38:78cf:3045:4a12:ed3a:4b8b]:18333 failed: Network is unreachable (101)
2018-07-21 16:13:39 New outbound peer connected: version: 70015, blocks=1354840, peer=4
2018-07-21 16:13:40 connect() to [2001:0:9d38:78cf:7e:9a64:4e77:6860]:18333 failed: Network is unreachable (101)
2018-07-21 16:13:40 socket recv error Connection reset by peer (104)
2018-07-21 16:13:41 71 addresses found from DNS seeds
2018-07-21 16:13:41 dnsseed thread exit
2018-07-21 16:13:56 New outbound peer connected: version: 70015, blocks=1354840, peer=7
2018-07-21 16:13:56 connect() to [2001:0:9d38:6ab8:1c98:3aee:89e0:f3d8]:18333 failed: Network is unreachable (101)
2018-07-21 16:13:57 New outbound peer connected: version: 70015, blocks=1354840, peer=8
2018-07-21 16:13:58 socket recv error Connection reset by peer (104)
2018-07-21 16:13:58 socket recv error Connection reset by peer (104)
2018-07-21 16:13:58 connect() to [2001:470:a:c13::2]:18333 failed: Network is unreachable (101)
2018-07-21 16:13:58 connect() to [2600:3c01::f03c:91ff:fe2d:799a]:18333 failed: Network is unreachable (101)
2018-07-21 16:14:02 New outbound peer connected: version: 70015, blocks=1354840, peer=9
2018-07-21 16:14:04 socket recv error Connection reset by peer (104)
2018-07-21 16:14:09 New outbound peer connected: version: 70015, blocks=1354840, peer=10
2018-07-21 16:14:10 connect() to [2001:0:9d38:90d7:8bc:1e0b:b275:ab14]:18333 failed: Network is unreachable (101)
2018-07-21 16:14:10 socket recv error Connection reset by peer (104)
2018-07-21 16:14:16 New outbound peer connected: version: 70014, blocks=1354840, peer=11
2018-07-21 16:14:16 connect() to [2001:41d0:800:664::]:18333 failed: Network is unreachable (101)
2018-07-21 16:14:17 New outbound peer connected: version: 70015, blocks=1354840, peer=12

[73blazer]

I restarted with a reindex-chainstate.
It's taking it's time, but at least the outbound connections ran up to 8 and are holding at 8 while it does it's re-index.
We'll see if it holds when it gets to current.....

[73blazer]

Well, that ran up to block 1354775, 4 blocks before where it was stuck before.
all my outbounds then issued me socket recv error Connection reset by peer and it's stuck sitting there again, makes a new outbound immediately gets a socket recv error Connection reset by peer back.

What's the next move? I'm tempted to remove a few blocks from the blocks dir but that's not as simple as it used to be 5 years ago the last time I had something go wrong with a core instance.

[73blazer]

It did manage to get back to 1354779 but it stops there and eventually all my outbounds disappear/disconnect after a while and nothing ever downloads past this block.
The blockhash matches that from blockchain.info for that testnet block, why's it having issues getting the next block from the network to get back in sync?

Where should I go with this?

[73blazer]

Alright, what gives. This is driving me nuts.
I tried
reindex-chainstate - again stop at block 1354779
reindex - again stop at block 1354779
removing blocks and chainstate dirs - again stop at block 1354779
removing the entire data dir and starting from scratch - again stop at block 1354779

WTF?!?!?!?!?!?!

[73blazer]

I turned on some higher level debugging, this is what I get:

2018-07-24 23:27:47 sending version (102 bytes) peer=13
2018-07-24 23:27:47 send version message: version 70015, blocks=1354779, us=[::]:0, peer=13
2018-07-24 23:27:47 sending verack (0 bytes) peer=13
2018-07-24 23:27:47 receive version message: /bcoin:v1.0.0-pre/: version 70015, blocks=1355070, us=71.13.92.62:18333, peer=13
2018-07-24 23:27:47 received: verack (0 bytes) peer=13
2018-07-24 23:27:47 sending sendheaders (0 bytes) peer=13
2018-07-24 23:27:47 sending sendcmpct (9 bytes) peer=13
2018-07-24 23:27:47 sending sendcmpct (9 bytes) peer=13
2018-07-24 23:27:47 sending ping (8 bytes) peer=13
2018-07-24 23:27:47 initial getheaders (1355313) to peer=13 (startheight:1355070)
2018-07-24 23:27:47 sending getheaders (1061 bytes) peer=13
2018-07-24 23:27:47 sending feefilter (8 bytes) peer=13
2018-07-24 23:27:47 received: addr (31 bytes) peer=13
2018-07-24 23:27:47 received: sendcmpct (9 bytes) peer=13
2018-07-24 23:27:47 received: getblocks (1061 bytes) peer=13
2018-07-24 23:27:47 getblocks -1 to end limit 500 from peer=13
2018-07-24 23:27:47 received: pong (8 bytes) peer=13
2018-07-24 23:27:47 received: headers (1783 bytes) peer=13
2018-07-24 23:27:47 Requesting block 000000000000000f49b212a0150642ebd2204b8b3d2eaa0dd58f83a0ab4e2ce2 (1354780) peer=13
2018-07-24 23:27:47 sending getdata (37 bytes) peer=13
2018-07-24 23:27:47 connection to 123.115.67.133:18333 timeout
2018-07-24 23:27:47 socket recv error Connection reset by peer (104)
2018-07-24 23:27:47 disconnecting peer=13
2018-07-24 23:27:47 Cleared nodestate for peer=13

Not much there, request the next block, and socket recv error. I can clear the whole testnete data dir and request next block works fine until 1354780...stops cold there. pulling my hair out!!! ANyone...suggestions........

[73blazer]

Well..I finally figured it out with the help of <gmaxwell> on irc bitcion chat.
He mentioned it could be some firewall somewhere that is matching something in that block to an antivirus signature
He was right, I turned off my sonicwall gateway antivirus and it started moving and fully sync'd right up.

So...an oddball one, hope it helps someone in the future. I turned back on my gateway antivirus after it sync'd. I'll report to sonicwall a false positive, but they likely won't do much about it