[POOL][Scrypt][Scrypt-N][X11] Profit switching pool - wafflepool.com

[comeonalready]

What I would suggest to anyone who was affected by this recent redirect attack:

Enable verbose logging to file going forward, and gather as much information as possible about the attack and forward it on to poolwaffle.  Include in your message:

- your btc address so he can check your share submissions in the database (and when they stopped in relation to the attack)
- the specific wafflepool server to which you were connected (e.g. useast, uswest, eu, whatever.)
- the time that you noticed the redirection attack was occurring, and the time you suspect it started if that information is available to you.
- your operating system type and version of only your affected miners (do not assume all of your miners were redirected, but only report the ones that you are certain were)
- the dns servers that are configured for your affected miners (again, check those dns server settings, do not make any assumptions)  If they are the internal ip address of your local router then supply it, and the external dns server addresses that your router points to as well
- type and version of your mining software
- whether your miner software was still showing something.wafflepool.com in its status display or an unexpected ip address and/or port number, and/or what it usually displays in that space
- whether you have any backup servers configured and whether they are other wafflepool servers or at another different pool
- if running cgminer, kalroth cgminer, or sgminer, whether you specify failover-only on the command line or in the configuration file
- anything else that seems relevant

I would also suggest that if you are using the aesthetic 'poolname' and/or 'name' feature of kalroth cgminer or sgminer, that you disable it as it might possibly be hiding the actual server host name or ip address to which your miner is actually connected.  (This is only an assumption and not verified to be the case.  But until it is either, then perhaps better to be on the safe side and able to notice a future redirection of your hashpower when it occurs again.)

[utahjohn]

Does anyone know where the stolen hashpower was sent to (besides just an ip address?)  Somebody had to be gaining from it!

[comeonalready]

Does anyone know where the stolen hashpower was sent to (besides just an ip address?)  Somebody had to be gaining from it!

If anyone has share logs and actually solved a block, we could possibly find it sitting in a block chain somewhere and trace it to a wallet address.  But associating a name with that wallet address would be considerably more difficult, especially since it is likely to have traveled through an exchange.

Did anyone note the difficulty level of the coin being mined?  Not the 256, 512, or 1024 number but the one underneath it followed by an M in the cgminer display to the right of block and the left of start time?

[utahjohn]

I tried to do a traceroute on the ip and it times out after a couple hops from my isp to max of 30 hops ...

anybody do a whois on it?  I don't know command on winblowz.  Suppose I could fire up LMDE in a Virtualbox and try ...

[comeonalready]

I tried to do a traceroute on the ip and it times out after a couple hops from my isp to max of 30 hops ...

anybody do a whois on it?  I don't know command on winblowz.  Suppose I could fire up LMDE in a Virtualbox and try ...

Sophisticated hackers don't necessarily point redirected traffic at themselves.  They are just as likely to find vulnerable equipment already sitting on the internet and compromise it too in order to obfuscate their involvement.  Of course, we could always be dealing with someone at that ip address, but as likely just another innocent victim.

[poolwaffle]

So a quick update on the hash-hijacking (best name I can come up with).

Sorry I wasn't more active on here (thanks comeon for relaying a ton of info), we we furiously trying to find what was causing the issue on IRC, and things were fast moving enough that if I would take the time to post it here, it wouldn't be relevant 3 minutes later.

Things we're confident it isn't:
1) DNS.  Theres no reason to suspect it was a DNS hijack at all.  It happened to multiple pools (us, multipool), and we don't host DNS at the same locations.
2) Hacked Server / Injected code.  Code running in production was diff'd against our daily backups, and nothing changed.  Plus, stratum code had not been restarted since I did it manually (times line up).  Also, this affected Multipool, and we don't run the same code

Things it could still be:
1) Miner malware.  It hits too many different configurations (OS's, miner versions, etc) to be in the miner itself.  Perhaps a bad wallet or something, but to accomplish what they're doing, it would need to be injecting packets into your TCP stack, which while possible, is unlikely.

What we think it is:
Our best guess at the current time is a MITM attack somewhere on the internet.  Because it is in a specific location, they would only be rerouting certain segments of traffic, not everyone, which lines up with what we're seeing.  This could either be done at the DNS level for those specific users, or at the BGP level (less likely).  What would happen in this case is that mining traffic from users affected would go _through_ this other user's IP before being relayed to WP.  He would let you associate and start mining, and at some point, inject a single packet into the stream that says "redirect your miner to this other address".  At which point your miner would listen and redirect to his address and start mining there.

The reason this is the most likely event is due to how we're seeing users switch pools.  This shows in logs:
[01:04:46] Reconnect requested from Waffle [WEST] to 206.223.224.225:3009
Which is the logline you'd see if a pool sent a "client.reconnect" message, and the user sees themselves connected to a different pool afterwards.

Our stratum server _never_ sends this line (we've never used the "client.reconnect" message, don't have any use for it), which makes it seem like it is being injected by a 3rd party somewhere along the way.  Also the fact that this is happening to Multipool is a good indicator that its network related (somewhere in the network between WP and the end user).

As for how to combat it, it really depends on how they're becoming the middlepoint, which we don't know yet.  Both Multipool and us have endpoints in the same datacenters, so it is possible that its something at the host, but seems a bit unlikely due to it only affecting some users, and that subset of users isn't a changing group for the most part (its not a random 1%, its a selected group).

At this point, we're still just digging, and don't have any more information that isn't posted here.  We're still looking into it, and we have a ton of people watching packet captures to get more information.  Sorry I don't know more...

[comeonalready]

So a quick update on the hash-hijacking (best name I can come up with).

Sorry I wasn't more active on here (thanks comeon for relaying a ton of info), we we furiously trying to find what was causing the issue on IRC, and things were fast moving enough that if I would take the time to post it here, it wouldn't be relevant 3 minutes later.

Things we're confident it isn't:
1) DNS.  Theres no reason to suspect it was a DNS hijack at all.  It happened to multiple pools (us, multipool), and we don't host DNS at the same locations.
2) Hacked Server / Injected code.  Code running in production was diff'd against our daily backups, and nothing changed.  Plus, stratum code had not been restarted since I did it manually (times line up).  Also, this affected Multipool, and we don't run the same code

Things it could still be:
1) Miner malware.  It hits too many different configurations (OS's, miner versions, etc) to be in the miner itself.  Perhaps a bad wallet or something, but to accomplish what they're doing, it would need to be injecting packets into your TCP stack, which while possible, is unlikely.

What we think it is:
Our best guess at the current time is a MITM attack somewhere on the internet.  Because it is in a specific location, they would only be rerouting certain segments of traffic, not everyone, which lines up with what we're seeing.  This could either be done at the DNS level for those specific users, or at the BGP level (less likely).  What would happen in this case is that mining traffic from users affected would go _through_ this other user's IP before being relayed to WP.  He would let you associate and start mining, and at some point, inject a single packet into the stream that says "redirect your miner to this other address".  At which point your miner would listen and redirect to his address and start mining there.

The reason this is the most likely event is due to how we're seeing users switch pools.  This shows in logs:
[01:04:46] Reconnect requested from Waffle [WEST] to 206.223.224.225:3009
Which is the logline you'd see if a pool sent a "client.reconnect" message, and the user sees themselves connected to a different pool afterwards.

Our stratum server _never_ sends this line (we've never used the "client.reconnect" message, don't have any use for it), which makes it seem like it is being injected by a 3rd party somewhere along the way.  Also the fact that this is happening to Multipool is a good indicator that its network related (somewhere in the network between WP and the end user).

As for how to combat it, it really depends on how they're becoming the middlepoint, which we don't know yet.  Both Multipool and us have endpoints in the same datacenters, so it is possible that its something at the host, but seems a bit unlikely due to it only affecting some users, and that subset of users isn't a changing group for the most part (its not a random 1%, its a selected group).

At this point, we're still just digging, and don't have any more information that isn't posted here.  We're still looking into it, and we have a ton of people watching packet captures to get more information.  Sorry I don't know more...

As both wafflepool and mutipool do have stratum servers located within the same data center, I would focus my search on a vulnerable piece of networking equipment through which both of your network traffic is passing, and where packets could be inspected to collect client ip addresses/tcp port address combinations, tcp sequence numbers, and even stratum message index numbers.  With that information, packets with forged source headers could be sent to miners from many other locations on the internet, independent of the data collection location, but much more easily from the collection point.  That both pools are affected might suggest that they are only inspecting packets based upon tcp port number, and that only some miners are affected could suggest that there is a certain amount of guessing done by their tcp injection process, and that many forged packets containing client.reconnect command messages are being discarded at their destination for having bad tcp sequence numbers (some home based routers incorporate stateful packet inspection, some don't), or out of range stratum message index numbers (which admittedly, I am not certain exactly what cgminer or other mining software would do with - I would hope discard them).

[rallasnackbar]

How come the reject rate is so high?


A:1100304  R:56368


thats 5% rejects.

[comeonalready]

How come the reject rate is so high?

A:1100304  R:56368

thats 5% rejects.

There are other much more important things going on right now.  Let's worry about that later!

[Meeho]

Things it could still be:
1) Miner malware.  It hits too many different configurations (OS's, miner versions, etc) to be in the miner itself.  Perhaps a bad wallet or something, but to accomplish what they're doing, it would need to be injecting packets into your TCP stack, which while possible, is unlikely.

What we think it is:
Our best guess at the current time is a MITM attack somewhere on the internet.

I do not believe it is miner malware because it is happening with various miners on various operating systems and, for me, it is only happening when connected to Wafflepool.

I use Kalroth's 3.7.3 from http://k-dev.net/cgminer/ on Win7, Asus router with Tomato FW and remote access disabled, and my ISP's DNS. DNS checks were always resolved correctly to Wafflepool's IP when I checked. My miner went to backup pool completely because of --failover-only and firewall blocking the [probable] stratum redirect.

You said your code didn't change. Is it possible it is a sort of malware resident in server's memory?

[utahjohn]

I just observed similar behaviour on dmd.cryptroll.com ... Miner chugging away and 0 shares showing on frontend page.  How do I enable detailed logging in cgminer.conf

I am running with 8 pools quota based with failover-only.

[comeonalready]

Things it could still be:
1) Miner malware.  It hits too many different configurations (OS's, miner versions, etc) to be in the miner itself.  Perhaps a bad wallet or something, but to accomplish what they're doing, it would need to be injecting packets into your TCP stack, which while possible, is unlikely.

What we think it is:
Our best guess at the current time is a MITM attack somewhere on the internet.

I do not believe it is miner malware because it is happening with various miners on various operating systems and, for me, it is only happening when connected to Wafflepool.

I use Kalroth's 3.7.3 from http://k-dev.net/cgminer/ on Win7, Asus router with Tomato FW and remote access disabled, and my ISP's DNS. DNS checks were always resolved correctly to Wafflepool's IP when I checked. My miner went to backup pool completely because of --failover-only and firewall blocking the stratum redirect.

You said your code didn't change. Is it possible it is a sort of malware resident in server's memory?

You have some very important information in this post.  How do you know your firewall blocked a client.redirect command message?  Do you have more data?

[Meeho]

I use Kalroth's 3.7.3 from http://k-dev.net/cgminer/ on Win7, Asus router with Tomato FW and remote access disabled, and my ISP's DNS. DNS checks were always resolved correctly to Wafflepool's IP when I checked. My miner went to backup pool completely because of --failover-only and firewall blocking the stratum redirect.

You have some very important information in this post.  How do you know your firewall blocked a client.redirect command message?  Do you have more data?

Ah, sorry if I misled, I didn't observe it directly as I didn't have cgminer logging enabled. I was deducing from PW's observations and log reports, my rig's behaviour and the most likely scenarios discussed here.

[comeonalready]

What we think it is:
Our best guess at the current time is a MITM attack somewhere on the internet.  Because it is in a specific location, they would only be rerouting certain segments of traffic, not everyone, which lines up with what we're seeing.  This could either be done at the DNS level for those specific users, or at the BGP level (less likely).  What would happen in this case is that mining traffic from users affected would go _through_ this other user's IP before being relayed to WP.  He would let you associate and start mining, and at some point, inject a single packet into the stream that says "redirect your miner to this other address".  At which point your miner would listen and redirect to his address and start mining there.

As for how to combat it, it really depends on how they're becoming the middlepoint, which we don't know yet.  Both Multipool and us have endpoints in the same datacenters, so it is possible that its something at the host, but seems a bit unlikely due to it only affecting some users, and that subset of users isn't a changing group for the most part (its not a random 1%, its a selected group).

I cannot help but wonder that if it is a true man in the middle attack, then why would he even bother allowing miners to initiate and authorize a stratum connection to the intended pool server in the first place, instead of just rewriting the destination headers in the incoming tcp packets from the clients to include the desired server ip address and tcp port within the incoming packets themselves, as he would be also able to rewrite source headers within the return traffic?  By doing that, the miners would still see wafflepool listed on their cgminer display, as opposed to the rogue server ip address.  This could suggest that he is only able to inspect the network traffic but not rewrite it, so therefore tcp packets with forged source headers containing client.redirect command messages are being sent to miners because he is not relaying traffic but only inspecting it.

Though in support of a true man in the middle attack, for a day or two you had been searching for a reason why miners were not receiving work from the servers quickly enough, such that some miners (specifically noticed by cudaminers) were going idle.  At that point he might have still been setting up shop but not yet begun his attack.  It is unfortunate that the stratum server code on the servers had only been running briefly, and that you cannot be absolutely certain that some as yet unknown problem does not still reside within it. (a performance related problem, not a vulnerability)

[Meeho]

I cannot help but wonder that if it is a true man in the middle attack, then why would he even bother allowing miners to initiate and authorize a stratum connection to the intended pool server in the first place, instead of just rewriting the destination headers in the incoming tcp packets from the clients to include the desired server ip address and tcp port within the incoming packets themselves, as he would be also able to rewrite source headers withing the return traffic?  By doing that, the miners would still see wafflepool listed on their cgminer display, as opposed to the rogue server ip address.  This could suggest that he is only able to inspect the traffic but not rewrite it, so therefore tcp packets with forged source headers are being sent to miners because he is not relaying traffic but only inspecting it.

Though in support of a true man in the middle attack, for a day or two you had been searching for a reason why miners were not receiving work from the servers quickly enough, such that some miners (specifically cudaminers) we going idle.  At that point he might have still been setting up shop but not yet begun his attack.

Could it be such a clever attack, that they would not hijack the whole mining session, but only select packets as to divert a small portion of hash power to their servers, so people would not see a complete drop in their waffle stats and become too suspicious. I remember seeing some reports of too small hashing power reported by the stats in this thread. Though some have lost their whole work, so maybe this theory has no merit.

[comeonalready]

I cannot help but wonder that if it is a true man in the middle attack, then why would he even bother allowing miners to initiate and authorize a stratum connection to the intended pool server in the first place, instead of just rewriting the destination headers in the incoming tcp packets from the clients to include the desired server ip address and tcp port within the incoming packets themselves, as he would be also able to rewrite source headers withing the return traffic?  By doing that, the miners would still see wafflepool listed on their cgminer display, as opposed to the rogue server ip address.  This could suggest that he is only able to inspect the traffic but not rewrite it, so therefore tcp packets with forged source headers are being sent to miners because he is not relaying traffic but only inspecting it.

Though in support of a true man in the middle attack, for a day or two you had been searching for a reason why miners were not receiving work from the servers quickly enough, such that some miners (specifically cudaminers) we going idle.  At that point he might have still been setting up shop but not yet begun his attack.

Could it be such a clever attack, that they would not hijack the whole mining session, but only select packets as to divert a small portion of hash power to their servers, so people would not see a complete drop in their waffle stats and become too suspicious. I remember seeing some reports of too small hashing power reported by the stats in this thread. Though some have lost their whole work, so maybe this theory has no merit.

Not likely.  Pool stratum servers salt the work handed out to miners so that block solutions cannot be submitted directly to the blockchain by unscrupulous miners.  From the miner perspective, it's an all or nothing takeover.  For anything along the lines of what you are suggesting to work, the attacker would need access to the salt string, which never leaves the pool stratum server.  (unless the server is compromised.)

[poolwaffle]

The only reason I can think of for a redirect rather than just a hijacking is to allow him to repoint to various compromised servers.  Enable a MITM for a few seconds, redirect some traffic to a compromised box, turn off MITM.  Very difficult to see/catch the MITM happening if its only there for a few seconds, and the results (the redirected miners) will continue happily along for a while.

The idling miners turned out to be a different issue entirely unfortunately.  We re-send the exact same work request if we haven't sent a work update after 30 seconds (we had seen some miners timing out after 30 seconds of no new work), and some miners are seeing a duplicate work request (30 seconds later) and idling for some reason.

Don't think they're related.

[comeonalready]

The only reason I can think of for a redirect rather than just a hijacking is to allow him to repoint to various compromised servers.  Enable a MITM for a few seconds, redirect some traffic to a compromised box, turn off MITM.  Very difficult to see/catch the MITM happening if its only there for a few seconds, and the results (the redirected miners) will continue happily along for a while.

The idling miners turned out to be a different issue entirely unfortunately.  We re-send the exact same work request if we haven't sent a work update after 30 seconds (we had seen some miners timing out after 30 seconds of no new work), and some miners are seeing a duplicate work request (30 seconds later) and idling for some reason.

Don't think they're related.

One of the miners really needs to capture a client.redirect packet for analysis.  I will enable port mirroring on my switch and set up to capture relevant packets outside of the firewall on my end, but I might just never see one.  Can you clue me into the most likely server(s) whose network traffic is being inspected or redirected?

[ycsi]

The only reason I can think of for a redirect rather than just a hijacking is to allow him to repoint to various compromised servers.  Enable a MITM for a few seconds, redirect some traffic to a compromised box, turn off MITM.  Very difficult to see/catch the MITM happening if its only there for a few seconds, and the results (the redirected miners) will continue happily along for a while.

Check this out: https://bitcointalk.org/index.php?topic=434464.msg5848594#msg5848594

It seems that Betarigs miners are having similar problem with stratum reconnect/hi-jacking?

[comeonalready]

To anyone and everyone using cgwatcher, whether your hashpower has been hijacked yet or not, you can configure cgwatcher to send an e-mail or perform another action when it detects a pool change in cgminer/kcgminer/sgminer.  To do this, click on the schedule tab, then add a scheduled action, and use the settings shown in the following image.  Then notify poolwaffle as soon as possible after your miner gets hijacked, and if you really want to be helpful, don't fix it until you hear back from him!