Forum's Cloudflare turned to always on? (S.O.S bots)

[TryNinja]

Today I woke up and saw this post on the BitcoinTalk SuperNotifier thread:

Mayday mayday mayday…. Bot is down… I repeat Telegram bot is down!

Edit:

Oh I see now I konw why it's down... cloudflare thank you theymos... looks like its updated to not only for login page but on every first or new request to bitcointalk.

I started debugging and it looks like every request fails due to Cloudflare. I tried bypassing it but have had no success so far.

@theymos, is this here to stay?

loyce's and bpip's scraper also seems to be down? (paging @LoyceV, @suchmoon, @ibminer).

[Zaguru12]

I was on my way tryna post this, I have switched between Firefox and Chrome all day trying to figure out may be its my connection or the browser. I have seen some error messages logining before, but I haven't encounter this since signing up here in the forum. I just hope it's a temporary error and it's fixed quick

[TryNinja]

Looks like we are back! Requests aren't failing anymore.

Thanks, theymos?

[LoyceV]

loyce's and bpip's scraper also seems to be down?

I hadn't noticed yet, but indeed, I miss all posts between posts 61426048 and 61427300. That's 1251 missing posts in 5 hours and 40 minutes.

(paging @LoyceV, @suchmoon, @ibminer).

That didn't work

Thanks, theymos?

Theymos hasn't been online.

[shahzadafzal]

loyce's and bpip's scraper also seems to be down? (paging @LoyceV, @suchmoon, @ibminer).

I hadn't noticed yet, but indeed, I miss all posts between posts 61426048 and 61427300. That's 1251 missing posts in 5 hours and 40 minutes.

Yes looks like it's working now, but at first I thought that 1 sec time has cost me a ban to my IP

But later saw Telegram Bot and ChartBuddy also down then realized its not only me.

Theymos hasn't been online.

That means cloudflare itself did this or some rule triggered it... It's quite possible in future it might be turned on again or may be permanently.

Is there way to bypass it or somehow configure it to use our existing (live) session or pass cookies for the scripts?

[LoyceV]

Yes looks like it's working now, but at first I thought that 1 sec time has cost me a ban to my IP

But later saw Telegram Bot and ChartBuddy also down then realized its not only me.

So.... you're saying you broke it for everyone

Is there way to bypass it or somehow configure it to use our existing (live) session or pass cookies for the scripts?

I don't feed cookies to my scraper, so it doesn't index the Investigations board. I use cookies for other tasks, but none of them were running so I can't tell if it would have worked.

[TryNinja]

Is there way to bypass it or somehow configure it to use our existing (live) session or pass cookies for the scripts?

I already pass the bot's session (cookies) to my scraper, but that didn't help. I may have found a possible workaround, but I didn't have the time to test it.

1. If Cloudflare pops up, calls a function that invokes a puppeteer page (an instance of chrome) to the forum and waits for CF to clear out.
2. Grabs the cookie from CF that lets you use the forum normally and passes it to all my other headless requests.

There is a chance that CF could ask me to solve a captcha though, so that would be a dead end.

[PrivacyG]

I am still having issues with Cloud Flare.  I still get the page asking me to enable Java Script every now and then.  Got it for about 3 times in the last hour while browsing the forum.  Weirdly enough, it typically appears if I go to the Home page of the forum.

-
Regards,
PrivacyG

[digaran]

I was on an endless checking if your connection is secure loop, closed the site, came back 10 min later to see I'm logged in.

[drwhobox]

After 10 minutes of waiting, I logged in this time. Something happened to the server, or it's just because of the user load?
But now I confirmed it is not because of my connection.

[JeromeTash]

Happened to me earlier on, though, I was just accessing my account that was already logged in. The unusual check appeared, and I had initially thought it was just me. Nonetheless, it was not slow, so I was ok with everything. Perhaps it's the bots that were most affected.

[jackg]

I don't feed cookies to my scraper, so it doesn't index the Investigations board. I use cookies for other tasks, but none of them were running so I can't tell if it would have worked.

Bpip fed cookies under Vod's code (it probably still does) to get user info so we probably already know the issue still persisted during that time.

I wonder if this was resolved by whitelisting everything or just the bots, we probably won't find out similarly to if this was something automated by CF, an accidental kill switch triggered or was implemented for a reason (all are quite likely).

Based on the up time of this forum, there has to be multiple people with access to all connection information (there's probably more that could do defensive operations too - such as a controlled rerouting or changing certain cloudlfare settings to improve performance or limit some more intense pingers).

[suchmoon]

loyce's and bpip's scraper also seems to be down? (paging @LoyceV, @suchmoon, @ibminer).

True, I can't scrape anything - getting 503 errors.

1. If Cloudflare pops up, calls a function that invokes a puppeteer page (an instance of chrome) to the forum and waits for CF to clear out.

I have a prototype of a parser that works entirely via Playwright/Firefox and even solves captchas but it'd be quite absurd to use it. Not only because it needs an ungodly amount of RAM to work, but also because it looks like we're going against the policy (unwritten as it may be) of Cloudflare and/or Bitcointalk. This needs to be solved properly, i.e. by implementing some sort of whitelist/key/whatever. Not by trying to hack the damn thing.

Edit: my other parser that I use for plagiarism checks is still running albeit intermittently, not sure what the difference is that makes this one kinda sorta work.

[shahzadafzal]

loyce's and bpip's scraper also seems to be down? (paging @LoyceV, @suchmoon, @ibminer).

True, I can't scrape anything - getting 503 errors.

Strange it may seem but this is what I figured out for now if you navigate to https://bitcointalk.org/index.php you will still get 503 error. However if you navigate to to any other page or only https://bitcointalk.org/ it bypasses Cloudflare

For example:
https://bitcointalk.org/index.php                         Error 503
https://bitcointalk.org/index.php?action=stats      Error 503

https://bitcointalk.org/index.php?action=profile;u=1634314          200 OK
https://bitcointalk.org/index.php?topic=5422131.0                       200 OK
https://bitcointalk.org/index.php?action=merit;stats=recent          200 OK

I think only theymos can guide here more.

Edit: Strangely enough you can bypass Cloudflare for the home page just by adding any parameter e.g. https://bitcointalk.org/index.php?s=1

[ABCbits]

Perhaps it's the bots that were most affected.

I use Tor Browser and yesterday i was also seriously affected with CloudFlare security check page. I managed to reduce frequency of check page by only opening 1 tab, change Tor circuit occasionally and save post text/draft on text editor. But it wasn't pleasant experience.

Edit: even now, i still see CloudFlare security check page on few occasion.

[PrivacyG]

shahzadafzal is correct.  I am only running my browser with Java Script off and I can log in to Bitcoin Talk using ccode but Home page does not load.  The rest seems to work properly.  I am lucky I know 'index.php?action=profile' means viewing my own profile and that is how I skip the error page.

-
Regards,
PrivacyG

[suchmoon]

Strange it may seem but this is what I figured out for now if you navigate to https://bitcointalk.org/index.php you will still get 503 error. However if you navigate to to any other page or only https://bitcointalk.org/ it bypasses Cloudflare

Unfortunately I'm getting a 503 on https://bitcointalk.org/index.php?action=login2;ccode=... which makes it impossible for the parser to log in. But at least it's something I can experiment with and see if tweaking some parameters would help.

[PrivacyG]

Unfortunately I'm getting a 503 on https://bitcointalk.org/index.php?action=login2;ccode=... which makes it impossible for the parser to log in. But at least it's something I can experiment with and see if tweaking some parameters would help.

I log in using https://bitcointalk.org/index.php?action=login;ccode= and it works well, is there a difference between login and login2?

-
Regards,
PrivacyG

[ShowOff]

Edit: even now, i still see CloudFlare security check page on few occasion.

I don't know what the problem is, I also experienced it.
It's frequent and I've experienced CloudFlare security check 2 times in the last hour. Indeed, at this time my internet is not smooth where several times I have to turn off the wifi device because the internet is disconnected, but I'm sure this is not an internet problem.

I hope there is a solution to this problem quickly, I will be bother if the frequency is getting shorter even this CloudFlare security check doesn't take much time ''only 3-5 seccond".

[suchmoon]

Unfortunately I'm getting a 503 on https://bitcointalk.org/index.php?action=login2;ccode=... which makes it impossible for the parser to log in. But at least it's something I can experiment with and see if tweaking some parameters would help.

I log in using https://bitcointalk.org/index.php?action=login;ccode= and it works well, is there a difference between login and login2?

login2 happens when you click the "Login" button. I guess it redirects to index.php and throws the error, but at that point the login is completed and cookies should be good so I'll try to ignore that error and see if it can load any other pages afterwards. It just sucks having to code all these sketchy workarounds that will probably break the next time Cloudflare tweaks something.