Title: What i discovered about webscraping Bitcointalk.org Post by: Zilon on April 03, 2023, 08:36:03 PM Hello mates i tried doing some fun stuffs with python BeautifulSoup library to scrap some information and possibly save them in a variable maybe to get to see the anchor tags and also scrap to see users with the highest activity in the last 20 days, play around with some informations scrapped from bitcointalk url but unfortunately i got an error message. I tried the code on a few other sited and it worked well but that of the forum gave me this error
https://i.ibb.co/C54XJ85/Screenshot-2023-04-03-at-20-30-35.png (https://ibb.co/TBJnK2B) https://i.ibb.co/PcgmG9S/Screenshot-2023-04-03-at-20-31-04.png (https://ibb.co/wS40hwv) i tried the same code on a few other sites like analytics I was able to get all the href and anchor tags from the sites https://i.ibb.co/8NxQcMN/Screenshot-2023-04-03-at-20-17-19.png (https://ibb.co/RY6Jy2Y) i did similar for facebook and it worked so i kept wondering why it didn't work for Bitcointalk url. I will be glad if some one can educate me why i can't scrap information from the forum. https://i.ibb.co/x5mr0rC/Screenshot-2023-04-03-at-21-31-52.png (https://ibb.co/F7sdMdx) Title: Re: What i discovered about webscraping Bitcointalk.org Post by: jackg on April 04, 2023, 03:03:37 AM Where's your code? Are you doing any looping (trying to load the website multiple times a second will result in an error, not sure if there's something else too as you've not added your code - feel free to dm if you don't want to post it publicly but remove login details if there are any).
time.sleep(1000) would be enough to add to a loop to stop the error - the time is in milliseconds if you want to edit it. Title: Re: What i discovered about webscraping Bitcointalk.org Post by: Zilon on April 04, 2023, 06:30:02 AM Where's your code? Are you doing any looping (trying to load the website multiple times a second will result in an error, not sure if there's something else too as you've not added your code - feel free to dm if you don't want to post it publicly but remove login details if there are any). my code is on the <img> element i posted but i will still type them if it is not visibletime.sleep(1000) would be enough to add to a loop to stop the error - the time is in milliseconds if you want to edit it. Code: ! pip install BeautifulSoup Code: HTTPError: HTTP Error 403: Forbidden Title: Re: What i discovered about webscraping Bitcointalk.org Post by: OmegaStarScream on April 04, 2023, 07:40:22 AM There are two options here:
To use requests instead of urllib: Code: import urllib Or add a user-agent to the request you're making: Code: import urllib Either way, make sure you're not sending requests too often[1]. You should use time.sleep but that function takes seconds in Python, and not milliseconds. [1] https://bitcointalk.org/index.php?topic=953815.msg10442011#msg10442011 Title: Re: What i discovered about webscraping Bitcointalk.org Post by: NotATether on April 04, 2023, 07:58:23 AM Try using Requests library to read the data instead of URLlib3.
Although I no longer have the code sample to show you, my implementation of a post scraper using Requests worked magnificently well, with a timeout of 1 second. You're probably running into issues with Cloudflare though, hence the 403. Maybe you should chain an anti-captcha browser or service to the library as well. Title: Re: What i discovered about webscraping Bitcointalk.org Post by: Zilon on April 04, 2023, 08:45:51 AM .... Thank you it solved the problem well.. And worked just finehttps://i.ibb.co/crr3VZv/Screenshot-2023-04-04-at-09-34-41.png (https://ibb.co/Tgg2d7T) https://i.ibb.co/C8GpZzm/Screenshot-2023-04-04-at-09-43-27.png (https://ibb.co/SyCD8P5) |