Bitcoin Forum

Other => Meta => Topic started by: Lavender on July 11, 2013, 01:16:15 AM



Title: How can I download all posts from a user?
Post by: Lavender on July 11, 2013, 01:16:15 AM
I'm interested in a way to download all posts from a user (not me). This way, I would like to search one large document by date of post or by keywords.


Title: Re: How can I download all posts from a user?
Post by: 🏰 TradeFortress 🏰 on July 11, 2013, 01:19:43 AM
You'll need to web scrap. Shouldn't be too hard, you can probably get by with file_get_contents and splitting strings


Title: Re: How can I download all posts from a user?
Post by: Bitsky on July 11, 2013, 08:32:30 AM
I could write a scraper and store the posts in e.g. a XML file.

PM me if you're interested.


Title: Re: How can I download all posts from a user?
Post by: jackjack on July 11, 2013, 09:47:23 AM
https://bitcointalk.org/index.php?action=profile;u=67058;sa=showPosts
   +
Python
   =
Done with less than 20 lines


Title: Re: How can I download all posts from a user?
Post by: Lavender on July 13, 2013, 05:43:17 PM
I am familiar with the terms Python, XML, and web scraping, but I have not actually used any of them. Any suggestions on where to start?


Title: Re: How can I download all posts from a user?
Post by: Bitsky on July 14, 2013, 09:38:09 AM
I am familiar with the terms Python, XML, and web scraping, but I have not actually used any of them. Any suggestions on where to start?
There's a difference between being familiar with the terms and being able to do it.
Basically your options are to either use this as a project to learn a bit about programming, or pay someone else to do it for you.


Title: Re: How can I download all posts from a user?
Post by: metacoin on July 15, 2013, 03:13:34 PM
I am familiar with the terms Python, XML, and web scraping, but I have not actually used any of them. Any suggestions on where to start?
http://stackoverflow.com/questions/2190502/screen-scraping-with-python (http://stackoverflow.com/questions/2190502/screen-scraping-with-python)
http://www.crummy.com/software/BeautifulSoup/ (http://www.crummy.com/software/BeautifulSoup/)
http://scrapy.org/ (http://scrapy.org/)