|
Title: fetch bitcointalk data Post by: lahm-44 on December 14, 2015, 01:19:45 PM do i can fetch bitcointalk data like post count and activity like bit-x campaingn. i know they dont have api feature but i see some site/campaingns are fetching it
Title: Re: fetch bitcointalk data Post by: mocacinno on December 14, 2015, 01:21:40 PM you can write a parser/scraper to just load the user's profile page, and parse their recent posts up untill the very first post..
A lot of work, but still doable ;) Title: Re: fetch bitcointalk data Post by: OmegaStarScream on December 14, 2015, 01:25:02 PM I'm not sure how they do it without an API but this should be more then enough to explain it to you : https://github.com/achow101/BitcointalkAccountPricer
Title: Re: fetch bitcointalk data Post by: mocacinno on December 14, 2015, 01:27:48 PM I'm not sure how they do it without an API but this should be more then enough to explain it to you : https://github.com/achow101/BitcointalkAccountPricer I think they do this by parsing the profile page, and then parsing all post history (like i said in my first post) I didn't browse the full sourcecode, but i immediately noticed following string on their github page: Code: Document profileSummary = Jsoup.connect("https://bitcointalk.org/index.php?action=profile;u=" + userId + ";sa=summary").get();Title: Re: fetch bitcointalk data Post by: lahm-44 on December 14, 2015, 01:42:19 PM thanks for the link i saw the codey they haveused html,css,php,xml and java too and honestly it has confused me even more i have read about scraping on google but never saw anything about it . hope i fan figure this this out and more hope is welcome
Title: Re: fetch bitcointalk data Post by: minifrij on December 14, 2015, 01:59:05 PM thanks for the link i saw the codey they haveused html,css,php,xml and java too and honestly it has confused me even more i have read about scraping on google but never saw anything about it . hope i fan figure this this out and more hope is welcome If you want to scrape a webpage, I suggest using PHP or another server side language. If you want to use PHP, I suggest that you use this framework (http://simplehtmldom.sourceforge.net/); I have used it in the past and as long as you have a basic idea of PHP data types it's very useful and easy to use.Also, it should be mentioned that if you want to scrape the forum you should only do it once per second or your server/bot will be IP banned. Title: Re: fetch bitcointalk data Post by: lahm-44 on December 14, 2015, 02:12:33 PM thanks for the link i saw the codey they haveused html,css,php,xml and java too and honestly it has confused me even more i have read about scraping on google but never saw anything about it . hope i fan figure this this out and more hope is welcome If you want to scrape a webpage, I suggest using PHP or another server side language. If you want to use PHP, I suggest that you use this framework (http://simplehtmldom.sourceforge.net/); I have used it in the past and as long as you have a basic idea of PHP data types it's very useful and easy to use.Also, it should be mentioned that if you want to scrape the forum you should only do it once per second or your server/bot will be IP banned. |