fetch bitcointalk data

[lahm-44]

do i can fetch bitcointalk data like post count and activity like bit-x campaingn. i know they dont have api feature but i see some site/campaingns are fetching it

[mocacinno]

you can write a parser/scraper to just load the user's profile page, and parse their recent posts up untill the very first post..

A lot of work, but still doable

[OmegaStarScream]

I'm not sure how they do it without an API but this should be more then enough to explain it to you : https://github.com/achow101/BitcointalkAccountPricer

[mocacinno]

I'm not sure how they do it without an API but this should be more then enough to explain it to you : https://github.com/achow101/BitcointalkAccountPricer

I think they do this by parsing the profile page, and then parsing all post history (like i said in my first post)

I didn't browse the full sourcecode, but i immediately noticed following string on their github page:

Code:

Document profileSummary = Jsoup.connect("https://bitcointalk.org/index.php?action=profile;u=" + userId + ";sa=summary").get();

[lahm-44]

thanks for the link i saw the codey they haveused html,css,php,xml and java too and honestly it has confused me even more i have read about scraping on google but never saw anything about it . hope i fan figure this this out and more hope is welcome

[minifrij]

thanks for the link i saw the codey they haveused html,css,php,xml and java too and honestly it has confused me even more i have read about scraping on google but never saw anything about it . hope i fan figure this this out and more hope is welcome

If you want to scrape a webpage, I suggest using PHP or another server side language. If you want to use PHP, I suggest that you use this framework; I have used it in the past and as long as you have a basic idea of PHP data types it's very useful and easy to use.

Also, it should be mentioned that if you want to scrape the forum you should only do it once per second or your server/bot will be IP banned.

[lahm-44]

thanks for the link i saw the codey they haveused html,css,php,xml and java too and honestly it has confused me even more i have read about scraping on google but never saw anything about it . hope i fan figure this this out and more hope is welcome

If you want to scrape a webpage, I suggest using PHP or another server side language. If you want to use PHP, I suggest that you use this framework; I have used it in the past and as long as you have a basic idea of PHP data types it's very useful and easy to use.

Also, it should be mentioned that if you want to scrape the forum you should only do it once per second or your server/bot will be IP banned.

thinks for the link finally i got a point from where i can start researching about it and i hope html dom are not only the single thing which is not taught to me on the free learning sites anyway thanks for the help.i am keeping this thread open so if anyone like to give any other helpful reference then u are welcome i will study it