Exporting data from the forum (trust network)

[Sukrim]

Now that the "trust" system is in effect for some time already, I thought it would be nice to take a closer look at it.

However, I don't want to go and write a crawler + parser to extract all info of ~330k accounts if there is another way. It might also save bitcointalk from quite a bit of traffic too. Is there some machine-readable way to export the data on the trust page (e.g. https://bitcointalk.org/index.php?action=trust;u=3) and/or are there limits or rules in place that could keep me from crawling these pages? Has someone already done this maybe? Searches for "trust" often just bring up pirateat40s scam...

[1714598213]

1714598213

[1714598213]

1714598213

[1714598213]

1714598213

[1714598213]

1714598213

[Baitty]

You would have to be exporting the data easy so often because of the trust is forever changing for each user. What you could do is restrict your search. so don't get data from users who have 0 activity, 0 posts and are newbies etc.

[Sukrim]

Well, depending on what one does of course something like a crawler (look at everyone account X trusts and is trusted by, then expand from there) will very easily find the largest connected network in the forum trust system and the remaining few ones can be then set aside or less often scraped... Also it might make sense anyways just to look at the "main" trust network, not at some off-the-grid ones that might be likely uninteresting anyways (unless you want to see someone creating a dozen sock puppets to trust his main account) - once someone scams, sooner or later somebody in the main network will anyways also point it out in the trust system and pull the scammer in.

My question is rather if there is something like https://bitcointalk.org/index.php?action=trust;u=3;format=json instead of me having to parse everything, similar to the wap2 switch (https://bitcointalk.org/index.php?action=trust;u=3;wap2).

[mprep]

The only possible way would be to add user1 to your trust list, then in the Trust settings page, bellow the text Trust network and Full you will see the people he trusts and the people his trusted people trust and so on. Then remove user1 and add user2, and so on. Hovever, be weary: the forum allows only one request per second. If you abuse it, you're IP will get banned. This was a problem for Bitcointa.lk as well.

[Sukrim]

Hm, as far as I understand it, this would anyways only extract the data of who sees whose "feedback". While that may be interesting too, I am more interested in the "feedback" itself (for example to see who really is good in calling out scammers and thus would be good to have in my own list).

The 1 request/second limit is very good to know, thanks for that info!

It seems like the trust pages are anyways locked to anonymous users, so I probably need my own bot account (hopefully there are not more restrictions for noobs regarding page requests?) to crawl...