Bitcoin Forum
October 24, 2020, 03:40:13 PM *
News: Latest Bitcoin Core release: 0.20.0 [Torrent]
 
   Home   Help Search Login Register More  
Pages: [1]
  Print  
Author Topic: Exporting data from the forum (trust network)  (Read 565 times)
Sukrim
Legendary
*
Offline Offline

Activity: 2562
Merit: 1002


View Profile
June 21, 2014, 01:08:22 PM
 #1

Now that the "trust" system is in effect for some time already, I thought it would be nice to take a closer look at it.

However, I don't want to go and write a crawler + parser to extract all info of ~330k accounts if there is another way. It might also save bitcointalk from quite a bit of traffic too. Is there some machine-readable way to export the data on the trust page (e.g. https://bitcointalk.org/index.php?action=trust;u=3) and/or are there limits or rules in place that could keep me from crawling these pages? Has someone already done this maybe? Searches for "trust" often just bring up pirateat40s scam...

https://www.coinlend.org <-- automated lending at various exchanges.
https://www.bitfinex.com <-- Trade BTC for other currencies and vice versa.
Mail me at Bitmessage: BM-BbiHiVv5qh858ULsyRDtpRrG9WjXN3xf
1603554013
Hero Member
*
Offline Offline

Posts: 1603554013

View Profile Personal Message (Offline)

Ignore
1603554013
Reply with quote  #2

1603554013
Report to moderator
1603554013
Hero Member
*
Offline Offline

Posts: 1603554013

View Profile Personal Message (Offline)

Ignore
1603554013
Reply with quote  #2

1603554013
Report to moderator
1603554013
Hero Member
*
Offline Offline

Posts: 1603554013

View Profile Personal Message (Offline)

Ignore
1603554013
Reply with quote  #2

1603554013
Report to moderator
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1603554013
Hero Member
*
Offline Offline

Posts: 1603554013

View Profile Personal Message (Offline)

Ignore
1603554013
Reply with quote  #2

1603554013
Report to moderator
1603554013
Hero Member
*
Offline Offline

Posts: 1603554013

View Profile Personal Message (Offline)

Ignore
1603554013
Reply with quote  #2

1603554013
Report to moderator
Baitty
Hero Member
*****
Offline Offline

Activity: 532
Merit: 500

Currently held as collateral by monbux


View Profile
June 21, 2014, 01:12:05 PM
 #2

You would have to be exporting the data easy so often because of the trust is forever changing for each user. What you could do is restrict your search. so don't get data from users who have 0 activity, 0 posts and are newbies etc.

Currently held as collateral by monbux
Sukrim
Legendary
*
Offline Offline

Activity: 2562
Merit: 1002


View Profile
June 21, 2014, 01:38:11 PM
 #3

Well, depending on what one does of course something like a crawler (look at everyone account X trusts and is trusted by, then expand from there) will very easily find the largest connected network in the forum trust system and the remaining few ones can be then set aside or less often scraped... Also it might make sense anyways just to look at the "main" trust network, not at some off-the-grid ones that might be likely uninteresting anyways (unless you want to see someone creating a dozen sock puppets to trust his main account) - once someone scams, sooner or later somebody in the main network will anyways also point it out in the trust system and pull the scammer in.

My question is rather if there is something like https://bitcointalk.org/index.php?action=trust;u=3;format=json instead of me having to parse everything, similar to the wap2 switch (https://bitcointalk.org/index.php?action=trust;u=3;wap2).

https://www.coinlend.org <-- automated lending at various exchanges.
https://www.bitfinex.com <-- Trade BTC for other currencies and vice versa.
Mail me at Bitmessage: BM-BbiHiVv5qh858ULsyRDtpRrG9WjXN3xf
mprep
Diamond Member
Global Moderator
Legendary
*
Offline Offline

Activity: 2744
Merit: 1732


In a world of peaches, don't ask for apple sauce


View Profile WWW
June 21, 2014, 07:09:58 PM
 #4

The only possible way would be to add user1 to your trust list, then in the Trust settings page, bellow the text Trust network and Full you will see the people he trusts and the people his trusted people trust and so on. Then remove user1 and add user2, and so on. Hovever, be weary: the forum allows only one request per second. If you abuse it, you're IP will get banned. This was a problem for Bitcointa.lk as well.

Sukrim
Legendary
*
Offline Offline

Activity: 2562
Merit: 1002


View Profile
June 21, 2014, 07:25:00 PM
 #5

Hm, as far as I understand it, this would anyways only extract the data of who sees whose "feedback". While that may be interesting too, I am more interested in the "feedback" itself (for example to see who really is good in calling out scammers and thus would be good to have in my own list).

The 1 request/second limit is very good to know, thanks for that info!

It seems like the trust pages are anyways locked to anonymous users, so I probably need my own bot account (hopefully there are not more restrictions for noobs regarding page requests?) to crawl...

https://www.coinlend.org <-- automated lending at various exchanges.
https://www.bitfinex.com <-- Trade BTC for other currencies and vice versa.
Mail me at Bitmessage: BM-BbiHiVv5qh858ULsyRDtpRrG9WjXN3xf
Pages: [1]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!