|
Title: Can we have one small change on Recent Merits page? Post by: hatshepsut93 on October 02, 2019, 11:31:27 PM I would like to scrape https://bitcointalk.org/index.php?action=merit;stats=recent to make real-time merit api and notifications, but this page lacks one crucial piece of data - the name and link to the profile of a user who received merit. Right now the only way to get this is to follow the link to the post that was merited and do additional scraping, but it just creates unnecessary load and is messy to implement.
Title: Re: Can we have one small change on Recent Merits page? Post by: hosseinimr93 on October 03, 2019, 12:06:45 AM http://loyce.club/Merit/merit.html which has been created by LoyceV contains all merit transactions, merit senders and merit receivers. But the problem is that the data is not up to date. I think they are updated weekly.
Title: Re: Can we have one small change on Recent Merits page? Post by: tranthidung on October 03, 2019, 12:15:24 AM http://loyce.club/Merit/merit.html which has been created by LoyceV contains all merit transactions, merit senders and merit receivers. But the problem is that the data is not up to date. I think they are updated weekly. It is weekly updated because the data source comes from weekly update of theymosHere you go: https://bitcointalk.org/merit.txt.xz Similar to trust.txt.xz, it'll be updated weekly. It will show only the last 120 days of data; someone else should archive the old ones if you want them. I am especially interested in analyses of this data which could point to sub-communities where the initial sMerit is exhausted and new sources are necessary, and people who might be good merit sources. Edit: Note that for a little while I had user_to and user_from as names, but I decided to change it to IDs. Title: Re: Can we have one small change on Recent Merits page? Post by: Vod on October 03, 2019, 12:28:20 AM I would like to scrape https://bitcointalk.org/index.php?action=merit;stats=recent to make real-time merit api and notifications, but this page lacks one crucial piece of data - the name and link to the profile of a user who received merit. Right now the only way to get this is to follow the link to the post that was merited and do additional scraping, but it just creates unnecessary load and is messy to implement. I sent this in a PM to Theymos when merit was released, but no reply. I don't think he wants to make it too easy to scrape the data. Meanwhile... https://bpip.org/report.aspx?r=meritratio ---> Merit sent/received ratio https://bpip.org/report.aspx?r=meritcircle ---> Merit circle (number of distinct profiles merit sent to or received from) https://bpip.org/report.aspx?r=mostmerit ---> Most overall merit (initial and received) https://bpip.org/report.aspx?r=mostmerited ---> Most merit received https://bpip.org/report.aspx?r=meritsent ---> Most merit sent https://bpip.org/smerit.aspx?to=&from=&start=&end=&min=&max= ---> Real Time Historical sMerit Transfer. See link for instructions https://bpip.org/r/mostmerited.aspx ---> List of most merited posts for a certain time frame Title: Re: Can we have one small change on Recent Merits page? Post by: hatshepsut93 on October 03, 2019, 12:44:32 AM http://loyce.club/Merit/merit.html which has been created by LoyceV contains all merit transactions, merit senders and merit receivers. But the problem is that the data is not up to date. I think they are updated weekly. I know, my project is actually based on it - https://bitcointalk.org/index.php?topic=5181082 I sent this in a PM to Theymos when merit was released, but no reply. I don't think he wants to make it too easy to scrape the data. I was under an impression that this kind of pages exists specifically to be scraped, am I wrong? Did theymos ever said anything about scraping in general - what's his stance on it? https://bpip.org/smerit.aspx?to=&from=&start=&end=&min=&max= ---> Real Time Historical sMerit Transfer. See link for instructions Doesn't work, says "to or from is required", which is not what I need - I don't want to poll this page for every subscribed user, this wouldn't be scalable. Title: Re: Can we have one small change on Recent Merits page? Post by: LoyceV on October 03, 2019, 01:30:30 PM Right now the only way to get this is to follow the link to the post that was merited and do additional scraping, but it just creates unnecessary load and is messy to implement. What's really going to get you, is if the Merited post was made on a hidden board (most likely the VIP or Staff board). It'll look like the post was deleted, and you'll have to correct it after theymos' weekly Merit data dump (https://bitcointalk.org/merit.txt.xz).Did theymos ever said anything about scraping in general - what's his stance on it? I haven't read anything about it, but if you go faster than 1 request per second, you'll hit a timeout. |