hatshepsut93 (OP)
Legendary
Offline
Activity: 3010
Merit: 2148
|
|
September 02, 2019, 04:27:21 PM |
|
I have created a public API for merit data, based on LoyceV's log of merit transactions, which is based on theymos's merit dumps. The data is updated once per week. I didn't tune the rate limits yet, cause I don't know what kind of load can I expect, so refer to the headers if needed.
|
|
|
|
hatshepsut93 (OP)
Legendary
Offline
Activity: 3010
Merit: 2148
|
|
September 17, 2019, 04:58:07 PM |
|
Here's a userscript that utilizes my API do display earned merit of every poster just bellow their total merit score. // ==UserScript== // @name Display earned merit // @version 1.0 // @grant none // @include https://bitcointalk.org/index.php?topic=* // @run-at document-end // ==/UserScript==
const getMeritsFromApi = async function(ids) { return await fetch(`https://bct-quote-notifier.herokuapp.com/api/merit/v1/profiles/?ids=${ids.join(",")}`).then(r => r.json() ) }
const insertEmerit = function(meritNode, eMerit) { try { let eMeritNode = document.createElement("text") meritNode.after(eMeritNode) meritNode.after(document.createElement("br")) eMeritNode.textContent = `eMerit: ${eMerit}` } catch (e) { console.error(e) } }
const main = async function() { let posters = new Map([]) for (const node of document.querySelectorAll(".poster_info > .smalltext")) { const meritNode = [...node.childNodes].find(e => e.textContent.match(/Merit/))
const userId = [...node.childNodes].find(e => e.href && e.href.match(/action=profile;u=/)).href.split(";u=")[1]
if (meritNode && userId) { let poster = posters.get(userId) if (poster) { poster.meritNodes.push(meritNode) } else { posters.set(userId, { meritNodes: [meritNode] }) } } }
let earnedMerits = await getMeritsFromApi([...posters].map(poster => poster[0]))
earnedMerits .filter(e => e !== null) .forEach(e => { let poster = posters.get(e.id.toString()) if (poster) { poster.earned = e.earned } })
for (let [, poster] of posters) { try { poster.meritNodes.forEach(node => { insertEmerit(node, poster.earned || 0) }) } catch (e) { console.error(e) } } }
main()
|
|
|
|
LoyceV
Legendary
Offline
Activity: 3360
Merit: 16969
Thick-Skinned Gang Leader and Golden Feather 2021
|
Are you still using my weekly merit.all.txt for this, or are you also scraping Recent merits on a regular basis to show near real-time results? Every 10 minutes will probably be enough for this.
|
|
|
|
hatshepsut93 (OP)
Legendary
Offline
Activity: 3010
Merit: 2148
|
|
September 17, 2019, 05:44:10 PM |
|
Are you still using my weekly merit.all.txt for this, or are you also scraping Recent merits on a regular basis to show near real-time results? Every 10 minutes will probably be enough for this. I'm using your merit.txt, didn't even know we finally have a better merit log. Are you going to continue to update your merit.txt? I could change my API to scraping recent merits if there was some interest from the public, so far I got 0 requests in the last week
|
|
|
|
DdmrDdmr
Legendary
Offline
Activity: 2366
Merit: 10869
There are lies, damned lies and statistics. MTwain
|
|
September 17, 2019, 05:55:38 PM |
|
<…> I could change my API to scraping recent merits if there was some interest from the public, so far I got 0 requests in the last week <…>
The issue is probably more on the usability side, than the readiness of it being up-to-date right down to the most recent TXs. The profile and TX data references the inner forum IDs, which is fine, but anyone who is willing to exploit the data still needs to retrieve/scrape the mappings of the UserIDs to names and message IDs to the message hierarchy (i.e. forum path (section, subsection, etc.), post name).
|
|
|
|
LoyceV
Legendary
Offline
Activity: 3360
Merit: 16969
Thick-Skinned Gang Leader and Golden Feather 2021
|
|
September 17, 2019, 06:30:15 PM |
|
I'm using your merit.txt, didn't even know we finally have a better merit log. Are you going to continue to update your merit.txt? As far as I know, it's been there since the introduction of Merit. But I'll keep doing my weekly updates anyway.
|
|
|
|
hatshepsut93 (OP)
Legendary
Offline
Activity: 3010
Merit: 2148
|
|
December 31, 2019, 02:17:32 PM |
|
I have started scraping recent merit transactions, so the accuracy of my API will be significantly improved. I still don't guarantee it being 100% accurate, and at worst you can expect it to be only accurate up to the previous week.
|
|
|
|
tranthidung
Legendary
Offline
Activity: 2324
Merit: 4100
Farewell o_e_l_e_o
|
|
January 01, 2020, 02:24:04 AM |
|
I appreciated your decent time to code the scripts for users but if I was you, I would not going to this one. Users, personally know how many sMerits they have at specific point of time. While if one user is curious how many sMerits left the others have, they might choose to use your scripts and your JSON API for merit data.
I don't say negatively about it but I wonder what is the true reasons to look at the others' unused sMerits (to beg for merits, something like that).
BPIP provided that statistic during a very long period till the day it stopped showing that one. I don't know why Vod decided to stop providing it. As I remembered, BPIP did not provide that statistic before it shown issues with data scraping recent months.
Anyway, thank you for the script and happy new year, @hatshepsut93.
|
|
|
|
hacker1001101001
|
|
January 01, 2020, 07:49:46 AM |
|
I don't say negatively about it but I wonder what is the true reasons to look at the others' unused sMerits (to beg for merits, something like that).
I agree, sMerits data scraping is Ok, but making that public to the level of viewing it under user's profile seems a bit off the track. Theymos would always appreciate keeping it as much as hidden as he can if you see. I will not be posting a definitive list of merit sources (so that people don't bug them too much), though you'll soon figure out who they are if you pay attention.
Still a good peice of work, and I appreciate the efforts of the OP. But ya making more info easily accessible is not always profitable to the community overall. Merit Sources would just be fed up with begging requests.
|
|
|
|
hatshepsut93 (OP)
Legendary
Offline
Activity: 3010
Merit: 2148
|
|
January 01, 2020, 12:33:06 PM |
|
I appreciated your decent time to code the scripts for users but if I was you, I would not going to this one. Users, personally know how many sMerits they have at specific point of time. While if one user is curious how many sMerits left the others have, they might choose to use your scripts and your JSON API for merit data.
I don't say negatively about it but I wonder what is the true reasons to look at the others' unused sMerits (to beg for merits, something like that).
BPIP provided that statistic during a very long period till the day it stopped showing that one. I don't know why Vod decided to stop providing it. As I remembered, BPIP did not provide that statistic before it shown issues with data scraping recent months.
Anyway, thank you for the script and happy new year, @hatshepsut93.
Huh? Where did you get the idea that I provide any data related to sMerit? I only provide data for merit transactions and merit profiles, and merit profiles have earned and sent merit, and that's all. As for BPIP, the difference is that this is an API rather than an app, the goal of API is for other people to build on top of it. I have created a userscript that displays user's earned merit below their actual merit in threads, by requesting this data from this API. One can create some merit statistics tool based on my API without having to do any scraping themselves.
|
|
|
|
tranthidung
Legendary
Offline
Activity: 2324
Merit: 4100
Farewell o_e_l_e_o
|
|
January 01, 2020, 02:23:22 PM |
|
Huh? Where did you get the idea that I provide any data related to sMerit? I only provide data for merit transactions and merit profiles, and merit profiles have earned and sent merit, and that's all.
As for BPIP, the difference is that this is an API rather than an app, the goal of API is for other people to build on top of it. I have created a userscript that displays user's earned merit below their actual merit in threads, by requesting this data from this API. One can create some merit statistics tool based on my API without having to do any scraping themselves.
I am deeply sorry to you. I mis-remembered what I saw in the image you posted with your script, in the second post. That shows eMerits, not sMerits. Once again, I am sorry because of my carelessness. Ok, now I understood you do. As being said, even if you provide sMerits with your script, there is nothing wrong from you. When people (users) want to use any tool, they can do it positively or negatively, you can control that.
|
|
|
|
akhjob
|
|
January 02, 2020, 05:57:08 PM |
|
Here's a userscript that utilizes my API do display earned merit of every poster just bellow their total merit score. -skip- [img]https://i.imgur.com/wOHHo8P.png[/img]
I think the data provided by the userscript is wrong for almost all users. Take your profile for example: It shows Merit = 1040, eMerit = 529, that means the airdropped merit is 1040-529=511, which is not possible. My Profile shows Merit = 220, eMerit = 210, so Airdropped Merits = 10, but that's not the case, I started with 0 merits
|
|
|
|
LoyceMobile
|
|
January 02, 2020, 07:00:36 PM |
|
I think the data provided by the userscript is wrong for almost all users.
It's updated once a week, so the difference is the merit earned since last Friday morning.
|
|
|
|
dzungmobile
Sr. Member
Offline
Activity: 770
Merit: 394
I stand with Ukraine!
|
|
January 02, 2020, 11:24:47 PM |
|
OP has started to scrap recent merit transactions to improve the accuracy, and not only depends on weekly merit data dumps from LoyceV - that is what OP intitially depended upon.
|
|
|
|
|