Bitcoin Forum
February 28, 2020, 05:05:29 AM *
News: Latest Bitcoin Core release: 0.19.0.1 [Torrent]
 
   Home   Help Search Login Register More  
Pages: [1]
  Print  
Author Topic: JSON API for merit data  (Read 322 times)
hatshepsut93
Legendary
*
Offline Offline

Activity: 1442
Merit: 1137


( ͡° ͜ʖ ͡°)


View Profile
September 02, 2019, 04:27:21 PM
 #1


I have created a public API for merit data, based on LoyceV's log of merit transactions, which is based on theymos's merit dumps. The data is updated once per week.

I didn't tune the rate limits yet, cause I don't know what kind of load can I expect, so refer to the headers if needed.

1582866329
Hero Member
*
Offline Offline

Posts: 1582866329

View Profile Personal Message (Offline)

Ignore
1582866329
Reply with quote  #2

1582866329
Report to moderator
1582866329
Hero Member
*
Offline Offline

Posts: 1582866329

View Profile Personal Message (Offline)

Ignore
1582866329
Reply with quote  #2

1582866329
Report to moderator
AWARD-WINNING
CRYPTO CASINO
ASKGAMBLERS
PLAYERS CHOICE 2019
PROUD
PARTNER OF
1500+
GAMES
2 MIN
CASH-OUTS
24/7
SUPPORT
100s OF
FREE SPINS
PLAY NOW
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1582866329
Hero Member
*
Offline Offline

Posts: 1582866329

View Profile Personal Message (Offline)

Ignore
1582866329
Reply with quote  #2

1582866329
Report to moderator
1582866329
Hero Member
*
Offline Offline

Posts: 1582866329

View Profile Personal Message (Offline)

Ignore
1582866329
Reply with quote  #2

1582866329
Report to moderator
1582866329
Hero Member
*
Offline Offline

Posts: 1582866329

View Profile Personal Message (Offline)

Ignore
1582866329
Reply with quote  #2

1582866329
Report to moderator
hatshepsut93
Legendary
*
Offline Offline

Activity: 1442
Merit: 1137


( ͡° ͜ʖ ͡°)


View Profile
September 17, 2019, 04:58:07 PM
Merited by LoyceV (10), hugeblack (2), ETFbitcoin (1), o_e_l_e_o (1), tranthidung (1), Husna QA (1)
 #2

Here's a userscript that utilizes my API do display earned merit of every poster just bellow their total merit score.

Code:
// ==UserScript==
// @name     Display earned merit
// @version  1.0
// @grant    none
// @include        https://bitcointalk.org/index.php?topic=*
// @run-at         document-end
// ==/UserScript==

const getMeritsFromApi = async function(ids) {
    return await fetch(`https://bct-quote-notifier.herokuapp.com/api/merit/v1/profiles/?ids=${ids.join(",")}`).then(r =>
        r.json()
    )
}

const insertEmerit = function(meritNode, eMerit) {
    try {
        let eMeritNode = document.createElement("text")
        meritNode.after(eMeritNode)
        meritNode.after(document.createElement("br"))
        eMeritNode.textContent = `eMerit: ${eMerit}`
    } catch (e) {
        console.error(e)
    }
}

const main = async function() {
    let posters = new Map([])
    for (const node of document.querySelectorAll(".poster_info > .smalltext")) {
        const meritNode = [...node.childNodes].find(e => e.textContent.match(/Merit/))

        const userId = [...node.childNodes].find(e => e.href && e.href.match(/action=profile;u=/)).href.split(";u=")[1]

        if (meritNode && userId) {
            let poster = posters.get(userId)
            if (poster) {
                poster.meritNodes.push(meritNode)
            } else {
                posters.set(userId, { meritNodes: [meritNode] })
            }
        }
    }

    let earnedMerits = await getMeritsFromApi([...posters].map(poster => poster[0]))

    earnedMerits
        .filter(e => e !== null)
        .forEach(e => {
            let poster = posters.get(e.id.toString())
            if (poster) {
                poster.earned = e.earned
            }
        })

    for (let [, poster] of posters) {
        try {
            poster.meritNodes.forEach(node => {
                insertEmerit(node, poster.earned || 0)
            })
        } catch (e) {
            console.error(e)
        }
    }
}

main()



LoyceV
Legendary
*
Offline Offline

Activity: 1778
Merit: 5875


Thick-Skinned Gang Leader


View Profile WWW
September 17, 2019, 05:38:03 PM
Merited by hatshepsut93 (1)
 #3

Are you still using my weekly merit.all.txt for this, or are you also scraping Recent merits on a regular basis to show near real-time results? Every 10 minutes will probably be enough for this.

hatshepsut93
Legendary
*
Offline Offline

Activity: 1442
Merit: 1137


( ͡° ͜ʖ ͡°)


View Profile
September 17, 2019, 05:44:10 PM
 #4

Are you still using my weekly merit.all.txt for this, or are you also scraping Recent merits on a regular basis to show near real-time results? Every 10 minutes will probably be enough for this.

I'm using your merit.txt, didn't even know we finally have a better merit log. Are you going to continue to update your merit.txt?

I could change my API to scraping recent merits if there was some interest from the public, so far I got 0 requests in the last week  Embarrassed

DdmrDdmr
Hero Member
*****
Offline Offline

Activity: 784
Merit: 3418


There are lies, damned lies and statistics. MTwain


View Profile WWW
September 17, 2019, 05:55:38 PM
 #5

<…> I could change my API to scraping recent merits if there was some interest from the public, so far I got 0 requests in the last week <…>
The issue is probably more on the usability side, than the readiness of it being up-to-date right down to the most recent TXs. The profile and TX data references the inner forum IDs, which is fine, but anyone who is willing to exploit the data still needs to retrieve/scrape the mappings of the UserIDs to names and message IDs to the message hierarchy (i.e. forum path (section, subsection, etc.), post name).

LoyceV
Legendary
*
Offline Offline

Activity: 1778
Merit: 5875


Thick-Skinned Gang Leader


View Profile WWW
September 17, 2019, 06:30:15 PM
 #6

I'm using your merit.txt, didn't even know we finally have a better merit log. Are you going to continue to update your merit.txt?
As far as I know, it's been there since the introduction of Merit. But I'll keep doing my weekly updates anyway.

hatshepsut93
Legendary
*
Offline Offline

Activity: 1442
Merit: 1137


( ͡° ͜ʖ ͡°)


View Profile
December 31, 2019, 02:17:32 PM
 #7

I have started scraping recent merit transactions, so the accuracy of my API will be significantly improved. I still don't guarantee it being 100% accurate, and at worst you can expect it to be only accurate up to the previous week.

tranthidung
Hero Member
*****
Online Online

Activity: 742
Merit: 1255


Manager? See my profile page, please


View Profile WWW
January 01, 2020, 02:24:04 AM
 #8

I appreciated your decent time to code the scripts for users but if I was you, I would not going to this one. Users, personally know how many sMerits they have at specific point of time. While if one user is curious how many sMerits left the others have, they might choose to use your scripts and your JSON API for merit data.

I don't say negatively about it but I wonder what is the true reasons to look at the others' unused sMerits (to beg for merits, something like that).

BPIP provided that statistic during a very long period till the day it stopped showing that one. I don't know why Vod decided to stop providing it. As I remembered, BPIP did not provide that statistic before it shown issues with data scraping recent months.

Anyway, thank you for the script and happy new year, @hatshepsut93.




▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄    ▄▄▄▄                  ▄▄▄   ▄▄▄▄▄        ▄▄▄▄▄   ▄▄▄▄▄▄▄▄▄▄▄▄    ▄▄▄▄▄▄▄▄▄▄▄▄▄▄   ▄▄▄▄▄▄▄▄▄▄▄▄▄▄   ▄▄▄▄▄▄▄▄▄▄▄
 ▀████████████████▄  ████                 █████   ▀████▄    ▄████▀  ▄██████████████   ████████████▀  ▄█████████████▀  ▄█████████████▄
              ▀████  ████               ▄███▀███▄   ▀████▄▄████▀               ████   ████                ████                   ▀████
   ▄▄▄▄▄▄▄▄▄▄▄█████  ████              ████   ████    ▀██████▀      ██████████████▄   ████████████▀       ████       ▄▄▄▄▄▄▄▄▄▄▄▄████▀
   ██████████████▀   ████            ▄███▀     ▀███▄    ████        ████        ████  ████                ████       ██████████████▀
   ████              ████████████▀  ████   ██████████   ████        ████████████████  █████████████▀      ████       ████      ▀████▄
   ▀▀▀▀              ▀▀▀▀▀▀▀▀▀▀▀   ▀▀▀▀   ▀▀▀▀▀▀▀▀▀▀▀▀  ▀▀▀▀        ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀   ▀▀▀▀▀▀▀▀▀▀▀▀        ▀▀▀▀       ▀▀▀▀        ▀▀▀▀▀

#1 CRYPTO CASINO & SPORTSBOOK
  WELCOME
BONUS
.INSTANT & FAST.
.TRANSACTION.....
.PROVABLY FAIR.
......& SECURE......
.24/7 CUSTOMER.
............SUPPORT.
BTC      |      ETH      |      LTC      |      XRP      |      XMR      |      BNB      |     more
hacker1001101001
Sr. Member
****
Offline Offline

Activity: 742
Merit: 356


OBJECTIVE STANDARDS GUILD : https://bit.ly/2PgqLR9


View Profile
January 01, 2020, 07:49:46 AM
 #9

I don't say negatively about it but I wonder what is the true reasons to look at the others' unused sMerits (to beg for merits, something like that).

I agree, sMerits data scraping is Ok, but making that public to the level of viewing it under user's profile seems a bit off the track. Theymos would always appreciate keeping it as much as hidden as he can if you see.

I will not be posting a definitive list of merit sources (so that people don't bug them too much), though you'll soon figure out who they are if you pay attention.

Still a good peice of work, and I appreciate the efforts of the OP. But ya making more info easily accessible is not always profitable to the community overall. Merit Sources would just be fed up with begging requests.
hatshepsut93
Legendary
*
Offline Offline

Activity: 1442
Merit: 1137


( ͡° ͜ʖ ͡°)


View Profile
January 01, 2020, 12:33:06 PM
 #10

I appreciated your decent time to code the scripts for users but if I was you, I would not going to this one. Users, personally know how many sMerits they have at specific point of time. While if one user is curious how many sMerits left the others have, they might choose to use your scripts and your JSON API for merit data.

I don't say negatively about it but I wonder what is the true reasons to look at the others' unused sMerits (to beg for merits, something like that).

BPIP provided that statistic during a very long period till the day it stopped showing that one. I don't know why Vod decided to stop providing it. As I remembered, BPIP did not provide that statistic before it shown issues with data scraping recent months.

Anyway, thank you for the script and happy new year, @hatshepsut93.

Huh? Where did you get the idea that I provide any data related to sMerit? I only provide data for merit transactions and merit profiles, and merit profiles have earned and sent merit, and that's all.

As for BPIP, the difference is that this is an API rather than an app, the goal of API is for other people to build on top of it. I have created a userscript that displays user's earned merit below their actual merit in threads, by requesting this data from this API. One can create some merit statistics tool based on my API without having to do any scraping themselves.

tranthidung
Hero Member
*****
Online Online

Activity: 742
Merit: 1255


Manager? See my profile page, please


View Profile WWW
January 01, 2020, 02:23:22 PM
 #11

Huh? Where did you get the idea that I provide any data related to sMerit? I only provide data for merit transactions and merit profiles, and merit profiles have earned and sent merit, and that's all.

As for BPIP, the difference is that this is an API rather than an app, the goal of API is for other people to build on top of it. I have created a userscript that displays user's earned merit below their actual merit in threads, by requesting this data from this API. One can create some merit statistics tool based on my API without having to do any scraping themselves.
I am deeply sorry to you. I mis-remembered what I saw in the image you posted with your script, in the second post. That shows eMerits, not sMerits. Once again, I am sorry because of my carelessness.  Cry


Ok, now I understood you do.

As being said, even if you provide sMerits with your script, there is nothing wrong from you. When people (users) want to use any tool, they can do it positively or negatively, you can control that.




▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄    ▄▄▄▄                  ▄▄▄   ▄▄▄▄▄        ▄▄▄▄▄   ▄▄▄▄▄▄▄▄▄▄▄▄    ▄▄▄▄▄▄▄▄▄▄▄▄▄▄   ▄▄▄▄▄▄▄▄▄▄▄▄▄▄   ▄▄▄▄▄▄▄▄▄▄▄
 ▀████████████████▄  ████                 █████   ▀████▄    ▄████▀  ▄██████████████   ████████████▀  ▄█████████████▀  ▄█████████████▄
              ▀████  ████               ▄███▀███▄   ▀████▄▄████▀               ████   ████                ████                   ▀████
   ▄▄▄▄▄▄▄▄▄▄▄█████  ████              ████   ████    ▀██████▀      ██████████████▄   ████████████▀       ████       ▄▄▄▄▄▄▄▄▄▄▄▄████▀
   ██████████████▀   ████            ▄███▀     ▀███▄    ████        ████        ████  ████                ████       ██████████████▀
   ████              ████████████▀  ████   ██████████   ████        ████████████████  █████████████▀      ████       ████      ▀████▄
   ▀▀▀▀              ▀▀▀▀▀▀▀▀▀▀▀   ▀▀▀▀   ▀▀▀▀▀▀▀▀▀▀▀▀  ▀▀▀▀        ▀▀▀▀▀▀▀▀▀▀▀▀▀▀▀   ▀▀▀▀▀▀▀▀▀▀▀▀        ▀▀▀▀       ▀▀▀▀        ▀▀▀▀▀

#1 CRYPTO CASINO & SPORTSBOOK
  WELCOME
BONUS
.INSTANT & FAST.
.TRANSACTION.....
.PROVABLY FAIR.
......& SECURE......
.24/7 CUSTOMER.
............SUPPORT.
BTC      |      ETH      |      LTC      |      XRP      |      XMR      |      BNB      |     more
akhjob
Sr. Member
****
Offline Offline

Activity: 616
Merit: 259


View Profile
January 02, 2020, 05:57:08 PM
 #12

Here's a userscript that utilizes my API do display earned merit of every poster just bellow their total merit score.

Code:
-skip-
[img]https://i.imgur.com/wOHHo8P.png[/img]

I think the data provided by the userscript is wrong for almost all users. Take your profile for example:



It shows Merit = 1040, eMerit = 529, that means the airdropped merit is 1040-529=511, which is not possible.



My Profile shows Merit = 220, eMerit = 210, so Airdropped Merits = 10, but that's not the case, I started with 0 merits
LoyceMobile
Full Member
***
Offline Offline

Activity: 677
Merit: 156


Hover mouse above my sig to find many useful links


View Profile WWW
January 02, 2020, 07:00:36 PM
 #13

I think the data provided by the userscript is wrong for almost all users.
It's updated once a week, so the difference is the merit earned since last Friday morning.

dzungmobile
Jr. Member
*
Online Online

Activity: 31
Merit: 2


View Profile
January 02, 2020, 11:24:47 PM
 #14

OP has started to scrap recent merit transactions to improve the accuracy, and not only depends on weekly merit data dumps from LoyceV - that is what OP intitially depended upon.

tranthidung's topic list (https://bitcointalk.org/gettopics.php?user=1292764)
Pages: [1]
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!