Goodbye to Merit History...

[Vod]

All good things must come to an end.

Today marks 120 days since the merit system was introduced, and as Theymos said, old entries are going to start dropping off the file:

https://bitcointalk.org/merit.txt.xz

If you want a complete list of merit entries, you now need to keep your own historical records.

Oldest entries (120+ days) will also start disappear off of individual merit pages.

[Wheelige]

And this means that anyone that merited themselves through alt accounts that havent been caught yet will get away with it.
Why was there a 120 day limit put on it?

[Piggy]

I think many people in here have a list for statistical purposes or some back up. In any case there would be many ways to preserve it online, the file is growing but is nothing enormously big.

[LoyceMobile]

I've been keeping records for a while now, with weekly updates. Assuming my script works correct, see here:

Weekly update (2018-05-18_Fri_10.14h)


theymos' data
Sample (format: time    amount  msg     user_from       user_to):

Code:

1526611812	1	3896497.msg37564005	55384	397737
1526611668	1	3904890.msg37566177	55384	101872
1526609537	1	3920469.msg37618243	35	1638760
1526609519	1	3920469.msg37621805	35	1809400
1526609504	1	3920469.msg37626593	35	347141
1526609497	1	3920469.msg37626533	35	140584
1526609479	1	3876298.msg37573164	153634	1920993
1526609465	2	3876298.msg37564969	153634	1920993
1526609438	1	3714531.msg37456319	153634	1920993
1526609410	10	2968050.msg30499459	175302	325061
1526609123	1	3920469.msg37618957	140584	78147
1526608852	1	3765354.msg37106943	51173	459836
1526608397	1	3459858.msg37625859	33083	1551663
1526607990	1	3466518.msg36033078	537726	347141
1526607011	1	3459858.msg37624785	33083	32660
1526606705	1	3221666.msg33554736	1180530	1987256
1526606439	1	3902111.msg37586978	553678	487418
1526606086	1	2360806.msg37339067	255027	1182581
1526606068	1	2360806.msg37329293	255027	976204
1526605974	1	3449802.msg35929703	698159	1068078
1526605845	1	3916891.msg37598078	1892084	1649515
1526604791	1	603757.msg27659572	993728	1030671
1526604580	3	3852312.msg37619660	696532	136484
1526603644	1	529294.msg37621428	988740	35
1526603356	5	3345156.msg37620639	234771	1893840
1526603337	1	3920469.msg37611087	988740	35
1526603316	1	3920469.msg37611040	988740	35
1526603280	1	3920469.msg37618957	988740	78147
1526602914	1	3511922.msg36840357	1713155	1910038
1526602486	2	529294.msg37621428	140584	35
1526601791	1	3910520.msg37577860	140827	1000199
1526601167	1	2972259.msg37617969	349247	1153843
1526601015	1	2544574.msg37527601	131361	1021018
1526600766	1	2048414.msg37359622	6822	199420
1526600743	1	3776827.msg37282360	1069571	1180530
1526600399	1	3896497.msg37536433	1069571	1180530
1526600188	1	3857601.msg37425062	1180530	1069571
1526600086	3	3326310.msg34750654	696532	771962
1526599859	3	3921159.msg37612948	696532	121796
1526599542	1	3917625.msg37613688	443338	1117904
1526599520	3	3922830.msg37617374	1000199	1143728
1526599279	1	3920469.msg37618957	84521	78147
1526598580	1	3875992.msg37585019	68364	187673
1526598540	1	178336.msg37617836	113106	569455
1526598395	2	3917884.msg37601937	917365	1521430
1526598156	1	178336.msg37617836	183757	569455
1526598144	1	3910520.msg37577860	1291828	1000199
1526598052	2	3922830.msg37617374	1000199	1143728
1526597700	1	3922020.msg37615090	370611	1291828
1526596027	1	3772728.msg37180165	1381048	1381008
.......
.......
.......

Full version^* (4 MB) (not limited to 120 days).
Also available in human readable format^* (21 MB).

^* This file will be overwritten by newer versions

Update: I only now see my upload failed, I'll fix that when I'm back at my PC.

[vit05]

I did not know about it, thank you for the warning. I will start saving the records to always remember who gave me merit and for what reason. Unfortunately, this might help some people that do merit farm. Since attempts to combat them are still being tested.

[LoyceV]

Unfortunately, this might help some people that do merit farm.

If needed, I can make a thread in Reputation where I grant Merit-history-requests for any user.

[Thirio]

Tried to open the link, downloaded something and then i checked the content. Man, i did not understand any of those. Other than having a backup file/record for the merit history, are there any reason why are we doing this?

[DdmrDdmr]

Today marks 120 days since the merit system was introduced, and as Theymos said, old entries are going to start dropping off the file:
https://bitcointalk.org/merit.txt.xz

Seen it. The Merit.txt file already has some initial hours trimmed off:

This Weeks:
nMerit     nTx        nFrom   nTo      minDate                              maxDate
156729   70826   14056   15682   2018-01-25 02:52:04.000   2018-05-25 02:45:40.000

Last Weeks:
nMerit     nTx        nFrom   nTo      minDate                               maxDate
154986   69364   13904   15511   2018-01-24 22:12:21.000   2018-05-18 02:50:12.000

The chopped off registers add up to 2359 sMerits (610 wacky deflowering the system Txs).

The weird thing I've found are a set of 9 Txs that were in last week's file and not on this weeks, belonging to the common period:

time         amount   msg                           user_from    user_to   converted_date
1526466697   1   45212.msg37354230       1192397   209286    2018-05-16 10:31:37.000
1526427398   1   3297659.msg37378125   452769     1168027   2018-05-15 23:36:38.000
1526417463   1   1747305.msg37390368   1160524   1012481   2018-05-15 20:51:03.000
1526412210   1   3297659.msg37378125   1627661   1168027   2018-05-15 19:23:30.000
1526403535   2   2671650.msg37344086   234915     501852    2018-05-15 16:58:55.000
1526391452   2   2671650.msg37344086   1765178   501852    2018-05-15 13:37:32.000
1523884328   2   3315165.msg34758186   974425     668651    2018-04-16 13:12:08.000
1518181690   1   2669342.msg29300364   1141229   1495332   2018-02-09 13:08:10.000
1518181563   1   2669342.msg29300364   1141229   1495332   2018-02-09 13:06:03.000

I've seen this happen before, but now it is a (very minor) issue, since we have to merge files.
The above 9 TXs should be in both files. I tried to check some of the messages and they are probably deleted messages. Nevertheless, as with all Merit.txt files, the TX should remain in the file.

Any idea why this happens to those 9 TXs ?

[TheBeardedBaby]

Yeah, my first merit (from 24th of Jan) is now in the History , gone.
Good that I have a back-up and I was thinking to make a thread to keep track on the merit and also for reference.
Time flies ....

January 24, 2018, 11:46:59 PM: 1 from poptok1 for Re: What is the function of the "Merit" score?

[TMAN]

So now is the time the scammers with a slight amount of intelligence are found out. The guys who noticed the 120 day limit will now start trading back the merits they received... luckily the detectives have the data dump

[LoyceV]

The weird thing I've found are a set of 9 Txs that were in last week's file and not on this weeks, belonging to the common period:
~
Any idea why this happens to those 9 TXs ?

I checked all time stamps, and they all show up in this weeks's data file.

[Thirdspace]

If you want a complete list of merit entries, you now need to keep your own historical records.

is there a way around to make archive.is able to capture my merit page?
it seems that merit page can only be viewed by registered users only, any reason for this?
It would be great if theymos can change so viewing the page does not require login

[DdmrDdmr]

I checked all time stamps, and they all show up in this weeks's data file.

Thanks, I think I know what's going on with them:

time         amount       msg                         user_from    user_to  
1526466697   1   4097266.msg37354230   1192397   209286
1526427398   1   4075132.msg37378125   452769   1168027
1526417463   1   3979143.msg37390368   1160524   1012481
1526412210   1   4075132.msg37378125   1627661   1168027
1526403535   2   4032536.msg37344086   234915   501852
1526391452   2   4032536.msg37344086   1765178   501852
1523884328   2   4169555.msg34758186   974425   668651
1518181690   1   4029745.msg29300364   1141229   1495332
1518181563   1   4029745.msg29300364   1141229   1495332

The timestamp is in the current file as you say, but the messageID has changes from one file to the other. The first record (4097266.msg37354230) still exists and reading the post seems to have moved section. The remaining 8 are now deleted (propably after moving them first).

I'm using as a conceptual PK all the fields in the record, not just the time, therefore showing those records in the new file as non-existant in relation to last week's file.

I checked and there are 499 cases of timestamps with 2 or more associated TXs in the file, therefore going for the complete record PK.

Edit:
What this entails is that changes to records older than 120 days may not be consistent in our ALT aggregate databases.
For example, as seen above, some merited TXs may change section or get deleted, but the ALT aggregate database will still point to the original message once the TX is outside of the 120 window frame.

It should be a minor issue, since in a week only 9 awarded Txs have suffered a change of associated message Id. Just a bug for accuracy, but not a problem on the whole.

[mocacinno]

If you want a complete list of merit entries, you now need to keep your own historical records.

is there a way around to make archive.is able to capture my merit page?
it seems that merit page can only be viewed by registered users only, any reason for this?
It would be great if theymos can change so viewing the page does not require login

there are ways around this...

Code:

curl 'https://bitcointalk.org/index.php?action=merit;u=".$uid."' -H 'accept-encoding: gzip, deflate, br' -H 'accept-language: en-US,en;q=0.9' -H 'upgrade-insecure-requests: 1' -H 'user-agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/64.0.3282.167 Chrome/64.0.3282.167 Safari/537.36' -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8' -H 'referer: https://bitcointalk.org/index.php?action=unreadreplies' -H 'authority: bitcointalk.org' ".$cookie." -H 'if-modified-since: Thu, 22 Mar 2018 14:20:56 GMT' --compressed > output.html

The only thing you need are the cookies of a logged in session, you'll probably need to refresh this cookie every hour, and you shouldn't make more than 30 requests/minute
Even bypasses cloudflare

[Emilyearl]

Many people have the merit statistical analysis backed up somewhere. Something like this shouldn't be allowed to just wipe away like that. It has to be preserved so that in time to come, if the merit system persists one can look back and say when merit was first introduced, it was like or like that.

[DdmrDdmr]

Many people have the merit statistical analysis backed up somewhere. Something like this shouldn't be allowed to just wipe away like that. It has to be preserved so that in time to come, if the merit system persists one can look back and say when merit was first introduced, it was like or like that.

There's alway this (took me sometime to retrieve the reference, but I had it in the back of my mind):

Mainly I do these date-limitation things just because it's always a massive pain to add pagination to things...

For the data dumps, I've been thinking that I might provide historical data (across multiple files), though.

So the current design of the forum does not seem to facilitate paging on our Merit History track.

[MountainConqueror]

In my opinion merit is needed for bounty hunters only, if you're visiting that forum for chatting only there is no difference if you're a Member or a Hero.
Anyway I predict a tons of whine in next couple of days  

[mobilazy]

I wish initial merits be gone along with history after 120 days. Now we'll see wave 2 of initial merits "distribution".

Who else got fooled by the title? 

[LoyceV]

The timestamp is in the current file as you say, but the messageID has changes from one file to the other. The first record (4097266.msg37354230) still exists and reading the post seems to have moved section. The remaining 8 are now deleted (propably after moving them first).

This is annoying I use this to add new lines to my old file:

Code:

comm -1 -3 <(sort merit.all.txt) <(sort merit.today.txt) > merit.new.txt

But, this indeed fails on the lines you've mentioned:

Code:

1526466697  1     45212.msg37354230 1192397     209286
1526466697  1     4097266.msg37354230     1192397     209286

It refers to this post, and my script outputs this:

Code:

  1650. 18 Merit received by aleix (#209286) from 3 unique users in 9 transactions

In reality, aleix has received 17 Merit.

I checked and there are 499 cases of timestamps with 2 or more associated TXs in the file, therefore going for the complete record PK.

Some of these can be explained by double clicking the Merit button, like these:

Two times 1 Merit from Syndikat to undercomander13 for this post.
Two times 4 Merit from minerbro to Piston_82 for this post.
~more~

What this entails is that changes to records older than 120 days may not be consistent in our ALT aggregate databases.
For example, as seen above, some merited TXs may change section or get deleted, but the ALT aggregate database will still point to the original message once the TX is outside of the 120 window frame.

I'm not sure yet how to fix my "all time copy" of theymos' data. I don't really mind of a few links are incorrect, but the amounts should add up exactly. I think I just have to find where the files overlap and copy it without using the comm command.

[TheBeardedBaby]

@LoyceV, can you check my profile according your "all time data" if my merit stats are correct? Trying to troubleshoot a problem here so it will be useful to know if there is something wrong with the data dumps.