Bitcoin Forum
November 22, 2026, 11:17:01 PM *
News: Latest Bitcoin Core release: 32.1 [Torrent]
 
   Home   Help Search Login Register More  
Pages: « 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 [39]
  Print  
Author Topic: [TOOL] ⭐Bitcointalk BRDb score  (Read 12524 times)
LoyceV
Legendary
*
Offline

Activity: 4228
Merit: 23204


Thick-Skinned Gang Leader and Golden Feather 2021


View Profile WWW
November 18, 2026, 09:17:36 PM
 #761

So post #5000500 should be the point where the scraping date and the near-actual date of the posts on BitcoinTalk overlap?
No, it was just a random example to show how to estimate 99.9% of the post times.
My near real time scraping started at post 51904009, anything before that was scraped a year later.

¡uʍop ǝpᴉsdn pɐǝɥ ɹnoʎ ɥʇᴉʍ ʎuunɟ ʞool no⅄
[MUSIC] The Ballad of LoyceV
*Ace* (OP)
Hero Member
*****
Offline

Activity: 602
Merit: 1227


brdb.cc


View Profile
November 18, 2026, 10:48:50 PM
 #762

So post #5000500 should be the point where the scraping date and the near-actual date of the posts on BitcoinTalk overlap?
No, it was just a random example to show how to estimate 99.9% of the post times.
My near real time scraping started at post 51904009, anything before that was scraped a year later.

Okay, I understand, I'm starting to sample the message IDs. It certainly won't be 100% accurate, but at least it'll be the right day. Then I'll stop at the place where your RealTime scraping starts. It should be an acceptable compromise for Filippone too.  Cool
But I'm sure he won't settle, he wants the best of the best.  Grin

*Ace* (OP)
Hero Member
*****
Offline

Activity: 602
Merit: 1227


brdb.cc


View Profile
November 19, 2026, 08:16:11 AM
Merited by LoyceV (4), fillippone (3), dkbit98 (1)
 #763

Ok, I did a general rearrangement of the post dates, I interpolated dates by taking the dates on posts taken from Bitcointalk and placing anchors as close as possible in the range taken as a sample. The match wasn't perfect, but it wasn't disastrous either. I'm sharing the results for posts from 2014 to 2019, but by mistake, 1.7 million posts from 2012-13 were also included. which have been anchored to 2014. They will be fixed as soon as possible

The following tables were created by DeepSeek because I'm too lazy to do it by hand  Grin


BRDb — Post Date Accuracy Verification
Random sample of 200 posts, DB date vs Bitcointalk real date

msg_idDB date (interpolated)Real date (Bitcointalk)Error (hours)Error (days)
479837052018-11-17 12:40:102018-11-17 12:57:310.30.01
222195102017-09-25 23:00:312017-09-25 21:18:231.70.07
228627532017-10-11 15:22:052017-10-11 14:57:560.40.02
482569102018-11-27 00:56:302018-11-27 02:18:231.40.06
406301552018-06-22 01:11:582018-06-22 01:16:050.10.00
256209392017-12-02 17:35:482017-12-02 17:26:280.20.01
172227112016-12-18 16:49:322016-12-18 15:39:091.20.05
480922722018-11-21 06:54:162018-11-21 07:22:520.50.02
193307072017-06-02 19:58:422017-06-02 17:54:172.10.09
496848292019-02-11 03:34:192019-02-11 03:52:570.30.01
326859482018-03-19 16:21:112018-03-19 16:12:400.10.01
230958512017-10-16 15:36:302017-10-16 16:14:100.60.03
307033302018-02-20 17:37:352018-02-20 17:34:440.00.00
413454272018-07-02 08:11:452018-07-02 09:19:441.10.05
453737292018-09-08 09:33:272018-09-08 09:35:380.00.00
118572912015-07-12 07:55:452015-07-12 07:37:540.30.01
138423282016-02-10 22:56:022016-02-10 21:41:571.20.05
123527432015-09-07 13:52:152015-09-07 14:37:350.80.03
494104642019-01-25 03:50:202019-01-25 04:19:110.50.02
460461652018-09-23 02:49:202018-09-23 02:14:440.60.02
279266162018-01-11 17:52:392018-01-11 17:45:010.10.01
278935762018-01-11 07:46:302018-01-11 08:23:360.60.03
291365442018-01-29 01:07:352018-01-28 23:58:001.20.05
64339012014-04-28 06:22:302014-04-28 06:35:320.20.01
356550592018-04-26 16:56:362018-04-26 16:52:430.10.00
371698152018-05-13 18:05:322018-05-13 18:05:200.00.00
386452252018-05-26 20:00:192018-05-26 19:58:410.00.00
199705012017-07-06 08:24:112017-07-06 10:04:121.70.07
190065852017-05-13 18:53:002017-05-13 16:54:152.00.08
185772272017-04-13 20:32:402017-04-13 18:57:231.60.07

Full sample of 200 posts available on request. Max error observed: 3.72 hours.

Error Statistics (sample of 200 posts)

MetricHoursDays
Minimum0.000.00
Median0.570.02
Mean0.760.03
P901.920.08
P992.800.12
Maximum3.720.16

Error distribution

Error rangePostsPercentage
< 1 hour13567.5%
1 – 6 hours6532.5%
6 – 24 hours00.0%
1 – 3 days00.0%
> 3 days00.0%

100% of sampled posts have an error below 4 hours.

Post History Distribution by Year (after fix)

YearPosts
20123
201330
20143,244,192
20151,430,165
20161,818,552
20174,592,224
20187,838,356
20192,621,270
20201,785,834
20211,708,046
20221,375,629
20231,447,405
20241,258,188
20251,217,969
20261,066,354
TOTAL31,398,110

Note: 2014 includes ~1.69M posts from 2012-2013, so they were assigned the earliest available anchor date, 2014-02-07, This data will be fixed as soon as possible.

fillippone
Legendary
*
Online Online

Activity: 3080
Merit: 22008


Duelbits.com - Rewarding, beyond limits.


View Profile WWW
November 19, 2026, 10:06:45 PM
Merited by *Ace* (3)
 #764

Wow, this was a very clever solution.
And yes, I want the best, but I also like these very smart approaches. And, given the conditions, I think LoyceV solution is very elegant.
Kudos to both of you.

LoyceV
Legendary
*
Offline

Activity: 4228
Merit: 23204


Thick-Skinned Gang Leader and Golden Feather 2021


View Profile WWW
November 20, 2026, 06:43:32 AM
Merited by *Ace* (3)
 #765

Max error observed: 3.72 hours.[/i]
Maybe checking every 1000 posts isn't enough when the forum is quiet. In my example, it took only 49 minutes to make 1000 posts. But if you notice the time between posts is - say - 4 hours, it wouldn't hurt to get one more data point by checking another post in the middle.

¡uʍop ǝpᴉsdn pɐǝɥ ɹnoʎ ɥʇᴉʍ ʎuunɟ ʞool no⅄
[MUSIC] The Ballad of LoyceV
*Ace* (OP)
Hero Member
*****
Offline

Activity: 602
Merit: 1227


brdb.cc


View Profile
November 20, 2026, 07:31:08 AM
 #766

Max error observed: 3.72 hours.[/i]
Maybe checking every 1000 posts isn't enough when the forum is quiet. In my example, it took only 49 minutes to make 1000 posts. But if you notice the time between posts is - say - 4 hours, it wouldn't hurt to get one more data point by checking another post in the middle.

Yes, the 1000 post range worked well for recent years, but for the years 2009-2013, it had some errors lasting longer than 6 hours. I just ran the script again. But this time, with anchors every 500 posts, they should realign with a more acceptable margin of error. I'll see the results in a couple of hours.

*Ace* (OP)
Hero Member
*****
Offline

Activity: 602
Merit: 1227


brdb.cc


View Profile
November 20, 2026, 10:15:50 PM
 #767

BRDb — Post Date Fix: Final Results

Post distribution by year (after full fix)

YearPosts
2009215
201021,494
2011232,787
2012320,727
2013914,571
20141,749,507
20151,432,219
20161,819,835
20174,592,889
20187,838,553
20192,621,486
20201,786,053
20211,708,147
20221,375,712
20231,447,497
20241,258,202
20251,217,969
20261,071,626
TOTAL31,400,000+

Accuracy verification (sample of 200 posts)

MetricHoursMinutes
Minimum0.000
Median~0.25~15
Typical range0.0 – 0.70 – 42
99th percentile< 2.0< 120
Outliers (>24h)0 / 2000%

Here’s what I’ve done today with the date fixes. As always, I asked DeepSeek to generate the tables for me; my little programme does a good job  Grin
The average error is around 15 minutes, which is acceptable, I’d say, but the test was carried out on a sample of 200 posts, and none of these showed errors exceeding 24 hours.
I should check all the posts stored in the database, but that would take a long time.
The error is acceptable, let’s say, but if you find any inconsistencies, please report them here.

LoyceV
Legendary
*
Offline

Activity: 4228
Merit: 23204


Thick-Skinned Gang Leader and Golden Feather 2021


View Profile WWW
November 21, 2026, 09:53:18 AM
Merited by *Ace* (1)
 #768

none of these showed errors exceeding 24 hours.
If the maximum time between some of the data points is more than 24 hours, you could just sample another one in between Smiley

¡uʍop ǝpᴉsdn pɐǝɥ ɹnoʎ ɥʇᴉʍ ʎuunɟ ʞool no⅄
[MUSIC] The Ballad of LoyceV
*Ace* (OP)
Hero Member
*****
Offline

Activity: 602
Merit: 1227


brdb.cc


View Profile
Today at 10:24:05 PM
 #769

none of these showed errors exceeding 24 hours.
If the maximum time between some of the data points is more than 24 hours, you could just sample another one in between Smiley

I hope there aren't any. I honestly haven't checked because I'd have to check practically every post one by one! I'd have done better to fix the dates directly by scraping them one by one. But if they were to appear, I would prefer to pin that single post directly rather than placing other anchors.

Pages: « 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 [39]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!