LoyceV
Legendary

Activity: 4228
Merit: 23202
Thick-Skinned Gang Leader and Golden Feather 2021
|
 |
November 18, 2026, 09:17:36 PM |
|
So post #5000500 should be the point where the scraping date and the near-actual date of the posts on BitcoinTalk overlap? No, it was just a random example to show how to estimate 99.9% of the post times. My near real time scraping started at post 51904009, anything before that was scraped a year later.
|
|
|
|
|
*Ace* (OP)
|
 |
November 18, 2026, 10:48:50 PM |
|
So post #5000500 should be the point where the scraping date and the near-actual date of the posts on BitcoinTalk overlap? No, it was just a random example to show how to estimate 99.9% of the post times. My near real time scraping started at post 51904009, anything before that was scraped a year later. Okay, I understand, I'm starting to sample the message IDs. It certainly won't be 100% accurate, but at least it'll be the right day. Then I'll stop at the place where your RealTime scraping starts. It should be an acceptable compromise for Filippone too. But I'm sure he won't settle, he wants the best of the best. 
|
|
|
|
|
*Ace* (OP)
|
Ok, I did a general rearrangement of the post dates, I interpolated dates by taking the dates on posts taken from Bitcointalk and placing anchors as close as possible in the range taken as a sample. The match wasn't perfect, but it wasn't disastrous either. I'm sharing the results for posts from 2014 to 2019, but by mistake, 1.7 million posts from 2012-13 were also included. which have been anchored to 2014. They will be fixed as soon as possible The following tables were created by DeepSeek because I'm too lazy to do it by hand  BRDb — Post Date Accuracy Verification Random sample of 200 posts, DB date vs Bitcointalk real date | msg_id | DB date (interpolated) | Real date (Bitcointalk) | Error (hours) | Error (days) | | 47983705 | 2018-11-17 12:40:10 | 2018-11-17 12:57:31 | 0.3 | 0.01 | | 22219510 | 2017-09-25 23:00:31 | 2017-09-25 21:18:23 | 1.7 | 0.07 | | 22862753 | 2017-10-11 15:22:05 | 2017-10-11 14:57:56 | 0.4 | 0.02 | | 48256910 | 2018-11-27 00:56:30 | 2018-11-27 02:18:23 | 1.4 | 0.06 | | 40630155 | 2018-06-22 01:11:58 | 2018-06-22 01:16:05 | 0.1 | 0.00 | | 25620939 | 2017-12-02 17:35:48 | 2017-12-02 17:26:28 | 0.2 | 0.01 | | 17222711 | 2016-12-18 16:49:32 | 2016-12-18 15:39:09 | 1.2 | 0.05 | | 48092272 | 2018-11-21 06:54:16 | 2018-11-21 07:22:52 | 0.5 | 0.02 | | 19330707 | 2017-06-02 19:58:42 | 2017-06-02 17:54:17 | 2.1 | 0.09 | | 49684829 | 2019-02-11 03:34:19 | 2019-02-11 03:52:57 | 0.3 | 0.01 | | 32685948 | 2018-03-19 16:21:11 | 2018-03-19 16:12:40 | 0.1 | 0.01 | | 23095851 | 2017-10-16 15:36:30 | 2017-10-16 16:14:10 | 0.6 | 0.03 | | 30703330 | 2018-02-20 17:37:35 | 2018-02-20 17:34:44 | 0.0 | 0.00 | | 41345427 | 2018-07-02 08:11:45 | 2018-07-02 09:19:44 | 1.1 | 0.05 | | 45373729 | 2018-09-08 09:33:27 | 2018-09-08 09:35:38 | 0.0 | 0.00 | | 11857291 | 2015-07-12 07:55:45 | 2015-07-12 07:37:54 | 0.3 | 0.01 | | 13842328 | 2016-02-10 22:56:02 | 2016-02-10 21:41:57 | 1.2 | 0.05 | | 12352743 | 2015-09-07 13:52:15 | 2015-09-07 14:37:35 | 0.8 | 0.03 | | 49410464 | 2019-01-25 03:50:20 | 2019-01-25 04:19:11 | 0.5 | 0.02 | | 46046165 | 2018-09-23 02:49:20 | 2018-09-23 02:14:44 | 0.6 | 0.02 | | 27926616 | 2018-01-11 17:52:39 | 2018-01-11 17:45:01 | 0.1 | 0.01 | | 27893576 | 2018-01-11 07:46:30 | 2018-01-11 08:23:36 | 0.6 | 0.03 | | 29136544 | 2018-01-29 01:07:35 | 2018-01-28 23:58:00 | 1.2 | 0.05 | | 6433901 | 2014-04-28 06:22:30 | 2014-04-28 06:35:32 | 0.2 | 0.01 | | 35655059 | 2018-04-26 16:56:36 | 2018-04-26 16:52:43 | 0.1 | 0.00 | | 37169815 | 2018-05-13 18:05:32 | 2018-05-13 18:05:20 | 0.0 | 0.00 | | 38645225 | 2018-05-26 20:00:19 | 2018-05-26 19:58:41 | 0.0 | 0.00 | | 19970501 | 2017-07-06 08:24:11 | 2017-07-06 10:04:12 | 1.7 | 0.07 | | 19006585 | 2017-05-13 18:53:00 | 2017-05-13 16:54:15 | 2.0 | 0.08 | | 18577227 | 2017-04-13 20:32:40 | 2017-04-13 18:57:23 | 1.6 | 0.07 | Full sample of 200 posts available on request. Max error observed: 3.72 hours.Error Statistics (sample of 200 posts) | Metric | Hours | Days | | Minimum | 0.00 | 0.00 | | Median | 0.57 | 0.02 | | Mean | 0.76 | 0.03 | | P90 | 1.92 | 0.08 | | P99 | 2.80 | 0.12 | | Maximum | 3.72 | 0.16 |
Error distribution | Error range | Posts | Percentage | | < 1 hour | 135 | 67.5% | | 1 – 6 hours | 65 | 32.5% | | 6 – 24 hours | 0 | 0.0% | | 1 – 3 days | 0 | 0.0% | | > 3 days | 0 | 0.0% |
100% of sampled posts have an error below 4 hours. Post History Distribution by Year (after fix) | Year | Posts | | 2012 | 3 | | 2013 | 30 | | 2014 | 3,244,192 | | 2015 | 1,430,165 | | 2016 | 1,818,552 | | 2017 | 4,592,224 | | 2018 | 7,838,356 | | 2019 | 2,621,270 | | 2020 | 1,785,834 | | 2021 | 1,708,046 | | 2022 | 1,375,629 | | 2023 | 1,447,405 | | 2024 | 1,258,188 | | 2025 | 1,217,969 | | 2026 | 1,066,354 | | TOTAL | 31,398,110 | Note: 2014 includes ~1.69M posts from 2012-2013, so they were assigned the earliest available anchor date, 2014-02-07, This data will be fixed as soon as possible.
|
|
|
|
fillippone
Legendary
Online
Activity: 3080
Merit: 22005
Duelbits.com - Rewarding, beyond limits.
|
 |
November 19, 2026, 10:06:45 PM |
|
Wow, this was a very clever solution. And yes, I want the best, but I also like these very smart approaches. And, given the conditions, I think LoyceV solution is very elegant. Kudos to both of you.
|
|
|
|
LoyceV
Legendary

Activity: 4228
Merit: 23202
Thick-Skinned Gang Leader and Golden Feather 2021
|
 |
November 20, 2026, 06:43:32 AM |
|
Max error observed: 3.72 hours.[/i] Maybe checking every 1000 posts isn't enough when the forum is quiet. In my example, it took only 49 minutes to make 1000 posts. But if you notice the time between posts is - say - 4 hours, it wouldn't hurt to get one more data point by checking another post in the middle.
|
|
|
|
|
*Ace* (OP)
|
 |
November 20, 2026, 07:31:08 AM |
|
Max error observed: 3.72 hours.[/i] Maybe checking every 1000 posts isn't enough when the forum is quiet. In my example, it took only 49 minutes to make 1000 posts. But if you notice the time between posts is - say - 4 hours, it wouldn't hurt to get one more data point by checking another post in the middle. Yes, the 1000 post range worked well for recent years, but for the years 2009-2013, it had some errors lasting longer than 6 hours. I just ran the script again. But this time, with anchors every 500 posts, they should realign with a more acceptable margin of error. I'll see the results in a couple of hours.
|
|
|
|
|
*Ace* (OP)
|
 |
November 20, 2026, 10:15:50 PM |
|
BRDb — Post Date Fix: Final Results Post distribution by year (after full fix) | Year | Posts | | 2009 | 215 | | 2010 | 21,494 | | 2011 | 232,787 | | 2012 | 320,727 | | 2013 | 914,571 | | 2014 | 1,749,507 | | 2015 | 1,432,219 | | 2016 | 1,819,835 | | 2017 | 4,592,889 | | 2018 | 7,838,553 | | 2019 | 2,621,486 | | 2020 | 1,786,053 | | 2021 | 1,708,147 | | 2022 | 1,375,712 | | 2023 | 1,447,497 | | 2024 | 1,258,202 | | 2025 | 1,217,969 | | 2026 | 1,071,626 | | TOTAL | 31,400,000+ |
Accuracy verification (sample of 200 posts) | Metric | Hours | Minutes | | Minimum | 0.00 | 0 | | Median | ~0.25 | ~15 | | Typical range | 0.0 – 0.7 | 0 – 42 | | 99th percentile | < 2.0 | < 120 | | Outliers (>24h) | 0 / 200 | 0% |
Here’s what I’ve done today with the date fixes. As always, I asked DeepSeek to generate the tables for me; my little programme does a good job  The average error is around 15 minutes, which is acceptable, I’d say, but the test was carried out on a sample of 200 posts, and none of these showed errors exceeding 24 hours. I should check all the posts stored in the database, but that would take a long time. The error is acceptable, let’s say, but if you find any inconsistencies, please report them here.
|
|
|
|
LoyceV
Legendary

Activity: 4228
Merit: 23202
Thick-Skinned Gang Leader and Golden Feather 2021
|
 |
November 21, 2026, 09:53:18 AM |
|
none of these showed errors exceeding 24 hours. If the maximum time between some of the data points is more than 24 hours, you could just sample another one in between 
|
|
|
|
|