LoyceV's Merit data analysis (full data since Jan. 24, 2018; not just 120 days)

[GazetaBitcoin]

Hey Loyce, I noticed something for a while... the post titles are not displayed anymore when accessing these 2 links from your website: Full Merit transaction history (including graphs) for any user and Complete list of all Merit transactions including post titles.

What I am seeing is this:





As you can see in the second image, some post titles are displayed though, but only very few of them. 99% of post titles are not displayed... And I remember that a while ago, all post titles were displayed, so something changed recently. Do you know why this is happening?

[LoyceV]

Do you know why this is happening?

No
It's been like this ever since I moved to another server (about a year ago). Some older titles are working, so I guess it's a "weird character" somewhere messing things up. It's annoyed me too, but searching the problem was more annoying, so I gave up. That was a year ago

[GazetaBitcoin]

Do you know why this is happening?

No

This is sad indeed...

It's been like this ever since I moved to another server (about a year ago). Some older titles are working, so I guess it's a "weird character" somewhere messing things up.

But... there are so many post titles not shown... perhaps it's more than one weird character...? I mean, looking for one weird character may be like searching for a needle in a haystack... but if the issue affects more characters, maybe it would be easier to locate it...?

It's annoyed me too, but searching the problem was more annoying, so I gave up. That was a year ago

Those post titles were really useful in searching some old posts... at least for me... I wish I could help somehow in fixing this issue, because for me it was very useful to see all those post titles... Maybe if we do some collective brain storming some good ideas can come...?

[joker_josue]

Those post titles were really useful in searching some old posts... at least for me... I wish I could help somehow in fixing this issue, because for me it was very useful to see all those post titles... Maybe if we do some collective brain storming some good ideas can come...?

Give me a few hours, maybe I'll figure out the problem... 

[LoyceV]

But... there are so many post titles not shown... perhaps it's more than one weird character...? I mean, looking for one weird character may be like searching for a needle in a haystack... but if the issue affects more characters, maybe it would be easier to locate it...?

It's spaghetti code I created this years ago, and it got bigger and bigger. I'll have a look again when I have some time. If only I'd use databases, this would be so much easier....

[joker_josue]

But... there are so many post titles not shown... perhaps it's more than one weird character...? I mean, looking for one weird character may be like searching for a needle in a haystack... but if the issue affects more characters, maybe it would be easier to locate it...?

It's spaghetti code I created this years ago, and it got bigger and bigger. I'll have a look again when I have some time. If only I'd use databases, this would be so much easier....

From what I can analyze, it includes the topic link, just leaves the title blank.


If it is not in the database, it may be difficult to repurchase this data. It all depends on how you store the information.

[LoyceV]

But... there are so many post titles not shown... perhaps it's more than one weird character...? I mean, looking for one weird character may be like searching for a needle in a haystack... but if the issue affects more characters, maybe it would be easier to locate it...?

It turns out my list of Merited post titles was last updated September 21 last year, around the time I moved servers. My guess for one "weird" character was because that sometimes gets something stuck.
I've waited long enough, so I'll just have to figure it out now. It's especially annoying since I have quite an inefficient system, and each tiny test takes a few hours to process the data. Step 1 is running now, I'll continue when it's ready.
It doesn't help that I created my spaghetti code years ago.

Those post titles were really useful in searching some old posts... at least for me...

I use it to find back good posts, by searching for posts I Merited in the past.

From what I can analyze, it includes the topic link, just leaves the title blank.

Correct. My title array is the problem.

If it is not in the database

I don't use a database.

it may be difficult to repurchase this data.

It's free data

It all depends on how you store the information.

It's one file:

Code:

0:<DEL>deleted post</DEL>
10:<DEL>deleted post</DEL>
28:Welcome to the new Bitcoin forum!
29:Repost: Bitcoin Maturation
30:Repost: Request: Make this anonymous?
31:Re: Repost: Bitcoin Maturation
33:Repost: How anonymous are bitcoins?
36:Repost: Linux/UNIX compile
40:A few suggestions
43:Questions about Bitcoin
46:Re: Questions about Bitcoin
48:Re: Questions about Bitcoin
49:Re: Questions about Bitcoin
67:Re: A few suggestions
73:Bitcoin 0.2 released!
82:Re: New Exchange Service: "BTC 2 PSC"
100:New exchange (Bitcoin Market)
104:TOR and I2P
111:Re: New Exchange Service: "BTC 2 PSC"
118:A newb's test - anyone want to buy a picture for $1?
121:Re: A newb's test - anyone want to buy a picture for $1?
137:Blocks never stop generating?
139:Re: Blocks never stop generating?
142:Re: Blocks never stop generating?
144:We accept Bitcoins [moved to bitcoin.it/wiki/Trade]
160:Re: Blocks never stop generating?
172:Re: Payment server
188:Re: Welcome to the new Bitcoin forum!
200:Re: Welcome to the new Bitcoin forum!
223:Re: TOR and I2P
227:Re: Questions about Addresses
238:Bitcoin Currency Symbol &#3647;
239:Re: Bitcoin Currency Symbol &#3647;
240:Re: Bitcoin Currency Symbol &#3647;
243:Re: Bitcoin Currency Symbol &#3647;
249:Proof-of-work difficulty increasing
.......
.......
.......
62880800:Re: How to reduce the price of goods 
62880928:História das criptomoedas: O dedo gordo que derrubou o Bitcoin de 65k para 8k
62880944:Re: Gewinnspiel: Flashmob für Willis Blockparty
62880948:Re: &#1055;&#1086;&#1089;&#1083;&#1077;&#1076;&#1085;&#1080;&#1081; &#1074;&#1072;&#1075;&#1086;&#1085; &#1085;&#1072; &#1089;&#1077;&#1074;&#1077;&#1088;
62880971:Re: How Bitcoin can provide Subscription Auto Renewal solution
62880978:Re: Bitcoin kwoledge is not complete without bitcoin investment
62880991:Re: BITCOIN PUMP!
62881039:Re: [TELEGRAM] Bot de Notificações e Tópicos do Fórum
62881053:Re: Buy the DIP, and HODL!
62881101:Vermögen auslagern Familienstiftung vs. Bitcoin
62881132:Re: AGTI-&#1080;&#1085;&#1074;&#1077;&#1089;&#1090;&#1080;&#1094;&#1080;&#1086;&#1085;&#1085;&#1099;&#1081; &#1090;&#1086;&#1082;&#1077;&#1085; &#1076;&#1083;&#1103; &#1057;/&#1061;
62881151:Re: Concursuri/tombole pe bitcointalk (sau pe aprope)
62881424:Re: [Guide] Solo mine testnet bitcoins with cgminer, Bitcoin Core, and a Compac F
62881470:Re: It ain't meant to be easy
62881481:Re: [H] BTC [W] 200 EUR PayPal 1:1
62881545:Re: Ruger New Model Single Six .22WMRF
62881601:Re: CoinEx - &#1087;&#1083;&#1072;&#1090;&#1092;&#1086;&#1088;&#1084;&#1072; &#1087;&#1086; &#1086;&#1073;&#1084;&#1077;&#1085;&#1091; &#1094;&#1080;&#1092;&#1088;&#1086;&#1074;&#1099;&#1084;&#1080; &#1074;&#1072;&#1083;&#1102;&#1090;&#1072;&#1084;&#1080;
62881613:Re: [Competition]: Bounty Manager Run - Prize: 2mBTC
62881668:Re: Wall Observer BTC/USD - Bitcoin price movement tracking &amp; discussion
62881695:Re: &#128640;-Sportsbet.io-&#128640; Official Partner of Newcastle United Football Club
62881711:Re: Wall Observer BTC/USD - Bitcoin price movement tracking &amp; discussion
62881723:Re: [ANN] CoinPlaza.it: non-custodial instant crypto exchange - Bitcoin &#9889; Enabled
62881766:Re: &#127942;Rollbit.com | Champions League Pool | $11,000+ in Prizes &#9917;&#65039; Discussion Thread
62881857:Re: Is FullRBF allowing double spend?
62881893:Re: Wall Observer BTC/USD - Bitcoin price movement tracking &amp; discussion
62881976:Re: &#127942;Rollbit.com | Champions League Pool | $11,000+ in Prizes &#9917;&#65039; Discussion Thread
62882024:Re: [Competition]: Bounty Manager Run - Prize: 3mBTC
62882096:Re: Report Malware and Suspicious Links here so Mods can take Action !
62882278:Re: NERDMINER: Bitcoin lottery miners
62882296:[TRADUÇÃO] Len Sassaman e Satoshi: Uma história cypherpunk
62882325:Re: Preparing for opportunities 
62882407:Re: Betting significant amount
62882452:Re: Buy the DIP, and HODL!
62882458:Re: Betting significant amount
62882537:Re: It ain't meant to be easy
62882604:Re: Overview of Bitcointalk Signature-Ad Campaigns [Last update: 17-Sep-23]
62882616:Re: Buy the DIP, and HODL!
62882633:Re: How to reduce the price of goods 
62882784:Re: [HOLD] vbetFTN.com | Leading Online Gambling Operator Signature Campaign | Sr. +
62882820:Re: BITCOIN PUMP!
62882830:Re: Der Aktuelle Kursverlauf
62882905:Re: Wall Observer BTC/USD - Bitcoin price movement tracking &amp; discussion
62882934:Re: Betting significant amount
62883050:Re: Wall Observer BTC/USD - Bitcoin price movement tracking &amp; discussion
62883102:Re: Falha Winrar permite que hackers roubem criptomoedas
62883159:Re: Paxos recovers its $500,000 'fat finger' Bitcoin transaction fee
62883191:Re: 2023 NBA Season
62883234:Re: [SORTEIO2] 0.00062 BTC por um novo tópico
62883256:Re: 2023 NBA Season
62883307:Re: Success or Fail? - Talking about Gambling as a major income 
62883373:Re: &#127944;&#127944; The American Football Discussion Thread &#127944;&#127944;
62883380:Re: &#127944;&#127944; The American Football Discussion Thread &#127944;&#127944;

For years, it's been on my mind to "redo" everything, utilizing a database instead of static files. But I never learned how to do that, and the longer I wait, the bigger my "lock-in" gets.

[joker_josue]

From what I can analyze, it includes the topic link, just leaves the title blank.

Correct. My title array is the problem.

If it is not in the database

I don't use a database.

it may be difficult to repurchase this data.

It's free data

It all depends on how you store the information.

It's one file:


For years, it's been on my mind to "redo" everything, utilizing a database instead of static files. But I never learned how to do that, and the longer I wait, the bigger my "lock-in" gets.

So, if you have a file that has all the topic titles, and has the link to each topic, it won't be that complicated.
You just have to create a way to combine the two information.

But, of course, the problem is in the way the html file is created. How is done?

[LoyceV]

So, if you have a file that has all the topic titles, and has the link to each topic, it won't be that complicated.
You just have to create a way to combine the two information.

That's what you'd think  I can't even find how one file gets copied, and yet, it gets copied. Like I said: spaghetti code
My test run ended in "<DEL>deleted post</DEL>" on every post I scrape, so there's more going wrong.
Update: I can blame PowerGlove for the "deleted post" message. But that still doesn't explain my other problems.

[PowerGlove]

Update: I can blame PowerGlove for the "deleted post" message.

Don't blame me too hard...

LoyceV and TryNinja would probably appreciate some notice before you merge it, so that they can tweak their scrapers if necessary.

I have so many (old) scrapers, I can't tell if this is going to break something. I'll just see when it happens.

Also, if it was the Unicode arrow (style B) that broke things, then:

I think, if it were up to me, I might just go with C: it's nice and simple, with plenty of contrast, so it's really easy to pick out the "marked" post, and it still looks SMFish to my eyes.

(In all seriousness, I'm sorry to have created more work for you: if you estimate that I can help out in any way, feel free to PM me.)

[LoyceV]

Don't blame me too hard...

Don't worry about it. Getting the part I need from the forum's HTML is always tricky. And sometimes I find an exception on some local board months later. This part is working again. Of course, I should have taken the thread title instead of the post title from the start, but those titles I have on a different server.

I still don't know what happens to that one file. I'm running another test. If all goes well, it's going to download the last year of Merited post titles.

[joker_josue]

So, if you have a file that has all the topic titles, and has the link to each topic, it won't be that complicated.
You just have to create a way to combine the two information.

That's what you'd think  I can't even find how one file gets copied, and yet, it gets copied. Like I said: spaghetti code
My test run ended in "<DEL>deleted post</DEL>" on every post I scrape, so there's more going wrong.
Update: I can blame PowerGlove for the "deleted post" message. But that still doesn't explain my other problems.

This is very frustrating when it happens.

The HTML file has to be created at some point, it has to reach that point. I know it's easier said than said. But, I just want to try to help.

[LoyceV]

This is very frustrating when it happens.

The HTML file has to be created at some point, it has to reach that point.

Searching for the name or command doesn't show the file, and yet, it gets copied. I'll have to go through all code I guess, but don't have time for that now.

The update is still running. If it works when I manually start it, but fails from a cronjob, it could have to do with different environment variables. I've had that problem in the past too.
I hope it'll finish catching up on downloading post titles by tomorrow.

[LoyceV]

Hey Loyce, I noticed something for a while... the post titles are not displayed anymore when accessing these 2 links from your website: Full Merit transaction history (including graphs) for any user and Complete list of all Merit transactions including post titles.

I fixed the title scraping and updated everything, but that wasn't the reason the other titles didn't show up. I manually ran my script, and everything works! See:

Merit sent by GazetaBitcoin (Trust list) from January 24, 2018 until June 21, 2024 (source)


  1341. Thu Jun 20 10:51:58 2024: 3 (3320) to criptoevangelista (Trust list) (history) for Re: [LOG] The ranked up members - Congratulations!
  1340. Thu Jun 20 09:38:24 2024: 5 (3317) to dansus021 (Trust list) (history) for Bantu Ukraina dengan Bitcoin!
  1339. Wed Jun 19 12:44:54 2024: 1 (3312) to cygan (Trust list) (history) for [Poradnik Linux] Sygnatura PGP - Szyfrowanie/deszyfrowanie wiadomo&#347;ci
  1338. Wed Jun 19 07:36:27 2024: 1 (3311) to iwantmyhomepaidwithbtc2 (Trust list) (history) for [FREE RAFFLE] - Custom eXch Cryptosteel Capsule (#13)!
  1337. Tue Jun 18 10:26:58 2024: 1 (3310) to iwantmyhomepaidwithbtc2 (Trust list) (history) for [FREE RAFFLE] Blackjack.fun &#128420; Prize $50 in Bitcoins | Round #5
  1336. Tue Jun 18 10:26:34 2024: 1 (3309) to iwantmyhomepaidwithbtc2 (Trust list) (history) for [FREE RAFFLE] BC.Game | Bonus code for $50 | Round #4
  1335. Sun Jun 16 15:56:23 2024: 4 (3308) to NeuroticFish (Trust list) (history) for Re: Concursuri/tombole pe bitcointalk (sau pe aprope)
  1334. Fri Jun 14 16:46:21 2024: 4 (3304) to MeCsc (history) for Re: Info-thread: Translation of Useful English topics to Naija languages, pidgin etc
  1333. Fri Jun 14 15:57:02 2024: 1 (3300) to NeuroticFish (Trust list) (history) for Re: Concursuri/tombole pe bitcointalk (sau pe aprope)
  1332. Wed Jun 12 09:25:35 2024: 1 (3299) to r_victory (Trust list) (history) for [Tutorial] Assinatura PGP - Criptografar/Descriptografar mensagem (Linux)
  1331. Mon Jun 10 17:20:08 2024: 1 (3298) to cygan (Trust list) (history) for [Linux Anleitung] PGP Signature - Nachricht verschlüsseln/entschlüsseln

So somehow it fails from a cronjob, which makes me think it's caused by environment variables. I tried to change that, but now my cronjob is spitting out other errors. I'll try again.

Do you know why this is happening?

Still no