[project] api.loyce.club: crowdsourcing Bitcointalk data

[LoyceV]

In [BRAINSTORM] Any requests for custom extensions/user scripts for BitcoinTalk?, the suggestion for a Merit data API came up. User CryptoNeed made a PHP script, and to stop going off-topic there, I've created this topic.

api.loyce.club
I wouldn't know how to build an API, and have no experience with MySQL or PHP to store data. But I've collected a lot of data on Merit, Trust and bans (Nuke or Autoban) and wouldn't mind putting that to good use.

Join?
If anyone is interested in crowdsourcing additions, I'll be happy to host them. I would like it if any code contributions can be as free as possible, allowing anyone to build upon it and improve it. I won't be the best judge of all code, so I appreciate input on what's safe, efficient and worth using.

No spam
All my threads are now self-moderated to stop signature spam. I will remove all irrelevant posts. If you quote the entire OP, your entry will be deleted.

[LoyceV]

Currently, I use http://api.loyce.club/crowdsourcing/ for testing. CryptoNeed's code is there, but I'm not sure how to use it.

I would expect this to work, but it doesn't:
http://api.loyce.club/crowdsourcing/meritjson.php?donor=459836

[CryptoNeed]

Currently, I use http://api.loyce.club/crowdsourcing/ for testing. CryptoNeed's code is there, but I'm not sure how to use it.

I would expect this to work, but it doesn't:
http://api.loyce.club/crowdsourcing/meritjson.php?donor=459836

Not quite sure as it runs on my local servers without problems.
I noticed that the URL actually is http://api.loyce.club/crowdsourcing/merit.all.txt. Try to change the $url variable in the script. ($url = "merit.all.txt"; should be enough because script and source data is stored in the same directory).

I also want to credit Initscri. He is going to work on a combined version of this script.
Maybe we should add some error handling aswell. 

[Initscri]

Currently, I use http://api.loyce.club/crowdsourcing/ for testing. CryptoNeed's code is there, but I'm not sure how to use it.

I would expect this to work, but it doesn't:
http://api.loyce.club/crowdsourcing/meritjson.php?donor=459836

You have an error_log, feel free to shoot me a PM with the most recent error if you wish. I can take a look.

Try using the modified code I suggested in the other thread. I'm assuming you're running that as a linux server, so the code modification may be required.

[TryNinja]

^ in that case, he will also need to log every merit transaction in the mysql database. It’s possible to make an API without an sql server.

Idk if Loyce knows mysql. But he already doesn’t know PHP, so that would just complicate his life.

I’ll give a shot on making the API.

[CryptoNeed]

^ in that case, he will also need to log every merit transaction in the mysql database. It’s possible to make an API without an sql server.
Idk if Loyce knows mysql. But he already doesn’t know PHP, so that would just complicate his life.
I’ll give a shot on making the API.

As far as I know he doesnt know SQL. I think he is putting all the data together file by hand.
It seems that Loyce did not have uploaded the adjusted the code as under the given URL it is responsing with the old version.

[Initscri]

Have you install or/and configure MySQL? It's definitely required to build API services.

If we need simplicity, we could do it with only a table (which shown below).

Code:

CREATE TABLE MERIT (
    ID INT(11) PRIMARY KEY NOT NULL,
    QTY INT(2) NOT NULL,
    MSG VARCHAR(25) NOT NULL,
    SENDER INT(8) NOT NULL,
    RECEIVER INT(8) NOT NULL
);

Short demo (with MariaDB, but should be fully compatible with MySQL) :

Code:

MariaDB [test]> CREATE TABLE MERIT (
    ->     ID INT(11) PRIMARY KEY NOT NULL,
    ->     QTY INT(2) NOT NULL,
    ->     MSG VARCHAR(25) NOT NULL,
    ->     SENDER INT(8) NOT NULL,
    ->     RECEIVER INT(8) NOT NULL
    -> );
Query OK, 0 rows affected (0.029 sec)

MariaDB [test]> DESC MERIT;
+----------+-------------+------+-----+---------+-------+
| Field    | Type        | Null | Key | Default | Extra |
+----------+-------------+------+-----+---------+-------+
| ID       | int(11)     | NO   | PRI | NULL    |       |
| QTY      | int(2)      | NO   |     | NULL    |       |
| MSG      | varchar(25) | NO   |     | NULL    |       |
| SENDER   | int(8)      | NO   |     | NULL    |       |
| RECEIVER | int(8)      | NO   |     | NULL    |       |
+----------+-------------+------+-----+---------+-------+
5 rows in set (0.001 sec)

MariaDB [test]> SELECT * FROM MERIT;
Empty set (0.000 sec)

MariaDB [test]> INSERT INTO MERIT VALUES (1561085310, 2, "178336.msg51543068", 452769, 239406);
Query OK, 1 row affected (0.008 sec)

MariaDB [test]> SELECT * FROM MERIT;
+------------+-----+--------------------+--------+----------+
| ID         | QTY | MSG                | SENDER | RECEIVER |
+------------+-----+--------------------+--------+----------+
| 1561085310 |   2 | 178336.msg51543068 | 452769 |   239406 |
+------------+-----+--------------------+--------+----------+
1 row in set (0.000 sec)

P.S. I learn SQL a bit, but i've no idea about MySQL security or performance.

It's not required to build the API service + the script CryptoNeed built simply just runs off the system Loyce has already built.

I think he is putting all the data together file by hand.

I'm assuming he uses a script to build it.

[CryptoNeed]

I'm assuming he uses a script to build it.

Yes for sure. At least I hope so. 
By hand I meant sth like fetching data from the website via script.

I pm'd him an "updated" version to have forms at the html one, so that the loaded script doesn't appear totally empty.
How is your progress on the combined script with RegEx, Initscri?

Code:

<html>
	<head>
		<title>Merit LookUp</title>
	</head>
	<body>
		<form action="merithtml.php" method="get">
			Donor: <input type="text" name="donor" default="0"><input type="submit"><br />
		</form>
		<form action="merithtml.php" method="get">
			Recipient: <input type="text" name="recipient"><input type="submit"><br />	
		</form>
<?php
//GET Request by: merithtml.php?donor=USERID

$url = "merit.all.txt"; 
$file = file($url);
$file = str_replace("	", ",",$file);
$file = str_replace("\r\n", "",$file);
// put the array together
foreach ($file as $line => $value) {
	$value = explode(",", $value);
    $bigArray[] = array("Donor" => $value[3], "Recipient" => $value[4], "Merits" => $value[1], "Msg" => $value[2], "DTG" => $value[0]);
}

function searchForDonor($uid, $array) {
	echo "<table>";
	foreach ($array as $key => $val) {
		if($val["Donor"] === $uid) {
			//print_r($array[$key]);
			echo "<tr>
					<td>".date("d.m.Y H:i:s", $val["DTG"])."</td>
					<td><a href='https://bitcointalk.org/index.php?topic=".$val["Msg"]."#". strstr($val["Msg"], "msg")."'>".$val["Msg"]."</a></td>
					<td><a href='https://bitcointalk.org/index.php?action=profile;u=".$val["Donor"]."'>".$val["Donor"]."</a></td>
					<td>-(".$val["Merits"].")-></td>
					<td><a href='https://bitcointalk.org/index.php?action=profile;u=".$val["Recipient"]."'>".$val["Recipient"]."</a></td>
				</tr>";
		} 
	}
	echo "</table>";
	return null;
}

function searchForRecipient($uid, $array) {
	echo "<table>";
	foreach ($array as $key => $val) {
		if($val["Recipient"] === $uid) {
			//print_r($array[$key]);
			echo "<tr>
					<td>".date("d.m.Y H:i:s", $val["DTG"])."</td>
					<td><a href='https://bitcointalk.org/index.php?topic=".$val["Msg"]."#". strstr($val["Msg"], "msg")."'>".$val["Msg"]."</a></td>
					<td><a href='https://bitcointalk.org/index.php?action=profile;u=".$val["Recipient"]."'>".$val["Recipient"]."</a></td>
					<td><-(".$val["Merits"].")-</td>
					<td><a href='https://bitcointalk.org/index.php?action=profile;u=".$val["Donor"]."'>".$val["Donor"]."</a></td>
				</tr>";
		} 
	}
	echo "</table>";
	return null;
}

if(isset($_GET["donor"])) {
	searchForDonor($_GET["donor"], $bigArray);
} 

if(isset($_GET["recipient"])) {
	searchForRecipient($_GET["recipient"], $bigArray);

}

?>

	</body>
</html>

Also, I decided to regain some of my PHP skills. Have forgotten how much fun it is. 

[Initscri]

I'm assuming he uses a script to build it.

Yes for sure. At least I hope so. 
By hand I meant sth like fetching data from the website via script.

I pm'd him an "updated" version to have forms at the html one, so that the loaded script doesn't appear totally empty.
How is your progress on the combined script with RegEx, Initscri?

Code:

<html>
	<head>
		<title>Merit LookUp</title>
	</head>
	<body>
		<form action="merithtml.php" method="get">
			Donor: <input type="text" name="donor" default="0"><input type="submit"><br />
		</form>
		<form action="merithtml.php" method="get">
			Recipient: <input type="text" name="recipient"><input type="submit"><br />	
		</form>
<?php
//GET Request by: merithtml.php?donor=USERID

$url = "merit.all.txt"; 
$file = file($url);
$file = str_replace("	", ",",$file);
$file = str_replace("\r\n", "",$file);
// put the array together
foreach ($file as $line => $value) {
	$value = explode(",", $value);
    $bigArray[] = array("Donor" => $value[3], "Recipient" => $value[4], "Merits" => $value[1], "Msg" => $value[2], "DTG" => $value[0]);
}

function searchForDonor($uid, $array) {
	echo "<table>";
	foreach ($array as $key => $val) {
		if($val["Donor"] === $uid) {
			//print_r($array[$key]);
			echo "<tr>
					<td>".date("d.m.Y H:i:s", $val["DTG"])."</td>
					<td><a href='https://bitcointalk.org/index.php?topic=".$val["Msg"]."#". strstr($val["Msg"], "msg")."'>".$val["Msg"]."</a></td>
					<td><a href='https://bitcointalk.org/index.php?action=profile;u=".$val["Donor"]."'>".$val["Donor"]."</a></td>
					<td>-(".$val["Merits"].")-></td>
					<td><a href='https://bitcointalk.org/index.php?action=profile;u=".$val["Recipient"]."'>".$val["Recipient"]."</a></td>
				</tr>";
		} 
	}
	echo "</table>";
	return null;
}

function searchForRecipient($uid, $array) {
	echo "<table>";
	foreach ($array as $key => $val) {
		if($val["Recipient"] === $uid) {
			//print_r($array[$key]);
			echo "<tr>
					<td>".date("d.m.Y H:i:s", $val["DTG"])."</td>
					<td><a href='https://bitcointalk.org/index.php?topic=".$val["Msg"]."#". strstr($val["Msg"], "msg")."'>".$val["Msg"]."</a></td>
					<td><a href='https://bitcointalk.org/index.php?action=profile;u=".$val["Recipient"]."'>".$val["Recipient"]."</a></td>
					<td><-(".$val["Merits"].")-</td>
					<td><a href='https://bitcointalk.org/index.php?action=profile;u=".$val["Donor"]."'>".$val["Donor"]."</a></td>
				</tr>";
		} 
	}
	echo "</table>";
	return null;
}

if(isset($_GET["donor"])) {
	searchForDonor($_GET["donor"], $bigArray);
} 

if(isset($_GET["recipient"])) {
	searchForRecipient($_GET["recipient"], $bigArray);

}

?>

	</body>
</html>

Also, I decided to regain some of my PHP skills. Have forgotten how much fun it is. 

Haven't had a chance to make any modifications as of yet. Been swamped.

Is that the new code you're working off of?

[CryptoNeed]

I'm assuming he uses a script to build it.

Yes for sure. At least I hope so.  
By hand I meant sth like fetching data from the website via script.

I pm'd him an "updated" version to have forms at the html one, so that the loaded script doesn't appear totally empty.
How is your progress on the combined script with RegEx, Initscri?

Code:

<html>
	<head>
		<title>Merit LookUp</title>
	</head>
	<body>
		<form action="merithtml.php" method="get">
			Donor: <input type="text" name="donor" default="0"><input type="submit"><br />
		</form>
		<form action="merithtml.php" method="get">
			Recipient: <input type="text" name="recipient"><input type="submit"><br />	
		</form>
<?php
//GET Request by: merithtml.php?donor=USERID

$url = "merit.all.txt"; 
$file = file($url);
$file = str_replace("	", ",",$file);
$file = str_replace("\r\n", "",$file);
// put the array together
foreach ($file as $line => $value) {
	$value = explode(",", $value);
    $bigArray[] = array("Donor" => $value[3], "Recipient" => $value[4], "Merits" => $value[1], "Msg" => $value[2], "DTG" => $value[0]);
}

function searchForDonor($uid, $array) {
	echo "<table>";
	foreach ($array as $key => $val) {
		if($val["Donor"] === $uid) {
			//print_r($array[$key]);
			echo "<tr>
					<td>".date("d.m.Y H:i:s", $val["DTG"])."</td>
					<td><a href='https://bitcointalk.org/index.php?topic=".$val["Msg"]."#". strstr($val["Msg"], "msg")."'>".$val["Msg"]."</a></td>
					<td><a href='https://bitcointalk.org/index.php?action=profile;u=".$val["Donor"]."'>".$val["Donor"]."</a></td>
					<td>-(".$val["Merits"].")-></td>
					<td><a href='https://bitcointalk.org/index.php?action=profile;u=".$val["Recipient"]."'>".$val["Recipient"]."</a></td>
				</tr>";
		} 
	}
	echo "</table>";
	return null;
}

function searchForRecipient($uid, $array) {
	echo "<table>";
	foreach ($array as $key => $val) {
		if($val["Recipient"] === $uid) {
			//print_r($array[$key]);
			echo "<tr>
					<td>".date("d.m.Y H:i:s", $val["DTG"])."</td>
					<td><a href='https://bitcointalk.org/index.php?topic=".$val["Msg"]."#". strstr($val["Msg"], "msg")."'>".$val["Msg"]."</a></td>
					<td><a href='https://bitcointalk.org/index.php?action=profile;u=".$val["Recipient"]."'>".$val["Recipient"]."</a></td>
					<td><-(".$val["Merits"].")-</td>
					<td><a href='https://bitcointalk.org/index.php?action=profile;u=".$val["Donor"]."'>".$val["Donor"]."</a></td>
				</tr>";
		} 
	}
	echo "</table>";
	return null;
}

if(isset($_GET["donor"])) {
	searchForDonor($_GET["donor"], $bigArray);
} 

if(isset($_GET["recipient"])) {
	searchForRecipient($_GET["recipient"], $bigArray);

}

?>

	</body>
</html>

Also, I decided to regain some of my PHP skills. Have forgotten how much fun it is.  

Haven't had a chance to make any modifications as of yet. Been swamped.

Is that the new code you're working off of?

Nah, currently not working on it as I have some social events coming up this week. I just added a form and thats all.
Will have some time next week.

[LoyceV]

Not quite sure as it runs on my local servers without problems.
I noticed that the URL actually is http://api.loyce.club/crowdsourcing/merit.all.txt. Try to change the $url variable in the script. ($url = "merit.all.txt"; should be enough because script and source data is stored in the same directory).

It was:

Code:

CryptoNeed.html:$url = "merit.all.txt"; 
CryptoNeed.php:$url = "http://loyce.club/Merit/merit.all.txt";
CryptoNeed2.php:$url = "http://loyce.club/Merit/merit.all.txt";

That location still exists. I've implemented your suggested changes, but the error is the same.

You have an error_log, feel free to shoot me a PM with the most recent error if you wish. I can take a look.

I've renamed it to error_log.txt, so you can click it.

Have you install or/and configure MySQL? It's definitely required to build API services.

My shared hosting:

Code:

Apache Version 	2.4.39
PHP Version 	5.6.40
MySQL Version 	5.7.26-cll-lve
Architecture 	x86_64
Operating System 	linux

Idk if Loyce knows mysql. But he already doesn’t know PHP, so that would just complicate his life.

I don't know either one

I’ll give a shot on making the API.

Thanks

As far as I know he doesnt know SQL.

Correct.

I think he is putting all the data together file by hand.

Incorrect (I update 350,000 files per week now)

It seems that Loyce did not have uploaded the adjusted the code as under the given URL it is responsing with the old version.

I postponed it until I had a bit more time. I'll try the suggestions now.

I pm'd him an "updated" version to have forms at the html one, so that the loaded script doesn't appear totally empty.

I've uploaded it to http://api.loyce.club/crowdsourcing/CryptoNeed.html but it gives some errors. http://api.loyce.club/crowdsourcing/CryptoNeed3.php.

[CryptoNeed]

I've uploaded it to http://api.loyce.club/crowdsourcing/CryptoNeed.html but it gives some errors.

I pm'd you. File extension should be .php instead of .html.
(I called it html version because of it's output.)

[Initscri]

Not quite sure as it runs on my local servers without problems.
I noticed that the URL actually is http://api.loyce.club/crowdsourcing/merit.all.txt. Try to change the $url variable in the script. ($url = "merit.all.txt"; should be enough because script and source data is stored in the same directory).

It was:

Code:

CryptoNeed.html:$url = "merit.all.txt"; 
CryptoNeed.php:$url = "http://loyce.club/Merit/merit.all.txt";
CryptoNeed2.php:$url = "http://loyce.club/Merit/merit.all.txt";

That location still exists. I've implemented your suggested changes, but the error is the same.

You have an error_log, feel free to shoot me a PM with the most recent error if you wish. I can take a look.

I've renamed it to error_log.txt, so you can click it.

Have you install or/and configure MySQL? It's definitely required to build API services.

My shared hosting:

Code:

Apache Version 	2.4.39
PHP Version 	5.6.40
MySQL Version 	5.7.26-cll-lve
Architecture 	x86_64
Operating System 	linux

Idk if Loyce knows mysql. But he already doesn’t know PHP, so that would just complicate his life.

I don't know either one

I’ll give a shot on making the API.

Thanks

As far as I know he doesnt know SQL.

Correct.

I think he is putting all the data together file by hand.

Incorrect (I update 350,000 files per week now)

It seems that Loyce did not have uploaded the adjusted the code as under the given URL it is responsing with the old version.

I postponed it until I had a bit more time. I'll try the suggestions now.

I pm'd him an "updated" version to have forms at the html one, so that the loaded script doesn't appear totally empty.

I've uploaded it to http://api.loyce.club/crowdsourcing/CryptoNeed.html but it gives some errors. http://api.loyce.club/crowdsourcing/CryptoNeed3.php.

You may need to implement my changes here: https://bitcointalk.org/index.php?topic=5156357.msg51564286#msg51564286

[LoyceV]

You may need to implement my changes here: https://bitcointalk.org/index.php?topic=5156357.msg51564286#msg51564286

Results:
This works: http://api.loyce.club/crowdsourcing/CryptoNeed_Initscri_tab.php?donor=35
This doesn't work: http://api.loyce.club/crowdsourcing/CryptoNeed_Initscri_tab.php?recipient=35

The one that works is quite slow to generate, I'm not sure how useful this is going to be for an API that loads data on many users.

[CryptoNeed]

You may need to implement my changes here: https://bitcointalk.org/index.php?topic=5156357.msg51564286#msg51564286

Results:
This works: http://api.loyce.club/crowdsourcing/CryptoNeed_Initscri_tab.php?donor=35
This doesn't work: http://api.loyce.club/crowdsourcing/CryptoNeed_Initscri_tab.php?recipient=35

The one that works is quite slow to generate, I'm not sure how useful this is going to be for an API that loads data on many users.

Can't follow all the changes anymore, but at least one versions seems to be working. Can you paste the code of the file here again?

It's so slow because of the .txt file. And the larger the .txt file becomes, the longer it will need to load.
The Algo is as follows:
1. Open/Download the file
2. Read the file (line by line)
3. Delete special chars
4. Build an array
5. Go through the whole array
6. Display matches with user id.


What we can do against this is:
Write a script to to import the whole txt into a SQL-DB. This would increase the speed to "infinity". But once the DB is established you need to keep care of it and save all new data to this DB.
I am not sure how you process all the data. Maybe you can tell us how exactly you get the data and how you are saving it into your .txt file.

[LoyceV]

Can't follow all the changes anymore, but at least one versions seems to be working. Can you paste the code of the file here again?

I've copied each .php to .txt at http://api.loyce.club/crowdsourcing/ so you can read them.

I am not sure how you process all the data. Maybe you can tell us how exactly you get the data and how you are saving it into your .txt file.

This one is actually quite simple:
1. I download and extract https://bitcointalk.org/merit.txt.xz
2. I overwrite the last ~113 days in merit.all.txt with the new ~120 days
The format is the same as the forum's data dump, I only add historic records.

[CryptoNeed]

Can't follow all the changes anymore, but at least one versions seems to be working. Can you paste the code of the file here again?

I've copied each .php to .txt at http://api.loyce.club/crowdsourcing/ so you can read them.

I will have a look deeper look into it when I have some time.

I am not sure how you process all the data. Maybe you can tell us how exactly you get the data and how you are saving it into your .txt file.

This one is actually quite simple:
1. I download and extract https://bitcointalk.org/merit.txt.xz
2. I overwrite the last ~113 days in merit.all.txt with the new ~120 days
The format is the same as the forum's data dump, I only add historic records.

Ahh okay.
You know what? I skip the deeper look into the script. As soon as I am back home, I will provide you with:
1. a new script which creates a SQL DB with a table.
2. some slim browser interface to paste the effected lines from the .txt-file into.
3. some SQL magic 

[Initscri]

You may need to implement my changes here: https://bitcointalk.org/index.php?topic=5156357.msg51564286#msg51564286

Results:
This works: http://api.loyce.club/crowdsourcing/CryptoNeed_Initscri_tab.php?donor=35
This doesn't work: http://api.loyce.club/crowdsourcing/CryptoNeed_Initscri_tab.php?recipient=35

The one that works is quite slow to generate, I'm not sure how useful this is going to be for an API that loads data on many users.

You can probably speed it up by caching the parsed $bigArray variable in a cache file.

I can write this into my code when I work on it.
It may not be by much, but it will drop a step.

@Crypto, can you send me the current code you have now?

[LoyceV]

I'll take the discussion from another topic here:

I like the idea !

But like to automate things, I think it would not be very complicated to automate the process with a bit of PHP code for example

If it interests you, I would be ready to help you for that !

Oh, okay, You are self-hosting maybe ? If so which OS ( Raspberry, Other ? ) ?

The data is created on my old netbook, then uploaded to my shared hosting:

Code:

Apache Version 	2.4.41
PHP Version 	7.2.23
MySQL Version 	5.7.28-cll-lve
Architecture 	x86_64
Operating System 	linux

Let's continue in this topic

[lulucrypto]

Okay, so if I understand correctly, you still have a VPS ( Shared hosting ) behind the domain "api.loyce.club" ?

So, I do not understand too much the story with the Netbook ^^

Why not do everything on the VPS at once ?