[ANN - IN CONSTRUCTION] PhysiTrack - A General Tracker for Physical Bitcoin

[TheMagnat]

Hello everyone,

I introduce myself, I'm currently working in Yogg's company as an intern developer.
I am working on a range of tools around the Bitcoin Core and the blockchain.
He tasked me with a few projects including an algorithm that generates a file containing the whole set of Bitcoin addresses that exist and have a balance. This algorithm is theoretically finished but is still going on parsing the blockchain as it is huge. We think the computation should be finished in around 15 days if everything rolls out as planned.

During this time I also started to produce a web application at the request of Yogg,that aims to list as many bitcoin collectible items that have existed, exist and will exist. This project is still in developpement and is unusable without the result of the first algorithm but we have a first version to show you to give you an idea of this app.

Here is the temporary URL we use for the development of the tool :

http://dev-physitrack.bitcoin.paris

We would love you to check it out, and especially to have your feedback on the fonctionnality that are actually available and which ones you would like us to add.

Please note that like I said, the data that is displayed is innacurate due to the fact that our address list w/ balances is still getting generated by our algorithm.
Also the design of the app is still pretty basic but we're going to ask for the help of a front-end developper to make the perfect physical bitcoin tracker which include a wide range of physical bitcoin to track.

[1715512112]

1715512112

[1715512112]

1715512112

[1715512112]

1715512112

[1715512112]

1715512112

[yogg]

Hey there !
Yes, I confirm everything stated in OP is correct.

I am glad to welcome TheMagnat as a developer intern in my company.
He's doing a hell of a coding job since he started !!

I taught him some of the inner works of Bitcoin and it's blockchain so he can realize that script whichs outputs a file containing the whole set of BTC addresses that have a balance. Basically we recoded an internal blockchain explorer from scratch.

He told me it's much easier to get ahold of BTC once you understand the underlying tech, but this is another topic ...

We tried his script from block 1 to 304 000 (since block were rather empty so fast processing);
and the sum of BTC in the file was only ~0.02% off compared to the theoritical amount of BTC in existence by that block. We explain this as our script do not take multisig transactions and non-standard transaction into account. (Which isn't a problem for a physical bitcoin tracker)

Now as block got full around block #400,000, the process is heavier and takes more time but we'll soon have it all and will refresh the tracker website that Magnat works on.

While this operation gets finalized, we are working on the tracker backend. Once we are satisfied, we'll hire a front-end developer to make it nicer and smoother to browse.
The current version is far away from what this tracker will be like once it is finalized.

Our aim is to make the ultimate, global, physical bitcoin tracker.
Eventually, it will also serve as a global reference for the current and already circulating physical Bitcoin.

We welcome any feedback about our project.
If you have an idea or a remark, or a feature you'd love to see included, please let us know !


Suggestions / To-do list :

- Notify in a visible way when a physical bitcoin is supposedly a scam and/or compromised.
- Add the notable physical bitcoin makers to PhysiTrack. List all the "one time" runs as "Others".
  Great reference with many public addresses : https://bitcointalk.org/index.php?topic=2022902.0 (Thank you so much minerjones!)

[krogothmanhattan]

This is great! Thanks guys and also welcome to the forum!

[Edits]

Holy shit. Go big or go home. This will be an epic tracker.

As a noob (and from personal experience), one that stands out to me as difficult would be lealana coins. There doesnt seem to be public lists posted anywhere for the BTC coins, the addresses of each coin are just passed from seller to buyer.

This will be a great reference for collectors if/when you complete it. Good idea and good luck guys, I look forward to using the final result.

[Room101]

Well this is awesome! I actually started a thread a month or two back:

https://bitcointalk.org/index.php?topic=5249112.msg54448236#msg54448236

about setting up a provenance database of all physical coins that have been sold.

Love, love, love what you are doing! Not sure if you would have the time or bandwidth to add a provenance section, it would involve a lot of website searching and scraping, but it would be awesome if you could. Plus if you haven't learnt it already, website scraping is a cool and surprisingly helpful tool to have in your skill set.

I did not have time to fully flesh anything out, so it may not be super feasible, but figured there were a few ways to go about the physical side of things. Get the public lists of public keys that exist for most of the physicals, and search the internet, but this forum specifically, for mentions of them, look for the [WTS] or [AUCTION] tag in the heading, and then get a timestamp and seller, or it might be easier to do it the other way round, search for the above tags, and scrap any addresses you find the posts under that header. You may have thought of a much better way than that though! Will only get a percentage of coins sold obviously, but it would also raise the collectible section of bitcointalk's profile, and hopefully help it become the only place people want to sell!

I would love to be able to look up the provenance of coins, not only would knowing the history be super fucking cool, but it would help with fake coins, and raise the value of coins with good history.

I'm pretty flat out at the moment, but if you need any advice on setting up huge EC2 instances as cheaply as possible to do any number crunching etc, PM me, I'd be happy to help, it's half my day job.

Super happy to hear this, you'll learn a lot I'm sure, sounds like you landed yourself a cool internship, thanks also to Yogg for doing this, love to see people helping the next generation start their careers! And their bitcoin addiction

[Mbitr]

Wow, looks great already !
Definitely got one hell of a job on your hands !!
If this could be the ultimate guide/reference site then it would be a remarkable achievement  
Thankyou and fingers crossed that it’s doable  

[cygan]

a very good site, which will make a lot of things easier for collectors.
here you have now all collections at a glance and can access your personal "collections" very quickly
respect, thumbs up and the merits are on the way 

welcome to the forum by the way

[Room101]

another thing I thought of was giving users the ability to take a pic of the coin and upload it, add their PGP key, or forum name etc. Lot of privacy issues etc, could very well be a horrible terrible no-good idea, but thought I would throw it out there

[ChiBitCTy]

Damn Yoggy you really have been a busy bee of late, continuing to surprise us with awesome stuff left and right.  Kind of surprised you didn't ask me to take up the coding project here though  . 

Is there any plans to do the final version in English? I think this would be ideal being that some browsers suck at translating and others like say Duck Duck don't even offer the option. 

Random thought..Any chance you could add Satori custom chips?  Bitflyer, OmiseGo, Zaif, and MissBitcoin Japan being the customs made in "bulk" that I know of.. I can provide at least one address or first keys for each set.  We could then track down on the blockchain how many were made of each pretty easily no? FYI Moto ( Satori owner/founder) will not give us the addresses as he wouldn't even tell us how many of each were made for these exchanges. 

[chronicsky]

That's some good shit right there.
Surely needed a all in one physical tracker.

-snip-
one that stands out to me as difficult would be lealana coins. There doesnt seem to be public lists posted anywhere for the BTC coins, the addresses of each coin are just passed from seller to buyer.

-snip-

Definitely true! there's no record of lealana coins, so if this can help with that, it's gonna be golden.
though it still won't show the unloaded coins address.

[MoparMiningLLC]

I found my satori there and it had the right address but the wrong balance - I assume as this is still a work in progress that it will change to show the correct balance when you are finished.

[TheMagnat]

Thanks everyone for your welcome and for all this positivity and enthusiasm with regard to this project ! it motivates us even more to lead this project to the end

We're going to take in count all your feedbacks and will add as many feature we can to make it the ultimate physical tracker you need !

@MoparMiningLLC actually our balance database is empty, only some coldkey got a balance value that is not empty but it's for debug purpose so do not worry about it !

[COVID-19]

He tasked me with a few projects including an algorithm that generates a file containing the whole set of Bitcoin addresses that exist and have a balance.

Check this out:  https://bitcointalk.org/index.php?topic=5254914.0

[buwaytress]

Awesome project guys, just found you from LoyceV's merit list, so dumped my spare ones at you. I think it's a really interesting project, and collectors would love this but, my question is: aren't many physical Bitcoins anyway unfunded? That was my assumption anyway, I have bought the odd collectible or so, but only to give away to others as gifts so I don't myself own any now but I recall everything I'd ever bought as unfunded.

So how does that work with this project? Still awesome anyway and I'm looking forward, 15 days means it's 4 more days coming!

@OP How could you never have had a forum account before!? But welcome =)

[yogg]

Awesome project guys, just found you from LoyceV's merit list, so dumped my spare ones at you. I think it's a really interesting project, and collectors would love this but, my question is: aren't many physical Bitcoins anyway unfunded? That was my assumption anyway, I have bought the odd collectible or so, but only to give away to others as gifts so I don't myself own any now but I recall everything I'd ever bought as unfunded.

So how does that work with this project? Still awesome anyway and I'm looking forward, 15 days means it's 4 more days coming!

@OP How could you never have had a forum account before!? But welcome =)

Thanks everyone for your kind words.

Yes, it is very interesting !

There are many physical Bitcoins that are unfunded, but also there are many that are funded and there is only a few trackers that are very specific to one kind of item.

4 more days coming ... Well ...
Our script is parsing the whole dataset of Bitcoin Core's blockchain.
This is where we are at (just checked now) :


As blocks got completely full at around block #400,000; the process became much more lenghtly.
1000 blocks means that 1GB of data was processed. I'm not sure how to enhance the performances.

I'm pretty flat out at the moment, but if you need any advice on setting up huge EC2 instances as cheaply as possible to do any number crunching etc, PM me, I'd be happy to help, it's half my day job.

Thanks !
I will reach out to you, maybe you can give us a couple advise ? That'd be great.

He tasked me with a few projects including an algorithm that generates a file containing the whole set of Bitcoin addresses that exist and have a balance.

Check this out:  https://bitcointalk.org/index.php?topic=5254914.0

Thanks for pointing this thread ! Yes, absolutely.
We have access to Blockchair's data-dumps with an API key. They do a lot of them !

However Magnat's task was to find a way to come up with one of these dumps, and this is what we're compiling as of now.
Actually, that's why it displays balances for Coldkeys.
I gave him Blockchair's dump and a list of Coldkeys and asked him to retrieve the amounts from the dump.
We do it step by step.

When all the existing blocks will be processed, we'll have the state of all addresses "in-use" and their "balances" in BTC's blockchain.
We will process new blocks on the fly to always have an up-to-date state of addresses.
This data is precious and I can already think of more uses for it when the parsing is complete.

But yeah, this tracker will function only with our in-house tools.

Here's a link to many/most of the public maker's address lists: https://bitcointalk.org/index.php?topic=2022902.0

Good luck

Thanks for the great ressource !
Magnat's been compiling many makers lists for now and will continue to do so, among other things.

That's some good shit right there.
Surely needed a all in one physical tracker.

-snip-
one that stands out to me as difficult would be lealana coins. There doesnt seem to be public lists posted anywhere for the BTC coins, the addresses of each coin are just passed from seller to buyer.
-snip-

Definitely true! there's no record of lealana coins, so if this can help with that, it's gonna be golden.
though it still won't show the unloaded coins address.

Thanks Sky !
Yeah... I'm not sure how to figure that out.

Can someone share a Lealana BTC coin public address ?
Maybe we can figure it out by following the trails in the blockchain but without a proper list it's going to be tough.

[LoyceV]

Get the public lists of public keys that exist for most of the physicals, and search the internet, but this forum specifically, for mentions of them

I still haven't finished downloading all Bitcointalk posts, but if it helps, I can share what I have. It's currently around 60 GB and it takes about half a year to collect all data from the forum (with 1 request per forum).

When all the existing blocks will be processed, we'll have the state of all addresses "in-use" and their "balances" in BTC's blockchain.
We will process new blocks on the fly to always have an up-to-date state of addresses.

Have you considered what's going to happen to your database if you process a new block that orphans afterwards?

[yogg]

Get the public lists of public keys that exist for most of the physicals, and search the internet, but this forum specifically, for mentions of them

I still haven't finished downloading all Bitcointalk posts, but if it helps, I can share what I have. It's currently around 60 GB and it takes about half a year to collect all data from the forum (with 1 request per forum).

That'd be interesting, but I cannot think of a efficient way to extract all the useful info from that dump.

When all the existing blocks will be processed, we'll have the state of all addresses "in-use" and their "balances" in BTC's blockchain.
We will process new blocks on the fly to always have an up-to-date state of addresses.

Have you considered what's going to happen to your database if you process a new block that orphans afterwards?

Thanks a lot for pointing that out !!

Well, no. I haven't thought about it, but I have a simple kinda "quick & dirty" fix :
Process all blocks until :

Code:

nbCurrentBlock - 6

It can be 12, or even 144.

Our routine to calculate the balance of each physical will most likely be updated once every 24hr;
I don't think it is a big issue for the tracker to be constantly 24hr behind.

[buwaytress]

4 more days coming ... Well ...
Our script is parsing the whole dataset of Bitcoin Core's blockchain.
This is where we are at (just checked now) :

As blocks got completely full at around block #400,000; the process became much more lenghtly.
1000 blocks means that 1GB of data was processed. I'm not sure how to enhance the performances.

Ouf so that's about 230++GB of data left to process then, so definitely not four more days -- but good to see you're at least already at the 2016 mark, right about the time I got my first proper BTC anyway! Keep us posted!

Can someone share a Lealana BTC coin public address ?
Maybe we can figure it out by following the trails in the blockchain but without a proper list it's going to be tough.

This is one such full public address from this thread: 1Btc1qJg2gQmbFFjBbNs1LwgFQ3VyLoYgG suggests it's a vanity one, and from this batch of all beginning with "1Btc"

This one has a partial address starting with "12CBvVtq..."

I wonder if you've tried contacting the user smoothie. They do say in a previous post they stopped funding the coins after early 2014!

[Room101]

That'd be interesting, but I cannot think of a efficient way to extract all the useful info from that dump.

hmmmm..... that could be a fun ML problem for someone to solve. Start with Collectibles, hand label the first 50 pages or so, see what it spits out, it's the kind of structured data it might work well on, especially with [auction] and [wts] labels already exisiting, then numeric bids and offers, almost always with a PK in the first post.

You could probably do a pretty good job just with a basic python script to be honest, especially if you added a "Report Mistake" button to the website, so you could effectively crowd source the labeling problem.

[LoyceV]

That'd be interesting, but I cannot think of a efficient way to extract all the useful info from that dump.

If you're looking for posted Bitcoin addresses, it's quite easy to find. If you have to get the information from images, I don't know how to automate that.

Well, no. I haven't thought about it, but I have a simple kinda "quick & dirty" fix :
Process all blocks until :

Code:

nbCurrentBlock - 6

That solves it

It can be 12, or even 144.

You won't need more than 6 confirmations. If that gets rewinded, we have much bigger problems