.NET Blockchain Parser

[lontivero]

Hi,

I've just uploaded a little project that can parse the blockchain (the raw blk??.dat files) and returns the block's data (block, transactions, inputs and outputs) while it walks the blockchain. It is a very very simple piece of code and works pretty well.

As an example, in the same github repo there is a tool that reads the blockchain and save its blocks data in a ms sql database. I called it BlockchainToSql. I know there is a similar tool available for MySql but we, the .NET devs, are as we are.

the github repo: https://github.com/lontivero/BlockchainParser

If you find it interesting and want to collaborate then please do it.

[doof]

Awesome.  This has been on my todo list for a while.

[doof]

Ill make a pull request tonight, but here's some feedback.

• No Unit tests, the base 58 class is ideal to unit test and important its tested.
• Parts are really Models IMO
• I would make BlockchainToSql a class library so we can use it in our projects
• Separate the SQL code out, to maybe a provider.  Then inject the provider using something like IRepsoitory interface.  This will enable devs to write providers for other databases etc.
• For timings, use System.Diagnostics.Stopwatch class instead of Date.UtcNow
• Not sure why you are explicitly calling dbContext.Dispose(), when you are using a "using"

Good work tho!  Need some c# code for peer to peer getwork stuff now.

[lontivero]

Ill make a pull request tonight, but here's some feedback.

• No Unit tests, the base 58 class is ideal to unit test and important its tested.
• Parts are really Models IMO
• I would make BlockchainToSql a class library so we can use it in our projects
• Separate the SQL code out, to maybe a provider.  Then inject the provider using something like IRepsoitory interface.  This will enable devs to write providers for other databases etc.
• For timings, use System.Diagnostics.Stopwatch class instead of Date.UtcNow
• Not sure why you are explicitly calling dbContext.Dispose(), when you are using a "using"

Good work tho!  Need some c# code for peer to peer getwork stuff now.

@doof, I agree with all what you say, of course. It's just that I prefer to share earlier code than share better code.

[doof]

Ill make a pull request tonight, but here's some feedback.

• No Unit tests, the base 58 class is ideal to unit test and important its tested.
• Parts are really Models IMO
• I would make BlockchainToSql a class library so we can use it in our projects
• Separate the SQL code out, to maybe a provider.  Then inject the provider using something like IRepsoitory interface.  This will enable devs to write providers for other databases etc.
• For timings, use System.Diagnostics.Stopwatch class instead of Date.UtcNow
• Not sure why you are explicitly calling dbContext.Dispose(), when you are using a "using"

Good work tho!  Need some c# code for peer to peer getwork stuff now.

@doof, I agree with all what you say, of course. It's just that I prefer to share earlier code than share better code.

Happy to help work on this project, I need something similar.  Also need to change the models to have properties instead of vars.

[itod]

While we are at this subject, there are two remarkable and similar C# previous projects:
- http://www.codeproject.com/Articles/784519/NBitcoin-How-to-scan-the-Blockchain (large and in detail)
- https://github.com/kria/BlockChainParser (more like this one)

Except these two, there are two excellent code projects in C which also work directly on the blockchain without the RPC interface:
- http://codesuppository.blogspot.com/2013/07/bitcoin-code-snippets.html (you've mentioned John Ratcliff on GitHub page)
- https://github.com/mcdee/blockparser (lightning fast and very useful, but demands 8GB of RAM)

Hope this helps.

[doof]

While we are at this subject, there are two remarkable and similar C# previous projects:
- http://www.codeproject.com/Articles/784519/NBitcoin-How-to-scan-the-Blockchain (large and in detail)
- https://github.com/kria/BlockChainParser (more like this one)

Except these two, there are two excellent code projects in C which also work directly on the blockchain without the RPC interface:
- http://codesuppository.blogspot.com/2013/07/bitcoin-code-snippets.html (you've mentioned John Ratcliff on GitHub page)
- https://github.com/mcdee/blockparser (lightning fast and very useful, but demands 8GB of RAM)

Hope this helps.

Good tips, I've re-written a lot of code just so I get a good understanding of how it works.  Would be good if people contributed to one c# project and make that rock solid, rather than a lot of half baked ones :S

[itod]

While we are at this subject, there are two remarkable and similar C# previous projects:
- http://www.codeproject.com/Articles/784519/NBitcoin-How-to-scan-the-Blockchain (large and in detail)
- https://github.com/kria/BlockChainParser (more like this one)

Except these two, there are two excellent code projects in C which also work directly on the blockchain without the RPC interface:
- http://codesuppository.blogspot.com/2013/07/bitcoin-code-snippets.html (you've mentioned John Ratcliff on GitHub page)
- https://github.com/mcdee/blockparser (lightning fast and very useful, but demands 8GB of RAM)

Hope this helps.

Good tips, I've re-written a lot of code just so I get a good understanding of how it works.  Would be good if people contributed to one c# project and make that rock solid, rather than a lot of half baked ones :S

Agree, specially if that project can have decent tests and be modular. http://bitcoinj.github.io/ is a good example how it should be done, it's so good that many developers recommend it over bitcoind reference implementation for use in corporate environments.

[Nicolas Dorier]

For NBitcoin, the interesting part for you is the "Manual scanning" from http://www.codeproject.com/Articles/784519/NBitcoin-How-to-scan-the-Blockchain

Install-Package NBitcoin

then,

BlockStore store = new BlockStore("E:\\Bitcoin\\blocks", Network.Main);
foreach(var block in store.Enumerate())
{
...
}

If needed, you can index block headers and blocks location in the directory into an SqlLite database as explained in the article.

Agree, specially if that project can have decent tests and be modular. http://bitcoinj.github.io/ (http://bitcoinj.github.io/) is a good example how it should be done, it's so good that many developers recommend it over bitcoind reference implementation for use in corporate environments.

Not for throwing flower to myself, but NBitcoin largely match it, and is spreading fast these days.

[lontivero]

Yes, I know your work Nicolas and it's excellent really. I am a big fan of NBitcoin but I didn't know about the scanning capabilities (I can see it is a recent development).

[Nicolas Dorier]

oh lontivero, I did not see you were the guy writing the question !
Sure you know, I remember that you contributed on the NAT part

When you did, I think the BlockStore was already developed, this is an old class.

Anyway, if you need a feature that you think is missing from NBitcoin, you can ask me. If it can be useful to other, I don't care developing it.

[doof]

For NBitcoin, the interesting part for you is the "Manual scanning" from http://www.codeproject.com/Articles/784519/NBitcoin-How-to-scan-the-Blockchain

Install-Package NBitcoin

then,

BlockStore store = new BlockStore("E:\\Bitcoin\\blocks", Network.Main);
foreach(var block in store.Enumerate())
{
...
}

If needed, you can index block headers and blocks location in the directory into an SqlLite database as explained in the article.

Agree, specially if that project can have decent tests and be modular. http://bitcoinj.github.io/ (http://bitcoinj.github.io/) is a good example how it should be done, it's so good that many developers recommend it over bitcoind reference implementation for use in corporate environments.

Not for throwing flower to myself, but NBitcoin largely match it, and is spreading fast these days.

Flowers are pretty!  Nice, I've seen this article before, very good too.  Would be good to have a WCF c# project to do the networking / peer to peer.  Then either write to disk or to a db.  Kinda a project im working on.

[lontivero]

The couple WPF and Peer-2-Peer sound great but it is no real. If you want a p2p app you have to open a tcp socket.

Then either write to disk or to a db

NBitcoin can be useful for you, it doesn't have a way to write to a db (i think so) but it can write to disk.

[Nicolas Dorier]

Nice, I've seen this article before, very good too.

Thanks, take a look at my new one on TransactionBuilder, I earned a Tee-Shirt from Codeproject for it. (http://www.codeproject.com/Articles/835098/NBitcoin-Build-Them-All)

Would be good to have a WCF c# project to do the networking / peer to peer.  Then either write to disk or to a db.  Kinda a project im working on.

I created an blockchain indexer (NBitcoin.Indexer), I wrote an article about it, and using it in prod. (Colored Coins Compatible recently)
It uses Azure Storage. I will surely provide a JSON and SOAP interface to it in the future. But for now this is at the bottom of my stackoverflow of work.

NBitcoin can be useful for you, it doesn't have a way to write to a db (i think so) but it can write to disk.

That's why I am using Sqlite for some basic indexing, it is a good compromise between full db and ease of use. (even if I don't like the package dependency I took on Sqlite)

[doof]

The couple WPF and Peer-2-Peer sound great but it is no real. If you want a p2p app you have to open a tcp socket.

WCF, not WPF.  WCF on IIS for a bitcoind / daemon replacement.

http://msdn.microsoft.com/en-us/library/cc297274.aspx

[doof]

I created an blockchain indexer (NBitcoin.Indexer), I wrote an article about it, and using it in prod. (Colored Coins Compatible recently)
It uses Azure Storage. I will surely provide a JSON and SOAP interface to it in the future. But for now this is at the bottom of my stackoverflow of work.

Looking at it now.  I'm doing a similar thing with coloured coins.

[Nicolas Dorier]

I created an blockchain indexer (NBitcoin.Indexer), I wrote an article about it, and using it in prod. (Colored Coins Compatible recently)
It uses Azure Storage. I will surely provide a JSON and SOAP interface to it in the future. But for now this is at the bottom of my stackoverflow of work.

Looking at it now.  I'm doing a similar thing with coloured coins.

Don't hesitate to ask, I have not documented everything yet.
The idea is that the indexer does not deal with fork, the client does. I did lots of unit test you can run directly by setting up your own storage.
The fact that the indexer does not deal with forks make it horizontally scalable.

[doof]

FYI Im getting a null exception on block 215.  Ill raise an issue and try fix tonight and make a pull request.

[Nicolas Dorier]

I took a look at your code, I think you can win lots of time by switching to raw SQL instead of EF and trying BULK Insert.
http://msdn.microsoft.com/fr-fr/library/ms188365.aspx

I won't be surprised about a x10 perf improvement, maybe even more.

[lontivero]

The couple WPF and Peer-2-Peer sound great but it is no real. If you want a p2p app you have to open a tcp socket.

WCF, not WPF.  WCF on IIS for a bitcoind / daemon replacement.

http://msdn.microsoft.com/en-us/library/cc297274.aspx

Yes, I made a mistake when I said WPF instead of WCF however I am right when I say WCF is not a real option. You can share with me a lot of links to msdn and that won't change my opinion. Have you ever seen a p2p app developed with wcf? No real software uses wcf, they use sockets instead.