Bitcoin Forum

Hi,

I've just uploaded a little project that can parse the blockchain (the raw blk?????.dat files) and returns the block's data (block, transactions, inputs and outputs) while it walks the blockchain. It is a very very simple piece of code and works pretty well.

As an example, in the same github repo there is a tool that reads the blockchain and save its blocks data in a ms sql database. I called it BlockchainToSql. I know there is a similar tool available for MySql but we, the .NET devs, are as we are.

the github repo: https://github.com/lontivero/BlockchainParser

If you find it interesting and want to collaborate then please do it.

Awesome.  This has been on my todo list for a while.

Ill make a pull request tonight, but here's some feedback.

• No Unit tests, the base 58 class is ideal to unit test and important its tested.
• Parts are really Models IMO
• I would make BlockchainToSql a class library so we can use it in our projects
• Separate the SQL code out, to maybe a provider.  Then inject the provider using something like IRepsoitory interface.  This will enable devs to write providers for other databases etc.
• For timings, use System.Diagnostics.Stopwatch class instead of Date.UtcNow
• Not sure why you are explicitly calling dbContext.Dispose(), when you are using a "using"

Good work tho!  Need some c# code for peer to peer getwork stuff now.

@doof, I agree with all what you say, of course. It's just that I prefer to share earlier code than share better code.

Happy to help work on this project, I need something similar.  Also need to change the models to have properties instead of vars.

While we are at this subject, there are two remarkable and similar C# previous projects:
- http://www.codeproject.com/Articles/784519/NBitcoin-How-to-scan-the-Blockchain (http://www.codeproject.com/Articles/784519/NBitcoin-How-to-scan-the-Blockchain) (large and in detail)
- https://github.com/kria/BlockChainParser (https://github.com/kria/BlockChainParser) (more like this one)

Except these two, there are two excellent code projects in C which also work directly on the blockchain without the RPC interface:
- http://codesuppository.blogspot.com/2013/07/bitcoin-code-snippets.html (http://codesuppository.blogspot.com/2013/07/bitcoin-code-snippets.html) (you've mentioned John Ratcliff on GitHub page)
- https://github.com/mcdee/blockparser (https://github.com/mcdee/blockparser) (lightning fast and very useful, but demands 8GB of RAM)

Hope this helps.

Good tips, I've re-written a lot of code just so I get a good understanding of how it works.  Would be good if people contributed to one c# project and make that rock solid, rather than a lot of half baked ones :S

Agree, specially if that project can have decent tests and be modular. http://bitcoinj.github.io/ (http://bitcoinj.github.io/) is a good example how it should be done, it's so good that many developers recommend it over bitcoind reference implementation for use in corporate environments.

For NBitcoin, the interesting part for you is the "Manual scanning" from http://www.codeproject.com/Articles/784519/NBitcoin-How-to-scan-the-Blockchain (http://www.codeproject.com/Articles/784519/NBitcoin-How-to-scan-the-Blockchain)


then,

If needed, you can index block headers and blocks location in the directory into an SqlLite database as explained in the article.


Not for throwing flower to myself, but NBitcoin largely match it, and is spreading fast these days. ;)

Yes, I know your work Nicolas and it's excellent really. I am a big fan of NBitcoin but I didn't know about the scanning capabilities (I can see it is a recent development).

oh lontivero, I did not see you were the guy writing the question ! :D
Sure you know, I remember that you contributed on the NAT part ;)

When you did, I think the BlockStore was already developed, this is an old class.

Anyway, if you need a feature that you think is missing from NBitcoin, you can ask me. If it can be useful to other, I don't care developing it.

Flowers are pretty!  Nice, I've seen this article before, very good too.  Would be good to have a WCF c# project to do the networking / peer to peer.  Then either write to disk or to a db.  Kinda a project im working on.

The couple WPF and Peer-2-Peer sound great but it is no real. If you want a p2p app you have to open a tcp socket.


NBitcoin can be useful for you, it doesn't have a way to write to a db (i think so) but it can write to disk.

Thanks, take a look at my new one on TransactionBuilder, I earned a Tee-Shirt from Codeproject for it. :D (http://www.codeproject.com/Articles/835098/NBitcoin-Build-Them-All (http://www.codeproject.com/Articles/835098/NBitcoin-Build-Them-All))

I created an blockchain indexer (NBitcoin.Indexer), I wrote an article about it, and using it in prod. (Colored Coins Compatible recently)
It uses Azure Storage. I will surely provide a JSON and SOAP interface to it in the future. But for now this is at the bottom of my stackoverflow of work. :(

That's why I am using Sqlite for some basic indexing, it is a good compromise between full db and ease of use. (even if I don't like the package dependency I took on Sqlite)

WCF, not WPF.  WCF on IIS for a bitcoind / daemon replacement.

http://msdn.microsoft.com/en-us/library/cc297274.aspx

Looking at it now.  I'm doing a similar thing with coloured coins.

Don't hesitate to ask, I have not documented everything yet.
The idea is that the indexer does not deal with fork, the client does. I did lots of unit test you can run directly by setting up your own storage.
The fact that the indexer does not deal with forks make it horizontally scalable.

FYI Im getting a null exception on block 215.  Ill raise an issue and try fix tonight and make a pull request.

I took a look at your code, I think you can win lots of time by switching to raw SQL instead of EF and trying BULK Insert.
http://msdn.microsoft.com/fr-fr/library/ms188365.aspx (http://msdn.microsoft.com/fr-fr/library/ms188365.aspx)

I won't be surprised about a x10 perf improvement, maybe even more.

Yes, I made a mistake when I said WPF instead of WCF however I am right when I say WCF is not a real option. You can share with me a lot of links to msdn and that won't change my opinion. Have you ever seen a p2p app developed with wcf? No real software uses wcf, they use sockets instead.

True, I havent seen any p2p wcf in the wild.

Can you put a sample block in github?

I have an abandoned project that is not the best example https://github.com/lontivero/peer2net (I have to redo it some day)

Ok.  Ill be using a project like this and bitcoind in the short term.  What happens if the processor is rerun, will duplicates be added?  Could the block hash be a primary key to stop duplicates?

Difficulty isn't correct

This project has saved me a bunch of time, and we seem to both have the same goal.

Not a lot of btc, but I sent you a tip.  Thanks.

I saw it and appreciate it. Thx

I wrote this C# code to calculate the hash too https://bitcointalk.org/index.php?topic=888078.0

Might be worth adding a column in the block table to contain the hash.  Seperate thread can hash the block, and then could reference one block to another.

Thank you very much. I like the license!

Nice code friend and I did a hatchet job on the code but it does not work on the block.dat
I am using because I think it's been change over these days to use Berkeley-DB format
for the files but useful code none the less and well written

See https://bitcointalk.org/index.php?topic=2748620.new#new

Small, simple and easy to follow and is a good starting point for anyone going down this road