Bitcoin Forum
November 22, 2017, 07:13:15 PM *
News: Latest stable version of Bitcoin Core: 0.15.1  [Torrent].
 
   Home   Help Search Donate Login Register  
Pages: [1] 2 »  All
  Print  
Author Topic: .NET Blockchain Parser  (Read 4258 times)
lontivero
Full Member
***
Offline Offline

Activity: 157

Amazing times are coming


View Profile
November 19, 2014, 09:52:27 PM
 #1

Hi,

I've just uploaded a little project that can parse the blockchain (the raw blk?Huh?.dat files) and returns the block's data (block, transactions, inputs and outputs) while it walks the blockchain. It is a very very simple piece of code and works pretty well.

As an example, in the same github repo there is a tool that reads the blockchain and save its blocks data in a ms sql database. I called it BlockchainToSql. I know there is a similar tool available for MySql but we, the .NET devs, are as we are.

the github repo: https://github.com/lontivero/BlockchainParser

If you find it interesting and want to collaborate then please do it.
1511377995
Hero Member
*
Offline Offline

Posts: 1511377995

View Profile Personal Message (Offline)

Ignore
1511377995
Reply with quote  #2

1511377995
Report to moderator
1511377995
Hero Member
*
Offline Offline

Posts: 1511377995

View Profile Personal Message (Offline)

Ignore
1511377995
Reply with quote  #2

1511377995
Report to moderator
Join ICO Now Coinlancer is Disrupting the Freelance marketplace!
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
1511377995
Hero Member
*
Offline Offline

Posts: 1511377995

View Profile Personal Message (Offline)

Ignore
1511377995
Reply with quote  #2

1511377995
Report to moderator
1511377995
Hero Member
*
Offline Offline

Posts: 1511377995

View Profile Personal Message (Offline)

Ignore
1511377995
Reply with quote  #2

1511377995
Report to moderator
doof
Hero Member
*****
Offline Offline

Activity: 765


View Profile WWW
November 19, 2014, 10:53:16 PM
 #2

Awesome.  This has been on my todo list for a while.
doof
Hero Member
*****
Offline Offline

Activity: 765


View Profile WWW
November 19, 2014, 11:03:12 PM
 #3

Ill make a pull request tonight, but here's some feedback.

  • No Unit tests, the base 58 class is ideal to unit test and important its tested.
  • Parts are really Models IMO
  • I would make BlockchainToSql a class library so we can use it in our projects
  • Separate the SQL code out, to maybe a provider.  Then inject the provider using something like IRepsoitory interface.  This will enable devs to write providers for other databases etc.
  • For timings, use System.Diagnostics.Stopwatch class instead of Date.UtcNow
  • Not sure why you are explicitly calling dbContext.Dispose(), when you are using a "using"

Good work tho!  Need some c# code for peer to peer getwork stuff now.
lontivero
Full Member
***
Offline Offline

Activity: 157

Amazing times are coming


View Profile
November 20, 2014, 01:55:28 AM
 #4

Ill make a pull request tonight, but here's some feedback.

  • No Unit tests, the base 58 class is ideal to unit test and important its tested.
  • Parts are really Models IMO
  • I would make BlockchainToSql a class library so we can use it in our projects
  • Separate the SQL code out, to maybe a provider.  Then inject the provider using something like IRepsoitory interface.  This will enable devs to write providers for other databases etc.
  • For timings, use System.Diagnostics.Stopwatch class instead of Date.UtcNow
  • Not sure why you are explicitly calling dbContext.Dispose(), when you are using a "using"

Good work tho!  Need some c# code for peer to peer getwork stuff now.

@doof, I agree with all what you say, of course. It's just that I prefer to share earlier code than share better code.
doof
Hero Member
*****
Offline Offline

Activity: 765


View Profile WWW
November 20, 2014, 03:02:25 AM
 #5

Ill make a pull request tonight, but here's some feedback.

  • No Unit tests, the base 58 class is ideal to unit test and important its tested.
  • Parts are really Models IMO
  • I would make BlockchainToSql a class library so we can use it in our projects
  • Separate the SQL code out, to maybe a provider.  Then inject the provider using something like IRepsoitory interface.  This will enable devs to write providers for other databases etc.
  • For timings, use System.Diagnostics.Stopwatch class instead of Date.UtcNow
  • Not sure why you are explicitly calling dbContext.Dispose(), when you are using a "using"

Good work tho!  Need some c# code for peer to peer getwork stuff now.

@doof, I agree with all what you say, of course. It's just that I prefer to share earlier code than share better code.

Happy to help work on this project, I need something similar.  Also need to change the models to have properties instead of vars.
itod
Legendary
*
Online Online

Activity: 1092


^ will code for bitcoins


View Profile
November 20, 2014, 07:10:20 PM
 #6

While we are at this subject, there are two remarkable and similar C# previous projects:
- http://www.codeproject.com/Articles/784519/NBitcoin-How-to-scan-the-Blockchain (large and in detail)
- https://github.com/kria/BlockChainParser (more like this one)

Except these two, there are two excellent code projects in C which also work directly on the blockchain without the RPC interface:
- http://codesuppository.blogspot.com/2013/07/bitcoin-code-snippets.html (you've mentioned John Ratcliff on GitHub page)
- https://github.com/mcdee/blockparser (lightning fast and very useful, but demands 8GB of RAM)

Hope this helps.

EMPTY SIGNATURE SPACE
[Click here to make an offer]
doof
Hero Member
*****
Offline Offline

Activity: 765


View Profile WWW
November 20, 2014, 10:43:53 PM
 #7

While we are at this subject, there are two remarkable and similar C# previous projects:
- http://www.codeproject.com/Articles/784519/NBitcoin-How-to-scan-the-Blockchain (large and in detail)
- https://github.com/kria/BlockChainParser (more like this one)

Except these two, there are two excellent code projects in C which also work directly on the blockchain without the RPC interface:
- http://codesuppository.blogspot.com/2013/07/bitcoin-code-snippets.html (you've mentioned John Ratcliff on GitHub page)
- https://github.com/mcdee/blockparser (lightning fast and very useful, but demands 8GB of RAM)

Hope this helps.
Good tips, I've re-written a lot of code just so I get a good understanding of how it works.  Would be good if people contributed to one c# project and make that rock solid, rather than a lot of half baked ones :S
itod
Legendary
*
Online Online

Activity: 1092


^ will code for bitcoins


View Profile
November 21, 2014, 02:26:48 PM
 #8

While we are at this subject, there are two remarkable and similar C# previous projects:
- http://www.codeproject.com/Articles/784519/NBitcoin-How-to-scan-the-Blockchain (large and in detail)
- https://github.com/kria/BlockChainParser (more like this one)

Except these two, there are two excellent code projects in C which also work directly on the blockchain without the RPC interface:
- http://codesuppository.blogspot.com/2013/07/bitcoin-code-snippets.html (you've mentioned John Ratcliff on GitHub page)
- https://github.com/mcdee/blockparser (lightning fast and very useful, but demands 8GB of RAM)

Hope this helps.
Good tips, I've re-written a lot of code just so I get a good understanding of how it works.  Would be good if people contributed to one c# project and make that rock solid, rather than a lot of half baked ones :S

Agree, specially if that project can have decent tests and be modular. http://bitcoinj.github.io/ is a good example how it should be done, it's so good that many developers recommend it over bitcoind reference implementation for use in corporate environments.

EMPTY SIGNATURE SPACE
[Click here to make an offer]
Nicolas Dorier
Hero Member
*****
Offline Offline

Activity: 686


View Profile
November 22, 2014, 01:40:45 AM
 #9

For NBitcoin, the interesting part for you is the "Manual scanning" from http://www.codeproject.com/Articles/784519/NBitcoin-How-to-scan-the-Blockchain

Quote
Install-Package NBitcoin

then,
Quote
BlockStore store = new BlockStore("E:\\Bitcoin\\blocks", Network.Main);
foreach(var block in store.Enumerate())
{
...
}

If needed, you can index block headers and blocks location in the directory into an SqlLite database as explained in the article.

Quote
Agree, specially if that project can have decent tests and be modular. http://bitcoinj.github.io/ (http://bitcoinj.github.io/) is a good example how it should be done, it's so good that many developers recommend it over bitcoind reference implementation for use in corporate environments.

Not for throwing flower to myself, but NBitcoin largely match it, and is spreading fast these days. Wink

Bitcoin address 15sYbVpRh6dyWycZMwPdxJWD4xbfxReeHe
lontivero
Full Member
***
Offline Offline

Activity: 157

Amazing times are coming


View Profile
November 22, 2014, 04:54:02 PM
 #10

Yes, I know your work Nicolas and it's excellent really. I am a big fan of NBitcoin but I didn't know about the scanning capabilities (I can see it is a recent development).

Nicolas Dorier
Hero Member
*****
Offline Offline

Activity: 686


View Profile
November 23, 2014, 12:15:14 AM
 #11

oh lontivero, I did not see you were the guy writing the question ! Cheesy
Sure you know, I remember that you contributed on the NAT part Wink

When you did, I think the BlockStore was already developed, this is an old class.

Anyway, if you need a feature that you think is missing from NBitcoin, you can ask me. If it can be useful to other, I don't care developing it.

Bitcoin address 15sYbVpRh6dyWycZMwPdxJWD4xbfxReeHe
doof
Hero Member
*****
Offline Offline

Activity: 765


View Profile WWW
November 26, 2014, 11:09:29 PM
 #12

For NBitcoin, the interesting part for you is the "Manual scanning" from http://www.codeproject.com/Articles/784519/NBitcoin-How-to-scan-the-Blockchain

Quote
Install-Package NBitcoin

then,
Quote
BlockStore store = new BlockStore("E:\\Bitcoin\\blocks", Network.Main);
foreach(var block in store.Enumerate())
{
...
}

If needed, you can index block headers and blocks location in the directory into an SqlLite database as explained in the article.

Quote
Agree, specially if that project can have decent tests and be modular. http://bitcoinj.github.io/ (http://bitcoinj.github.io/) is a good example how it should be done, it's so good that many developers recommend it over bitcoind reference implementation for use in corporate environments.

Not for throwing flower to myself, but NBitcoin largely match it, and is spreading fast these days. Wink

Flowers are pretty!  Nice, I've seen this article before, very good too.  Would be good to have a WCF c# project to do the networking / peer to peer.  Then either write to disk or to a db.  Kinda a project im working on.
lontivero
Full Member
***
Offline Offline

Activity: 157

Amazing times are coming


View Profile
November 27, 2014, 01:55:06 AM
 #13

The couple WPF and Peer-2-Peer sound great but it is no real. If you want a p2p app you have to open a tcp socket.

Then either write to disk or to a db

NBitcoin can be useful for you, it doesn't have a way to write to a db (i think so) but it can write to disk.
Nicolas Dorier
Hero Member
*****
Offline Offline

Activity: 686


View Profile
November 27, 2014, 02:12:36 AM
 #14

Quote
Nice, I've seen this article before, very good too.
Thanks, take a look at my new one on TransactionBuilder, I earned a Tee-Shirt from Codeproject for it. Cheesy (http://www.codeproject.com/Articles/835098/NBitcoin-Build-Them-All)

Quote
Would be good to have a WCF c# project to do the networking / peer to peer.  Then either write to disk or to a db.  Kinda a project im working on.
I created an blockchain indexer (NBitcoin.Indexer), I wrote an article about it, and using it in prod. (Colored Coins Compatible recently)
It uses Azure Storage. I will surely provide a JSON and SOAP interface to it in the future. But for now this is at the bottom of my stackoverflow of work. Sad

NBitcoin can be useful for you, it doesn't have a way to write to a db (i think so) but it can write to disk.
That's why I am using Sqlite for some basic indexing, it is a good compromise between full db and ease of use. (even if I don't like the package dependency I took on Sqlite)

Bitcoin address 15sYbVpRh6dyWycZMwPdxJWD4xbfxReeHe
doof
Hero Member
*****
Offline Offline

Activity: 765


View Profile WWW
November 27, 2014, 04:47:09 AM
 #15

The couple WPF and Peer-2-Peer sound great but it is no real. If you want a p2p app you have to open a tcp socket.

WCF, not WPF.  WCF on IIS for a bitcoind / daemon replacement.

http://msdn.microsoft.com/en-us/library/cc297274.aspx
doof
Hero Member
*****
Offline Offline

Activity: 765


View Profile WWW
November 27, 2014, 04:49:01 AM
 #16

I created an blockchain indexer (NBitcoin.Indexer), I wrote an article about it, and using it in prod. (Colored Coins Compatible recently)
It uses Azure Storage. I will surely provide a JSON and SOAP interface to it in the future. But for now this is at the bottom of my stackoverflow of work. Sad

Looking at it now.  I'm doing a similar thing with coloured coins.
Nicolas Dorier
Hero Member
*****
Offline Offline

Activity: 686


View Profile
November 27, 2014, 01:10:15 PM
 #17

I created an blockchain indexer (NBitcoin.Indexer), I wrote an article about it, and using it in prod. (Colored Coins Compatible recently)
It uses Azure Storage. I will surely provide a JSON and SOAP interface to it in the future. But for now this is at the bottom of my stackoverflow of work. Sad

Looking at it now.  I'm doing a similar thing with coloured coins.

Don't hesitate to ask, I have not documented everything yet.
The idea is that the indexer does not deal with fork, the client does. I did lots of unit test you can run directly by setting up your own storage.
The fact that the indexer does not deal with forks make it horizontally scalable.

Bitcoin address 15sYbVpRh6dyWycZMwPdxJWD4xbfxReeHe
doof
Hero Member
*****
Offline Offline

Activity: 765


View Profile WWW
December 01, 2014, 11:09:20 PM
 #18

FYI Im getting a null exception on block 215.  Ill raise an issue and try fix tonight and make a pull request.
Nicolas Dorier
Hero Member
*****
Offline Offline

Activity: 686


View Profile
December 02, 2014, 12:20:45 AM
 #19

I took a look at your code, I think you can win lots of time by switching to raw SQL instead of EF and trying BULK Insert.
http://msdn.microsoft.com/fr-fr/library/ms188365.aspx

I won't be surprised about a x10 perf improvement, maybe even more.

Bitcoin address 15sYbVpRh6dyWycZMwPdxJWD4xbfxReeHe
lontivero
Full Member
***
Offline Offline

Activity: 157

Amazing times are coming


View Profile
December 02, 2014, 01:50:16 AM
 #20

The couple WPF and Peer-2-Peer sound great but it is no real. If you want a p2p app you have to open a tcp socket.

WCF, not WPF.  WCF on IIS for a bitcoind / daemon replacement.

http://msdn.microsoft.com/en-us/library/cc297274.aspx

Yes, I made a mistake when I said WPF instead of WCF however I am right when I say WCF is not a real option. You can share with me a lot of links to msdn and that won't change my opinion. Have you ever seen a p2p app developed with wcf? No real software uses wcf, they use sockets instead.
Pages: [1] 2 »  All
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!