C# Node

[pmlyon]

Hi, are there any other developers out there interested in or working on a bitcoin c# node? I've been working on one for a couple of months and I think I'm starting to see some promising results. I'd love to connect with anyone else interested in this work.

The node is still very, very basic; it can't really be a called a proper node in any sense. I've only been working on laying the foundational pieces.

I currently have it so that I can calculate the UTXO from block 0 to block ~218,000 in 6 minutes, which involves a bit under 12 million transactions. I'm using a SQLite backend on an SSD to get those times. This isn't doing any rules validation like coinbase rewards, it's just calculating the UTXO. It is making sure that there are no double spends and that each transaction input is in the UTXO of the previous step in the blockchain. At every height increment along the way, I also have an immutable snapshot of the UTXO at that height that I can hand off instantly. I think that's a respectable time to calculate the UTXO, but I'd love to get some feedback.

Also, does anyone know where I'd be able to find the size of the current UTXO? I'd like to compare to make sure that I'm getting the same result.

Thanks,

Paul

[pmlyon]

Link here: https://www.dropbox.com/s/zc88mn292lx865a/BitSharp.zip

Make that: https://github.com/pmlyon/BitSharp

[DeathAndTaxes]

The reference client stores the UXTO in the data directory under the subfolder chainstate.  It is stored in leveldb. 

[piotr_n]

Also, does anyone know where I'd be able to find the size of the current UTXO? I'd like to compare to make sure that I'm getting the same result.

it's 6540789 records, at block #245466
how much it takes on disk - it is very db specific.
I think its currently below 300MB on disk, in the official client, but it uses leveldb that has some built in compression.
and I am not quite sure if it even has the outputs scripts there, or only references to the blockchain db, where it gets the outputs from when needed.
normally (with the content of unspent out scripts) it might add up to almost 1GB, at the current height.

[pmlyon]

Sweet, that matches the number I have exactly. Thanks!

[piotr_n]

if you have a new satoshi client running, there is RPC command "gettxoutsetinfo" - it can tell you all you need.

[pmlyon]

I've added an initial overview of where I'm at with my node implementation:

https://github.com/pmlyon/BitSharp/wiki

[pmlyon]

Here's an initial screenshot, this UI is just to support the development work. I've included a simple block navigator to try out some very basic data visualization. I'm only showing the UTXO changes on the screen, but at each step along the way I have the entire UTXO calculated and immediately available for hand-off. It's able to navigate through close to a hundred blocks per second at today's UTXO size on my i5 hardware.

[DeathAndTaxes]

Very nice I will have to grab a copy and play around with it this weekend.  I know the answer is probably read the code but what are you using for a datastore/database and what libraries are you using for the functions crypto (Native Framework? OpenSSL? )?

[pmlyon]

I'm using SQLite for the database and BouncyCastle for the ECDsa verification and for SHA-256 and RIPEMD-160 I'm using the built-in .Net libaries.

The ECDsa verification is awful right now. It takes me 1/10th of a second just to verify a single signature, I have the verification disabled for the moment because of the speed. I haven't had any luck with OpenSSL yet to try it out.

I don't have much in the way of overall code documentation yet, I'll be working on that. Everything is put together just in broad strokes right now so that I have something usable to work with. There is a ton of design and clean-up to do.

Anyway, thanks for checking it out!

[pmlyon]

Anyone have any recommendations on a good embedded database I can use from C#? SQLite is turning out to be awful for concurrent writes, and Firebird does not seem to support binary data properly.

[2112]

Anyone have any recommendations on a good embedded database I can use from C#? SQLite is turning out to be awful for concurrent writes, and Firebird does not seem to support binary data properly.

I'm curious why you insist on an embedded database? Why not use one of the many database abstraction layers available on Windows or simply the C# LINQ?

[pmlyon]

For development purposes, I want the solution to be ready to run as-is with no installation required.

[2112]

For development purposes, I want the solution to be ready to run as-is with no installation required.

Thanks for the concise and honest answer.

In my experience people and organizations who choose to develop in C# do that primarily because of the tremendous choice of the data storage layer backends. By excluding one of the strongest features of C# you'll also exclude the majority of the C# developers who would otherwise be very interested in your project.

Hi, are there any other developers out there interested in or working on a bitcoin c# node? I've been working on one for a couple of months and I think I'm starting to see some promising results. I'd love to connect with anyone else interested in this work.

[pmlyon]

Oh, you'll absolutely be able to plug in a different SQL or whatever back-end. For unit tests I have a "back-end" that just stores everything in memory in a dictionary, for example. I just want a decent out-of-box solution.

I have FireBird close to working with binary columns, running into a snag now where it's not accepting nulls. I have to say I'm pretty shocked that FireBird doesn't have proper binary column support.

[DeathAndTaxes]

Oh, you'll absolutely be able to plug in a different SQL or whatever back-end. For unit tests I have a "back-end" that just stores everything in memory in a dictionary, for example. I just want a decent out-of-box solution.

I have FireBird close to working with binary columns, running into a snag now where it's not accepting nulls. I have to say I'm pretty shocked that FireBird doesn't have proper binary column support.

SQL Express w/ local DB option?  It is what I usually use for rapid prototyping.  SQL Server Compact Edition is another option.

It is possible (although a pain in the ass) to compile google's leveldb (same db used by bitcoind) as a visual studio project so if you want something lighter weight that is an option.

[2112]

Oh, you'll absolutely be able to plug in a different SQL or whatever back-end. For unit tests I have a "back-end" that just stores everything in memory in a dictionary, for example. I just want a decent out-of-box solution.

I have FireBird close to working with binary columns, running into a snag now where it's not accepting nulls. I have to say I'm pretty shocked that FireBird doesn't have proper binary column support.

Well, for the majority of the C# developers "plug in a different SQL or whatever back-end" means "change the connection string". I have yet to meet any C# developer who didn't already have MS-SQL or Oracle either installed on the same machine or immediately available by just giving the program the name of the server. In fact one of the more common problems working with C# developers was that they have so many different backends installed (or at least connectable with no prompting) that this becomes the source of confusion and seemingly irreproducible results.

We have certainly have a very different development background.

[pmlyon]

Oh, you'll absolutely be able to plug in a different SQL or whatever back-end. For unit tests I have a "back-end" that just stores everything in memory in a dictionary, for example. I just want a decent out-of-box solution.

I have FireBird close to working with binary columns, running into a snag now where it's not accepting nulls. I have to say I'm pretty shocked that FireBird doesn't have proper binary column support.

Well, for the majority of the C# developers "plug in a different SQL or whatever back-end" means "change the connection string". I have yet to meet any C# developer who didn't already have MS-SQL or Oracle either installed on the same machine or immediately available by just giving the program the name of the server. In fact one of the more common problems working with C# developers was that they have so many different backends installed (or at least connectable with no prompting) that this becomes the source of confusion and seemingly irreproducible results.

We have certainly have a very different development background.

It's more of a personal principle. I like all development environments that I create to be runnable with zero configuration, or as close to as humanly possible. Sensible defaults out of box to get you started, and then allow you to tweak the environment from there.

[DeathAndTaxes]

Not sure if anyone encountered this problem but using git the solution file (and other files in root directory as a well as 4 of the project files) were not cloned.    Not sure why and it may have been my setup but downloading the zip from github, extract, and making it a new repo worked.

[pmlyon]

Oh, you'll absolutely be able to plug in a different SQL or whatever back-end. For unit tests I have a "back-end" that just stores everything in memory in a dictionary, for example. I just want a decent out-of-box solution.

I have FireBird close to working with binary columns, running into a snag now where it's not accepting nulls. I have to say I'm pretty shocked that FireBird doesn't have proper binary column support.

SQL Express w/ local DB option?  It is what I usually use for rapid prototyping.  SQL Server Compact Edition is another option.

It is possible (although a pain in the ass) to compile google's leveldb (same db used by bitcoind) as a visual studio project so if you want something lighter weight that is an option.

SQL Express was actually my first option, unfortunately the size limit means I can't fit the full blockchain in it.

I'm actually super close to figuring out the semantics I need to use to get FireBird working. Even if I don't, I think I'll probably end up going with it anyway, and just store all my numbers as strings I guess.