Bitcoin Forum

Hi, are there any other developers out there interested in or working on a bitcoin c# node? I've been working on one for a couple of months and I think I'm starting to see some promising results. I'd love to connect with anyone else interested in this work.

The node is still very, very basic; it can't really be a called a proper node in any sense. I've only been working on laying the foundational pieces.

I currently have it so that I can calculate the UTXO from block 0 to block ~218,000 in 6 minutes, which involves a bit under 12 million transactions. I'm using a SQLite backend on an SSD to get those times. This isn't doing any rules validation like coinbase rewards, it's just calculating the UTXO. It is making sure that there are no double spends and that each transaction input is in the UTXO of the previous step in the blockchain. At every height increment along the way, I also have an immutable snapshot of the UTXO at that height that I can hand off instantly. I think that's a respectable time to calculate the UTXO, but I'd love to get some feedback.

Also, does anyone know where I'd be able to find the size of the current UTXO? I'd like to compare to make sure that I'm getting the same result.

Thanks,

Paul

Link here: https://www.dropbox.com/s/zc88mn292lx865a/BitSharp.zip

Make that: https://github.com/pmlyon/BitSharp

The reference client stores the UXTO in the data directory under the subfolder chainstate.  It is stored in leveldb. 

it's 6540789 records, at block #245466
how much it takes on disk - it is very db specific.
I think its currently below 300MB on disk, in the official client, but it uses leveldb that has some built in compression.
and I am not quite sure if it even has the outputs scripts there, or only references to the blockchain db, where it gets the outputs from when needed.
normally (with the content of unspent out scripts) it might add up to almost 1GB, at the current height.

Sweet, that matches the number I have exactly. Thanks!

if you have a new satoshi client running, there is RPC command "gettxoutsetinfo" - it can tell you all you need.

I've added an initial overview of where I'm at with my node implementation:

https://github.com/pmlyon/BitSharp/wiki

Here's an initial screenshot, this UI is just to support the development work. I've included a simple block navigator to try out some very basic data visualization. I'm only showing the UTXO changes on the screen, but at each step along the way I have the entire UTXO calculated and immediately available for hand-off. It's able to navigate through close to a hundred blocks per second at today's UTXO size on my i5 hardware.

https://raw.github.com/pmlyon/BitSharp/master/.wiki/BitSharp.png

Very nice I will have to grab a copy and play around with it this weekend.  I know the answer is probably read the code but what are you using for a datastore/database and what libraries are you using for the functions crypto (Native Framework? OpenSSL? )?

I'm using SQLite for the database and BouncyCastle for the ECDsa verification and for SHA-256 and RIPEMD-160 I'm using the built-in .Net libaries.

The ECDsa verification is awful right now. It takes me 1/10th of a second just to verify a single signature, I have the verification disabled for the moment because of the speed. I haven't had any luck with OpenSSL yet to try it out.

I don't have much in the way of overall code documentation yet, I'll be working on that. Everything is put together just in broad strokes right now so that I have something usable to work with. There is a ton of design and clean-up to do. :)

Anyway, thanks for checking it out!

Anyone have any recommendations on a good embedded database I can use from C#? SQLite is turning out to be awful for concurrent writes, and Firebird does not seem to support binary data properly.

I'm curious why you insist on an embedded database? Why not use one of the many database abstraction layers available on Windows or simply the C# LINQ?

For development purposes, I want the solution to be ready to run as-is with no installation required.

Thanks for the concise and honest answer.

In my experience people and organizations who choose to develop in C# do that primarily because of the tremendous choice of the data storage layer backends. By excluding one of the strongest features of C# you'll also exclude the majority of the C# developers who would otherwise be very interested in your project.

Oh, you'll absolutely be able to plug in a different SQL or whatever back-end. For unit tests I have a "back-end" that just stores everything in memory in a dictionary, for example. I just want a decent out-of-box solution.

I have FireBird close to working with binary columns, running into a snag now where it's not accepting nulls. I have to say I'm pretty shocked that FireBird doesn't have proper binary column support.

SQL Express w/ local DB option?  It is what I usually use for rapid prototyping.  SQL Server Compact Edition is another option.

It is possible (although a pain in the ass) to compile google's leveldb (same db used by bitcoind) as a visual studio project so if you want something lighter weight that is an option.

Well, for the majority of the C# developers "plug in a different SQL or whatever back-end" means "change the connection string". I have yet to meet any C# developer who didn't already have MS-SQL or Oracle either installed on the same machine or immediately available by just giving the program the name of the server. In fact one of the more common problems working with C# developers was that they have so many different backends installed (or at least connectable with no prompting) that this becomes the source of confusion and seemingly irreproducible results.

We have certainly have a very different development background.

It's more of a personal principle. I like all development environments that I create to be runnable with zero configuration, or as close to as humanly possible. Sensible defaults out of box to get you started, and then allow you to tweak the environment from there.

Not sure if anyone encountered this problem but using git the solution file (and other files in root directory as a well as 4 of the project files) were not cloned.    Not sure why and it may have been my setup but downloading the zip from github, extract, and making it a new repo worked.

SQL Express was actually my first option, unfortunately the size limit means I can't fit the full blockchain in it.

I'm actually super close to figuring out the semantics I need to use to get FireBird working. Even if I don't, I think I'll probably end up going with it anyway, and just store all my numbers as strings I guess.

I finally got FireBird working. Here's one of the more interesting things I had to do to read a byte array from the database. :)

        public static byte[] GetCharBytes(this DbDataReader reader, int i)
        {
            var value = reader.GetString(i);

            //TODO for the love of...
            Guid guid;
            if (value.Length == 36 && Guid.TryParse(value, out guid))
            {
                return guid.ToByteArray();
            }
            else
            {
                //TODO make sure this won't actually mangle anything, see Guid above
                return value.Select(x => (byte)x).ToArray();
            }
        }

Got an out of memory exception at around ~ block 216,057.  I had all three caches set to 100 however the system has 16GB of memory, roughly 12GB free.  Looks like BitSharp.Client was using 958.6MB of memory at the time of the exception


Was able to restart and it resumed without issue.  Stranglely it is running as a 32bit process so that may have had something to do with it.  Need to check configuration settings as it should be building as Any CPU.

Yeah, I've had a rough time with running it in 32-bit as well. I do my best not to hold onto any objects for too long, but it seems that the GC struggles to keep up with the amount of objects going in and out of memory. I'm not a GC expert at all though, so I can't really say for sure what exactly's going in. It seems that I'm stressing things with the way I'm using ImmutableHashSet.

I'd really like to have this optimized for being able to quickly access the data, even if it does even up requiring 64-bit mode to run well. I'd like it to be really simple to program against the node easily for things like querying data. At the moment I'm trying to figure out how I'm going to store historical transaction data in a manner that will allow for that that kind of querying access. I want the end result to really be able to scale, though. I'd like to make this as high throughput as I can... lots of work ahead. :)

The issue wasn't so much 32bit process running out of memory.  I can't seem to figure out why it was loading as a 32 bit process.  Architecture was "AnyCPU".

The System.Data.SQLite assembly is "AnyCPU" (PE=PE32 & 32BITREQ=0) with no native code (ILONLY=1).


For some reason System.Data.SQLite is referencing the x86 not the x64 version of SQLite.Interop.dll forcing the entire assembly to run as a 32 bit process.  Forcing x64 as architecture results in a "BadImageFormatException" exception on System.Data.SQLite so once again for some unknown reason it is loading the assembly as 32 bit.  I am not sure why as the point of the split System.Data.SQLite.dll (managed AnyCPU wrapped) and SQLite.Interop.dll (native dll in both x86 and x64 versions) is to allow both x86 and x64 projects from the same reference right?

Anyways I bypassed the issue by just using the mixed mode x64 assembly.
http://system.data.sqlite.org/index.html/doc/trunk/www/downloads.wiki

Still not sure exactly why it didn't work.

Agreed.  For my purposes x86 compatibility is a non-issue and as time goes on the data will only get larger.  32 bit general purpose clients make sense where the client is really only interested in the UXTO but for a general purpose parser to build blockchain related tools I see little value in trying to make x86 work.


Well a nice start so far. On my system it synced the blockchain from the genesis block significantly faster than bitcoind did.  

You may want to use the repository pattern to abstract data access.
http://msdn.microsoft.com/en-us/library/ff649690.aspx

Make it testable with inversion of control and dependency injection.
http://msdn.microsoft.com/en-us/library/aa973811.aspx

Blockchain size is not finished growing and it's pretty large to be storing in a relational data store.  I'd be tempted to just store it in the file system.  Most people are running a logging file system these days so making a backup when doing any work might be sufficient.

With it in the file system then you can use async file access to limit how much memory you need to do your work.
http://msdn.microsoft.com/en-us/library/jj155757.aspx

In this case then your data layer would actually be helper functions for common action you need to take on the the file.

for other stuff you could store it in a simple data store since the needs for capacity would be much less.  Unless you are using Mono to target other platforms you'll be limited to windows, so stick with that stuff.  if SQL Compact is not your choice, you might try ravendb.
http://ravendb.net/

Thought about the blockchain size issue again and thought if this is for sure on windows you may want to look at the built in ISAM database in windows known as the Extensible Storage System.  This is the engine behind Active Directory and Exchange Server and provides ACID data storage for up to terabytes and it comes with Windows.

More info here:
http://msdn.microsoft.com/en-us/library/gg269259(v=exchg.10).aspx

And since this is a system API using a non-managed .DLL here is a codeplex project wrapping the .dll in a managed code wrapper.
http://managedesent.codeplex.com/documentation

For a standard node you are right there likely is very little use to store the full blocks in a database.  For efficiency full nodes generally just validate the block, store the header, and use the block to update the UXTO. In essence using full blocks just to build the UXTO.  Full nodes normally never need the historical blockchain except to build the UXTO in a trustless manner.  For most nodes a flat file is more than sufficient and this why the reference client does just that.

However I think it IS useful as a development platform to parse and store the blockchain in a database.  This is useful to building higher level tools for analysis.  I imagine that is how sites like blockexplorer and blockchain.info work. 

That's definitely the direction I'm coming at things from, although in the future I'd like to support lower trust levels as well.

Tonight I wired up the build to generate documentation, which I've placed on the wiki: http://pmlyon.github.io/BitSharp/apidocs/Index.html. There's no actual documentation yet, but now that it's integrated into the build I can work away at it.

I'd like to integrate code coverage results into the unit tests, but Visual Studio Express doesn't support generating code coverage results. Any recommendations out there on a good, free tool that I could use?

You aren't by any chance a student are you?

Unfortunately no. I may see about using my work computer to run unit tests with the full version of Visual Studio, I just want to be careful about keeping this segregated from my work where I want everything released in the public domain.

Haven't used it myself but this appears promising: https://github.com/sawilde/opencover

If you find nothing else our company has additional visual studio enterprise licenses which are not being used.  I imagine we could work something out (i.e. if necessary hire you as a contractor for 1 bitcent per year for the sole purpose of developing c# open source code to the public domain or under copyleft license, renewable annually).  I need to speak to counsel about it but if interested send me a PM. 

No ulterior motive here.  The public codebase for Bitcoin + .NET has been pretty much been non-existent.  C# is my preferred language so everything I have written has needed to be custom built (use a modified headless version of bitcoind running as a windows service with all interaction through a c# to JSON/RPC wrapper.    It would be useful to get a solid managed code library going (stop reinventing the wheel) if I can support that with a license it works for me.

That looks very interesting, particularly the PersistentDictionary, thanks!

Yeah very nice set of capabilities.  The fact that it is "built into windows" and requires no installation makes it useful for rapid deployment at least in windows environment.   With a data store abstraction layer a variety of storage systems could be supported.  ESENT could just be the default (providing no install instant availability for windows users).

Performance even on random lookups seems to be exceptional.  The bad news is that it looks like only certain datatypes can be used for keys.  Would be nice to store blocks with blockhash in binary (256 bit BigInteger) as the key.  Not sure if this is a limit of ESENT or just the current implementation of the managed code wrapper).  Still if performance using a hexadecimal string representation of that block hash has good enough performance it many be a non-issue.

One thing that may be worth considering is use ESENT for storing the blockchain and then use something like SQL Server (w/ Express for users without installed RDBMS) to store: wallet data (keypairs, tx history, user metadata), the UXTO, and a smaller table of just block headers.  The UXTO is less than 250MB, block headers since genesis block are about 15MB and most wallets are under a couple MB.  We have one massive (8,000+ keys, 20,000 txs) wallet and it is only 40MB.  All that should fit fine within the constraints of Express.

I am making a project to dump the blockchain into ESENT and benchmark performance (sequential inserts, random reads).

Thanks for the offer! I'll let you know what I find out with my own company, no ulterior motive suspected. ;)

I had started this project as a learning exercise to understand the Bitcoin tech better, but now I think that I've made enough progress to turn it into a real library. Wrapping it up in a proper Windows service and enabling RPC (presumably JSON) are definitely on my TODO list.

What kind of things are you currently using bitcoind for?  Understanding some use cases would really help me flesh things out. All of the current code represents prototyping, not a fully thought-out design.

Well currently our company (Tangible Cryptography) services are halted as we dance with the state but we ran the site fastcash4bitcoins.com The bitcoind interface was used to track customer deposits and update a SQL database.  All high level functions are between the application and the RDBMS.  A service polls bitcoind periodically and updated the database with low level transaction data.

Thanks for the info. Sorry to hear about your troubles with the government, that sucks. Any light at the end of the tunnel?

I took a look at dotCover and I'm able to generate a test coverage report with the trial version of that. They have an open source license but I don't meet the standards yet, it's for more established projects. I'll see about applying for that eventually and I can use the trial version in the meantime.

Whatever you use, keep it to standards so that it remains portable.  I would be even tempted to use whatever database you do as a simple key, value store.  Assuming this remains a client tool and not some massive server undertaking you don't need any data level optimizations.  Without the need for data level optimizations you can keep it very portable as far as the back end.

Have you looked at stuff like WAL? (http://docs.xojo.com/index.php/SQLiteDatabase.MultiUser)

I'm trying to keep my database layer as unintelligent as possible, the main thing I rely on the database for outside of key/value is finding holes in the data. You could write a simplified storage layer that didn't hold onto all historical data and didn't account for holes in the data and then wire that up to the blockchain calculator, if you wanted.

I have, thanks. I reworked my SQLite storage provider a bit so that I have an application wide reader-writer lock on a single connection and I've enabled WAL as well. It seems to be performing much better now.

I just found this horrible bug in my == method for 32-byte hashes. I guess it goes to show how unique these hashes really are that I only discovered this now. :) Note the || operator instead of &&.

        public static bool operator ==(UInt256 left, UInt256 right)
        {
            return left.part1 == right.part1 || left.part2 == right.part2 || left.part3 == right.part3 || left.part4 == right.part4;
        }

I've done some work to split my block storage into headers and transactions, I'm able to download and chain the block headers separately from full blocks now. I'm also able to pull individual transactions out of the database, whereas before I was retrieving full blocks just to get one transaction. The initial syncing is working much faster now, I think I should eventually be able to download and sync the full blockchain within an hour, given a proper storage back-end. (not SQLite on a spinning disk :))

I'm starting to feel pretty confident in the approach I've taken to my storage system and feeding it into the blockchain calculator. I have a lot of testing and documentation to catch up on now that I've proven that out. I also want to get a very basic lite storage system up and running at some point so I can make sure that I'm designing the blockchain calculator in a flexible enough manner to support it. I'll need to return some additional data so that a lite storage provider can determine what data it can remove, and I need to make sure I don't have any accidental dependencies on full historical storage in the calculator.

The way that I'm currently calculating the UTXO isn't actually correct so I need to get that taken care of. I'll need to properly implement BIP 30, for example. General correctness with regards to bitcoind I've left mostly off the table at this point anyway. That is going to be a huge undertaking that will require significant testing, so I don't want to be approaching that piecemeal. Correctness should ideally be isolated to the blockchain calculator and the scripting projects. The scripting project is essentially unimplemented at this point, I've just done the bare minimum to get me up to about block 158,000.

That's where things stand today, for any of you following along. :)

Paul

This weekend I'd like to start getting the blocktester (https://github.com/TheBlueMatt/test-scripts/) setup for when I start implementing things exactly. Does anyone have any experience integrating with this tool?

Hi!

Do you still work on this?

Hey, work is definitely still ongoing. There haven't been any source code updates for a few weeks... I've had two other developers join the project, which I'm super excited about! :)

We're all currently busy getting caught up with each other before making any more updates. Going forward we'll have much more up front design work and documentation; the current source code is a prototype.

I'm planning on posting an update to the site over the weekend. Thanks for checking in!

Paul