C# Node

[pmlyon]

I finally got FireBird working. Here's one of the more interesting things I had to do to read a byte array from the database.

        public static byte[] GetCharBytes(this DbDataReader reader, int i)
        {
            var value = reader.GetString(i);

            //TODO for the love of...
            Guid guid;
            if (value.Length == 36 && Guid.TryParse(value, out guid))
            {
                return guid.ToByteArray();
            }
            else
            {
                //TODO make sure this won't actually mangle anything, see Guid above
                return value.Select(x => (byte)x).ToArray();
            }
        }

[DeathAndTaxes]

Got an out of memory exception at around ~ block 216,057.  I had all three caches set to 100 however the system has 16GB of memory, roughly 12GB free.  Looks like BitSharp.Client was using 958.6MB of memory at the time of the exception

BlockDataCache: DataCache encountered fatal exception in StorageWorker: "Exception of type 'System.OutOfMemoryException' was thrown."

at System.IO.MemoryStream.set_Capacity(Int32 value)\r\n   at System.IO.MemoryStream.EnsureCapacity(Int32 value)\r\n   at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)\r\n   at System.IO.BinaryWriter.Write(Byte[] buffer)\r\n   at...

Was able to restart and it resumed without issue.  Stranglely it is running as a 32bit process so that may have had something to do with it.  Need to check configuration settings as it should be building as Any CPU.

[pmlyon]

Got an out of memory exception at around ~ block 216,057.  I had all three caches set to 100 however the system has 16GB of memory, roughly 12GB free.  Looks like BitSharp.Client was using 958.6MB of memory at the time of the exception

BlockDataCache: DataCache encountered fatal exception in StorageWorker: "Exception of type 'System.OutOfMemoryException' was thrown."

at System.IO.MemoryStream.set_Capacity(Int32 value)\r\n   at System.IO.MemoryStream.EnsureCapacity(Int32 value)\r\n   at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)\r\n   at System.IO.BinaryWriter.Write(Byte[] buffer)\r\n   at...

Was able to restart and it resumed without issue.  Stranglely it is running as a 32bit process so that may have had something to do with it.  Need to check configuration settings as it should be building as Any CPU.

Yeah, I've had a rough time with running it in 32-bit as well. I do my best not to hold onto any objects for too long, but it seems that the GC struggles to keep up with the amount of objects going in and out of memory. I'm not a GC expert at all though, so I can't really say for sure what exactly's going in. It seems that I'm stressing things with the way I'm using ImmutableHashSet.

I'd really like to have this optimized for being able to quickly access the data, even if it does even up requiring 64-bit mode to run well. I'd like it to be really simple to program against the node easily for things like querying data. At the moment I'm trying to figure out how I'm going to store historical transaction data in a manner that will allow for that that kind of querying access. I want the end result to really be able to scale, though. I'd like to make this as high throughput as I can... lots of work ahead.

[DeathAndTaxes]

Got an out of memory exception at around ~ block 216,057.  I had all three caches set to 100 however the system has 16GB of memory, roughly 12GB free.  Looks like BitSharp.Client was using 958.6MB of memory at the time of the exception

BlockDataCache: DataCache encountered fatal exception in StorageWorker: "Exception of type 'System.OutOfMemoryException' was thrown."

at System.IO.MemoryStream.set_Capacity(Int32 value)\r\n   at System.IO.MemoryStream.EnsureCapacity(Int32 value)\r\n   at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)\r\n   at System.IO.BinaryWriter.Write(Byte[] buffer)\r\n   at...

Was able to restart and it resumed without issue.  Stranglely it is running as a 32bit process so that may have had something to do with it.  Need to check configuration settings as it should be building as Any CPU.

Yeah, I've had a rough time with running it in 32-bit as well. I do my best not to hold onto any objects for too long, but it seems that the GC struggles to keep up with the amount of objects going in and out of memory. I'm not a GC expert at all though, so I can't really say for sure what exactly's going in. It seems that I'm stressing things with the way I'm using ImmutableHashSet.

The issue wasn't so much 32bit process running out of memory.  I can't seem to figure out why it was loading as a 32 bit process.  Architecture was "AnyCPU".

The System.Data.SQLite assembly is "AnyCPU" (PE=PE32 & 32BITREQ=0) with no native code (ILONLY=1).

corflags System.Data.SQLite.dll
Microsoft (R) .NET Framework CorFlags Conversion Tool.  Version  4.0.30319.17929

Copyright (c) Microsoft Corporation.  All rights reserved.

Version   : v4.0.30319
CLR Header: 2.5
PE        : PE32
CorFlags  : 0x9
ILONLY    : 1
32BITREQ  : 0
32BITPREF : 0
Signed    : 1

For some reason System.Data.SQLite is referencing the x86 not the x64 version of SQLite.Interop.dll forcing the entire assembly to run as a 32 bit process.  Forcing x64 as architecture results in a "BadImageFormatException" exception on System.Data.SQLite so once again for some unknown reason it is loading the assembly as 32 bit.  I am not sure why as the point of the split System.Data.SQLite.dll (managed AnyCPU wrapped) and SQLite.Interop.dll (native dll in both x86 and x64 versions) is to allow both x86 and x64 projects from the same reference right?

Anyways I bypassed the issue by just using the mixed mode x64 assembly.
http://system.data.sqlite.org/index.html/doc/trunk/www/downloads.wiki

Still not sure exactly why it didn't work.

I'd really like to have this optimized for being able to quickly access the data, even if it does even up requiring 64-bit mode to run well.

Agreed.  For my purposes x86 compatibility is a non-issue and as time goes on the data will only get larger.  32 bit general purpose clients make sense where the client is really only interested in the UXTO but for a general purpose parser to build blockchain related tools I see little value in trying to make x86 work.

I'd like it to be really simple to program against the node easily for things like querying data. At the moment I'm trying to figure out how I'm going to store historical transaction data in a manner that will allow for that that kind of querying access. I want the end result to really be able to scale, though. I'd like to make this as high throughput as I can... lots of work ahead.

Well a nice start so far. On my system it synced the blockchain from the genesis block significantly faster than bitcoind did.  

[mike8675309]

You may want to use the repository pattern to abstract data access.
http://msdn.microsoft.com/en-us/library/ff649690.aspx

Make it testable with inversion of control and dependency injection.
http://msdn.microsoft.com/en-us/library/aa973811.aspx

Blockchain size is not finished growing and it's pretty large to be storing in a relational data store.  I'd be tempted to just store it in the file system.  Most people are running a logging file system these days so making a backup when doing any work might be sufficient.

With it in the file system then you can use async file access to limit how much memory you need to do your work.
http://msdn.microsoft.com/en-us/library/jj155757.aspx

In this case then your data layer would actually be helper functions for common action you need to take on the the file.

for other stuff you could store it in a simple data store since the needs for capacity would be much less.  Unless you are using Mono to target other platforms you'll be limited to windows, so stick with that stuff.  if SQL Compact is not your choice, you might try ravendb.
http://ravendb.net/

[mike8675309]

Thought about the blockchain size issue again and thought if this is for sure on windows you may want to look at the built in ISAM database in windows known as the Extensible Storage System.  This is the engine behind Active Directory and Exchange Server and provides ACID data storage for up to terabytes and it comes with Windows.

More info here:
http://msdn.microsoft.com/en-us/library/gg269259(v=exchg.10).aspx

And since this is a system API using a non-managed .DLL here is a codeplex project wrapping the .dll in a managed code wrapper.
http://managedesent.codeplex.com/documentation

[DeathAndTaxes]

Blockchain size is not finished growing and it's pretty large to be storing in a relational data store.  I'd be tempted to just store it in the file system.  Most people are running a logging file system these days so making a backup when doing any work might be sufficient.

For a standard node you are right there likely is very little use to store the full blocks in a database.  For efficiency full nodes generally just validate the block, store the header, and use the block to update the UXTO. In essence using full blocks just to build the UXTO.  Full nodes normally never need the historical blockchain except to build the UXTO in a trustless manner.  For most nodes a flat file is more than sufficient and this why the reference client does just that.

However I think it IS useful as a development platform to parse and store the blockchain in a database.  This is useful to building higher level tools for analysis.  I imagine that is how sites like blockexplorer and blockchain.info work. 

[pmlyon]

Blockchain size is not finished growing and it's pretty large to be storing in a relational data store.  I'd be tempted to just store it in the file system.  Most people are running a logging file system these days so making a backup when doing any work might be sufficient.

For a standard node you are right there likely is very little use to store the full blocks in a database.  For efficiency full nodes generally just validate the block, store the header, and use the block to update the UXTO. In essence using full blocks just to build the UXTO.  Full nodes normally never need the historical blockchain except to build the UXTO in a trustless manner.  For most nodes a flat file is more than sufficient and this why the reference client does just that.

However I think it IS useful as a development platform to parse and store the blockchain in a database.  This is useful to building higher level tools for analysis.  I imagine that is how sites like blockexplorer and blockchain.info work.  

That's definitely the direction I'm coming at things from, although in the future I'd like to support lower trust levels as well.

Tonight I wired up the build to generate documentation, which I've placed on the wiki: http://pmlyon.github.io/BitSharp/apidocs/Index.html. There's no actual documentation yet, but now that it's integrated into the build I can work away at it.

[pmlyon]

I'd like to integrate code coverage results into the unit tests, but Visual Studio Express doesn't support generating code coverage results. Any recommendations out there on a good, free tool that I could use?

[DeathAndTaxes]

I'd like to integrate code coverage results into the unit tests, but Visual Studio Express doesn't support generating code coverage results. Any recommendations out there on a good, free tool that I could use?

You aren't by any chance a student are you?

[pmlyon]

I'd like to integrate code coverage results into the unit tests, but Visual Studio Express doesn't support generating code coverage results. Any recommendations out there on a good, free tool that I could use?

You aren't by any chance a student are you?

Unfortunately no. I may see about using my work computer to run unit tests with the full version of Visual Studio, I just want to be careful about keeping this segregated from my work where I want everything released in the public domain.

[mike8675309]

Any recommendations out there on a good, free tool that I could use?

Haven't used it myself but this appears promising: https://github.com/sawilde/opencover

[DeathAndTaxes]

I'd like to integrate code coverage results into the unit tests, but Visual Studio Express doesn't support generating code coverage results. Any recommendations out there on a good, free tool that I could use?

You aren't by any chance a student are you?

Unfortunately no. I may see about using my work computer to run unit tests with the full version of Visual Studio, I just want to be careful about keeping this segregated from my work where I want everything released in the public domain.

If you find nothing else our company has additional visual studio enterprise licenses which are not being used.  I imagine we could work something out (i.e. if necessary hire you as a contractor for 1 bitcent per year for the sole purpose of developing c# open source code to the public domain or under copyleft license, renewable annually).  I need to speak to counsel about it but if interested send me a PM. 

No ulterior motive here.  The public codebase for Bitcoin + .NET has been pretty much been non-existent.  C# is my preferred language so everything I have written has needed to be custom built (use a modified headless version of bitcoind running as a windows service with all interaction through a c# to JSON/RPC wrapper.    It would be useful to get a solid managed code library going (stop reinventing the wheel) if I can support that with a license it works for me.

[pmlyon]

Thought about the blockchain size issue again and thought if this is for sure on windows you may want to look at the built in ISAM database in windows known as the Extensible Storage System.  This is the engine behind Active Directory and Exchange Server and provides ACID data storage for up to terabytes and it comes with Windows.

More info here:
http://msdn.microsoft.com/en-us/library/gg269259(v=exchg.10).aspx

And since this is a system API using a non-managed .DLL here is a codeplex project wrapping the .dll in a managed code wrapper.
http://managedesent.codeplex.com/documentation

That looks very interesting, particularly the PersistentDictionary, thanks!

[DeathAndTaxes]

Thought about the blockchain size issue again and thought if this is for sure on windows you may want to look at the built in ISAM database in windows known as the Extensible Storage System.  This is the engine behind Active Directory and Exchange Server and provides ACID data storage for up to terabytes and it comes with Windows.

More info here:
http://msdn.microsoft.com/en-us/library/gg269259(v=exchg.10).aspx

And since this is a system API using a non-managed .DLL here is a codeplex project wrapping the .dll in a managed code wrapper.
http://managedesent.codeplex.com/documentation

That looks very interesting, particularly the PersistentDictionary, thanks!

Yeah very nice set of capabilities.  The fact that it is "built into windows" and requires no installation makes it useful for rapid deployment at least in windows environment.   With a data store abstraction layer a variety of storage systems could be supported.  ESENT could just be the default (providing no install instant availability for windows users).

Performance even on random lookups seems to be exceptional.  The bad news is that it looks like only certain datatypes can be used for keys.  Would be nice to store blocks with blockhash in binary (256 bit BigInteger) as the key.  Not sure if this is a limit of ESENT or just the current implementation of the managed code wrapper).  Still if performance using a hexadecimal string representation of that block hash has good enough performance it many be a non-issue.

One thing that may be worth considering is use ESENT for storing the blockchain and then use something like SQL Server (w/ Express for users without installed RDBMS) to store: wallet data (keypairs, tx history, user metadata), the UXTO, and a smaller table of just block headers.  The UXTO is less than 250MB, block headers since genesis block are about 15MB and most wallets are under a couple MB.  We have one massive (8,000+ keys, 20,000 txs) wallet and it is only 40MB.  All that should fit fine within the constraints of Express.

I am making a project to dump the blockchain into ESENT and benchmark performance (sequential inserts, random reads).

[pmlyon]

If you find nothing else our company has additional visual studio enterprise licenses which are not being used.  I imagine we could work something out (i.e. if necessary hire you as a contractor for 1 bitcent per year for the sole purpose of developing c# open source code to the public domain or under copyleft license, renewable annually).  I need to speak to counsel about it but if interested send me a PM. 

No ulterior motive here.  The public codebase for Bitcoin + .NET has been pretty much been non-existent.  C# is my preferred language so everything I have written has needed to be custom built (use a modified headless version of bitcoind running as a windows service with all interaction through a c# to JSON/RPC wrapper.    It would be useful to get a solid managed code library going (stop reinventing the wheel) if I can support that with a license it works for me.

Thanks for the offer! I'll let you know what I find out with my own company, no ulterior motive suspected.

I had started this project as a learning exercise to understand the Bitcoin tech better, but now I think that I've made enough progress to turn it into a real library. Wrapping it up in a proper Windows service and enabling RPC (presumably JSON) are definitely on my TODO list.

What kind of things are you currently using bitcoind for?  Understanding some use cases would really help me flesh things out. All of the current code represents prototyping, not a fully thought-out design.

[DeathAndTaxes]

If you find nothing else our company has additional visual studio enterprise licenses which are not being used.  I imagine we could work something out (i.e. if necessary hire you as a contractor for 1 bitcent per year for the sole purpose of developing c# open source code to the public domain or under copyleft license, renewable annually).  I need to speak to counsel about it but if interested send me a PM. 

No ulterior motive here.  The public codebase for Bitcoin + .NET has been pretty much been non-existent.  C# is my preferred language so everything I have written has needed to be custom built (use a modified headless version of bitcoind running as a windows service with all interaction through a c# to JSON/RPC wrapper.    It would be useful to get a solid managed code library going (stop reinventing the wheel) if I can support that with a license it works for me.

Thanks for the offer! I'll let you know what I find out with my own company, no ulterior motive suspected.

I had started this project as a learning exercise to understand the Bitcoin tech better, but now I think that I've made enough progress to turn it into a real library. Wrapping it up in a proper Windows service and enabling RPC (presumably JSON) are definitely on my TODO list.

What kind of things are you currently using bitcoind for?  Understanding some use cases would really help me flesh things out. All of the current code represents prototyping, not a fully thought-out design.

Well currently our company (Tangible Cryptography) services are halted as we dance with the state but we ran the site fastcash4bitcoins.com The bitcoind interface was used to track customer deposits and update a SQL database.  All high level functions are between the application and the RDBMS.  A service polls bitcoind periodically and updated the database with low level transaction data.

[pmlyon]

Well currently our company (Tangible Cryptography) services are halted as we dance with the state but we ran the site fastcash4bitcoins.com The bitcoind interface was used to track customer deposits and update a SQL database.  All high level functions are between the application and the RDBMS.  A service polls bitcoind periodically and updated the database with low level transaction data.

Thanks for the info. Sorry to hear about your troubles with the government, that sucks. Any light at the end of the tunnel?

I took a look at dotCover and I'm able to generate a test coverage report with the trial version of that. They have an open source license but I don't meet the standards yet, it's for more established projects. I'll see about applying for that eventually and I can use the trial version in the meantime.

[mike8675309]

Whatever you use, keep it to standards so that it remains portable.  I would be even tempted to use whatever database you do as a simple key, value store.  Assuming this remains a client tool and not some massive server undertaking you don't need any data level optimizations.  Without the need for data level optimizations you can keep it very portable as far as the back end.

[chalbersma]

I'm using SQLite for the database and BouncyCastle for the ECDsa verification and for SHA-256 and RIPEMD-160 I'm using the built-in .Net libaries.

The ECDsa verification is awful right now. It takes me 1/10th of a second just to verify a single signature, I have the verification disabled for the moment because of the speed. I haven't had any luck with OpenSSL yet to try it out.

I don't have much in the way of overall code documentation yet, I'll be working on that. Everything is put together just in broad strokes right now so that I have something usable to work with. There is a ton of design and clean-up to do.

Anyway, thanks for checking it out!

Have you looked at stuff like WAL?