Bitcoin Forum

Hello to everybody,

TL;DR: i believe bitcoind is not capable of creating multiple/individualized wallets, how can i do that without third parties using only C#/C++/PHP/maybe python? are there any libraries that may be helpful? or is it easy enough to communicate directly with the nodes and write my own client? (in a perfect world, i want an api that is not owned by a third party like blockchain.info or coinbase, and id like a library that will not require jumping languages and using HTTP communication or any form of request/response type interactions)



------ My issue/goals/questions: -------

I have created past relatively well known financial services in the past, however i used higher level APIs to achieve my goals. i have a large team working for me but i have run in to a major problem on a recent project we are getting in to. The problem is, we dont know what we are doing, and if i dont understand something i cant teach the team.

So, i need to create a web application that has bitcoin functionality, of course, my requirements are

- low level transaction broadcasting/signing. bitcoind-RPC works for this with raw transaction broadcasting.
- Multi-sig support, 2 of 2 or 2 of 3. bitcoind has only partial implementation
- unlimited wallet creation, not accounts like bitcoind, but actual wallets with individual encryption, primarily for individualized multi-sig transactions/wallets. bitcoind is crippled on this front
- wallet encryption
-potentially the ability to generate predictable addresses using a seed (not required, may be too advanced to start with, but if i figure out the bitcoin system, this should be a piece of cake if api limitations are not an issue)

so as bitcoind is useless to me (I THINK. correct me if i can expand functionality somehow), i cant think of or find a lower level API besides some experimental c++ one. how did all these other sites do it? like blockchain.info that stores countless individualized wallets? how about coinbase? i doubt they used bitcoind but im also doubting that they actually do it 100% raw.  I in no way intend to develop anything as intensive as an exchange, but i do need similar functionality where user security is the only priority, and bitcoind/shared wallets are out of the question.  


I am currently developing in C# with asp.net, i would love to keep it all in c#, but i can very easily switch to php, i chose c# for security purposes. i will not do java unless it is the only option. i also am willing to write a server for executing the commands in c++, but i feel c++ may be too low level for such a networking intensive application and will triple the work load. and im sure if c++ can do it, so can c# since i dont expect a ton of low level requirements like memory allocations and byte manipulation.


------ if a 'ground-up' approach IS necessary: --------

So, if i do need to start doing this all without an api, where do i start? im sure i can dig through qt's source but wow... there is a LOT to look through. and ift his is the case, how are wallets created? are they even a thing or is it all a matter of just storing public and private keys, decrypting, deserializing responses from the nodes, and handling however i like?

what IS a wallet? does the blockchain/nodes/miners/etc... even recognize wallets or is it a construct we use for organizing the addresses like an account?

is there an API for communicating with the nodes? how does something actually get sent to the blockchain? simple post request/response over http? im lost here.


----- In conclusion: ------

So i need some help, and i hope one of you geniuses can assist me. if somebody can prove they are knowledgeable, i am more than happy to pay handsomely for your time and expertise. my goal here is more so to learn than it is to get the project done. i know i could hire somebody with the experience, but in my opinion, if i cant write and understand the code myself, im unfit to even oversee the project let alone be called its owner.


any insight will be greatly appreciated.

there is something really wrong with this post  ???, and it starts with the header. Then there is lot of bla bla about a big team. And the impression is given, that there is lots of experience and knowledge behind. Just to finalize with "code = doc?". And a certain lazyness attitude. This is all fairly ok for a newbie. But doesn't match with the intro words. You can certainly get better answers, if you stay away from storytails, and get to the points immedeatly.

Welcome to the crazy world of crypto currency. So you want to deep dive into crypto?
With C# using a windows machine??? Go and get some professional equipment first (Mac, Unix, OpenBSD,...), we are talking security here.

Yes, there are some Java libraries, python codes and more, when you dig through the forum.

Professionals usually start with reading before coding. This would bring you immedeatly to bitcoin.org, to understand the eco system (and it's code).
Then you could read probably the online book (or buy it) from Andreas: "Mastering Bitcoin", here: http://chimera.labs.oreilly.com/books/1234000001802/
I recommend especially the chapter 4 on wallets.
Also there is the very good post from Ken: http://www.righto.com/2014/02/bitcoins-hard-way-using-raw-bitcoin.html
On youtube you find Bitcoin 101, which explains alot.

A wallet is in its simplest form just your priv key and some code to generate a transaction. With this privkey you can sign transactions, and send it to the network. For sure, today's wallets provide more functionality. But this is the basics. So you have a key store and some code. Miners or blockchain do not see, what wallet was used to generate a transaction. To integrate a tx in a block, the tx must follow specific rules with a layout of data, and the wallet software would help to do that. This question shows some missing knowledge on the basics of the network. Review the reading links  ;)
Yup, and then there is a "send" function to the network. In this forum there was once some code, I just don't remember the link...  :-\
 
So please, re-edit your post a bit, correct some wording, read the docs, and start to code. Become a valuable member of the community, and share some of your developments. Hint: SegWit wallets are still rare :-)

A warm welcome to the community!
 

thanks for your answer about wallet functionality. however be careful, many will instantly dismiss you as a "pro" the second you bash c#/windows as a viable web application setup. wamp is a hugely popular setup, and C# is also very popular and a great php alternative for a lower level programming language if needed. but these are opinions and in no way reflect my, or your ability to make a viable piece of software.

I said i was going to look through the SOURCE of bitcoin qt, the actual source to see how they are doing everything and how it communicates with the network, not the source to learn how to use bitcoind and how to communicate using the json rpc, i can solve that with google if i needed to, i think you misunderstood. i am very familiar with how to use the bitcoind json rpc and i am well versed enough in c to read the source and understand it as it is open source, its just a lot to go through to find how they sign transactions and broadcast them and what protocols they are using.


so before i jump the gun here, can you explain what you mean by code = doc? i dont understand what that is in reference to. it sounds to me like you interpreted that i had an issue understanding bitcoind, and i claimed i would look at the source to figure out how to use it... as i explained already, that is not the case.

and laziness i must disagree with. im asking for pointers on what i should start digging in to, ive been reading for a decade straight, 90% of my job is reading and i am asking for more stuff to read. however, i have more experience with business than bitcoin based programming at this point, i have developed projects that most likely you or somebody you are close to has taken advantage of. but i agree, a team does not mean i have experience. But i am very experienced in many aspects of programming, and i dont beleive lacking knowledge in one field is deserving of a public shaming due to a misunderstanding of the term documentation and source (which i never did confuse to be clear).


after months of researching and learning how to accomplish what i needed, i find all i have learned is no longer viable because the project's scope changed. I also made it very clear i have minimal experience on this level aside from high level APIs/RPCs/libraries (like bitcoind). my reason to give my background was to show i am not looking to be spoon fed info, that i can compensate whoever is willing to help and as an introduction because it is my first post. if i was a 13 year old kid learning to program as a hobby, i would say that in my first post, instead i own a few tech related companies so that is what i put in my first post, maybe i will meet somebody of value here, maybe not.

Also i agree with your comment on linux, its a more secure OS out of the box, it allows more control to the user, it is less liekly to be hacked however a poorly managed linux server is far worse than a poorly managed windows server. and a properly managed windows server is just as capable as you have come to expect from linux/unix distros. however PHP is in no way a better language than c# for MY project (or in general, they are just different, PHP is a scripting language) if its better for the bitcoin side of things, i will use it basedo n the opinions of experienced forum members and my own research, but i doubt that as well. you did not say php though, i am just assuming based on your past posts, maybe you mean python/java. im expecting python/cpp/c# to be a good fit however. i chose c# as this project is very large, its not 100% bitcoin related either, i had other obstacles to overcome.

plus, any real developer can secure a windows server, it takes work but same applies to securing a linux distro.  


I have began coding, and ive made a lot of progress, i feel the wording is fine. bitcoind is not capable of managing enterprise level softwarre, i understand a lot about bitcoind, i have read the docs and im not looking for help on that.

If you truly mean a wallet, look at:
https://github.com/bitcoin/bitcoin/blob/0.15/doc/release-notes.md#multi-wallet-support
But I don't think that is what you want.

If you mean a construct to just group addresses together per user, using a HD wallet (https://en.bitcoin.it/wiki/Deterministic_wallet#Type_2_hierarchical_deterministic_wallet)  per user stored in a DB is how many do it.  Nodes/miners/block chain doesn't know about wallets, at root, just inputs and outputs.

Then talking to bitcoind to communicate is an option.

Btw, going somewhere stating "x is a mess" isn't usually a good way to introduce yourself to get help, imho.  :-)

Wait until v0.15 is released. Bitcoin Core Devs have implemented multiple wallet support.

Refer: https://github.com/bitcoin/bitcoin/blob/0.15/doc/release-notes.md#multi-wallet-support

no - sorry, no. I think this is too easy. Forces me into another comment (rant?  ::)):

Windows 10 comes per default with "calling home" functionlaity, and you can de-activate it. Yes, but that's not my point. Also not the ongoing automatic "security" updates (which proofs, that software wasn't written correctly - and this is from professinal enterprise software developpers). Without wanting to go into real details and the overhowling long history of vulnarabilities in the operating system, you cannot trust closed source systems and drivers. You almost never have the source code to evaluate the possibly of hidden malfunctions. Compare this to Unix or xBSD, deep down into the kernel.  Ahh, the kernel... the windoofs kernel integrates sooooo many functions and drivers, that the layer of rights can be easily violated with a single piece of software, and you don't know it. In Linux and xBSD and OSX the underlying OS has an onion type approach, for segregation of duties (best is OpenBSD IMHO).
Whereas in older times it was annoying, that you got a blue screen, and had to restart your machine, nowadays it became a threat. And this threat became very much clear with the audio driver on HP systems. The audio driver logged each (cool, eh?) keystroke. Well, one might ask, why does an audio driver need to log each keystroke, but that's not the point where I want to go to. Fact is, this happens all the time, and imagine you are typing in your privkey or seed, and it is logged into a data structure, where you don't know what happens with it. Bitcoin is well enough known today, to store enourmos amounts of money... How can someone evaluate a closed source driver update (for sure only for security reasons)?
What I try to say: Windows might be ok, as an operating system for companies and enterprises, cause they can afford to calculate the risk against the necessary security extensions. Same for professional or enterprise devs: how much additional effort do I need to take, to make it secure? This does not prevent developpers from providing bad code (as we have seen with many failing exchanges), due to financial constraints.
My personal view is: to store values as little as your fiat wallet, any wallet/OS like breadwalet or airbitz or even online wallets. I think this is ok. In the northern hermisphere, most of us can carry the risk to loose 50 or 100 Euros/Dollars/Pounds in Bitcoin. But when it comes to values like 1000 or 10.000 ? Then you won't trust blindly the underlying system. You want to take precaucious measurements. And as a developper you would want to provide (hopefully security audited) code to the community - and not a quick and dirty assembled piece of software error prone for the next buffer overflows.

ok, enough ranting around, let's go to work! It is better to start coding, and then improve, as to buid an entry barrier that would make people scary and avoid diving into the crypto space. So remains only my personal view, and is by no means inetnded to be offensive to anyone...
 

CR1776- thank you! it may not be but i will read and let you know! also i agree about the header, but i assure you i am not ignorant to what i was writing, it just did not come out the way i planned after editing my post. it does not come from a place of ignorancet hough, the "mess" was in reference to its current state as it is in a transation period. it is currently in the process of accounting being deprecated, with partially integrated multi-sig and soon-to-be partially implemented multi-wallet support. its only part way done in regards to the RPC. That is my opinion although i agree with your statement 100%, it does not make a good impression even though it was a focal point ofm y message before i changed it around.

HCP- i may wait, i just wonder how long, ill have to look in to it a bit more. thanks for the info!



im all for a good rant, im just hoping the misunderstanding is now clear.

i responded to your points just out of good fun, but it does help me as well since if i should change i will. im playing devils advocate a bit here as i do agree with you, but i still think c# is best for my project and there is no viable way to run c# on a linux os using apache since it can not handle the .net framework. so then what, python+php? c++ even has issues. i just dont know about that.

yo ucan not ignore the extreme lack of EXTENSIVE software developed for linux, the repositories are growing but it is a fraction of what we can get with windows.


I think you confuse anonymity and personal security with software security.

100% agreed,  however i will play devils advocate again for a moment.  you also can not trust open source just becausei t is open source, its all on a per case basis depending on procedures the developers have in place. its a fallacy if you believe open source is secure because many eyes are better than few eyes. we know from experience that users of open source all depend on the next guy to check out and verify it, but most people with the ability to audit a software dont have the time or motivation, so it goes un-audited. truecrypt was open source, it took YEARS to audit and many flaws were found. the only people iwth true motivation to go through the source are hackers, paid auditors (some use them, but not all, but do you really trust them either?), the developers, Gov, and some oddballs that are very interested and bored, or enterprise that really care about security. this leaves a lot of open source projects left un-checked with their source open for all to see... and break.  

 and we are not discussing windows 10 here, we are talking about windows server. different kernal + os all together (slightly, but still different).

and software is never written correctly from day 1, new exploits arise, it usually is not fault of the developers (well it is fault of the developers but it is impossible for projects of that complexity to have all current and potential holes covered. the more complex a system, the more ways it can be broken). it may be fault of a library used or a plugin, it may be the fault of a partner software. look at ubuntu, it is a bloatware filled OS, i personally feel it is more secure than windows 10, but it has the same issues. updates help this, and while an ANONYMITY concern, its not a security concern due to modern day signature verification of the updates.

unless you read and understand the source yourself, and i mean actually read it through, you can trust it about as much as closed source.


I love debian, its my favorite os in conjunction with kde. but that is my personal preference, linux has flaws. its more secure because less hackers understand it, and you have more control over its operations. linux with an uneducated user is as bad as windows. but regardless, how often have you looked through openBSDs source? id bet good money you have not read and understood more than a few pages unless you are involved in the project. you expect others have though, but who? open bsd is the most popular so has the most eyes, but do you trust those eyes?

you say openBSD may have hidden malfunctions, well if that is the case it was not developed correctly from the start (ofcourse) and who will catch it first, an attacker or the good guys?


again we are discussing personal anonymity, not corporate security. and again, this can easily be prevented using proper firewall rules, the logging is bad buit without access to teh network it is useless.

the good thing is, if i dont know what happens to it, neither does an attacker, would i consider hp an attacker? no, a threat yes but to freedom as a whole not the security of my software unless the software was in some way a target of hp or the gov, which at this point is not the case. if hp ever caused financial losses to me or the users, its like a built in insurance policy when they are sued in a class action.

I really am an anonymity nut, but in business i feel windows is a valid option. its very enterprise oriented. C# is an amazing language, i mean its not VB were talking about here. C# can be everybodies best friend. i know real dirt low level languages, im pretty experienced with ASM to a small extent even, i even know very high level like java. c# can be both, sure its .net but it still has c++ functionality. and nobody would call c++ an insecure language.

windows should be avoided at all costs, windows 10 disgusts me with how they treat users as advertising IDs and nothing more, they are a curse on teh technology community and will put an end to interest in IT as a whole. but ill be damned if they didnt produce a great way to develop in depth web applications and software. if only iis and microsoft sql were up to speed. Windows, apache, mysql, c# is my preferred setup now second to lamp.

i dont touch java, never plan to unless persuaded properly.


so we do agree here it seems...



well before you said read before working i think, but as i have said, im already working. im just at a stand still until i decide how to move forward. write my own wallet over a 6 month span, or find an API/RPC/library.

Bitcoinj is on Github, try it. It allows to download both full chain and light chain, has multiformat RPC.

genesis.vision:

i heard of bitcoinj in passing but dismissed it due to teh fact it is java based, i have minimal experience in java although your recommendation makes me think it deserves a second look. i bet i could pick the language up rather easily due to how similar it is to C++. i just worry about it in terms of security, anybody have any thoughts on this? is java the best route when security is top priority?


you say it has an RPC though, can anybody elaborate on this? if its a library how can it have an RPC? i assume by just giving classes that allow for an RPC to easily be created once compiled and ran in java? im asking here just for redundancy, i will dig in to it more, but so far i have found little about anything to do with an RPC.


I will dig in to this a lot more, thanks a bunch for recommendation!

I've seen plenty of companies that are willing to set up nodes and wallets for you, the only problem of course is functionality and customization, very hard to customize a corporation-built wallet with your own functions and qualities, kinda just a copy with them. Your choice what path you choose though!

There's a really good article written way back in 2014 which explains the full low level details with an example for creating and signing a transaction and sending it out to the bitcoin network without a wallet.

It's here 'Bitcoin the hard way' : http://www.righto.com/2014/02/bitcoins-hard-way-using-raw-bitcoin.html

Programs that provide this functionality can be written in any language that supports the cryptographic primitives, these are :

sha256
ripemd160
random numbers to create private keys - always in a crypto lib
secp256k1 elliptic curve operations, namely create pubkey from privkey along with signature creation and validation
base58 encode / decode - you might need to implement this yourself depending on which language you're using - it's not hard to do but requires bignums (large integer) libraries, there will be bignum functions inside the library which does the ECC math as it's all done using mpi/bignums.

Given the above you could write a Bitcoin client from scratch.

I did not expect all these great answers!



samson, you are the man, thank you!


That is what i was looking for, i couldnt find much. im somewhat educated with asymmetrical encryption but im sure i could get the rest taken care of pretty easily. i mean its not like i have to write the encryption algorithims, i just have to find a library with teh functionality built in.

however the random number generator thing seems scary to me. It makes me doubt my ability to keep a system like that secure. writing a non clock-based random numebr generator from scratch can be difficult, and the built in random functions in C are useless when trying to build entropy for a key since they can be broken with ease in most cases.

Regardless im just rambling, that is exactly what i needed! even just your few pointers gave me enough to get started with where to look. I think i will start with attempting to use bitcoinj then if it fails i will "do it the hard way" so to speak.


To the guy above:

I can not use a third party wallet provider, for a business of this scale. i will look in to it as maybe i dont 100% understand what exactly it is they offer, but as yo usay its limited functionality i doubt it would be of any use. i am not looking for a way to host a wallet to take payments and process payments, im looking for a way to have full control over everything from creating multisig individualized wallets, raw transactions, selecting which inputs to output, when to sign, when to broadcast, and so on. i doubt much else other than a library would solve my issue, or a REALLY good RPC (like bitcoind but with a few added features).

but if you think these third party providers could offer me this functionality please let me know! im excited to try anything and everything at this point.

You never have to write your own random number generator, every crypto library I've used (not that many) comes with a secure random number generator built in, they're used to generate keypairs and for ECDSA a random number is required when creating a signature.

In fact the 'normal' method of creating an ECC keypair is to call a single function which first generates the random private key and then calculates the public key and returns both of these at the same time in a 'context'. This is when using mbedTLS (a pretty good library I've found) or LibSodium (note this one does not support secp256k1 aka the bitcoin curve).

The Bitcoin developers put quite an effort into building their own libSecp256k1 which although I've never used should contain everything you need to create keypairs, signatures, etc.

Of course you could use any CSRNG for private key generation and link you project with say mbedTLS, LibSodium or some other crypto library.