RollerBot - Alpha testing begins soon!

[jondecker76]

Daily update, 03-05-2014
Current SVN revision: 522
Number of source code files: 171
Lines of Code: 70,759
Supported currencies: 140

Code:

42,ALF,ALP,AMC,ANC,ARG,ASC,AUR,BCX,BEN,BET,BFC,BOC,BQC,BTB,BTC,BTE,BTG,BTQ,BUK,CACH,CAP,CASH,CAT,CDC,CENT,CGB,CIN,CLR,CMC,CNC,COL,CPR,CRC,CSC,CTM,DBL,DEM,DGC,DMD,DOGE,DRK,DTC,DVC,EAC,ELC,ELP,EMD,EUR,EXC,EZC,FFC,FLO,FRC,FRE,FRK,FST,FTC,GDC,GLC,GLD,GLX,GME,GRW,HBN,HYC,I0C,ICN,IFC,IXC,JKC,KGC,LBW,LEAF,LK7,LKY,LOT,LTC,MAX,MEC,MEM,MEOW,MINT,MMC,MNC,MOON,MST,NAN,NBL,NEC,NET,NIB,NMC,NOBL,NRB,NUC,NVC,NXT,ORB,OSC,PHS,PPC,PTS,PWC,PXC,PYC,Points,QRK,RCH,RDD,REC,RED,RPC,RYC,SBC,SMC,SPT,SRC,STR,SXC,TAG,TEA,TEK,TGC,TIPS,TIX,TRC,UNO,USD,VLC,VTC,WDC,XJO,XNC,XPM,YAC,YBC,ZCC,ZEIT,ZET

Supported markets: 276

Code:

42-BTC,ALF-BTC,ALP-BTC,ALP-LTC,AMC-BTC,AMC-LTC,ANC-BTC,ANC-LTC,ARG-BTC,ARG-LTC,ASC-BTC,ASC-DOGE,ASC-LTC,ASC-XPM,AUR-BTC,AUR-LTC,BCX-BTC,BEN-BTC,BET-BTC,BET-LTC,BFC-BTC,BOC-BTC,BOC-LTC,BQC-BTC,BQC-CNY,BQC-LTC,BTB-BTC,BTB-CNY,BTC-BTC,BTC-CNY,BTC-EUR,BTC-RUR,BTC-USD,BTE-BTC,BTG-BTC,BTQ-CNY,BUK-BTC,CACH-BTC,CAP-BTC,CASH-BTC,CAT-BTC,CAT-DOGE,CAT-LTC,CDC-BTC,CENT-CNY,CENT-LTC,CGB-BTC,CGB-LTC,CIN-BTC,CLR-BTC,CMC-BTC,CMC-CNY,CNC-BTC,CNC-CNY,CNC-LTC,COL-LTC,COL-XPM,CPR-LTC,CRC-BTC,CSC-BTC,CTM-BTC,CTM-DOGE,CTM-LTC,DBL-LTC,DEM-BTC,DEM-LTC,DGC-BTC,DGC-CNY,DGC-LTC,DMD-BTC,DOGE-BTC,DOGE-CNY,DOGE-LTC,DRK-BTC,DTC-BTC,DTC-CNY,DVC-BTC,DVC-CNY,DVC-LTC,DVC-XPM,EAC-BTC,EAC-DOGE,EAC-LTC,ELC-BTC,ELP-BTC,ELP-LTC,EMD-BTC,EUR-BTC,EUR-RUR,EUR-USD,EXC-BTC,EXC-CNY,EZC-BTC,EZC-LTC,FFC-BTC,FFC-DOGE,FFC-LTC,FLO-BTC,FLO-LTC,FRC-BTC,FRC-CNY,FRC-LTC,FRE-BTC,FRK-BTC,FRK-LTC,FST-BTC,FST-LTC,FTC-BTC,FTC-CNY,FTC-LTC,GDC-BTC,GLC-BTC,GLC-LTC,GLD-BTC,GLD-LTC,GLX-BTC,GME-LTC,GRW-BTC,HBN-BTC,HYC-BTC,I0C-BTC,ICN-BTC,ICN-LTC,IFC-BTC,IFC-CNY,IFC-DOGE,IFC-LTC,IFC-XPM,IXC-BTC,JKC-BTC,JKC-LTC,KGC-BTC,KGC-LTC,LBW-BTC,LEAF-BTC,LK7-BTC,LKY-BTC,LOT-BTC,LOT-DOGE,LOT-LTC,LTC-BTC,LTC-CNY,LTC-EUR,LTC-RUR,LTC-USD,MAX-BTC,MAX-CNY,MEC-BTC,MEC-CNY,MEC-LTC,MEM-LTC,MEOW-BTC,MEOW-LTC,MINT-BTC,MINT-CNY,MMC-BTC,MMC-CNY,MNC-BTC,MOON-BTC,MOON-DOGE,MOON-LTC,MST-LTC,NAN-BTC,NBL-BTC,NEC-BTC,NET-BTC,NET-CNY,NET-LTC,NET-XPM,NIB-BTC,NMC-BTC,NMC-CNY,NMC-LTC,NMC-USD,NOBL-BTC,NOBL-DOGE,NOBL-LTC,NRB-BTC,NUC-BTC,NVC-BTC,NVC-USD,NXT-BTC,NXT-CNY,NXT-LTC,ORB-BTC,ORB-LTC,OSC-BTC,PHS-BTC,PPC-BTC,PPC-CNY,PPC-LTC,PPC-USD,PPC-XPM,PTS-BTC,PTS-CNY,PWC-BTC,PXC-BTC,PXC-LTC,PYC-BTC,Points-BTC,QRK-BTC,QRK-CNY,QRK-LTC,QRK-XPM,RCH-BTC,RCH-LTC,RDD-BTC,RDD-LTC,REC-BTC,REC-LTC,RED-BTC,RED-CNY,RED-LTC,RPC-BTC,RYC-LTC,SBC-BTC,SBC-LTC,SMC-BTC,SMC-DOGE,SMC-LTC,SPT-BTC,SRC-BTC,SRC-CNY,STR-BTC,SXC-BTC,SXC-DOGE,SXC-LTC,TAG-BTC,TAG-CNY,TAG-DOGE,TAG-LTC,TEA-BTC,TEA-DOGE,TEA-LTC,TEK-BTC,TGC-BTC,TGC-LTC,TIPS-CNY,TIPS-LTC,TIX-CNY,TIX-LTC,TIX-XPM,TRC-BTC,UNO-BTC,UNO-DOGE,USD-BTC,USD-RUR,VLC-BTC,VLC-LTC,VTC-BTC,VTC-CNY,VTC-DOGE,VTC-LTC,WDC-BTC,WDC-CNY,WDC-DOGE,WDC-LTC,XJO-BTC,XNC-BTC,XNC-LTC,XPM-BTC,XPM-CNY,XPM-LTC,YAC-BTC,YAC-CNY,YAC-LTC,YBC-BTC,ZCC-BTC,ZCC-CNY,ZEIT-BTC,ZEIT-DOGE,ZEIT-LTC,ZET-BTC,ZET-CNY,ZET-DOGE,ZET-LTC

Not a bad day again (feeling quite a bit better finally).  Mostly just work on verification testing  Here is the current status of the verifications:
Cryptsy: Fully tested
BTC-e: Fully tested
BTer: Fully tested
Vircurex: All good except for an issue getting trades. See: https://bitcointalk.org/index.php?topic=49383.msg5527803#msg5527803.  Hopefully they fix it soon.
Coins-e:  All good except for a small problem with their listorders API method.  I have reported the bug to them.
Coinex: Fully tested
Bittrex: All good except for a small bug I found with their market/getopenorders method.  It has been reported as well. Edit: Heard back from them, it will be fixed within a day!

Moving forward I'm going to put some energy into the website and user registration and work on getting the frontend services running

[ManeBjorn]

That is allot of exchanges I need to join. 
Good to hear you are feeling better.

[newquestions]

I'm ready to help you test. Please let me know when to start ...

[shaun2007182]

i am happy to see your feeling better.  I'm feeling pretty awful myself today.   Have you given me a virus lol. 

[bitcointodayand]

I am very happy to test it. Do you have any performance stats so far? This looks like an incredible system.

[jondecker76]

I am very happy to test it. Do you have any performance stats so far? This looks like an incredible system.

I did a lot of benchmarking of the system when I had it running as a personal project, and the results were astounding (making decisions at 10 second resolution across 6 exchanges at 5-10 markets per exchange with very little processor usage).  Things have changed considerably though now that I have started making the platform scalable.  For example, database queries may be a bit slower due to higher load from having multiple users, while at the same time there will be a nice speed-up from having common API calls shared among multiple users.  With that in mind, I can't really predict yet what the performance gain/hit will be from the recent changes until I get things back to a state where I can do new benchmarks.  I do however expect the performance to be on par with what I experienced running it as a stand alone system for myself, and early tests so far seem to support this as well.

The next couple weeks will tell a lot now that the backend system is running and I can see how the newly separated frontend services run.  I'll definitely be sharing my results here either way as I progress

[jondecker76]

Not a lot to report today

I'm still waiting to hear back from some of the exchanges in which I've identified bugs in their API (currently I've hear back from Bittrex with indication of a fix on the way, and ANX (anxbtc/anxpro) with interest in working on solutions).  Therefore there are no new significant changes for today.  I've also contacted some other potential industry partners in order to open clear lines of communication.

Tomorrow I will mostly work on web frontend stuff, and will report back here tomorrow with my progress

[jondecker76]

Today I'm mainly going to be working in Amazon AWS console getting things organized (VPC, elastic IP addresses, load balancers, etc), then work on the website starts.  I'll update the thread later with how things go.

[jondecker76]

Today was a great day.  A lot of progress is being made on final decisions when it comes to the actual specifics behind RollerBot deployment.  I've brought aboard someone with extensive knowledge in system administration and deployment, especially when it comes to AWS.  This weekend will be a pivotal point for RollerBot as the full groundwork is put into place that will empower the scalable architecture that I dreamed for this platform.  There should be some exciting news in the coming weeks as things are nearing being ready for alpha testing.

Stay tuned

[ManeBjorn]

I am glad it is going so well.
Also glad to hear you are feeling better.
Sounds like you are really rolling along.

Today was a great day.  A lot of progress is being made on final decisions when it comes to the actual specifics behind RollerBot deployment.  I've brought aboard someone with extensive knowledge in system administration and deployment, especially when it comes to AWS.  This weekend will be a pivotal point for RollerBot as the full groundwork is put into place that will empower the scalable architecture that I dreamed for this platform.  There should be some exciting news in the coming weeks as things are nearing being ready for alpha testing.

Stay tuned

[ninjaboon]

Not a lot to report today

I'm still waiting to hear back from some of the exchanges in which I've identified bugs in their API (currently I've hear back from Bittrex with indication of a fix on the way, and ANX (anxbtc/anxpro) with interest in working on solutions).  Therefore there are no new significant changes for today.  I've also contacted some other potential industry partners in order to open clear lines of communication.

Tomorrow I will mostly work on web frontend stuff, and will report back here tomorrow with my progress

I know the ANX team personally. Buzz me if they don't get back to you.

[jondecker76]

Not a lot to report today

I'm still waiting to hear back from some of the exchanges in which I've identified bugs in their API (currently I've hear back from Bittrex with indication of a fix on the way, and ANX (anxbtc/anxpro) with interest in working on solutions).  Therefore there are no new significant changes for today.  I've also contacted some other potential industry partners in order to open clear lines of communication.

Tomorrow I will mostly work on web frontend stuff, and will report back here tomorrow with my progress

I know the ANX team personally. Buzz me if they don't get back to you.

I've hear back from them once and sent another email.  So far they have been great to work with

[jondecker76]

3-8-2014

No code changes today.  All of today was spend in AWS with configuration.
I've recruited the help of an absolutely awesome system administrator to help guide RollerBot in the best direction possible when it comes to the underlying infrastructure.  Most of today was spent discussing future direction and getting things set up to provide the scalability and reliability needed for the trading platform.  A lot of discussion revolved around security and many things were put in place today to ensure a secure platform to build on.  I wish I could share more, but this is leading to the possibility of an exciting announcement in the near future.

I'll have more updates tomorrow

[BenAnh]

I'm interested to test it as I have many alt-coins as well as an active trader on many exchanges plus an linux administrator. So this fits very well. I've use AWS and Windows Azure somewhat, just let me know if you need anything.

[jondecker76]

Just wanted to drop in with some progress updates.

I didn't get as much done as I wanted to over the weekend.  Most time was spent on getting the rest of the infrastructure set up on AWS, which included making some important decisions that took some time to thoroughly evaluate.  I'm not going to release too many details, but as of right now RollerBot consists of 5 ec2 server instances (with more to come) and the important backend services are islanded and isolated from the Internet.  The only server attached to the Internet is the web server, which it's self doesn't even have database access.  Many steps have been taken and decisions made to ensure a secure platform.  Now with that out of the way, I can continue marching forward with the code.

Code-wise, I spent a lot of time yesterday cleaning out the source tree of old/antiquated files and making an effort to improve the documentation (through phpdoc). 
Today will be spend finishing off some of the backend services (daemon spawning and load balancing) and then working on some of the user services (private API call daemons, etc)

Thank you everyone for your continued interest and support.  Things are starting to get exciting as everything is slowly starting to pull together

[jondecker76]

Killer day today
I did a lot of work tightening up the backend code.  The cache daemons are running fast and smooth, and the backend plugin system is fired back up (just storing trade data ATM)
Made some changes that reduces the number of MySql calls per second dramatically as well.  Letting it run all night to see how things do

[jondecker76]

3-12-14 Swinging in for a quick update

Getting the infrastructure set up has turned out to be a much larger task than I expected.  Almost the entire last week has been spent testing configurations, running load tests, reconfiguring, testing again, etc.
I'm happy to say that I have finally found the mix of performance, redundancy and cost needed to efficiently and effectively run the platform.  I would even venture to say that I've spent more in server infrastructure so far in development than most (if not all) bots spend in a year.  I'm not quite sure anyone really understands the amount of power it takes to run such a large trading platform (currently 8 servers in total to get through development and alpha testing, and even more when it comes to full release).  I'm beginning to see why something of this scale has not been done yet!

With all that said, things are still moving forward, albeit a bit slower than I originally expected.  Due to costs, RollerBot will launch supporting 10 exchanges.  As a userbase builds, I'll be able to afford more infrastructure to enable support for even more exchanges (in blocks of 10, to keep costs reasonable).

I'll be able to get back into the programming that needs done over the next couple of days now that the final layout for the infrastructure is now set!

[coingun]

Great update! I was very curious how much processing power would be required. Sounds like you need an infrastructure guy to worry about all the easy stuff so you can get back to the coding and development. If you have any immediate needs or desires you should outline them. Perhaps donations to fund more development servers? Would you be interested in having help beyond all us pestering you with "let me be a part of the alpha test team so i can make easy money with all your hard work?"

[jondecker76]

Great update! I was very curious how much processing power would be required. Sounds like you need an infrastructure guy to worry about all the easy stuff so you can get back to the coding and development. If you have any immediate needs or desires you should outline them. Perhaps donations to fund more development servers? Would you be interested in having help beyond all us pestering you with "let me be a part of the alpha test team so i can make easy money with all your hard work?"

I've actually been asked about donations a few times now.  I honestly didn't think there would be anyone willing to donate..  I'll get a donation address set up for those interested and post back here.

Regarding infrastructure - I have found at least part-time help with this recently and it has been a tremendous help.  We're actually in talks about the legalities and specifics about retaining him - If we work something out I'll make an announcement.

Regarding other help, there may be a possibility to bring in some limited PHP help just for supporting new exchanges (writing the exchange class files, most of it would be copy and paste)- though to expose even this little bit of the source code to an individual would require some good references and standing in the community.

Another area of help would be someone fluent in the Chineese language (or any foreign language where exchange platforms exist) to help me get accounts set up at these exchanges and do some test trades etc.

Thanks for the great Ideas!

[jondecker76]

Updated my signature with a BTC donation address.

Thanks for the suggestions guys