Bitcoin Forum
November 20, 2017, 09:43:26 AM *
News: Latest stable version of Bitcoin Core: 0.15.1  [Torrent].
 
   Home   Help Search Donate Login Register  
Pages: « 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 [27] 28 29 »
  Print  
Author Topic: Isis ATP [Automated Trading Platform] - Discussion  (Read 44616 times)
Nova!
Full Member
***
Offline Offline

Activity: 140


View Profile
May 18, 2013, 03:18:12 PM
 #521

Talking to a friend of mine who performs security audits on enterprise systems, I've found that this can also be set by the encryption policy.  There is a java setting somewhere to tell java to accept invalid certs (it's a VM flag or something).  So maybe that will resolve the issue.  If not you will need to install the cert and we will have to keep this problem in mind should we ever choose to roll an installer.

Donate @ 1LE4D5ERPZ4tumNoYe5GMeB5p9CZ1xKb4V
Join ICO Now A blockchain platform for effective freelancing
Advertised sites are not endorsed by the Bitcoin Forum. They may be unsafe, untrustworthy, or illegal in your jurisdiction. Advertise here.
Nova!
Full Member
***
Offline Offline

Activity: 140


View Profile
May 18, 2013, 04:02:29 PM
 #522

There seems to be a huge amount of opportunity in JPY/GBP according to the arbitrage engine right now.
It's been alerting me for 5 hours about a chance for 150% profits if only I had some yen in MtGox.
Just to test it I sold 1BTC into Yen and restarted the engine, then waited. 
10 seconds later I had a bunch of GBP which amounted to $150 or so.(pulled the USD/GBP rate from my forex account to do the math)

Have to be careful with the arbitrage engine though. 
I wrote the dang thing and I still don't trust it. 
Among other things, it doesn't care about market depth and just tries to buy the whole dang market out, immediately.
It's the reason for a solid month I owned all the Rubles on MtGox and couldn't get rid of them, I had to sell my rubles at a rather significant loss in the end.

Anyways, if anyone has some Yen in MtGox, the arbitrage engine will turn that into a nice profit right now assuming you have the ability to do something with GBP later.


Donate @ 1LE4D5ERPZ4tumNoYe5GMeB5p9CZ1xKb4V
Nova!
Full Member
***
Offline Offline

Activity: 140


View Profile
May 18, 2013, 04:28:14 PM
 #523

I noticed that once the arbitrage engine fired and ran into a condition where I didn't have money in the from-currency wallet, that trend trading would continue on as normal but trend-trades would no longer show up in my account history, even though I could see the little engine vigorously trying to make me some money.

I've solved the problem.  There is an exception thrown in the ArbitrageEngine that is being caught, but there is also a boolean set/unset that wraps the call to execute.  This boolean disables trend trading temporarily, but with the exception being thrown before it has a chance to re-enable, the trend trading was being shut off and never turned back on.

The solution was to move the code to re-enable trendtrading to a finally block, since those are called whether or not an exception is thrown.

We should probably make sure that if we are doing a try / catch wrapping functionality that will disable something temporarily, that we either move the whole thing outside the try catch block, or at least place the re-enabling functionality into a finally block.  This should be our coding standard for the future.  

Most likely the cock-up was on my end.  I'm just surprised it went this long without being found out or noticed.  That probably translates to a lot of money that was left on the table and it makes me sad to know everyone would have missed out on it because of such a simple mistake.

I'm committing all of these changes to the OpenPay repo under the ATP directory once I can see this thing run for 24hrs.

Donate @ 1LE4D5ERPZ4tumNoYe5GMeB5p9CZ1xKb4V
tunafish
Member
**
Offline Offline

Activity: 95


View Profile WWW
May 18, 2013, 10:09:25 PM
 #524

Lovely...

Code:
2013-05-18 03:05:05,579 ←[31m:←[0;39m WARNING: Testing connection to exchange Mt
Gox
Exception in thread "Thread-1" si.mazi.rescu.HttpException: Problem POSTing (IO)

        at si.mazi.rescu.HttpTemplate.executeRequest(HttpTemplate.java:194)
        at si.mazi.rescu.HttpTemplate.executeRequest(HttpTemplate.java:93)
        at si.mazi.rescu.RestInvocationHandler.invokeHttp(RestInvocationHandler.
java:59)
        at si.mazi.rescu.RestInvocationHandler.invoke(RestInvocationHandler.java
:54)
        at com.sun.proxy.$Proxy6.getAccountInfo(Unknown Source)
        at com.xeiam.xchange.mtgox.v1.service.account.MtGoxPollingAccountService
.getAccountInfo(MtGoxPollingAccountService.java:75)
        at org.aido.atp.AccountManager.refreshAccounts(AccountManager.java:121)
        at org.aido.atp.AccountManager.run(AccountManager.java:101)
        at java.lang.Thread.run(Unknown Source)
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.Validator
Exception: PKIX path building failed: sun.security.provider.certpath.SunCertPath
BuilderException: unable to find valid certification path to requested target
        at sun.security.ssl.Alerts.getSSLException(Unknown Source)

I'm guessing gox is using a self signed cert on their SSL connection??

I've investigated this further and found that it's because mtgox cheaped out and started using a cert from a company called startcom.
Java does not recognize this company as a valid certificate authority and thus does not recognize the certificate as being valid.
You will need to visit https://data.mtgox.com/api/2 (you will get an error on the page but just ignore it).
You will need to download the cert and manually install it. You can do this in most cases by clicking the "lock" icon in your browser and going to "export".

I don't think there is a way to automate this particular process.  If someone comes up with one please let me know.
 

I'm also stuck at this point:
"Impl.checkServerTrusted(Unknown Sour......................................
..........Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to
 find valid certification path to requested target
        at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(Unknown
 Source)
        at java.security.cert.CertPathBuilder.build(Unknown Source)
        ... 27 more"


I've exported the cert from mtgox but I dunno what to do with it or how to install it.
I used keytool as outlined here (http://docs.oracle.com/javase/1.4.2/docs/tooldocs/windows/keytool.html?) to try and load it but to no avail...When I exported it it was saved as "www.mtgox.com" which i renamed as mtgox.cer and used "java keytool -import -alias gox -file mtgox.cer" but i get "Error: could not find or load main class". I've been doing everything from a command prompt in win7, should I be using some java distro or something instead?

can anybody detail exactly what cert(s) are needed? daybyter was nice enough to link me some guides that explained the installation process but I'm still not clear on the process. If anybody could share an example of the code needed w/ one of the certs it'd be greatly appreciated. I'm really not at all familiar with java but if I have the codes/certs i think i can follow easily enough. I tried saving: http://code.google.com/p/java-use-examples/source/browse/trunk/src/com/aw/ad/util/InstallCert.java as a .jar and running it from command prompt but clearly I don't know what the hell i'm doing Cheesy

I haven't tried using btc-e yet but I imagine I'll run into the same problem right? Are there any other specific certs I need for btc-e?


I know i've asked alot of questions but if anybody can even answer one of them i'd really appreciate it, thanks.

Feeling generous?
BTC: 1MiaKvvzhuCrbWJ2iXA9RceZfRpNDjwJTk
Aido
Sr. Member
****
Offline Offline

Activity: 249


1. Collect underpants 2. ? 3. Profit


View Profile
May 18, 2013, 10:15:50 PM
 #525

Most likely the cock-up was on my end.  I'm just surprised it went this long without being found out or noticed.

I will take some "credit" for this cock-up. I messed around with Nova's arbitrage code and how it was disabling the trend trading. The reason the problem wasn't caught earlier was because, like Nova, I didn't fully trust the arbitrage engine so mostly disabled it when running live.

I had given the arbitrage code a total overhaul and implemented a Bellman-Ford algorithm to calculate opportunities. From my brief testing, it found the same opportunities as the original code so I re-implemented the original again.

Interesting Bash command line, try it Wink:
bitcoin-cli sendtoaddress 1Aidan4r4rqoCBprfp2dVZeYosZ5ryVqH6 `bitcoin-cli getbalance`
Nova!
Full Member
***
Offline Offline

Activity: 140


View Profile
May 19, 2013, 01:47:22 AM
 #526

Lovely...

Code:
2013-05-18 03:05:05,579 ←[31m:←[0;39m WARNING: Testing connection to exchange Mt
Gox
Exception in thread "Thread-1" si.mazi.rescu.HttpException: Problem POSTing (IO)

        at si.mazi.rescu.HttpTemplate.executeRequest(HttpTemplate.java:194)
        at si.mazi.rescu.HttpTemplate.executeRequest(HttpTemplate.java:93)
        at si.mazi.rescu.RestInvocationHandler.invokeHttp(RestInvocationHandler.
java:59)
        at si.mazi.rescu.RestInvocationHandler.invoke(RestInvocationHandler.java
:54)
        at com.sun.proxy.$Proxy6.getAccountInfo(Unknown Source)
        at com.xeiam.xchange.mtgox.v1.service.account.MtGoxPollingAccountService
.getAccountInfo(MtGoxPollingAccountService.java:75)
        at org.aido.atp.AccountManager.refreshAccounts(AccountManager.java:121)
        at org.aido.atp.AccountManager.run(AccountManager.java:101)
        at java.lang.Thread.run(Unknown Source)
Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.Validator
Exception: PKIX path building failed: sun.security.provider.certpath.SunCertPath
BuilderException: unable to find valid certification path to requested target
        at sun.security.ssl.Alerts.getSSLException(Unknown Source)

I'm guessing gox is using a self signed cert on their SSL connection??

I've investigated this further and found that it's because mtgox cheaped out and started using a cert from a company called startcom.
Java does not recognize this company as a valid certificate authority and thus does not recognize the certificate as being valid.
You will need to visit https://data.mtgox.com/api/2 (you will get an error on the page but just ignore it).
You will need to download the cert and manually install it. You can do this in most cases by clicking the "lock" icon in your browser and going to "export".

I don't think there is a way to automate this particular process.  If someone comes up with one please let me know.
 

I'm also stuck at this point:
"Impl.checkServerTrusted(Unknown Sour......................................
..........Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to
 find valid certification path to requested target
        at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(Unknown
 Source)
        at java.security.cert.CertPathBuilder.build(Unknown Source)
        ... 27 more"


I've exported the cert from mtgox but I dunno what to do with it or how to install it.
I used keytool as outlined here (http://docs.oracle.com/javase/1.4.2/docs/tooldocs/windows/keytool.html?) to try and load it but to no avail...When I exported it it was saved as "www.mtgox.com" which i renamed as mtgox.cer and used "java keytool -import -alias gox -file mtgox.cer" but i get "Error: could not find or load main class". I've been doing everything from a command prompt in win7, should I be using some java distro or something instead?

can anybody detail exactly what cert(s) are needed? daybyter was nice enough to link me some guides that explained the installation process but I'm still not clear on the process. If anybody could share an example of the code needed w/ one of the certs it'd be greatly appreciated. I'm really not at all familiar with java but if I have the codes/certs i think i can follow easily enough. I tried saving: http://code.google.com/p/java-use-examples/source/browse/trunk/src/com/aw/ad/util/InstallCert.java as a .jar and running it from command prompt but clearly I don't know what the hell i'm doing Cheesy

I haven't tried using btc-e yet but I imagine I'll run into the same problem right? Are there any other specific certs I need for btc-e?


I know i've asked alot of questions but if anybody can even answer one of them i'd really appreciate it, thanks.


I've been up all night babysitting the ATP.  Gonna push a commit in the morning, lots of little changes things I'm surprised I didn't notice before mostly related to the arbi engine.  I'm going to gut the code to move us off market orders and instead use the orderbook and limit orders to do the trading.  It will be more cross exchange friendly and frankly I'm down $20 on the conservative algo and $10 on the risky algo??  Arbitrage made a bit, but then got to smoking something and started buying zlotkies and things that have no liquidity.  It's going to need a rethink too.

btc-e uses thawte they're recognized so no worries there.  google how to install a new ca chain

Donate @ 1LE4D5ERPZ4tumNoYe5GMeB5p9CZ1xKb4V
tunafish
Member
**
Offline Offline

Activity: 95


View Profile WWW
May 19, 2013, 02:38:28 AM
 #527



I've exported the cert from mtgox but I dunno what to do with it or how to install it.
I used keytool as outlined here (http://docs.oracle.com/javase/1.4.2/docs/tooldocs/windows/keytool.html?) to try and load it but to no avail...When I exported it it was saved as "www.mtgox.com" which i renamed as mtgox.cer and used "java keytool -import -alias gox -file mtgox.cer" but i get "Error: could not find or load main class". I've been doing everything from a command prompt in win7, should I be using some java distro or something instead?

can anybody detail exactly what cert(s) are needed? daybyter was nice enough to link me some guides that explained the installation process but I'm still not clear on the process. If anybody could share an example of the code needed w/ one of the certs it'd be greatly appreciated. I'm really not at all familiar with java but if I have the codes/certs i think i can follow easily enough. I tried saving: http://code.google.com/p/java-use-examples/source/browse/trunk/src/com/aw/ad/util/InstallCert.java as a .jar and running it from command prompt but clearly I don't know what the hell i'm doing Cheesy

I haven't tried using btc-e yet but I imagine I'll run into the same problem right? Are there any other specific certs I need for btc-e?


I know i've asked alot of questions but if anybody can even answer one of them i'd really appreciate it, thanks.


I've been up all night babysitting the ATP.  Gonna push a commit in the morning, lots of little changes things I'm surprised I didn't notice before mostly related to the arbi engine.  I'm going to gut the code to move us off market orders and instead use the orderbook and limit orders to do the trading.  It will be more cross exchange friendly and frankly I'm down $20 on the conservative algo and $10 on the risky algo??  Arbitrage made a bit, but then got to smoking something and started buying zlotkies and things that have no liquidity.  It's going to need a rethink too.

btc-e uses thawte they're recognized so no worries there.  google how to install a new ca chain

wooo! wish i tried btc-e sooner on it. I'd still like to get gox going so I can take advantage of arbitrage (guess i could use bitstamp or something but gox generally seems to be highest). I'm sure i'll figure out the certs sooner or later, i think I have most of it figured out---to anyone else stuck on this, I found this link to be the most useful so far: http://www.opentox.org/tutorials/q-edit/how-to-install-ssl-certificates
Not sure if I'll have to install a cert for each error message I got or if I just need a single one from gox but I think I'll figure it out. Thanks to those who've offered help.


PS
Perhaps this thread should be restarted as AidoATP? I've gone through all 30 pages here way too many times for bits and pieces of information. Just a thought since the platform now seems like its getting off the ground all and all and now discussions seem to be more regarding the fine tuning seems more to be logic oriented than structural.

Feeling generous?
BTC: 1MiaKvvzhuCrbWJ2iXA9RceZfRpNDjwJTk
Nova!
Full Member
***
Offline Offline

Activity: 140


View Profile
May 19, 2013, 04:20:28 AM
 #528



I've exported the cert from mtgox but I dunno what to do with it or how to install it.
I used keytool as outlined here (http://docs.oracle.com/javase/1.4.2/docs/tooldocs/windows/keytool.html?) to try and load it but to no avail...When I exported it it was saved as "www.mtgox.com" which i renamed as mtgox.cer and used "java keytool -import -alias gox -file mtgox.cer" but i get "Error: could not find or load main class". I've been doing everything from a command prompt in win7, should I be using some java distro or something instead?

can anybody detail exactly what cert(s) are needed? daybyter was nice enough to link me some guides that explained the installation process but I'm still not clear on the process. If anybody could share an example of the code needed w/ one of the certs it'd be greatly appreciated. I'm really not at all familiar with java but if I have the codes/certs i think i can follow easily enough. I tried saving: http://code.google.com/p/java-use-examples/source/browse/trunk/src/com/aw/ad/util/InstallCert.java as a .jar and running it from command prompt but clearly I don't know what the hell i'm doing Cheesy

I haven't tried using btc-e yet but I imagine I'll run into the same problem right? Are there any other specific certs I need for btc-e?


I know i've asked alot of questions but if anybody can even answer one of them i'd really appreciate it, thanks.


I've been up all night babysitting the ATP.  Gonna push a commit in the morning, lots of little changes things I'm surprised I didn't notice before mostly related to the arbi engine.  I'm going to gut the code to move us off market orders and instead use the orderbook and limit orders to do the trading.  It will be more cross exchange friendly and frankly I'm down $20 on the conservative algo and $10 on the risky algo??  Arbitrage made a bit, but then got to smoking something and started buying zlotkies and things that have no liquidity.  It's going to need a rethink too.

btc-e uses thawte they're recognized so no worries there.  google how to install a new ca chain

wooo! wish i tried btc-e sooner on it. I'd still like to get gox going so I can take advantage of arbitrage (guess i could use bitstamp or something but gox generally seems to be highest). I'm sure i'll figure out the certs sooner or later, i think I have most of it figured out---to anyone else stuck on this, I found this link to be the most useful so far: http://www.opentox.org/tutorials/q-edit/how-to-install-ssl-certificates
Not sure if I'll have to install a cert for each error message I got or if I just need a single one from gox but I think I'll figure it out. Thanks to those who've offered help.


PS
Perhaps this thread should be restarted as AidoATP? I've gone through all 30 pages here way too many times for bits and pieces of information. Just a thought since the platform now seems like its getting off the ground all and all and now discussions seem to be more regarding the fine tuning seems more to be logic oriented than structural.

I'm thinking this product is going to need it's own website in the near future.  The OpenPay version is going to be called just ATP, not to cut Aido out but to distinguish it so people don't get confused.  I plan to take it to the point it works reliably and then open it up to the broader community.  Fortunately I'm now at a spot in life again where I can do that.

Now as to the cert.  If your cert came down as www.mtgox.com you grabbed the wrong cert.  You need to grab the cert from the TOP of the certificate authority chain.  The exact directions I followed myself are here... http://codebistro.com/2010/03/25/adding-cacert-to-the-java-trusted-store

For firefox you click on the lock icon, go to "view certificate", click "details".  Then you will see the ca cert at the top of the list that's the one you want it won't be MtGox it will be StartCom or whatever.  Select that one then click export.  Then follow the directions for your platform for installing a new Certificate Authority cert.  In windows remember that you need to do this from a command line and you need to do it with a command shell elevated to administrator (type cmd, right click select "run as administrator").  For linux you will need su or sudo and I'm guessing it's similar for OSX.

You'll know you made a mistake if you get a "file access error" from the keytool command.

Sorry it's so complicated.  We may be able to package the CA cert as a resource and just have a "first run" state that must be run as root for setting this part up.  Again I'm only doing that if we decide to roll a proper installer.

BTW Aido, that section that was messed up is not your fault, it's mine.  I recognize my own code style there, I'm just shaking my head in disbelief that I didn't notice it earlier.  I know better and I would have caught something like that in a code review.
It would be a very good idea as we refactor to create a series of unit tests to validate the complete state of the application.  Also things like trade blocking should be set at the application level from here on out and the application should be refactored to be a bit more like a state machine.  I'm working out in my head the changes I plan to make to this, but it will happen in phases.  There won't be any instances where we throw the baby out with the bathwater anymore.

Ok, so back to the new design.  I think the idea of programmable logic is an admirable idea, but when someone is just setting up for the first time, there needs to be a logical set of defaults included.  In my fork, I've set the default logic to be VWAPCross, not because I believe it is in any way inherently better, but because it's the simplest and easiest to understand.  Last trade was below vwap we buy, above vwap we sell.  As it stands now if you don't put anything into the bid/ask logic fields it spits out a nasty stack trace and does nothing.  I've fixed that in my branch, but it's refusing to let me commit at the moment.  I'm sort of at a loss on that one and will try to massage it a bit tomorrow.

After we refactor to use the order book and limit orders, the next step will be to add in a proper AI.  Right now, all we are using are algorithmic traders.  The AI will be optional and until it's trained with a few weeks of data it runs a very real risk of making very expensive mistakes.  However it's "learning profile" is a stored data set that can be downloaded and updated regularly with new market information.

Because it will attempt to predict the market direction, it will necessarily be significantly different than the algo traders we have now.  

The AI is a predictive DSP with a genetic algorithm based on a project I wrote a few years back for a company that couldn't afford downtime (downtime meant lives lost not just dollars).  The AI would watch all systems globally and learn about which systems were failing.  At first it just watches everything and reports failures, then it begins to make predictions and give each prediction a weight based on if it comes true or not.  Eventually the strongest predictions survive and the weakest ones die out.
The goal here was to compile an alert of probabilities and send it as far in advance as possible.  
The longer it runs the smarter it gets.  With a failure rate of 1 machine per day it could predict up to 4 hours in advance with 90% certainty.

I've been running it against MtGox since I first created the Isis-ATP project and it's a bit novel in that it can deduce the lump sum direction of the algorithmic traders at play (after all that's what a DSP does it isolates signals from noise).

Since January it's been showing that the pricing structure is actually being supported by VWAPCross and someone somewhere appears to be running a stochastic oscillator and that someone has some serious dollars.  That's pretty much all the bots and it accounts for over 80% of the trading activity.  The remaining is human traders (random noise tends to be from human activity).  

This is MtGox data only.  I haven't tried it against any other exchange and the DDOS attacks a few weeks ago gave the poor thing brain damage for lack of a better word. I'm not sure how to reconcile the dataset so I'll start a new profile when I integrate it.

I will also be adding an internal wallet and a configurable "sweep" to allow you to get your profits out daily and still keep enough on hand for expected volume.

Finally I will also be adding peer to peer mode to allow nodes to specialize in exchanges and share resources.

If that doesn't make a lot of sense imagine this...
There is a significant arbitrage opportunity between btc-e & mtgox on BTC right now.  You can buy bitcoins cheaper on BTC-e and sell them for more on MtGox (feel free to check this).

Each node can be configured to specialize in a particular exchange and this is where the bulk of it's money is at.  

If we can predict the size and length of an arbitrage opportunity we can do the buying and selling individually and settle up the profits later.  This obviously only works if you are working with close friends and people you know.  But the fact is this can work in the right circumstances.

Imagine if I had $5000 USD on BTC-e and Aido (sorry you're volunteered), has a few hundred BTC on MtGox.  I can authorize my machine to spend the $5000 USD into BTC on BTC-e and Aido can authorize his machine to spend the BTC on MtGox into USD.  I am effectively lending him my USD while he in turn is loaning me his BTC.

Once the arbitrage window is over (or we run out of cash), then we settle and split the profits.

Obviously this system would require trust which is scare in the bitcoin community, but if the transactions were small enough and there were enough people running it and we added a "confidence" ranking of some sort based on previous interactions, this thing has real potential to make serious money for anyone who cares to participate while not requiring everyone to have a hundred thousand accounts everywhere to try and keep track of.  

It's optional and will be default disabled, but just be aware this is one really easy way to run arbitrage without so much risk from window expiry.
Hope you guys find it exciting.  There is quite a bit of work to do here, but I think it will all be worth it in the end.

Donate @ 1LE4D5ERPZ4tumNoYe5GMeB5p9CZ1xKb4V
tunafish
Member
**
Offline Offline

Activity: 95


View Profile WWW
May 19, 2013, 05:27:33 AM
 #529





I'm thinking this product is going to need it's own website in the near future.  The OpenPay version is going to be called just ATP, not to cut Aido out but to distinguish it so people don't get confused.  I plan to take it to the point it works reliably and then open it up to the broader community.  Fortunately I'm now at a spot in life again where I can do that.

Now as to the cert.  If your cert came down as www.mtgox.com you grabbed the wrong cert.  You need to grab the cert from the TOP of the certificate authority chain.  The exact directions I followed myself are here... http://codebistro.com/2010/03/25/adding-cacert-to-the-java-trusted-store

For firefox you click on the lock icon, go to "view certificate", click "details".  Then you will see the ca cert at the top of the list that's the one you want it won't be MtGox it will be StartCom or whatever.  Select that one then click export.  Then follow the directions for your platform for installing a new Certificate Authority cert.  In windows remember that you need to do this from a command line and you need to do it with a command shell elevated to administrator (type cmd, right click select "run as administrator").  For linux you will need su or sudo and I'm guessing it's similar for OSX.

You'll know you made a mistake if you get a "file access error" from the keytool command.

Sorry it's so complicated.  We may be able to package the CA cert as a resource and just have a "first run" state that must be run as root for setting this part up.  Again I'm only doing that if we decide to roll a proper installer.

BTW Aido, that section that was messed up is not your fault, it's mine.  I recognize my own code style there, I'm just shaking my head in disbelief that I didn't notice it earlier.  I know better and I would have caught something like that in a code review.
It would be a very good idea as we refactor to create a series of unit tests to validate the complete state of the application.  Also things like trade blocking should be set at the application level from here on out and the application should be refactored to be a bit more like a state machine.  I'm working out in my head the changes I plan to make to this, but it will happen in phases.  There won't be any instances where we through the baby out with the bathwater anymore.

Ok, so back to the new design.  I think the idea of programmable logic is an admirable idea, but when someone is just setting up for the first time, there needs to be a logical set of defaults included.  In my fork, I've set the default logic to be VWAPCross, not because I believe it is in any way inherently better, but because it's the simplest and easiest to understand.  Last trade was below vwap we buy, above vwap we sell.  As it stands now if you don't put anything into the bid/ask logic fields it spits out a nasty stack trace and does nothing.  I've fixed that in my branch, but it's refusing to let me commit at the moment.  I'm sort of at a loss on that one and will try to massage it a bit tomorrow.

After we refactor to use the order book and limit orders, the next step will be to add in a proper AI.  Right now, all we are using are algorithmic traders.  The AI will be optional and until it's trained with a few weeks of data it runs a very real risk of making very expensive mistakes.  However it's "learning profile" is a stored data set that can be downloaded and updated regularly with new market information.

Because it will attempt to predict the market direction, it will necessarily be significantly different than the algo traders we have now.  

The AI is a predictive DSP with a genetic algorithm based on a project I wrote a few years back for a company that couldn't afford downtime (downtime meant lives lost not just dollars).  The AI would watch all systems globally and learn about which systems were failing.  At first it just watches everything and reports failures, then it begins to make predictions and give each prediction a weight based on if it comes true or not.  Eventually the strongest predictions survive and the weakest ones die out.
The goal here was to compile an alert of probabilities and send it as far in advance as possible.  
The longer it runs the smarter it gets.  With a failure rate of 1 machine per day it could predict up to 4 hours in advance with 90% certainty.

I've been running it against MtGox since I first created the Isis-ATP project and it's a bit novel in that it can deduce the lump sum direction of the algorithmic traders at play (after all that's what a DSP does it isolates signals from noise).

Since January it's been showing that the pricing structure is actually being supported by VWAPCross and someone somewhere appears to be running a stochastic oscillator and that someone has some serious dollars.  That's pretty much all the bots and it accounts for over 80% of the trading activity.  The remaining is human traders (random noise tends to be from human activity).  

This is MtGox data only.  I haven't tried it against any other exchange and the DDOS attacks a few weeks ago gave the poor thing brain damage for lack of a better word. I'm not sure how to reconcile the dataset so I'll start a new profile when I integrate it.

I will also be adding an internal wallet and a configurable "sweep" to allow you to get your profits out daily and still keep enough on hand for expected volume.

Finally I will also be adding peer to peer mode to allow nodes to specialize in exchanges and share resources.

If that doesn't make a lot of sense imagine this...
There is a significant arbitrage opportunity between btc-e & mtgox on BTC right now.  You can buy bitcoins cheaper on BTC-e and sell them for more on MtGox (feel free to check this).

Each node can be configured to specialize in a particular exchange and this is where the bulk of it's money is at.  

If we can predict the size and length of an arbitrage opportunity we can do the buying and selling individually and settle up the profits later.  This obviously only works if you are working with close friends and people you know.  But the fact is this can work in the right circumstances.

Imagine if I had $5000 USD on BTC-e and Aido (sorry you're volunteered), has a few hundred BTC on MtGox.  I can authorize my machine to spend the $5000 USD into BTC on BTC-e and Aido can authorize his machine to spend the BTC on MtGox into USD.  I am effectively lending him my USD while he in turn is loaning me his BTC.

Once the arbitrage window is over (or we run out of cash), then we settle and split the profits.

Obviously this system would require trust which is scare in the bitcoin community, but if the transactions were small enough and there were enough people running it and we added a "confidence" ranking of some sort based on previous interactions, this thing has real potential to make serious money for anyone who cares to participate while not requiring everyone to have a hundred thousand accounts everywhere to try and keep track of.  

It's optional and will be default disabled, but just be aware this is one really easy way to run arbitrage without so much risk from window expiry.
Hope you guys find it exciting.  There is quite a bit of work to do here, but I think it will all be worth it in the end.

Thanks for all the help with the certs, I really appreciate it a great deal. Now to try my hand at some arbitrage between gox/btc-e Tongue

As for the predictive AI, I know you weren't talking just to me but everyone here but I'm curious would this platform learn for only one user or would it compile data from all users to build trading logic and improve efficiency(as per what you refrenced)? To be more exact, what I mean is would the "learning profile" learn from all users of the platform, who are all more or less using the same trade logic, or would it learn from users trying different strategies and then build off that or maybe even some mix between the two?

I think making a platform that includes logic based off p2p decisions is something alot of people have been waiting to see. I know it's an ugly comparison but: rather than having one user dictate broad manipulation plans ("Buy x-coin and sell in 8 hours") and users follow this and pool resources, having decisions automated within a platform would be much more equitable. I realize the example you gave is only two users engaging in arbitrage but that only raises the specter of: well what if it were built to handle many more users and used shared logic? then it would be...well you can imagine the results. I'm sure there's a question of morality since it would likely be considered collusion by traditional standards but as long as it isn't so strong as to be the largest controlling force of the market then hey "no harm, no foul".  Anyone have thoughts on this?

I wish I was better with programming and could do more to help this project grow. If it helps I'm an Econ major and know a thing or two about how markets react under specific circumstances. While bitcoin isn't a traditional market alot of the same theory, i've learned, still applies.

Again, thanks to the users here who help me not suck at java Tongue

Feeling generous?
BTC: 1MiaKvvzhuCrbWJ2iXA9RceZfRpNDjwJTk
Jasmin68k
Jr. Member
*
Offline Offline

Activity: 46


View Profile
May 19, 2013, 09:28:04 AM
 #530

In case anyone still needs them, simple scripts to add the StartSSL certs for MtGox to the keystore:

Linux:

http://www.ailis.de/~k/uploads/scripts/import-startssl

Windows:

https://github.com/haron/startssl-java
Nova!
Full Member
***
Offline Offline

Activity: 140


View Profile
May 19, 2013, 09:45:14 AM
 #531

Thanks but you missed something.  Read my post more carefully.  

The example I was giving was to show how 2 people could work it (with or without software).  
I then go on to state how I would do it with multiple people doing it.  
I doubt it would be collusion.  Collusion only occurs when 2 or more parties act against the market and attempt to keep others out.Although I would definitely have a reputation system involved, I would expect it to be generally open to the public.  Anyone could join so I don't see a problem here.

In this case, each node functions as an expert in the conditions of the particular exchanges it is tied to.  Like synaptic pathways in a brain, the more it is used, the stronger it gets.  It's co-operative multitasking with no single central coordinator.  The AI would share it's result, but not it's learning profile across nodes since individual results are dependent upon a host of factors, but outcomes are always the same.

That is to say, we may each execute a trade at the same time on the same exchange.  That trade teaches the individual AI's something, i.e. it represents a unit of knowledge.  However our particular reasons for the trade and even which side we are on, are very likely to be different.  You may be selling because you bought earlier and are looking to take a profit.  I may be buying to cover a short or because I believe the price will continue upwards, or even just to provide liquidity to the market.

The end result though is that a trade occurred and the market reacted accordingly.  We know that the majority of systems are algorithmic traders and not true AI, not in the sense that they are designed to learn by making mistakes anyways. We know that we can uncover the aggregate algorithms and their effect on the market this way.  Once we know the algorithms driving a given market, we can then do something similar to a monte-carlo simulation and play forward and backwards through the data to discover how the market will react to certain trades that are in our favor.  The trades are placed according to the probability spread and the system watches the fills, repeating the cycle.


Donate @ 1LE4D5ERPZ4tumNoYe5GMeB5p9CZ1xKb4V
Aido
Sr. Member
****
Offline Offline

Activity: 249


1. Collect underpants 2. ? 3. Profit


View Profile
May 19, 2013, 09:59:21 AM
 #532

I'm thinking this product is going to need it's own website in the near future.  The OpenPay version is going to be called just ATP, not to cut Aido out but to distinguish it so people don't get confused.  I plan to take it to the point it works reliably and then open it up to the broader community.

Perhaps this thread should be restarted as AidoATP? I've gone through all 30 pages here way too many times for bits and pieces of information. Just a thought since the platform now seems like its getting off the ground all and all and now discussions seem to be more regarding the fine tuning seems more to be logic oriented than structural.

Given Nova's big plans for ATP and issues around the usage of the name Isis I suggest creating a new thread just titled ATP or similar; dropping the Aido and Isis references. I may halt development on the Aido branch now that Nova is back in the game.

Interesting Bash command line, try it Wink:
bitcoin-cli sendtoaddress 1Aidan4r4rqoCBprfp2dVZeYosZ5ryVqH6 `bitcoin-cli getbalance`
Aido
Sr. Member
****
Offline Offline

Activity: 249


1. Collect underpants 2. ? 3. Profit


View Profile
May 19, 2013, 10:18:04 AM
 #533

OK, I created a new topic ATP [Advanced Trading Platform] for discussing future development of ATP.

IsisATP and AidoATP specific issues may still be discussed here in this topic although development will be slow or halted.

Interesting Bash command line, try it Wink:
bitcoin-cli sendtoaddress 1Aidan4r4rqoCBprfp2dVZeYosZ5ryVqH6 `bitcoin-cli getbalance`
Nova!
Full Member
***
Offline Offline

Activity: 140


View Profile
May 19, 2013, 10:45:56 AM
 #534

I'm thinking this product is going to need it's own website in the near future.  The OpenPay version is going to be called just ATP, not to cut Aido out but to distinguish it so people don't get confused.  I plan to take it to the point it works reliably and then open it up to the broader community.

Perhaps this thread should be restarted as AidoATP? I've gone through all 30 pages here way too many times for bits and pieces of information. Just a thought since the platform now seems like its getting off the ground all and all and now discussions seem to be more regarding the fine tuning seems more to be logic oriented than structural.

Given Nova's big plans for ATP and issues around the usage of the name Isis I suggest creating a new thread just titled ATP or similar; dropping the Aido and Isis references. I may halt development on the Aido branch now that Nova is back in the game.

Don't halt your work.  You've done great things.  The architecture is supposed to be modular enough we can work on separate things without stomping on one another.
Speaking of which I just saw a divide by 0 error from bigmoney go flying by as the market took a dip and came back.  Something to do with the weight being precisely 0, it caused the thread to die.  We should probably be actively looking for those sorts of things.

Donate @ 1LE4D5ERPZ4tumNoYe5GMeB5p9CZ1xKb4V
daybyter
Legendary
*
Offline Offline

Activity: 965


View Profile
May 19, 2013, 02:02:49 PM
 #535

I think, you should work on making this stuff more modular. It would be cool, if a trader could just be plugged in into the whole app. So you could use a neural network trader or a an algorithmic trader. Or even a mixture of several traders.

I would also separate the trade logic from the rest of the app, using a trading language.

tunafish
Member
**
Offline Offline

Activity: 95


View Profile WWW
May 19, 2013, 05:25:52 PM
 #536

Thanks but you missed something.  Read my post more carefully.  

The example I was giving was to show how 2 people could work it (with or without software).  
I then go on to state how I would do it with multiple people doing it.  
I doubt it would be collusion.  Collusion only occurs when 2 or more parties act against the market and attempt to keep others out.Although I would definitely have a reputation system involved, I would expect it to be generally open to the public.  Anyone could join so I don't see a problem here.

In this case, each node functions as an expert in the conditions of the particular exchanges it is tied to.  Like synaptic pathways in a brain, the more it is used, the stronger it gets.  It's co-operative multitasking with no single central coordinator.  The AI would share it's result, but not it's learning profile across nodes since individual results are dependent upon a host of factors, but outcomes are always the same.

That is to say, we may each execute a trade at the same time on the same exchange.  That trade teaches the individual AI's something, i.e. it represents a unit of knowledge.  However our particular reasons for the trade and even which side we are on, are very likely to be different.  You may be selling because you bought earlier and are looking to take a profit.  I may be buying to cover a short or because I believe the price will continue upwards, or even just to provide liquidity to the market.

The end result though is that a trade occurred and the market reacted accordingly.  We know that the majority of systems are algorithmic traders and not true AI, not in the sense that they are designed to learn by making mistakes anyways. We know that we can uncover the aggregate algorithms and their effect on the market this way.  Once we know the algorithms driving a given market, we can then do something similar to a monte-carlo simulation and play forward and backwards through the data to discover how the market will react to certain trades that are in our favor.  The trades are placed according to the probability spread and the system watches the fills, repeating the cycle.



Sorry I wasn't referring to what you mentioned as collusion. I was saying that it could lead others to think of/want/develop a platform that promotes easy collusion...think an ATP with 100 users connected to a central server (or many p2p which would be slower but would still work i'd suppose) that executes an order for all nodes to sell 10% of their btc holdings at market rate thus likely causing a (further than the initial 10%) dip in prices that then all nodes make use of to buy at a lower price.... I'm just musing here and it's not really all that relevant to the discussion. I think what you suggested (the 2 user p2p nodes pooling resources) would be hard to call collusion since it's just a nice quid pro quo system that I'd love to try frankly since there's almost always powerful arbitrage opprotunities between btc-e and gox and I can imagine scenarios where it would also be useful within a single market with users moving between different currencies where users could share profit (possibley without lending but I need to check my math since this is still in my head)

At any rate your suggestion the p2p resource pooling as well as the idea for a nueral AI sounds ambitious and could be a very powerful tool over time.

Feeling generous?
BTC: 1MiaKvvzhuCrbWJ2iXA9RceZfRpNDjwJTk
Nova!
Full Member
***
Offline Offline

Activity: 140


View Profile
May 19, 2013, 05:47:53 PM
 #537

I think, you should work on making this stuff more modular. It would be cool, if a trader could just be plugged in into the whole app. So you could use a neural network trader or a an algorithmic trader. Or even a mixture of several traders.

I would also separate the trade logic from the rest of the app, using a trading language.

Let's go ahead and move this to the new ATP thread. 

But yeah, the AI trader is just meant to be a module along with others that could be loaded and specified at runtime.
I originally set out to make the whole thing a collection of intercommunicating modules, but there were issues with the design that was causing unexpected behavior IIRC.
So instead I settled on a fully integrated design.  Aido solved the modularity problem and problems from thread deadlocks and i like the way he laid the application out when he refactored it.

BTW don't use the intra-currency arbitrage bot.  I've been looking really closely at the source and it makes several assumptions that are or can be invalid, such as all of your balance in currency A can be readily converted to currency B etc. Unfortunately, this is not the case, and managed to cost me half my bitcoins as I left it running over night unchecked.  (I didn't have THAT many bitcoins fortunately, but still it was an expensive way to learn.) 

When I wrote it originally, bitcoins were about $6 each, so losing a bunch of pips wasn't a big deal.  Unfortunately, BTC is in the $100s+ the loss tolerance of a couple of pips translates into significant money now.

Donate @ 1LE4D5ERPZ4tumNoYe5GMeB5p9CZ1xKb4V
BCB
CTG
VIP
Legendary
*
Offline Offline

Activity: 1036


BCJ


View Profile
May 22, 2013, 02:19:09 PM
 #538

did you guys see this.

Though it might be interesting.

http://btcmath.com/2013/05/bitcoin-2013-presentation/

The take away for me is that it is not possible to profit on intragox trading with a gox fee of .06% (1.2% round trip)  based on the historical price history.  The profit margins are too thin when you also account for the cost of getting capital into and out of the market.

Gox, or a high volume competitor needs to have a  lower its fee..
daybyter
Legendary
*
Offline Offline

Activity: 965


View Profile
May 22, 2013, 02:27:39 PM
 #539

use btc-e ...

paulthetafy
Hero Member
*****
Offline Offline

Activity: 813


View Profile
May 23, 2013, 10:27:18 AM
 #540

Okay finally turned on live mode and got this error straight away...
Code:
2013-05-23 19:16:14,100 : ERROR: Caught unexpected exception, shutting down MtGox trend following trading agent now!. Details are listed below.
java.lang.NullPointerException
at com.xeiam.xchange.mtgox.v1.service.trade.polling.MtGoxPollingTradeService.placeMarketOrder(MtGoxPollingTradeService.java:84)
at org.aido.atp.TrendTradingAgent.marketOrder(TrendTradingAgent.java:554)
at org.aido.atp.TrendTradingAgent.evalBid(TrendTradingAgent.java:532)
at org.aido.atp.TrendTradingAgent.run(TrendTradingAgent.java:346)
at java.lang.Thread.run(Thread.java:722)

Any thoughts? Using AidoAtp by the way

Pages: « 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 [27] 28 29 »
  Print  
 
Jump to:  

Sponsored by , a Bitcoin-accepting VPN.
Powered by MySQL Powered by PHP Powered by SMF 1.1.19 | SMF © 2006-2009, Simple Machines Valid XHTML 1.0! Valid CSS!