Gekko - a javascript trading bot and backtesting platform

[Starving_Marvin]

The candle size is 60 and history size is 100. I like these settings. Not perfect but better than my previous ones. I just don't understand why does all the historical data suddenly disappear if I close Gekko for a minute? Because it was trading a few hours ago. The same re-loading time doesn't seem to occur if I don't login into btc-e but just press ctrl+c and exit.

Can I please have some help on getting the log to work? So I could see my balance without logging into btc-e. The email system gives weird errors for me...

[cryptcollector]

I've eventually got the history I needed for CEXIO and when I came to activate the actual trading Gekko just started to give me this error.
 
 2014-06-21 01:07:26 (DEBUG):    cex.io returned an error, retrying.. Nonce must be incremented waiting for 10000 ms

No matter what I try to do it just keeps on going, if I revert to just simulation is goes away. The API key and Secret are in and correct, I've even checked it with a different API key and got the same effect.

Anybody had this problem, and more importantly, has anybody solved it?

I'm going nuts here, I cant find this problem anywhere with regerad to gekko, not even by googling

Thanks

[kuzetsa]

I've eventually got the history I needed for CEXIO and when I came to activate the actual trading Gekko just started to give me this error.
 
 2014-06-21 01:07:26 (DEBUG):    cex.io returned an error, retrying.. Nonce must be incremented waiting for 10000 ms

No matter what I try to do it just keeps on going, if I revert to just simulation is goes away. The API key and Secret are in and correct, I've even checked it with a different API key and got the same effect.

Anybody had this problem, and more importantly, has anybody solved it?

I'm going nuts here, I cant find this problem anywhere with regerad to gekko, not even by googling

Thanks

I haven't worked on gekko in several months, and neither has askmike. Fortunately, cexio was the one I was personally trying to support so I'll update you further if I'm able to get it working, and will hopefully be able to provide assistane if the current state of gekko's cexio API support isn't completely broken (again, broken AGAIN, sometimes the APIs get changed by the various exchanges and there's sometimes nothing we can do to fix our code)

I'll update this thread with status of gekko's cexio API support this week / over the weekend / before the end of the month... maybe today but no promises


Update:

Took me less than an hour to install a clean install of gekko and configure for cexio's API... So far, at or around noon, local NY time I have gekko working fine with cexio in "watcher" mode (not live trading... yet) but that doesn't mean much since you said the error was in trading mode (I have a little GHS and BTC on hand on my cexio account for testing, so we'll see what happens next time there's a trend)

[cryptcollector]

I haven't worked on gekko in several months, and neither has askmike. Fortunately, cexio was the one I was personally trying to support so I'll update you further if I'm able to get it working, and will hopefully be able to provide assistane if the current state of gekko's cexio API support isn't completely broken (again, broken AGAIN, sometimes the APIs get changed by the various exchanges and there's sometimes nothing we can do to fix our code)

I'll update this thread with status of gekko's cexio API support this week / over the weekend / before the end of the month... maybe today but no promises


Update:

Took me less than an hour to install a clean install of gekko and configure for cexio's API... So far, at or around noon, local NY time I have gekko working fine with cexio in "watcher" mode (not live trading... yet) but that doesn't mean much since you said the error was in trading mode (I have a little GHS and BTC on hand on my cexio account for testing, so we'll see what happens next time there's a trend)

Hi kuzetsa,

Cheers for this, if there's anything I can do to help eg, testing or anything feel free to ask.
I was beginning to think is was me  

Look forward to the fix.

---added
Sorry should have also said originally that the alternate API key/Secret i used were for different accounts on CEX.io and different installation from different systems as well. All were running under Windows.

Crypt

[kuzetsa]

The particular error you're having makes me wonder if maybe your computer's clock isn't accurately set.

This could affect the trade API, as the exact time is sent by gekko using the cexio API when a trade happens.

Only advice I have at this time:

• Check to make certain your system clock is accurate. Gekko's cexio trade API uses a timestamp, and even if this doesn't fix the issue it's good to have an accurate clock   ...because reasons^TM

• Try reinstalling gekko, complete with the "npm install" step
• Try a fresh config file (it's a bit of work, but there could theoretically be an issue with your settings if it's an old config)
• Try a fresh API key, since at some point in the past 10 months it looks cex.io added a new IP address whitelist feature, and possibly they also changed the format for API keys?
  (the new API key I generated today looks a little bit longer than the old one)

Code:

2014-06-21 14:48:46 (DEBUG):    inserting candle 1127 (2014-06-21 18:47:00 UTC) volume: 1.1627554099999997
2014-06-21 14:48:46 (DEBUG):    calculated MACD properties for candle:
2014-06-21 14:48:46 (DEBUG):             short: 0.00745433
2014-06-21 14:48:46 (DEBUG):             long: 0.00745423
2014-06-21 14:48:46 (DEBUG):             macd: 0.00000010
2014-06-21 14:48:46 (DEBUG):             signal: -0.00000007
2014-06-21 14:48:46 (DEBUG):             macdiff: 0.00000017
2014-06-21 14:48:46 (DEBUG):    In uptrend since 9 candle(s)
2014-06-21 14:48:46 (DEBUG):    Processed trades, sleeping for a minute...
2014-06-21 14:48:47 (INFO):     Attempting too BUY 0.04942629630374857 GHS at cex.io
2014-06-21 14:48:47 (DEBUG):    BUY 0.04942629 GHS @ 0.00745485 BTC
2014-06-21 14:48:47 (DEBUG):    BUY order placed.  Order ID 1562717868
2014-06-21 14:49:18 (INFO):     BUY was successfull

For now, all I can seem to figure is that   it just works^TM   and there's nothing wrong with gekko's cexio support other than not having the fee.
(cexio now has a fee on all trades, so I had to modify the API slightly)

At some point I can clean up my code a bit (instead of a quick and dirty hack) to push a new update to the github repo for my
"correctly supports cexio" fork of gekko with the updated code to account for cexio now having a trading fee.


Updated with more info:

The hardcoded code with "zero fee" is outdated, as is the code comment:

Code:

FROM FILE: ~/gekko/echanges/cex.io.js


Trader.prototype.getFee = function(callback) {
  // cexio does currently don't take a fee on trades
  // TODO: isn't there an API call for this?
  callback(false, 0.0);
}

However, when I changed the value from 0.0 to a non-zero value, I had some other issues with the fee STILL not being used:

Code:

FROM FILE: ~/gekko/core/portfolioManager.js

Manager.prototype.trade = function(what) {
  if(what !== 'BUY' && what !== 'SELL')
    return;

  this.action = what;

  var act = function() {
    var amount, price;

    if(what === 'BUY') {

      // do we need to specify the amount we want to buy?
      if(this.infinityOrderExchange)
        amount = 10000;
      else
        amount = this.getBalance(this.currency) / this.ticker.ask;

I couldn't even find evidence of the fee being used, so I changed the line to:

Code:

        amount = ((1 - this.fee) * this.getBalance(this.currency)) / this.ticker.ask;

also note, it appears as though the API doesn't use an exact 0.2% fee, so I actually used a higher value in ~/gekko/echanges/cex.io.js

Code:

  callback(false, 0.0075);

that value corresponds to is 0.75% instead of 0.2%, and even then it sometimes still complains about insufficient funds (though not always) so I think maybe there is a rounding error somewhere in gekko I haven't managed to track down... Otherwise, like I said, gekko works fine.

Hope this info helps

Note: Feel free to hack around in your own gekko code and do better, as my current solution to the fee not being used (was a hardcoded 0.0) really is a quick and dirty hack rather than having gekko read the value from the config... which I don't even think works correctly for ANY exchange, so that's not just an issue with gekko's cexio support

[cryptcollector]

kuzetsa

Computer clock was set okay and syncing time from the internet.

I decided to setup gekko on Ubunutu 12.04, but came across another issue, but found the solution in an earlier post and finally got it running right, I've also implemented your changes.

I created a new API Key as suggested as I was getting the same error as on Windows once i got gekko running right, and this solved the problem on linux. Will test the new key on the windows installation and update.

Thanks your help, Now i just have to wait for 13 hrs for the database to get up to speed again. Is there a quicker way of getting the data that gekko needs? or do we just need to wait it out?

[kuzetsa]

kuzetsa

((...snip...))

I created a new API Key as suggested as I was getting the same error as on Windows once i got gekko running right, and this solved the problem on linux. Will test the new key on the windows installation and update.

Thanks your help, Now i just have to wait for 13 hrs for the database to get up to speed again. Is there a quicker way of getting the data that gekko needs? or do we just need to wait it out?

Glad to hear a new API key was all you needed.

As for your other issue of building a database, there are two options:

• "the data that gekko needs" will vary depending on your chosen settings for the technical indicators you're running gekko with (MACD, PPO, etc. etc. etc.) but the MACD / PPO / other trading method defaults can be easily changed, as can the candle sizes. I'm personally using rather exotic settings on my own, and require less than an hour of data for gekko to work as I want it to.
• If you really need to use the current settings (which requires more data) then you could simply copy your databases from the old gekko installation. So long as there are no gaps in the data, this should work. Being able to copy databases between gekko bot installations was a design goal when the gekko database backend was changed several months ago. It was a major rewrite, but was worth it

Edited to add:

Oh, I just realized you mentioned the "13 hours to rebuild database" more than 13 hours ago. Nevermind then I guess, but for future reference, you can copy databases to a fresh install to speed things up.

[cryptcollector]

I noticed that there used to be a backtest for gekko, is this still there, I take it the only problem there is, is that the data in CSV format is not that easily obtainable.

Would be a good feature to still have, even if perhaps the accumulated DB data was to be used for this purpose natively.

But even if not, getting the info out of the DB data would be a good alternative.

I'm still playing about with settings, but ghs is really flat at the moment, so not getting much response. Not so sure what I should set as a good threshold for it. I've expanded to 30 min candles, so I'm waiting once more for the DB to catch up 😠...

I thank you for your time and taking an interest in my troubles.

When I get this thing going proper and start to see results I will gladly contribute in one way or other.

Thanks 😃

[kuzetsa]

I noticed that there used to be a backtest for gekko, is this still there, I take it the only problem there is, is that the data in CSV format is not that easily obtainable.

Would be a good feature to still have, even if perhaps the accumulated DB data was to be used for this purpose natively.

But even if not, getting the info out of the DB data would be a good alternative.

((...snip...))

I never touched the old backtesting code, mainly because I wasn't personally using it before the DB rewrite, and come to think of it you're right -- looks like it never got ported to the new gekko framework... And from what I can tell, the current state of gekko is entirely event driven & asynchronous so I'm not sure how porting the backtest feature into the current version of gekko could even be done.

Moving forward, my own fork of gekko is probably ONLY going to support cexio, but a big thing to consider with GHS is that a port of the old backtesting code would probably be even more invalid than it was in the first place since over time, the constant increase in your BTC balance is strongly linked to how much GHS you have in your portfolio. There is a problem with the idea of selling your GHS in hopes of a short-term gain from a strategy where you buy back the GHS at a lower price, since you will often LOSE more money because your portfolio would not have the same mining income gains you would have earned from simply holding onto your GHS.

Working backtesting support (which would need to take into account timestamps, and then simulate hypothetical mining based on the historical conditions for the bitcoin network difficulty) for cexio is not anything I'm planning on implementing. I'm far more concerned with taking care of cleaning up bugs which were introduced during the "LocalDB" branch...

... and then MAYBE adding a new trading method which accounts for the real value of GHS (mining income based on realtime network conditions and difficulty, as well as deducting the maintenance fee... which itself will require tracking the USD bitcoin exchange rate, as the current cexio maintenance fees are based on USD exchange rate for bitcoin)

It's not going to be easy to get cexio backtesting working in a realistic / sane way

[cryptcollector]

It's not going to be easy to get cexio backtesting working in a realistic / sane way

I hear you, i've kind of given up trying to play the GHS market at the moment, its just not viable, unless you actually sit there and watch it for the sudden drops and spikes that have a gap of about 0.0002 between top and bottom.

I've decided to point my gekko (so to speak  ) at btc-e and LTC/BTC market.

I'm going to see about using CEX GHS to reinvest in itself, but at a slightly lower price than market value (take advantage of the sudden price drops in any given candle to a set amount).

I'm also going to point 10% of my mining power from hardware to CEX to fuel the reinvest untill i can get a reasonable sustainment from the GHS itself.

Point of this? Just a suggestion, if you are going to concentrate on a solely CEX.IO for your own fork of gekko, it might be a nice function to have where you never sell GHS but take advantage of market dumps to get maximum ghs for your generated BTC.

Just my tuppence worth :-)

[kuzetsa]

Point of this? Just a suggestion, if you are going to concentrate on a solely CEX.IO for your own fork of gekko, it might be a nice function to have where you never sell GHS but take advantage of market dumps to get maximum ghs for your generated BTC.

Yep, I already got it working exactly like that, and I'll be pushing a commit with the new version to my repo soon since I seem to have squished all the relevant bugs which I've been able to find (less than 10 bugs, and most of them were related to inconsistent formatting / display / logging of information)


Edited to add:

And here it is: https://github.com/kuzetsa/gekko/releases/tag/initial.fork.2014.june
DO NOT USE!!! (see next post for more info)

Note that the current version of this fork can always be found at:

https://github.com/kuzetsa/gekko/releases/latest

which, as of 2014 June 25th 15:52, the current version is still:

" initial.fork.2014.june "

[kuzetsa]

Fixed a bug with the "enforced" state (old cex.io-only reinvest mode) where the code had some poorly defined behavior which could cause a stuck state with no possible exit condition. Also, spent the last 30-45 minutes auditing code in proximity to the now-fixed bug, and updated the release notes with the specific details in:

https://github.com/kuzetsa/gekko/releases/tag/critical.bugfix, which is now the latest release as of:
2014 June 25th 22:00 (local NY time)


... And https://github.com/kuzetsa/gekko/releases/latest now points to the "critical.bugfix" release.

[ernie-]

Fixed a bug with the "enforced" state (old cex.io-only reinvest mode) where the code had some poorly defined behavior which could cause a stuck state with no possible exit condition. Also, spent the last 30-45 minutes auditing code in proximity to the now-fixed bug, and updated the release notes with the specific details in:

https://github.com/kuzetsa/gekko/releases/tag/critical.bugfix, which is now the latest release as of:
2014 June 25th 22:00 (local NY time)


... And https://github.com/kuzetsa/gekko/releases/latest now points to the "critical.bugfix" release.

Thanks for the update.

How come it says cex.io trading fee will be: 0.75%? cex.io fee is 0.2% usually.

[kuzetsa]

Fixed a bug with the "enforced" state (old cex.io-only reinvest mode) where the code had some poorly defined behavior which could cause a stuck state with no possible exit condition. Also, spent the last 30-45 minutes auditing code in proximity to the now-fixed bug, and updated the release notes with the specific details in:

https://github.com/kuzetsa/gekko/releases/tag/critical.bugfix, which is now the latest release as of:
2014 June 25th 22:00 (local NY time)


... And https://github.com/kuzetsa/gekko/releases/latest now points to the "critical.bugfix" release.

Thanks for the update.

How come it says cex.io trading fee will be: 0.75%? cex.io fee is 0.2% usually.

It's a safety margin

I was able to get more consistent behavior with a fee setting of 0.75% (without having to go to a higher fee for an even larger safety margin) only after I changed gekko's default minimum GHS trade to a higher value than the minimum allowed by cex.io policies: Orders are considered valid as small as 6 decimal places (0.000001 GHS)

At the 0.000001 GHS micro-trading level, the actual fee (due to *granularity) will sometimes be absurdly high... It's really not a bug with the exchange's fee system, or anything "wrong" with their policy to let people do micro-trades where the rounding error / granularity issue is quite bad.


Example data:

Code:

2014-06-26 @ 06:14:43, Fee: 0.00000004, Bought 0.00199889 GHS at 0.0075191 BTC
2014-06-26 @ 06:14:43, -0.00001508 BTC (actual fee, 0.26525%)
2014-06-26 @ 00:50:43, Fee: 0.00000003, Bought 0.00198139 GHS at 0.00753505 BTC
2014-06-26 @ 00:50:43, -0.00001496 BTC (actual fee, 0.20053%)
2014-06-26 @ 00:24:47, Fee: 0.00000003, Bought 0.00195676 GHS at 0.00752258 BTC
2014-06-26 @ 00:24:47, -0.00001475 BTC (actual fee, 0.2028%)
2014-06-25 @ 23:51:34, Fee: 0.00000003, Bought 0.00191964 GHS at 0.00753259 BTC
2014-06-25 @ 23:51:34, -0.00001449 BTC (actual fee, 0.207039%)
2014-06-25 @ 23:23:20, Fee: 0.00000003, Bought 0.00172432 GHS at 0.00749274 BTC
2014-06-25 @ 23:23:20, -0.00001295 BTC (actual fee, 0.23166%)

--- yesterday's (unrelated) critical bugfix ---

Code:

*** the price on this trade, and others was "not good"

2014-06-25 @ 20:39:11, Fee: 0.00000003, Bought 0.00174368 GHS at 0.00752997 BTC
2014-06-25 @ 20:39:11, -0.00001316 BTC (actual fee, 0.22796%)
...etc. etc. etc.

--- historical data on smaller trades ---

Code:

2014-06-23 @ 05:55:23, Fee: 0.00000001, Bought 0.00033359 GHS at 0.00746761 BTC
2014-06-23 @ 05:55:23, -0.00000252 BTC (actual fee, 0.396825%)
2014-06-23 @ 08:54:15, Fee: 0.00000001, Bought 0.00029047 GHS at 0.00744862 BTC
2014-06-23 @ 08:54:15, -0.00000219 BTC (actual fee, 0.456621%)
2014-06-23 @ 10:40:36, Fee: 0.00000001, Bought 0.00017371 GHS at 0.00748319 BTC
2014-06-23 @ 10:40:36, -0.00000131 BTC (actual fee, 0.7633%)
2014-06-23 @ 13:29:47, Fee: 0.00000001, Bought 0.00010602 GHS at 0.00745005 BTC
2014-06-23 @ 13:29:47, -0.00000080 BTC (actual fee, 1.25%)
2014-06-23 @ 17:27:56, Fee: 0.00000001, Bought 0.00026981 GHS at 0.00748623 BTC
2014-06-23 @ 17:27:56, -0.00000203 BTC (actual fee, 0.49261%)
2014-06-23 @ 21:19:55, Fee: 0.00000001, Bought 0.00012709 GHS at 0.00747413 BTC
2014-06-23 @ 21:19:55, -0.00000096 BTC (actual fee, 1.04167%)
2014-06-23 @ 21:40:49, Fee: 0.00000001, Bought 0.00030082 GHS at 0.007479 BTC
2014-06-23 @ 21:40:49, -0.00000226 BTC (actual fee, 0.442478%)
2014-06-23 @ 22:27:46, Fee: 0.00000001, Bought 0.00010293 GHS at 0.00748 BTC
2014-06-23 @ 22:27:46, -0.00000078 BTC (actual fee, 1.282051%)
2014-06-23 @ 23:11:45, Fee: 0.00000001, Bought 0.00024405 GHS at 0.007498 BTC
2014-06-23 @ 23:11:45, -0.00000184 BTC (actual fee, 0.543478%)
2014-06-24 @ 01:53:20, Fee: 0.00000001, Bought 0.00028878 GHS at 0.007514 BTC
2014-06-24 @ 01:53:20, -0.00000218 BTC (actual fee, 0.4587%)
2014-06-24 @ 02:32:54, Fee: 0.00000001, Bought 0.00029409 GHS at 0.00751441 BTC
2014-06-24 @ 02:32:54, -0.00000222 BTC (actual fee, 0.450450450450%)

^ note that none of those trades are even anywhere close to the minimum 0.000001 GHS order allowed by cex.io


* Granularity, and the resulting "rounding errors" are sometimes frustrating, but since the fees are paid in whole satoshi amounts there's not much I can do. I set it at 0.75% because I didn't want there to be (as many) errors related to "insufficient funds" during small trades (and later, made some tweaks to gekko's fee calculation and rounding code because it's common for granularity / rounding errors to be done in the favor of the exchange or merchant)

Granularity is the sort of thing some users might not think about or care about, and someone will probably go in and set their minimum trade to the actual, exchange-enforced minimum 0.000001 GHS value, because the user(s) might feel a compelling urge to reinvest so aggressively that the safety margins still might not be enough.

(I was personally trying to use gekko to reinvest using micro-trades much closer to the exchange-enforced minimum 0.000001 GHS, and the granularity was causing my fees to sometimes be much much higher, sometimes 5%, 10%)


TL;DR

* Granularity causes rounding errors on the fees when you're reinvesting tiny amounts with micro trades...

Specifically: The "1 satoshi" granularity causes issues. (The exchange rounds up to the next whole satoshi, and it's a common practice never waive the fee under any circumstances.)

[ernie-]

(I was personally trying to use gekko to reinvest using micro-trades much closer to the exchange-enforced minimum 0.000001 GHS, and the granularity was causing my fees to sometimes be much much higher, sometimes 5%, 10%)


TL;DR

* Granularity causes rounding errors on the fees when you're reinvesting tiny amounts with micro trades...

Specifically: The "1 satoshi" granularity causes issues. (The exchange rounds up to the next whole satoshi, and it's a common practice never waive the fee under any circumstances.)

Interesting problem, I wouldn't have even thought about it if you hadn't explained it. Didn't realize it has so much effect.

[wizzardTim]

So can someone suggest me parameter values for use with bitstamp? eg EMA? I am new to trading.

[kuzetsa]

(I was personally trying to use gekko to reinvest using micro-trades much closer to the exchange-enforced minimum 0.000001 GHS, and the granularity was causing my fees to sometimes be much much higher, sometimes 5%, 10%)


TL;DR

* Granularity causes rounding errors on the fees when you're reinvesting tiny amounts with micro trades...

Specifically: The "1 satoshi" granularity causes issues. (The exchange rounds up to the next whole satoshi, and it's a common practice never waive the fee under any circumstances.)

Interesting problem, I wouldn't have even thought about it if you hadn't explained it. Didn't realize it has so much effect.

Yeah, as you can see from some examples I provided from live trades (when I was working on bugfixes and testing over the weekend), when the fee is less than 5 satoshi, the granularity starts to make the rounding errors worse very quickly.

[kuzetsa]

So can someone suggest me parameter values for use with bitstamp? eg EMA? I am new to trading.

This chart from the "Goomboo's Journal" thread has a nice chart showing examples of how the simulations worked out based on the data they personally backtested against.

The EMA settings themselves are based on TA (technical analysis) theory, so you might be able to figure out how certain settings perform by looking at historical data using a charting tool which supports TA indicators such as the charts provided on:

bitcoin wisdom's "bitstamp" charts
(if you look under settings, MACD is a popular choice, but it doesn't have all the same ones as the example ones included in gekko.)

Also, note the following:

• gekko currently does not support backtesting, and hasn't supported backtesting for several months
• the fork I maintain only supports cex.io in a strict "reinvest only" mode, and won't work on bitstamp since "sell" is disabled in my fork
• If you need support for anything other than cex.io, you'll have to use "not my fork" such as the main / original gekko code (open source, and MIT licensed which is why I was able to fork it and make my own version) but it still has some bugs since it hasn't been updated since March 2014

Gekko is free, and "out of the box" the default settings don't always work well...

... and it doesn't even work until you edit your config.js to enable this line at the bottom:

// set this to true if you understand that Gekko will
// invest according to how you configured the indicators.
// None of the advice in the output is Gekko telling you
// to take a certain position. Instead it is the result
// of running the indicators you configured automatically.
//
// In other words: Gekko automates your trading strategies,
// it doesn't advice on itself, only set to true if you truly
// understand this.
//

config['I understand that Gekko only automates MY OWN trading strategies'] = false;

I'm not going to try to manage your portfolio for you (not for free, and without a suitable contract)
nor would I want to "basically be a liar" by promising the defaults are suitable for your needs  

[cryptcollector]

Well I'm now using your modified Gekko, so far so good.

Though as I've only got 16GHS to play with its slow going, I've pushed another 60GHS from hardware to compensate.

So, I'm impressed by what you've done in such a short space of time.
Keep up the good work, and if there’s a place for some btc contributions feel free to post :-)
As soon as I have some spare btc I'll bung you a donation.

Crypto

[kuzetsa]

How come it says cex.io trading fee will be: 0.75%? cex.io fee is 0.2% usually.

Actually, that workaround is probably no longer needed...

If anyone wants to test this prerelease version (intentionally NOT flagged as "latest") I'd appreciate it:

https://github.com/kuzetsa/gekko/releases/tag/microtrade

I've had it running for a few minutes, but other people are probably using different settings than mine so it'd be good to have more than one tester.