In coming few weeks, we are planning to add an optional news filter to our system which will help to stop trading and close existing trades during major news events. News filter can be useful in many trading systems if it is properly applied.

In machine learning algorithms, irrespective of how much training it does when a sudden market event happens which doesn't match any of the trained data, then the trained models tend to fail for a limited period of time and it leads to wipe out of the accumulated profit or leading to bigger loss.

So we may consider to add a news filter in near future. For now we are just manually handing the news events to stop trading during news and enable trading after news.

However, it is also important that news filter has no use if a strategy depends on backtesting since news filters can be used only during forward testing. So we will be using it in forward testing only.

Since we are approaching towards the end of the year, we recommend our EA users to be careful while using our EA in coming next few weeks. Because the liquidity in the markets might be very low and spreads might get widened in most of the currency pairs.

Also, the markets might behave completely randomly. So it is better to stop trading for next couple of weeks or watch the trades live. Also, please make sure to use the spread filter to avoid bad entries of the EA during high spread. Especially, the spread will be helpful during these kind of market conditions.

In our experience, few brokers increase spreads up to 15 to 20 pips during such markets which might lead to a series of trades getting hit by stoploss if you are using a tight stoploss.

We are noticing that there are multiple myfxbook users who are clicking on the 'copy' tab of our trading system in order to copy the trades of the system. So we would like to repeat that if you are interested to copy the trades of the system, then you should write a comment in this thread and once we receive at least 20 to 30 requests from users, then we may thing of making the system available for signals.

There is no point in clicking on 'Copy' tab of the system since it is not available for copy yet. So if you are seriously interested to copy the trades, then you should make a comment in the thread and it will be easy for us to know how many REAL myfxbook users are interested to copy the trades of the system or the requests are coming from one or two users having multiple myfxbook profiles.

As we mentioned in one of our previous posts, we received around 3 to 4 user requests for trade copy yet. If we receive sufficient number of requests within a short time, then we might make the system available for copy.

We are continuously facing issues with our 'rediffmail' email address by which the emails are getting delayed or sometimes never received to us or to the client and hence, from now onward we will continue to use our gmail address '[email protected]' as our primary email address for any future communications.

So we suggest all our existing clients or to anyone who is trying to contact us to contact our support at our gmail address '[email protected]'.

Our support email: [email protected]

Like @bmigette, I'm also curious about your model. Why do you make a difference between ML and NN? Isn't NN just a subcategory of ML?

Do you mean that you use a deep spiking NN? If so, it would probably be one of the first ones to do this?

Why do you use SNN, compared to just MLP/RNN/LSTM?

Do you use back propagation with gradient descent, or how do you train your SNN model?

Why do you use GPU for training, since you are dealing with a time series? I don't see how you could parallelize the training of a time series to make use of a GPU?

Where does your RL model come into the picture and what input do you use for this? It sounds like your input to your RL model is the output from your SNN model?

I would also be interested in copying your signal if you list it here on Autotrade or on MQL5. Thanks for your input!

Yonex posted:
Like @bmigette, I'm also curious about your model. Why do you make a difference between ML and NN? Isn't NN just a subcategory of ML?

Do you mean that you use a deep spiking NN? If so, it would probably be one of the first ones to do this?

Why do you use SNN, compared to just MLP/RNN/LSTM?

Do you use back propagation with gradient descent, or how do you train your SNN model?

Why do you use GPU for training, since you are dealing with a time series? I don't see how you could parallelize the training of a time series to make use of a GPU?

Where does your RL model come into the picture and what input do you use for this? It sounds like your input to your RL model is the output from your SNN model?

I would also be interested in copying your signal if you list it here on Autotrade or on MQL5. Thanks for your input!

There are a lot of questions in one post:)) we will try to answer a few of them for now...

Yes, you are correct. NN is a part of Machine learning. But a sufficiently trained NN can be used independently without using any further training and that is what we initially thought of and that's why we called it as NN network instead of machine learning.

We don't like to call an algo as true machine learning until it learns from real time market data while trading to adopt under any markets. We know that there are other developers of machine learning who just optimise an algo by curve fitting with past data and call it as machine learning and we don't call that in that way. That kind of algo can be called as 'automatic optimisation', but not machine learning in our perspective of course.

But it is not simple as it seems to be initially. There is technically no such model available in our knowledge which can generalise in all market conditions and hence, the NN needs to be continuously trained to filter out noise and keeping some residue for good signals just like a human brain using a multi-layer neural network so that in the last layer it will generate high quality signals based on the latest trained model.

Now, here is the tricky part because it requires massive computational power to run even a 2 to 3 layer neural network properly. If we will add continuously learning to it, then it even consumes more resources and that is the reason we are planning to use GPU.

But soon we are realising that GPU is not going to be sufficient also, and we are planning to do everything in Python and we may use cloud TPU services for training and then, connect to MT5 or MT4.

In forex trading, though it is difficult to have a good model , but also just having a good model is not enough since there is no way to know whether the signals generated by the model are actually getting executed as trades or simply getting discarded due to server freezing or mt4 freezing. So we need to calculate ahead as how many calculations or for loop iterations the model is doing inside to end up with the signals and how much time we can expect it to complete the cycle.

For example, at a opening candle if the signal is generated after 5 minutes of processing time by the model, then the price might have already moved significantly within that time and hence, such signal become useless.That is the main reason we are working on Spiking Neural network to tackle such issue.

But such things are very complicated and not easy to implement and definitely not easy to implement in MQL4 using the existing libraries. Hence, it will take time to complete such things or we may simply work completely on python. So for now we are testing in multiple demo and live accounts whatever we finish till now to see how everything goes.

Thanks for taking the time to answer my questions @leo23!

Some of your comments get me scratching my head a little. I think even a fairly complex SNN model would be able to forward the input variables to the output layer almost instantaneously. I think if it takes you 5 minutes (even if it is just an example) for your network (that has already been trained) to make a prediction, then perhaps you should look into optimizing your code...?

In MT4/MT5, you could just write your model in C++ and compile it to a DLL and call it from MT4/MT5? I mean, I'm able to run stacked NN consisting of several hidden layers that all output their individual predictions to a classification model that in turn provides buy/sell signals on a normal CPU effortlessly. It shouldn't take you more than a few milliseconds/0-50ms to do that?

If you want your code to run fast and achieve faster training, then instead of looking to use python, I think you could look into using C++ instead.

Yonex posted:
Thanks for taking the time to answer my questions @leo23!

Some of your comments get me scratching my head a little. I think even a fairly complex SNN model would be able to forward the input variables to the output layer almost instantaneously. I think if it takes you 5 minutes (even if it is just an example) for your network (that has already been trained) to make a prediction, then perhaps you should look into optimizing your code...?

In MT4/MT5, you could just write your model in C++ and compile it to a DLL and call it from MT4/MT5? I mean, I'm able to run stacked NN consisting of several hidden layers that all output their individual predictions to a classification model that in turn provides buy/sell signals on a normal CPU effortlessly. It shouldn't take you more than a few milliseconds/0-50ms to do that?

If you want your code to run fast and achieve faster training, then instead of looking to use python, I think you could look into using C++ instead.

If we are not wrong, then probably you might be referring to a NN using a couple of indicators as input variables and trained with past couple of months or years of past data which generates a BUY or SELL signal. Please clarify if we are wrong.

For better understanding, can you please answer these questions:
1.What is the file extension you are using for storing the data of your models like '.txt', '.rl' etc?
2.How many models are you using simultaneously to produce an output signal or you are using only 1 trained model to produce one output signal?
3.What is the average file size of your models in Bytes or KB or MB or GB?
4.How much past trading history you are considering to train the models?
5.Do you have an approximate idea as how many total calculations or iterations are done by your algo during the process starting from giving input variables till it generates an output value and as you mentioned which takes few seconds to generate the signals?
6.Finally, how many input variables you are considering to generate the output which you mentioned as taking few milliseconds? We mean like how many indicators or open, close prices etc you use?

By the way, like we mentioned in one of our previous posts we are trying to imitate an algo in forex trading which is something like 'ALPHA ZERO' used by 'GOOGLE Deep Mind' for beating most of the games like GO, Chess etc once it is fully trained. But in forex, the market behaviour is not fixed which means there are no specific rules and current hardware is costly to implement such an algo which will never loose and even 'ALPHA ZERO' may not work in forex as well.

1. I don't see how file extension would be important, but I store historical data in a SQL database and import the training data into an array held in the RAM. I don't do any read/write-to-disk activities during training or when the model is online, all information is held in arrays in the RAM.
2. About 50 various NN models, and then classification models after that to collect and filter their outputs.
3. The model itself is about 10 MB (training model), execution model is about 2 MB, training data is a about 1 GB.
4. Depends, but all in all usually about 8-10 years.
5. I have not calculated, but I don't think this is of major importance. I cap the training iteration at 100,000 and use early stopping, pruning, weight limitations, etc.
6. Each NN has about 10 input nodes, the variables to each node is calculated from the price during every time step, not looking back more than 20 steps.

How is your model setup?

Yonex posted:
1. I don't see how file extension would be important, but I store historical data in a SQL database and import the training data into an array held in the RAM. I don't do any read/write-to-disk activities during training or when the model is online, all information is held in arrays in the RAM.
2. About 50 various NN models, and then classification models after that to collect and filter their outputs.
3. The model itself is about 10 MB (training model), execution model is about 2 MB, training data is a about 1 GB.
4. Depends, but all in all usually about 8-10 years.
5. I have not calculated, but I don't think this is of major importance. I cap the training iteration at 100,000 and use early stopping, pruning, weight limitations, etc.
6. Each NN has about 10 input nodes, the variables to each node is calculated from the price during every time step, not looking back more than 20 steps.

How is your model setup?

Yes, file extension doesn't matter. We just wanted to know what exactly you are doing because of which you are getting the executions so fast.

Though we are not sure about the entire process how you are doing it. But here we see why the execution time is small in your case. Because as you mentioned the execution model is around 2 MB and probably, you are using one model for output calculations.

In our case, for a 2 year period, training model the model file size varies between 20 to 30 MB and around 50 to 100 models are used to to calculate the output signals. Total training models can be a few thousands during training which can take a couple of days, but out of that we choose between 50 to 100 best models for live trading based on external filters.

So the algo needs to search at least 50 models of each file size of 20 MB for one signal and so it takes time and that is only for a 2 year trained data which is not sufficient of course.

Also, if we are not wrong, then you are feeding the input nodes directly with some indicator values or prices which is simple and this is another reason why your output signal gets executed instantly. But in our case all the price values go through a series of calculations through a complex mathematical formulas(which are a bunch of nested for loops) to transform the values before feeding to the neural network and after that the output signals are calculated based on these transformed values inputs to the NN and this consumes maximum amount of time and some of the cases the search inside the models takes less time than value transformations.

So one 'nested for loop' during value transformations or during search inside the models is enough to freeze the mt4 platform and so we are working with SNN as well as we are working to improve the algorithm. We are not using any database for now. Everything is stored in hard drive in '.txt' or '.rl' files. So how many loops the algos is searching for and how many iterations are there in each loop and how much total time it will take to complete the cycle, everything needs to be well calculated before even training the models.

As we mentioned before, for our kind of algorithm python is the best choice for now. In fact, there are ready made open source source code is available online which we can use directly instead of writing everything from scratch exclusively for forex. We mean that using python we can simply use traditional image classification methods directly applied to MT4 strategy tester to train the NN if we have sufficient computing power.

This seems to be the holy grail to forex trading for now, because we already have our own profitable algos which are better than machine learning for now and we just need to run a backtester of multiple algos and visually feed the data to a NN to generate the models for classification.

Wait, you mean you are training your SNN model in MT4? And then you are reading/writing to the disk during every calculation? Then no wonder your model behaves so slow... You could try to do some profiling to see where your model is slow, I believe it is most likely during the disk read/write process. MT4 is also extremely slow to run backtests compared to other methods. I train my models in Matlab and just use the trading platform for trade execution.

Yonex posted:
Wait, you mean you are training your SNN model in MT4? And then you are reading/writing to the disk during every calculation? Then no wonder your model behaves so slow... You could try to do some profiling to see where your model is slow, I believe it is most likely during the disk read/write process. MT4 is also extremely slow to run backtests compared to other methods. I train my models in Matlab and just use the trading platform for trade execution.

No, for any machine learning algos we don't use MT4. Yes, we tried in MT4, but left. We use only MT5 for now and we are aware about MT4 backtesting issues.

That's why we keep telling machine learning and NN separately.For us, NN means no learning at all and everything is simple and coded in MQL4 with no models or anything like that. All the input optimised weights are hard coded inside the MQL4 code with indicator values and it's internal algorithm etc and there are no trained models in it, but it has just optimised weights for neurons and we keep changing the weights time to time. Probably, you are confusing the weights of the neurons as models.

But when we say machine learning, we actually mean algorithm based separate training models in MT5 using SNN and, then either we create a bridge using libraries to trade in MT4 or we simply copy trades as well from MT5 to MT4 if we want to use MT4.

We are not sure what exactly you mean by 'disk read/write process' and how you can then search the trading conditions inside the trained model if you will not read. May be you are referring to loading the model to virtual memory or something like that permanently during the live trading.

By the way, we are not concerned about slowness during training. We are referring to slowness while trading.

We are perfectly fine if it takes a couple of days for training also and in fact, we already tried this as well. It doesn't matter if it takes a month to complete the training if it is a one time training and then, if the trained model can execute all trades perfectly, then it will trade like a holy grail. It is very much normal and in fact, we can directly write the 'Monte Carlo tree search' algo to train itself without any past trading history at all and without using any MT5 backtester and purely based on self training to itself. It is very simple, but the model size will grow exponentially large may be in terms of GB.

So it is mostly the computational power, because if more data will be trained or more filters will be added, then file size will be more and ultimately it will not execute the trades.Otherwise, it is very simple task for us if any file size models can be execute the trades in real live markets since the algos are already available.

Irrespective of wherever you keep the trained model, you need to feed the input variables to the model and search inside the model for the output signal and it is always a whole bunch of nested for or while loops of 4 to 5 layers in it with if else statements while searching inside the models. Each model is like a trained brain structure where the search operation need to happen before making a trading decision.

Note that we are not talking about feeding 4 to 5 input variables or indicator values to generate a output signal like a 2 to 3 layer NN. The input variables to the NN usually vary in order of few thousands which are calculated from formulas and each variable going through the search inside 50 different models. It is basically similar to trying to imitate the human brain based on which SNN functions considering a time factor.

We want to update our fund management clients that we are not going to trade in any live account during the christmas and New Year holiday period though we may continue to trade in this demo account. We already stopped trading before few weeks due to low liquidity in the market. The normal trading will be started during first or second week of January 2019.

Also, though we mentioned earlier we would like to repeat for our EA users to be careful while using the EA during this period since the spread might be very high and also, make sure to use stoploss in each trade. Also, people can switch to higher time-frame temporarily to reduce the frequency of trades and hence, bringing more safety to the account balance.

Trading in this demo account has been stopped for next few days due to Holidays and will be resumed shortly . Trading in the live accounts of our fund management clients will be resumed only in January first or second week depending on market conditions.

Merry Christmas and Happy New Year to all of our EA users, fund management clients and to all myfxbook users!!!!

Trading in this demo account has been resumed.

As per our previous post, we will resume trading in fund management clients accounts in January first or second week.

While we are waiting for the live testing of our machine learning algo in demo or live accounts which we might do after few weeks, in the mean while we have been continuously improving our algo and it's coding. Now we have got sufficient time to fully focus on backtesting and adding new concepts to our algo to try out since markets are unreliable now and there is no point in forward testing before first or second week of January.

Backtesting results seems to be very interesting. However, until we do forward testing in demo and finally forward test in a live account for a couple of months, we don't want to conclude anything.

Now, we are completely working on MQL5 since the backtesting is much more reliable in MT5 terminal as compared to MT4. We have added multiple timeframe filters to filter noise from the trading signals and the results seem to be improved significantly. Apart from that there are few new concepts we have added to our algo and they don't have any existing technical terms or descriptions available and solely developed by our team. We might publish a test live account within next couple of weeks or months.

We have received multiple emails from EA users asking for recommended brokers for our EA. Though our EA can work fine with most of the regulated brokers we are listing some of our recommended brokers for the EA.

The recommended brokers are as follows:
Pepperstone, IC Markets, FxChoice, Dukascopy, XM,FxPro,TickMill,Forex.com,Forex4you

Now the markets have resumed to normal conditions. So we recommend our EA users to resume the trading of our EA from next week in their respective mt4 accounts with default settings of the EA.

We will also resume trading in our fund management clients accounts by end of next week or beginning of the third week of January.

Recently there were not enough trading opportunity for our system. So we have not yet resumed trading in all our fund management clients.

For now we have been focusing mainly on backtesting of our system, demo trading and improving the trading system further.

Trading in the fund management accounts will be resumed from next week from Monday.