Reusing MQL indicators in Python / calculating them from file? | Forex Factory

Options

Permalinks
Print Thread

Similar Threads

Calculating correlation in python then sending to Metatrader 5 replies

Saving user defined Functions within a mql file? 6 replies

MQL Coding - Let’s talk about how to code mql 0 replies

getting value from CSV file to mql indi and plot it 5 replies

How can I Change the MQL file to EXE 1 reply

Reusing MQL indicators in Python / calculating them from file?

  • #1
  • First Post: Mar 24, 2021 6:48pm Mar 24, 2021 6:48pm
I am working on a Machine Learning based Python script that analyzes stock/forex prices. Currently, the script works fine using 15 basic financial indicators to make decisions on whether to open a position or not. However, I now want to use a wider selection of indicators. I had previously collected ~1000 of them, written in MQL4. I have been programming in Python for years now, but my knowledge of MQL4 is quite rudimentary. Still, I could probably program these indicators in Python, one by one, but I am hoping there is another way to go about it, since the amount is too large, and in the future we might want to further extend the selection.

My initial idea was the following: Instead of trying to "pythonize" the .mq4 files at all costs, what if I could keep them with some minor modification only, and make the Python script evaluate them?

For example, I would want to calculate the RSI for Apple (AAPL) stock prices. (Since this is a very common indicator, there are various Python modules out there that can handle this - treat this example purely as a demonstration.) The Python script would download Apple stock data (e.g. using the Yahoo Finance API module), save it in a csv file, launch the modified RSI.mq4 file to calculate relative strength index, and read the MQL script's outputs.

My doubt is: does this even make sense and is it realistic to do? I have never seen examples of indicators that were not meant to be placed on a trading window in MetaTrader, but that read in a csv, operated on the data (as in: calculating an indicator for instance), and then saved the results to another csv file.
  • #2
  • Mar 25, 2021 7:37am Mar 25, 2021 7:37am
Quoting levix95
my knowledge of MQL4 is quite rudimentary
And the language itself is.

Use TA-Lib. You have many indicators there.

You can connect with Python to MetaTrader 5 to download price historical data and to send orders. You can do all the heavy lifting in Python and just connect through MetaTrader 5 when required: Python in MetaTrader 5

If MetaTrader 5 is not an option and you have to go with MetaTrader 4, then run some backtests with different indicators and save signals etc. in a csv file. Work with those files on Python and conduct there your research.
Or just download data from MetaTrader 5 demo and do the research. Later, create a MetaTrader 4 script to communicate with your Python systems.

Hope this helps.
Always ready to help
  • #3
  • Mar 25, 2021 3:05pm Mar 25, 2021 3:05pm
Quoting PaulMF
{quote} And the language itself is. Use TA-Lib. You have many indicators there. You can connect with Python to MetaTrader 5 to download price historical data and to send orders. You can do all the heavy lifting in Python and just connect through MetaTrader 5 when required: Python in MetaTrader 5 If MetaTrader 5 is not an option and you have to go with MetaTrader 4, then run some backtests with different indicators and save signals...
Thanks a lot for your reply, PaulMF.

I'm aware of TA-lib and have used it thoroughly in previous projects. Amazing library for the most common stock/forex indicators indeed. However, my goal here would be to analyze indicators that are not yet implemented in TA-lib (and will probably never be). As far as I'm concerned, the vast majority of indicators out there are written in MQL4 which is a serious bottleneck. As I said, translating them one-by-one to Python would be a feasible, albeit extremely time- and money-consuming. Another goal would be to circumvent the problem of MetaTrader only having data for a couple hundred of instruments. I would like to be able to input a csv file and calculate any indicator's values that exist. I think developing an EA that accepts a csv file with OHLCV as input (e.g. "AAPL.csv", "EURUSD.csv", and saves a new csv with an extra column attached (e.g. a column with SMA values) is yet a relatively quick and easy task in MQL4. But this EA would still have to be written one-by-one for all the thousands of indicators. On the flip side though, once the output csv files are created, Python can access them.

What do you and others reading this thread think about such an approach? Does this makes sense - or would I better off translating indicators directly from MQL4 to C++ which can later be embedded more easily to Python (assuming here that translating MQL4 to C++ would be relatively straightforward)?
  • #4
  • Mar 25, 2021 4:04pm Mar 25, 2021 4:04pm
You may create a Python script entering in a loop that will create one EA for each indicator, compile the EA, run a test, and finally save the result in a new column(s) of a csv or pandas data frame.

Any option like recoding every indicator to any other language will be very time consuming. If those indicators are created by you, then that won't take you too much time. If those indicators aren't yours, then don't waste your time (my opinion ). Most of them are useless as trading signals (sadly I have had to deal with many). Many repainting, recalculating, etc. Others just create new lines, histograms and arrows based on common indicators. You won't reach any conclusion from a Machine Learning perspective. By chance, you may end up with some pattern but besides that, nothing really important to consider.

Anyway, that's my opinion.
Always ready to help
  • #5
  • Edited 8:18am Aug 20, 2021 7:39am | Edited 8:18am
I made an account specifically for this topic. I am in the exact same boat as you are @levix95
MetaTrader is so slow when it comes to testing which makes it a pain for ML. Heck, even a 5 indicator EA's optimization can take upwards of 200 hours to complete with a modest range on parameters and a 1 year data set.

I have an idea, tell me if it makes any sense:
- Recreate the MT4 "Tester" to feed data to the ex4 scripts or whatever the tester does in order to evaluate the files.
- Create (or use existing) algo that reads indicator parameters/results and spits out an equation.
- Use that equation as your Python version of each indi.

This would allow you to train all in Python using Google's super computers!

I am unsure, though, how one would go about making the "Tester" part.
If this is too challenging, we could try mixing @PaulMF's idea in. Code EA's and optimize on small data set, output to csv, generate equation.
This approach would be more time consuming, however, would save countless hours when compared to re-programming everything.

Please let me know if you want to work on this together or if you have made any progress. Either way, I am starting on it now. Wish me luck
  • #6
  • Aug 21, 2021 6:29am Aug 21, 2021 6:29am
Quoting jagel
I made an account specifically for this topic. I am in the exact same boat as you are @levix95 MetaTrader is so slow when it comes to testing which makes it a pain for ML. Heck, even a 5 indicator EA's optimization can take upwards of 200 hours to complete with a modest range on parameters and a 1 year data set. I have an idea, tell me if it makes any sense: - Recreate the MT4 "Tester" to feed data to the ex4 scripts or whatever the tester does in order to evaluate the files. - Create (or use existing) algo that reads indicator parameters/results...
Python is slower than MQL. The problem isn't the Strategy Tester. If you run the same type of test in Python, it will be slower. In MetaTrader, testing time depends on the Model you choose, so if you select and Every tick model, this will require more time. If you create a similar on Every tick model in Python, the testing time will be slower than MetaTrader. Google's super computers won't make a difference in this regard.

Most probably you are facing the problem of purely coded custom indicators.
Algos Coder
  • #7
  • Aug 30, 2021 5:50am Aug 30, 2021 5:50am
Quoting losja
{quote} Python is slower than MQL. The problem isn't the Strategy Tester. If you run the same type of test in Python, it will be slower. In MetaTrader, testing time depends on the Model you choose, so if you select and Every tick model, this will require more time. If you create a similar on Every tick model in Python, the testing time will be slower than MetaTrader. Google's super computers won't make a difference in this regard. Most probably you are facing the problem of purely coded custom indicators.
Ahh yes, thanks for correcting me and providing some insight!
MQL4 is very restrictive, though. Python can utilize more hardware (GPU's are pretty slick), so wouldn't this speed up the process?

If MT4 can backtest with my GPU or at least more than 10% of my CPU, can you please show me how?
  • #8
  • Last Post: Sep 3, 2021 6:23am Sep 3, 2021 6:23am
Quoting jagel
{quote} Ahh yes, thanks for correcting me and providing some insight! MQL4 is very restrictive, though. Python can utilize more hardware (GPU's are pretty slick), so wouldn't this speed up the process? If MT4 can backtest with my GPU or at least more than 10% of my CPU, can you please show me how?
If you want to perform some specific optimization, different than what MetaTrader does, then Python is a good alternative.

The problem you face is the custom indicators (as default ones are already available in Python). How are you going to move them to Python? That's the point! If you rewrite them in plain Python, then each indicator's calculation will be slower. If you rewrite them in C/C++ and use it through Python, then you are getting similar performance than what you have in MQL4.

Once, you have created all the indicators for Python, a new issue arises. Are you going to use the same type of optimization that MetaTrader uses? A bulk of parameters that go from a minimum to a maximum with steps and their combinations? Then that in Python will probably be slower. Python is an interpreted language. MQL4 is not. How much hardware do you need to speed up your tests in Python and surpass MetaTrader performance? I don't know.

When you run the Strategy Tester do you use Tick data or Open Prices Only? Open Prices Only may solve your issue! Is there a clear reason to use Tick data?

Finally, I would never move to Python to run tests based on MetaTrader optimization logic. I would move to Python to perform some machine learning algorithms relying on extracted data from MetaTrader custom indicators. Are you sure those custom indicators deserve such a quantity of job?
Algos Coder
0 traders viewing now