Forward Testing Discussion Thread

FT.........here is the requested data from CSI

After eight trading days using Strat#5 and VP, my demo account is up 4.3%, not bad considering at the end of last week (three trading days) I was down about 40%. So this week I was able to bring the account up substantially.

90 trades:
44 winners
(24 trades to median price only)
46 losers
(21 trades to median price only)

My profits were from the trades to the median price only, the 45 trades to the opposite price level basicly cancelled themselves out. Therefore, those 45 trades did not add to or subtract from my account.

However, I also used Strat #5 (median price level trades only!) and VP on four other currency pairs outside the forward test (eur/cad, eur/jpy, gbp/jpy, nzd/usd) and here's where I really made profits, particularly on the gbp/jpy. Since I started using VP, I have consistently profited on this pair during the Asian session.

In fact, it has been my observation that VP data is most useful in the first twelve hours after it becomes available. Many days after 7am est, I have watched trades in profit turn around and loose value or even become unprofitable and close for a loss. My account would show a greater profit if I had analyzed all open trades at 7am est and made selective decisions about what action to take with each trade using t/a as a basis for such decisions. But since this forward test is all about finding a "set & forget" system, I resisted the temptation and let all trades run to 5pm est, and my account IS up.

These observations are just that: early observations about how VP can be used effectively. We still have a lot of work left in our forward test before more definitive conclusions can be reached; I just wanted to share these random thoughts with those who are observing.

Good points about being more selective .. I have also noticed that when I check the open trades in the am they are more likely to be in profit....

Also I would think that range analysis would also help .. useing the Vegas range spreadsheet would give one a bias based on previouse days..

most of my trades are single mediam profit target as the range mediam usually cancels out the 50 pip target. This usually happens to 4 of 6 pairs.

CRB data for 6 pairs

One of the reasons for data discrepancies may be due to something I noticed in your history spreadsheet. All of the currencies were forex pairs except one. The BP-cash.hst does not have the same identity in VP as GBP-USD cash.hst output file. Now the numbers may be the same in each but I don't know if they are or are not. Regardless would you furnish the GBP-USD-cash.hst file instead of the bp-cash.hst file. I want to eliminate any possible opportunities for error.

Thanks for all your great work.

FT

FT..............I will have to contact VP about this discrepency on Monday. My 10 pairs only include the BP-cash and not the gbp/usd; I think they are the same, but will confirm and let you know.

My week 2 results are posted...sorry for the delay. I will try to have future data posted by Saturday evening.

Thx...looking forward to results of all.

drjoe

Hello everyone..............

I will try to have the first two weeks results posted tomorrow.
Since spudbox has gone missing, I am doing Strat #3 as well. Just 12 more trades, and I think we need all the data for our forward test.

Ran out of time this weekend to post about VP and "the knife", I'll have that later in the week.

FT.............I contacted VP this am and you are correct I have had the wrong market. Just had it changed to gbp/usd and the attached file is the history for that pair. Thanks for the observation!

I know I am not the only one who had an incredible day with VP!

Strat #5 took my demo account up almost 20% with "set & forget" trading. Monitoring trades earlier in the day would have produced a bigger gain.

However............I did monitor my 'real' account trading with Strat #5 and gained 28% with 'real' money.........................WOW!!!

This VP stuff is awesome!

Sounds great BB. I am assuming you are trading a very small dollar amount in your real trading at this time...? Anyways, this is promising to say the least...don't know my results yet as I am at my office and about to get done for the day.

Look forward to hearing more on the results and your evaluation with the KNIFE when you get the time...

drjoe

This put a giant smile on my face. There is no question that "set and forget" has a pretty good chance with VP. For many it will be the most profitable way because some people do a worse job when monitoring than when not.

Good trading,

FT

All data from the three providers CSI, GEN, CRB is now in. Thanks for posting it for me guys. I'll post the massaged results in the next day or so.

Again thanks for all your efforts. This has been an illuminating exercise.

FT

Thanks for all your help and direction, FT. Looking forward to the results.

Ya thats great .. looking at strat 4 .. it was actually was more profitable than strat 5 .. last week ... .

Both 4 & 5 have the best results with EUR/US and USD/CHF ... so far .... lets hope another couple of weeks they are consistent. ..

Ok so here is the comparison. I've run a few different scenarios through this data but I did not include them in the spreadsheet because they were cumbersome to explain. However the data as it sits I believe to be accurate for data and provider. For no other reason than the order in which I got the data -- I listed CRB in the first column. After that I simply compared the other data provders to CRB. This has nothing to do with any belief that CRB should be the standard to compare to. Without a reference standard -- It was just a lot less work than tripling the size of the spreadsheet and using each provider as the standard to compare the other two with -- which anyone can do with copy and paste if they so choose.

As for the data itself it is appalling that the differences between data providers are so diverse. Some being more than 80 pips out, lots around 50 pips or so, and the number of days that providers are out more than 10 pips of each other is astonishing. I'd like to find the explanation for such dramatic differences. I'm surprised that VP has not sorted this out long ago. However I'll reserve judgment until I get some explanations. Anyone?

Worth of note is that Genesis had only one missing day of data in each currency. CRB had at least three missing days of data in each currency. CSI had one missing day of data in each currency and 12 YES that's TWELVE missing days of GBP data. Each of you should complain to your data provider for not replacing those missing days of data. For the amount they charge more than 1.00 a day for a simple ohlc for each day it should be accurate. Also I suspect that missing data throws off the next predictions. We'd need to contact VP to find out a definite answer to that.

FT

FT.............thanks for taking the time to make this comparison!!!

After reviewing this spreadsheet, I am appalled at the difference in predicted h/l!!!

Can we take this data and compare with the actual h/l for each day to determine which provider's data provided the most accurate predictions?

The differences shown in this spreadsheet would be, could be, significant enough to have a genuine impact on a trading system, 'set & forget' or otherwise.

I would like to dig deeper into this!

I sent a note off to one of VP's directors and advised him of the issue. Let's see what they come up with. In the mean time each of you could lodge a complaint to your respective providers and ask them about the data discrepancies and the missing data.

FT

Enclosed is the answer to my query about discrepancies between providers of VP data:


______________________________________________________


Good Afternoon:<o></o>

Lane asked me to look into the problem you reported. In short, the prices reported by all three providers are different and this causes the predictions to be different. This is because there is no single exchange for FOREX but rather a conglomerate of banks and trading firms.<o></o>

<o></o>After analyzing the accuracy VantagePoint 6.3 predictions for U.S. Dollar / Canadian Dollar we have come to the following conclusions:<o></o>

<!--[if !supportLists]-->1.<!--[endif]-->The accuracy of the predictions is the same for all three data providers. <o></o>

<!--[if !supportLists]-->2.<!--[endif]-->The variance between providers is not significantly greater than the error within each provider.<o></o>

<table class="MsoNormalTable" style="width: 410.2pt; margin-left: 40.65pt; border-collapse: collapse;" border="0" cellpadding="0" cellspacing="0" width="547"> <tbody><tr style="height: 12.75pt;"> <td style="padding: 0in 5.4pt; height: 12.75pt;" nowrap="nowrap" valign="bottom"> USD/CAD 3/18/04 – 2/15/06<o></o>

</td> <td style="padding: 0in 5.4pt; height: 12.75pt;" nowrap="nowrap" valign="bottom">

PHIGH<o></o>

</td> <td style="padding: 0in 5.4pt; height: 12.75pt;" nowrap="nowrap" valign="bottom">

PLOW<o></o>

</td> <td style="padding: 0in 5.4pt; height: 12.75pt;" nowrap="nowrap" valign="bottom">

PTS<o></o>

</td> <td style="padding: 0in 5.4pt; height: 12.75pt;" nowrap="nowrap" valign="bottom">

PTM<o></o>

</td> </tr> <tr style="height: 12.75pt;"> <td style="padding: 0in 5.4pt; height: 12.75pt;" nowrap="nowrap" valign="bottom"> CSI Average Error<o></o>

</td> <td style="padding: 0in 5.4pt; height: 12.75pt;" nowrap="nowrap" valign="bottom">

0.0056902<o></o>

</td> <td style="padding: 0in 5.4pt; height: 12.75pt;" nowrap="nowrap" valign="bottom">

0.0058056<o></o>

</td> <td style="padding: 0in 5.4pt; height: 12.75pt;" nowrap="nowrap" valign="bottom">

0.006952<o></o>

</td> <td style="padding: 0in 5.4pt; height: 12.75pt;" nowrap="nowrap" valign="bottom">

0.0121582<o></o>

</td> </tr> <tr style="height: 12.75pt;"> <td style="padding: 0in 5.4pt; height: 12.75pt;" nowrap="nowrap" valign="bottom"> Genesis Average Error<o></o>

</td> <td style="padding: 0in 5.4pt; height: 12.75pt;" nowrap="nowrap" valign="bottom">

0.0057272<o></o>

</td> <td style="padding: 0in 5.4pt; height: 12.75pt;" nowrap="nowrap" valign="bottom">

0.005802<o></o>

</td> <td style="padding: 0in 5.4pt; height: 12.75pt;" nowrap="nowrap" valign="bottom">

0.006981<o></o>

</td> <td style="padding: 0in 5.4pt; height: 12.75pt;" nowrap="nowrap" valign="bottom">

0.012165<o></o>

</td> </tr> <tr style="height: 12.75pt;"> <td style="padding: 0in 5.4pt; height: 12.75pt;" nowrap="nowrap" valign="bottom"> CRB Average Error<o></o>

</td> <td style="padding: 0in 5.4pt; height: 12.75pt;" nowrap="nowrap" valign="bottom">

0.005643<o></o>

</td> <td style="padding: 0in 5.4pt; height: 12.75pt;" nowrap="nowrap" valign="bottom">

0.00579<o></o>

</td> <td style="padding: 0in 5.4pt; height: 12.75pt;" nowrap="nowrap" valign="bottom">

0.006978<o></o>

</td> <td style="padding: 0in 5.4pt; height: 12.75pt;" nowrap="nowrap" valign="bottom">

0.012168<o></o>

</td> </tr> <tr style="height: 12.75pt;"> <td style="padding: 0in 5.4pt; height: 12.75pt;" nowrap="nowrap" valign="bottom"> Average Variance<o></o>

</td> <td style="padding: 0in 5.4pt; height: 12.75pt;" nowrap="nowrap" valign="bottom">

0.005920363<o></o>

</td> <td style="padding: 0in 5.4pt; height: 12.75pt;" nowrap="nowrap" valign="bottom">

0.006075605<o></o>

</td> <td style="padding: 0in 5.4pt; height: 12.75pt;" nowrap="nowrap" valign="bottom">

0.002966532<o></o>

</td> <td style="padding: 0in 5.4pt; height: 12.75pt;" nowrap="nowrap" valign="bottom">

0.002125<o></o>

</td> </tr> </tbody> </table> Error = ABS(Predicted Price – Actual Price) <o></o>

<o></o>Variance = ABS(MAX(Predicted Price CSI, Predicted Price Genesis, Predicted Price CRB)-MIN(Predicted Price CSI, Predicted Price Genesis, Predicted Price CRB))<o></o>

If you read “What’s New in VantagePoint 6.3” in the online help you will see:<o></o>

Standardized data- Combined trading session data is used whenever it is available and supported by all three data vendors. In addition, the intermarkets use continuous contracts which are consistent among all three data providers. This standardization improves accuracy<o></o>

<o></o>This refers to a major effort on our part to make the data as consistent as possible across providers, resulting in VantagePoint 6.3 being incompatible with the data used for all previous versions of VantagePoint. In many cases, one provider was reporting prices for an electronic session while others were reporting the pit session. In other cases, they were computing continuous contracts differently. We coordinated with the data providers to make the numbers as consistent as possible. Then we tested every market with every data provider retraining as necessary to achieve at least 70% accuracy for the neural index.<o></o>

<o></o>The average price discrepancy for USD/CAD among the three data providers is .005. When confronted with this, we got the following answer: Basically that there is not one source of data for FOREX markets. It is a conglomerate of banks and trading firms. Therefore, the prices will always vary by a couple of ticks depending on the source used. As a matter of policy, the data providers do not provide the sources for their non-exchange traded data.<o></o>

<o></o>In conclusion, my advice is to choose the data provider whose price most closely matches the price wherever you buy and sell from. <o></o>

<st1ersonname st="on">Len Pollard</st1ersonname><o></o>

Software Development Team Leader<o></o>

(813) 973-0496 x117<o></o>

[email protected]<o></o>

<!--[endif]--><o></o> 5268 Village Market<o></o>

Wesley <st1lace st="on">Chapel, FL <st1ostalcode st="on">33543</st1ostalcode></st1lace><o></o>

http://www.tradertech.com <o></o>

fax: (813) 973-2700 <o></o>

Copyright (c) 2006 Market Technologies, LLC. All Rights Reserved.


______________________________________________________________


I"ve posted my response in the next post...FT<o></o>

I advised Len that the average differences in data discrepancies we are experiencing are not the problem but the anomolies over 10 pips are the real issue. I mentioned that such discrepancies do not occur in the real world with such frequency and magnitude or the arbitrage opportunities in forex would be huge -- but in fact they are not.

His response was to ask if we can provide EOD data from the companies we are getting our trading data from as a comparison. He also indicated that he cannot simply go to CRB and say that Genesis data is different because CRB would just say "we get our data from another source and that is why our figures are different".

At this point one thing we can do is check the EOD data from our own brokers or chart providers and determine which of the data sources (crb, CSI, GEN) are most closely in line with our own trading data (dealing station).

Regardless after further discussion I was pleased that Len agreed to compile actual data (as opposed to the predicted data) and look further into this issue on our behalf and I'll keep you posted of new developments.

One thing that did come out in our discussion that we will most likely find very useful is that there is a one to one correlation between input data and ph/pl output. Therefore I'd say this confirms one contributors previous observation in this forum -- that when there is a gap in market data over the weekend it is advisable to add or subtract the gap to the predictions for the day following the gap. For example if the predicted high for the euro is 1.2100 and the gap over the weekend was +80 pips then the predicted high for the euro should be 1.2180 and the predicted low should also have that amount added.

I have not seen any new posts here lately. I hope everyone is still continuing with their forward testing. This has been an enlightening experience so far.

FT