Intro

I took a break for part of this evening from the lab and thought I would run down some of the blogs that I have on my roll but, haven’t had a chance to review recently. After perusing, and amidst some chuckling, the repeated gongs of doom (Mish & KD), I ended up reading a rather interesting post over on Quantivity that states Naïve Backtesting is Bogus

Deconstruct: phase one

“The most frequently cited conventional wisdom of quant trading is backtesting, often summarized as:

Wise traders do as much backtesting as possible before starting to trade a system with real money”

Well, that’s possible and I’ve done a lot of backtesting but I would not consider myself a trader in that I am more interesting in teasing out the system than the rest of it, most of the details which Wood imparts with is “Confessions” series.

Which brings me to one of my criticisms: wise traders know that the backtesting only shows them what has happened – not what will happen.

Case in point – whenever I’ve presented as system I always test it against a coin toss distribution. This gives an indication as to how close the system may or may not be to a random number generator. While much of the systems are outside that distribution, I have stated, and will continue to state: a system that works today has a limited half-life. And the half-life is shorter than you think.

Anyone who was following my Twitter stream back in Jan/Feb will have noticed that I suddenly stopped reporting on systems developed using mean-reversion as classified by my Genetic Algorithms because the results stopped making sense.

In retrospect, I believe this has to do with a regime-change but, the fact remains that a plethora of systems simply failed right before the March dive.

In that regard, I was no more wise than the guy that notices that his car is making some odd noises and promptly takes it to the shop in a manner of paranoid safety.

Phase Two

The author then proceeds to list a pseudo system generating procedure with a fair amount of snark:

1. Indicator: choose indicator (whether fundamental, technical, or statistical)
2. Data: choose long panel of data for some instrument (usually as much data as possible)
3. Backtest: build strategy by optimizing entry and exit, given indicator, over data panel
4. Profit!

To be fair to the author of this post, there is a great deal of literature on the subject of system building that does suggest that the way to build a profitable system is to find as long and consistent a trend as possible and build your system around it.

What is missing though, and I’ve covered it in more detail on my posts regarding my Spreadsheet trend discovery mechanism, is that it’s essential for even a naive backtester to split the test data between a ‘train’ set and a ‘test’ set.

This essential break between the elements of the dataset is what takes the “I’ll buy that $999.99 software from some guy on the internet” backtester right out of the realm of the authors aim and into another playing field altogether. Make no mistake, there is another word for the process that he’s describing and it’s lobbed left and right at system creators who seem to have made a system that’s deemed “too good” and it’s curve fitting.

Phase Three

OK – at this point the snark gives way to smirk and he pretty much punches himself on the way to making his essentially very small, malformed point:

If traders have learned anything during either 2007 – 2009 (or 1998 – 2002), it should be that the fundamentals of economics and finance are not stable: nearly every statistical measure in common use across nearly all asset classes exhibited inconsistent behavior over this period (mean, variance, volatility, covariance, correlation, cointegration, principal components, skew, kurtosis, etc.).

Unfortunately, this complex compound sentence upon which the internets have been subjected with has so many conflicting errors it’s difficult to sort out them all reasonably. I sort of feel like the guy in the XKCD panel having to stay up late because “someone on the internet is wrong” when I read infantile analysis as listed.

The correction should be: The fundamentals of economics and finance are never stable.

While at times they may appear to be humming along in perfect accordance to whatever set of mathematical formulae some tweed jacket wearing scholar has summarised them with, the simple fact is that it’s all an illusion.

I find this bit of snarky fluff irritating because anyone with a reasonable understanding of the Scientific Method will reject his analysis based on one simple fact: any system designer with integrity knows that his system will work “until it doesn’t”.

That’s because the scientific method starts from a point of skepticism regarding the hypothesis. In plain English it’s a response which says “your bright idea is probably wrong but, let’s check it for proof”.

In Summation

Mr. Snark points out that people have made a lot of money with the wrong system and compares it to the infamous “thousand monkeys” quip. The fact of the matter is that, you can make tonnes of money by being wrong most of the time.

Tabbed Blogger Emeritus Alphadawg put it best as the “Costanza System” where you bet against your internal instincts. And if I recall his performance this year, he did quite well by assuming that his initial take on the market was, well – wrong.

There is no silver bullet here. I’ve never talked about one, and I hope that The Fly will close my account if I start prattling about as if I’ve found one. All theories, agendas, books, systems, quants, and random guesses will be wrong most of the time.

Babe Ruth went a long way towards being right only a little over 1/3 of the time.

Any Quant with integrity will say the same.