When I was first introduced to data mining and modeling, I felt like I had found the goose that laid the golden egg. I thought, erroneously, that I could create a predictive model that would be able to tell me what the closing price would be for a specific asset. I successfully modeled the S&P500 Spiders (SPY) and was able to predict the daily closing price within a 3% price range. I soon learned that this only worked well when the market was trending in a one direction. If the market turned on a dime, as it usually does, the model would fall apart.
So I scratched that pipe dream and focused on identifying macro trends instead.
Having successfully modeled currency, stock, and future trends, I decided to start fooling around with market timing. Iâ€™m a firm believer that market timing is critical to financial success and hereâ€™s why. When I was in MBA school, I had written an independent research paper about Hedge Funds and the various trading strategies they use. One strategy I discovered was a volatility based strategy that would invest money during times of extreme market volatility. I analyzed three fictional portfolios to see if the volatility based strategy was superior to a buy and hold, and dollar cost averaging strategy.
I used the $VIX as my volatility indicator and assumed each investor would buy into the S&P500. The results shocked me! I donâ€™t remember the exact percentages anymore (Iâ€™ll try to dig out the paper and post it) but a buy and hold investor would get a 14% return (not bad), a dollar cost averaging investor would get a 20% return (even better), and a volatility based investor would return over 100% over the same time period. Damn!
Then I read â€œThe (Mis)Behavior of Marketsâ€ and realized that its easier, and smarter, to model volatility instead of prices. If I were able to forecast and determine the magnitude of volatility for a future event, I would be ready to take profits or buy in. Now that would be truly profitable!
So I began working on a S&P500 Volatility Timing Model, which is in testing phase right now. Its not perfect and it has few bugs in it (only 66% correlated) but hereâ€™s a snapshot of volatility vs. the S&P500 over the last three months. See anything that could make you money? Note: 1 is very volatile and anything below 0 is low volatility.
I heard this first on Bloomberg Radio and then found the article. It's about the ever increasing use of data mining and AI in the financial markets.
In his cubicle overlooking the trading floor, Kearns, 44, consults with Lehman Brothers traders as Ph.D.s tap away at secret software. The programs they're writing are designed to sift through billions of trades and spot subtle patterns in world markets.
Kearns, a computer scientist who has a doctorate from Harvard University, says the code is part of a dream he's been chasing for more than two decades: to imbue computers with artificial intelligence, or AI.
That's precisely the strength of an AI model, the ability to find and learn subtle patterns and help you find an emerging (or ending) trend.
Financial service companies have already begun to deploy basic machine-learning programs, Kearns says. Such programs typically work in reverse to solve problems and learn from mistakes.
Like every move a player makes in a game of chess, every trade changes the potential outcome, Kearns says. Machine-learning algorithms are designed to examine possible scenarios at every point along the way, from beginning to middle to end, and figure out the best choice at each moment. [By Jason Kelly]
I firmly believe that data mining, AI, and machine learning trading will accelerate over the years. Who knows, maybe my little model will move markets one day! :)
On Friday morning I caught Wallstrip's chat with Tim Wolters, of Collective Intellect, who uses statistical models to extract knowledge from unstructured data sources. I really enjoyed this episode because it highlights how you can use data mining to create Event Driven Analysis (EDA). Coincidently, I had beers with the Market Doctor last night where he explained to me that part of his PhD thesis was based on EDA. Well that just opened up about an hour of technical discussion as we downed our favorite brews.
Its surprisingly easy to build a rudimentary model and evaluate press releases, earnings announcements, and other key fundamental data relative to the noise of the market. I'm quite interested in following up on EDA and have decided to build a "test" model after I finish writing and posting my YALE Lessons. I'll probably test the earnings and announcement releases of one or two companies (maybe competitors) against the S&P500 and see what I find.