Building an AI financial market model - Lesson I

Before you can begin with building your own AI Financial Market Model (machine learned), you have to decide on what software to use.  Since I wrote this article in 2007, many new advances have been made in machine learning. Notably the python module Scikit Learn came out and Hadoop was released into the wild.

I’m not overly skilled in coding and programming — I know enough to get by- I settled on RapidMiner.  RapidMiner is a very simple visual programming platform that let’s you drag and drop operators” into a design canvas. Each operator has a specific type of task related to ETL, modeling, scoring, and extending the features of RapidMiner.

There is a slight learning curve but, it’s not hard to learn if you follow along with this tutorial!

The AI Financial Market Model

First download RapidMiner Studio and then get your market data (OHLCV prices), merge them together, transform the dates, figure out the trends, and so forth. Originally these tutorials built a simple classification type of model that look to see if your trend was classified as being in an up-trend” or a down-trend.” The fallacy was they didn’t not take into account the time series nature of the market data and the resulting model was pretty bad.

For this revised tutorial we’re going to do a few things.

  1. Install the Finance and Economics, and Series Extensions
  2. Select the S&P500 weekly OHLCV data for a range of 5 years. We’ll visualize the closing prices and auto-generate a trend label (i.e. Up or Down)
  3. We’ll add in other market securities (i.e. Gold, Bonds, etc) and see if we can do some feature selection
  4. Then we’ll build a forecasting model using some of new H20.ai algorithms included in RapidMiner v7.2

All processes will be shared and included in these tutorials. I welcome your feedback and comments.

The Data

We’re going to use the adjusted closing prices of the S&P500, 10 Year Bond Yield, and the Philadelphia Gold Index from September 30, 2011 through September 20, 2016.

The raw data looks like this:


We renamed the columns (attributes) humanely by removing the ^” character from the stock symbols.

Next we visualized the adjusted weekly closing price of the S&P500 using the built in visualization tools of RapidMiner.


The next step will be to transform the S&P500 adjusted closing price into Up and Down trend labels. To automatically do this we have to install the RapidMiner Series Extension and use the Classify by Trend operator. The Classify by Trend operator can only work if you set the set the SP500_Adjusted_Close column (attribute) as a Label role.

The Label role in RapidMiner is your target variable. In RapidMiner all data columns come in as Regular” roles and a Label” role is considered a special role. It’s special in the sense that it’s what you want the machine learned model to learn to. To achieve this you’ll use the Set Role operator. In the sample process I share below I also set the Date to the ID role. The ID role is just like a primary key, it’s useful when looking up records but doesn’t get built into the model.

The final data transformation looks like this:


The GSPC_Adjusted_Close column is now transformed and renamed to the label column.

The resulting process looks like this:


That’s the end of Lesson 1 for your first AI financial market model. You can download the above sample process here. To install it, just go to File > Import Process. Lesson 2 will be updated shortly.

This is an update to my original 2007 YALE tutorials and are updated for RapidMiner v7.0. In the original set of posts I used the term AI when I really meant Machine Learning

Up next Forex Thoughts Sunday afternoon again! It’s been a busy day for us today, we spent most of it outside working in our garden. Now that the day is fading fast, I’m Aussie Dollar Drops & Yen Rallies! The Austrailian Dollar dropped this morning against the USD and I’m watching key support levels for possible entry points. So far, none of my limit
Latest posts The Ye Old Blog List Motorola: Then and Now EWM Redux Testing for mean reversion with Python & developing simple VIX system - Talaikis unsorted - Tadas Talaikis Blog Steps to calculate centroids in cluster using K-means clustering algorithm - Data Science Central Basics of Statistical Mean Reversion Testing - QuantStart Algorithmic trading in less than 100 lines of Python code - O’Reilly Media Interpreting Machine Learning Models Microsoft the AI Powerhouse Investing in the S&P500 still beats AI Trading Microsoft makes a push to simplify machine learning | TechCrunch 10 Great Articles On Python Development — Hacker Noon Introduction to Keras Democratising Machine learning with H2O — Towards Data Science Getting started with Python datatable | Kaggle Phone Addiction Version 12 Launches Today! Machine Learning Making Pesto Tastier 5 Dangerous Things You Should Let Your Kids Do The Pyschology of Writing Investing in 2019 and beyond TensorFlow and High Level APIs Driving Marketing Performance with H2O Driverless AI Machine Learning and Data Munging in H2O Driverless AI with datatable Making AI Happen Without Getting Fired Latest Musings from a Traveling Sales Engineer The Night before H2O World 2019 Why Forex Trading is Frustrating Functional Programming in Python Automatic Feature Engineering with Driverless AI Ray Dalio's Pure Alpha Fund