I’m reposting an old article from my former site about how to achieve success in any datamining or neural net/AI model development. These 9 steps were developed by my buddy, the Marketdoctor, and are in his book, Data Mining and Business Intelligence : A Guide to Productivity. If you are newbie at datamining and neural nets, I suggest picking up his book, its a straight forward and easy to understand read.
Step 1: Decide what you want to know
This is tougher than it seems. First you’ll say, oh I want to know what drives my sales but when you dig deeper you might really want to know what drives sales based on your marketing campaigns. Take the time to ask questions and really think about what you want to discover before you spend the time building the model!
Step 2: Select the Relevant Performance Measure
After you decided what you want to find out from your data, you have to identify the relevant performance measure. This essentially means what kind of metric you want to achieve for your output. Are you merely looking for a simple answer, such as is the trend UP or DOWN? Or do you want to know the age group of teenagers who buy a particular brand of your soap?
Step 3: Decide what Instance the Data will be
Next, you have to inspect the data you have at hand and decide the time frame you wish your results to be in. Do you want to know the monthly, weekly, or daily trends of your stock market models or quarterly results from your market campaign?
Step 4: Identify your Driving Variables
Once you have your data and its in the right format you want, you have to determine which variables are the likely drivers that explain what’s causing your events to occur. We discussed driving/input variables at length in Lesson I of Building an AI financial data model.
Step 5: Acquire the Data
After you’ve done all that, you can build your data warehouse. Now download and compile all your data into a spreadsheet or database. See how much thought goes into this if you want to do it right?
Step 6: Visually Inspect the Data
This is where you look for holes in your data. Often I’ve seen missing bits of data or corrupted data such as integers in a categorical columns. This gets really tedious if its volumes of data but t must be done. Tip: YALE alerts you if you have missing data!
Step 7: Transform the Data
Sometimes the raw data you have may not be presented in the best way for you to mine it and you may have to add additional calculations (standard deviations or % returns) to it. In other instances you identify the strange data spikes, called outliers, in the data sets (you should delete these).
Step 8: Mine the Data
Ah, at last! You mine the data!
Step 9: Inspect Your Results
Does the data mining output make sense? Did it meet your assumptions or did it give you something radically different. You should always review and carefully analyze your results because you never know if you made a big blunder or the discovery of the century!
There you have it folks, datamining and the building of a neural net/AI model in 9 easy steps!
[tags]YALE, Datamining, AI, NeuralNets, Howto, FAQ, Success, Development[/tags]
After my Random Thoughts About Trends post, I decided to take a deeper look at one of the Water ETF’s available. I did a quick check on the Powershares Water Resources ETF (PHO) and its holdings. Despite a small correction between March and June 2006, PHO has been steadily marching upward. I might consider adding this ETF to my 401K account.
