Rapidminer 5.0 Video Tutorial #10 – Financial Time Series Modeling Part 2

Posted on by

In this video we continue building a financial time series model, using S&P500 daily OHLCV data, and the windowing, sliding validation, and forecasting performance operator.  We test the model with some out of sample S&P500 data.

[flashvideo file=wp-content/uploads/2010/03/Rapidminer5-vid10.mp4 /]

This video can be viewed in HQ by clicking this link here.  Please make sure you have Quicktime or another MP4 capable reader installed in your browser.

Here are the XLS training and out of sample files.

  1. S&P500 Training XLS
  2. S&P500 Out of Sample XLS
Newer Comments →
  • JohnS

    I love it!!! Thanks so much and I’ve enjoyed the journey as well. Now, time to go play around with it :)

  • C1borg

    Im raising a beer to you, cheers! Great video as usual look forward to chatting on the forums, and swapping ideas.

  • Philmo

    Hey Tom,

    Quick question – If you have your sliding validation set to 20-5-20-5 (like in the video), would you want to be forecasting the next 5 out of sample data points? I noticed you used 7…is there any link between these numbers? Secondly, using a “filter example range” operator and a loop of some sort, would it be possible to run your whole model in a loop, and every time it iterates through, you change the train and test set (values in the filter operator), then store the forecasted data somewhere. That way you could analyze how well you would have actually done had you used this model to trade the next x periods through time. What do you think?

  • http://www.neuralmarkettrends.com Tom

    @Philmo: I just picked those numbers randomly and for the purpose of the tutorial. The way that operator works is that it opens a training window 20 examples wide and then trains on them. The training window slides down 5 example rows after the iteration is complete.

    In this example, the training window is 20 examples wide and the test window is 20 examples wide. The model uses these 20 test examples to validate the model. Then it slides down 5 examples. The horizon parameter in the sliding validation operator IS NOT the same as the horizon on the windowing operator

    The fooling around of those parameters does have an effect on the model accuracy for sure and what I’ve been fooling around with is RM’s parameter optimization operator. That operator allows you iterate different parameters, say 20-5-20-5, over a range of values to maximize your accuracy. However, it requires a lot of computer horsepower but its wicked cool.

    Regarding the looping, I know that RM has looping process operators and filters but I’ve never used them. In theory, what you want to do is possible and I’d have to spend some time figuring it out. Unless you want to fiddle with it and post your findings on the forums?

  • Fan Tom

    HOW TO MAKE A STEP INTO THE FUTURE?

  • http://www.neuralmarkettrends.com Tom

    Watch the video, I discuss it.

  • JohnS

    @Tom, I read your comment on using the parameter optimization operator and decided to play around with it some myself.

    In my dataset consisting of a date, close, 10 day SMA, and a 20 day SMA and my training set spanning 1 year, I was able to come up with “prediction_trend_accuracy: 0.630 +/- 0.069 (mikro: 0.630)”. The parameter numbers were crazy (43-2-49-5) and it took over an hour to come up with those numbers. The cool part is that it predicted the trend with an 86% accuracy.

  • JohnS

    adding “notify me” to this thread.

  • http://www.neuralmarkettrends.com Tom

    @JohnS: You can even refine your model more by adding in parameter optimization for our SVM or Neural Net model. You can have it find the best learning and momentum parameters or kernel value.

  • Philmo

    @Tom, I spent some time yesterday and managed to figure it out. My solution seems to be working and I will be double checking some of my results today. It utilizes macros and loops so it might be interesting for other users as well, especially for those who don’t know how these operators work. The only problem is that I currently have RapidMiner creating a separate .csv file each time it iterates through the loop. I wrote a quick python script to compile them, but there has to be a less hackity way of doing this…anyways check the forum if you want to see my model. Let me know if there is a better way of compiling the predictions.

  • Philmo

    @Tom,
    Tried to post a new topic but I can’t see it…do you have to approve it or something? If not I will re-post.

  • JohnS

    @Tom, I didn’t even think about that. In my test last night I setup the parameter optimization operator and ran it but then I found myself having to remove it and adding back in my performance operator and apply model operator to get my results again. I feel like I’m doing something wrong there.

    Here is what I did, taking the completed model as per your last tutorial, I removed the apply model and added an genetic parameter optimizer. I cut the performance operator out and added back inside the parameter optimizer.

    The reason I removed the apply model operator is because the output from the performance operator that previously went to the apply model opperator returned no results to the apply model from within the parameter optimizer. Even though I had the same output connected to the output of the parameter optimizer and then to the apply model, it would generate an error at run time.

    I was trying to figure out a way to derrive the optimum parameter configuration, apply them to the process, and then have the normal results spit out all in one process. I’m guessing it could be done, just don’t know enough yet to be able to figure it out.

    If you want, I can start a thread in the forum for this topic and we can continue the discussion there.

    Thanks
    John

  • http://www.neuralmarkettrends.com Tom

    @Philmo: I’ve been having “permission” problems on my forums, it seems that PHPBB isn’t as user friendly and I’ve been having a hard time granting all new users full posting access. I just approved all the topics and posts, sorry for the delay.

    @JohnS: You’ll need the performance and apply model operator regardless, that’s what tells your validation process to adjust during the iterations to bring your model to some level of convergence. The Parameter Optimization operator has subprocesses. That’s where you stick the sliding val operator, and its subprocesses).

  • JohnS

    @Tom:

    I was a bit confused by your response so I went back and had a look at what I said. I see now that I called the sliding window validator a performance operator.

    So, let me revise my statement…
    I cut the Sliding Window Validatator and pasted that inside the parameter optimizer. I left the apply model outside of the parameter optimizer and fed the model output of the sliding window validator to an output port of the parameter optimizer. I fed that same outpout to the model input port of the apply model. This is where I get the error about no data being fed to the apply model operator.

    I guess I need to feed the output of the windowing operator from my “out of scope” sample into the parameter optimization operator and then put the apply model inside the parameter optimizer operator with the sliding window validator. Then I guess I could feed the output of the apply model to an output of the parameter optimizer and then connect that to the output of the process.

    I’m going to play around with that some more tonight.
    Thanks
    John

  • Rozita Jamili

    hi , i like to get full series of videos or slides about Rapid Miner teaching , is it avialable ? please inform me if anyone knows .

  • Rozita Jamili

    Thanks alot for your very very good videos, i am starting to watch , and i am surprised how much is different rapid miner 5 and 4.1? i used before rapid miner 4.1 that was really different , but your videos are very good , becasue my research is related to data mining , i am sure will be help me alot.
    Thank alot

  • http://www.neuralmarkettrends.com Tom

    @Rozita: I’m glad you find these videos useful!

  • wendyjap

    Hi Tom, thanks so much for the very useful tutorial videos!

    I have a question for you regarding series prediction. I hope you could answer it. Here:

    You showed us in earlier tutorial that we can use ‘Predict Series’ operator to predict time series. Other than that, we can also use ‘Sliding Window Validation’ to build up and calibrate the learning model, to be then used for predicting out-of-sample data. How are these 2 operators different and which one is better applied?

    I’m thinking that using validation method should be better as the data go through training and testing stages. I tried myself to compare performance from the 2 predictions. And I found that the forecast results from ‘Predict Series’ seems to fit better with the actual data, compared to what the validation method had produced.

    Can you give some comments? Thanks!

  • http://www.neuralmarkettrends.com Tom

    @wendyjap: the Predict Series operator is just a basic operator that will make some basic predictions to see if a model can fit the predict line on top of the input data. Its better to use a model with a cross validation, or in the case of time series a sliding window validation operator, and test it with out of sample data.

  • wendyjap

    I see your point, Tom. Thanks!

  • Nilam

    Hi Thomas,
    Ur all 10 videos are just awesome…..
    Each video contains good amount of information…..
    In future hoping awesome videos from u…
    best luch for it…
    Keep doing ur nobel work like this only…
    Thank you..

  • Marvin

    Hi Tom,

    I’ve followed your tutorial to the T but for some reason, the “Forecasting Performance” operator won’t drag into the testing area, or anywhere for that matter. Any suggestions? Thanks.

  • http://www.neuralmarkettrends.com Tom

    @Marvin: I think there’s a bug for the Forecasting Performance operator in one of the RM 5.X to 5.(X+1) upgrades. If you can downgrade to say RM 5.0, I think it would work.

  • Marvin

    It worked! Thank you. It was driving me crazy.

  • Istvan Pintye

    Hi Tom

    Thanks for the useful introduction to RapidMiner,
    But are these videos also useful for trading?
    (sorry, but I had some doubt around it, and I made
    a test)

    I made everything as you said except, I used your
    method on the Hungarian futures Index (bux1012)
    [highly correlated with snp500-e-mini]

    I used End of Day data (OHLC + V) and tried to predict the
    T + 1 close price (Support Vector Machine)

    as you said (pretty difficult use this prediction
    if the output variable is continuous), so (as you
    said before) if the prediction is negative than
    the real data (intraday move) should be negative
    (and vice versa)

    so I made a scatterplot (x = real data, y = predicted)
    and (surprisingly they are absolutely uncorrelated, pearson’s correlation ~ 0.07)

    // see on the picture bellow
    http://imagerz.com/QEJHCktvAwJSU1tEQgVQ

    so I discretized the the variables (if they were
    negative predicted then became -1, (both, the real
    data and the predicted) if they were positive then
    became +1

    After then, if the prediction was equal then the
    real (for example both were +1 or both were -1)
    then the prediction was considered ‘correct’
    If they were not equal then ‘false’

    I’ve got 84 correct observations, 56 false observations
    (all together 140 (in the out of sample data))

    That means, this method slightly better then the
    random chose // after analysing the confusion matrix
    probability of Chi-square ~ 0.02 (which means significant
    and
    slightly better than nothing or the random chose)

    But are this method really useful for trading
    after consider the slippage, the spread, the
    maximum loses, etc,

    or is it a big effort for nothing?

  • Tom

    @Isvtan:You say that the Hungarian market is closely correlated to the S&P eminis. How much so and I’m assuming its correlated for direction of the market (i.e. S&P is up, Hungary market is up, etc) That leads me to ask, what parameters did you use to set up your window/step sizes for training and validation?

    That, and including the parameters you use for the SVM (or neural net), greatly affect the accuracy of the output.

    That said, its always been close to impossible to predict an accurate numerical value, you will fail all the time. Of course, when evaluating any trading model, if you can’t gain an edge (% the trade will go in your favor) after factoring in slippage, commissions, and other fees, you shouldn’t use it. The tutorials I recorded are to show people how to set up a model, checking its accuray and validity is up to you! =)

  • delbanco2000

    tom i read the formidable work you have done linking excel and IB. Currently i envision to apply an excel model and do not have the skills to link a cell which will be the order to be sent to my IB account could you please help me

  • http://www.neuralmarkettrends.com Tom

    Delbanco2000: I never got this too work because I’m not a coder per se and my time is very limited these days. Let me know if you made progress on this.

  • Netis

    Thanks so much!!! I love it!!!

  • gfyang

    Hi, Tom,

    I found a strange error when doing exactly the same experiments shown in your video: ReadFile could not work!

    However, if I directly output the data afer ReadFile, this operator works. If I connect this operator with other operators, then it will not work, which means Number_of_examples=0 in out.

    Why? It is so strange.

    Thanks.

    Sincerely yours,
    gfyang

  • http://www.neuralmarkettrends.com Tom

    @Gyfang: You’ll have to send me a screen shot or the XML file, you must be doing something wrong because I get no errors.

  • tony

    Just finished all ten. Truly wonderful job, Tom. Can’t wait to start messing around with RM 5.0 on my own. Thank you!

  • Jason

    Hello Tom,
    I’m a beginner with RapidMiner. Thanks for you tutorial. They are pretty helpful.

    My simple problem is:
    I followed your tutorial #10. Now I wanna use Linear Regression operator to replace SVM used in your work. However, I got the error message “The attribute ‘close-0′ does not exist”.

    I do not know what is the problem, since I had no problem to do the same thing with the previous processes in your tutorial #8 and #9.

    Thanks in advance,
    Jason

  • John

    I’d love to be able to download these tutorial so I could study them at work. I do a 12 hr night shift and I don’t have internet access but I have a laptop.

    thanks for taking the time to do this for us.

    John

  • Anonymous

    @John: I provide a download link to the video with sample data files in the body of this post.

  • BruceT

    These are great tutorials! So is it possible to configure the process flow to do a walk-forward test, where it trains on 1 year of data, then tests on the next month out of sample, then moves everything forward a month and repeats? And captures the oos results for analysis, of course.

  • http://twitter.com/neuralmarket Thomas Ott

    Yeah i think you can do that with the loop operator but I never tried.

  • Marya1362

    Thanks for good video.I have a question.what’s your email?

  • Anonymous

    Just post your question here.

  • Fxtrader2011

    First, is there any possibility to register for the forum ?

    I tried to use your helpful video to build a model for the EURUSD 1H.

    I exactly took the details regarding the windowing operator, learner and the validation-operator. I checked the Result for the prediction_trend_accuracy with the Example-Files. The Process is correct. But, if I want to use EURUSD Data, I get a prediction_trend_accuracy of 0.92 or 0.86 and I think, this cannot be correct.

    So I built a little Trading System in Java and simulate Trades with a Spread of 4 Pips, but the results are still not realistic.

    Is it correct that a WindowSize of 1 in the Windowing-Operator means, that the SVM only learns one row (OHLC) of Data ?

    I also get a prediction_trend_accuracy of 0.92, if I only use 20 Rows of Data for the Training-File.

  • Fxtrader2011

    thank you for the fast answer.

    I am not sure, but the results are suspicious. A prediction trend accurarcy of 0.92, even if I use a example set of 10-20 periods and a testing example set of 100 periods. How can the trained model know about 100 periods of data with a training data size of 10-20 rows ?

  • Thomas Ott

    I need to install a new module to the forum first to give the me ability to invite people. I closed it down because I was getting too many spambots registering, be patient please!

    Setting the windowing parameters is critical to getting good results. The training and testing window width do affect your prediction accuracy. I suggest you do a parameter optimization run and vary those parameters (window training/testing widths) for your SVM process.

    Often, I’ve found on small time frames, is that because of the volatile nature of the OHLC time series data (especially in forex), the training window width will be very small. However, the converse is true for long time periods, like weekly data time frames.

  • http://www.neuralmarkettrends.com Thomas Ott

    You’d get bad results if you trained on 10-20 examples and then tested on 100 examples and you normally wouldn’t do that.

    What I suggest you investigate is using a training window width of 5 and a testing window width of 3 to start out with. The sliding window operator will slide 5 example rows per training and then test it against the next 3 examples, then slide another 5 ahead, etc.

    In my volatility presentation at RCOMM 2010, my volatility model was optimized using only training window of 2 weeks (2 example observations) because, well, volatility is volatile. You can see my RCOMM presentation on my linkedin acct, you have to connect to me first though.

  • Fxtrader2011

    I tried the settings. Now I get a Prediction Trend Accuracy of 0.69.
    The java-program with the trading system still shows a excellent equity curve.
    I used the settings from the video for defining the operator configuration.

    the training windowing operator with a horizon, window-size and step-size of 1 and create label = close

    the testing windowing operator with a horizon of 0 and no create-label

    the sliding window operator uses : training window width 5 , training window-step size 1 testing width = 3 , horizon 1

    Which parameter or configuration could lead to “Data Snooping” ?
    if I increase the window size of the windowing operator , the calculation time increases , but also the Prediction trend accuracy decreases …
    Is the SVM learning from only one row of data ?
    I thought the window size is responsible for defining the SVM-Training-Length, but if I increase the window size to 2 or 3 or 10 the Losses are getting bigger …
    This seems paradox.

  • http://www.neuralmarkettrends.com Thomas Ott

    The model will be optimized (i.e. best prediction trend accuracy) with some combination of parameters. The optimized parameters will change with the type of data set (transactional, time series, etc) and every time new data is added to the original trained set.

    That said, perhaps for your 1 hr OHLC forex data, the best windowing parameters are small and that if the windowing parameters are increased your prediction trend accuracy could very well decrease. I don’t know enough about your data to give you the right parameters, I can only make suggestions as a starting point.

    Try running a parameter optimization on your training data and vary the window training and testing widths. Try varying the kernel types for the SVM, etc. It will be vastly more time intensive but the results will tell you, for that particular trained data, what the best trend prediction accuracy would be using the resulting parameters.

  • Fxtrader2011

    it is standard 1 hour period data from EURUSD with Open High Low Close , same format as in the video , but not 900 examples only 100 or like I said 10-20 .

    If I save the model and restart RM and build a new Process :

    Read Excel (the testing data) -> Set Role -> Windowing (parameter : horizon = 0 , window size = 1 , step size = 1 and only “create single attributes” as marked true) -> Load Model -> Apply Model -> Result

    Is this free of Data Snooping and are the prediction results possible ?

    I know it is difficult to say ,but in general what settings are important to prevent those unrealistic prediction-accuracy ?

    I would like to know that, before I start a long term demo-trading-test.

    I also want to thank you for your help and the tutorial videos. I learned a lot in the last days.

  • http://www.neuralmarkettrends.com Thomas Ott

    If I remember correctly, your horizon parameter affects data snooping. If its set to 0, there is not distance between the training and testing set so technically the last row used training is part of the testing set (I think I set my horizon to 1 in the video).

    To avoid this problem, set the horizon to 1 or greater. The larger your horizon distance the less your prediction trend accuracy should be, but less data snooping.

    Additionally, training on 10-20 rows of data probably isn’t enough, you should check the “degrees of freedom” on your training data to make sure you have enough observations,.

  • Fxtrader2011

    I uploaded the project and the used training and testing data-files.

    Could you please check the process , if it is working correct ?

    That would be great.

    I also add a new attribute “direction”, which shows the predicted direction of a example.

    formula : ( prediction>close , 1 , 0 )

    but mostly it shows 1 or 0 and no other values

    http://fxtest.kilu.at/forex_test.zip

  • http://www.neuralmarkettrends.com Thomas Ott

    FX: I’d have to charge you for this level of help. If you’re interested we can discuss further,

  • Nicolás

    Hi Thomas,

    Your tutorial videos are excelent!….thanks a lot.

    I have a question for you, is there any way (in rapidminer) to create a model (times series) and update it in regular periods of time?. By example, create a model with the Monday’s data, and the next day, train (again) the same model with the Tuesday’s data.

    Greetings
    Nick.

Newer Comments →