Rapidminer 5.0 Video Tutorial #7 – Evolutionary Weighting Example

Posted on by

In this tutorial, we highlight Rapidminer's weighting operator using an evolutionary approach.  We use financial data to preweight inputs before we feed them into a neural network model to try to better classify a gold trend.

Video download link (HQ): Rapidminer 5.0 Video Tutorial #7

  • http://www.neuralmarkettrends.com Tom

    If you listen real carefully, you can hear my kids in the background.

  • scbhush

    Hi Tom
     
    Once again.  Thank you
    Rapid miner team must pay you that,  you are make rapid miner easy to understand.
     
    regards

  • Frank Gunseor

    Did you use a new data set for this tutorial or the same one you used previously?
    Thank you, and again a great turtorial!

  • http://www.neuralmarkettrends.com Tom

    @scbhush: Please consider joining my new forums! I haven't officially announced it yet but will soon!
    @Frank: Yes, this is new data, starting in January 2010 through early March I believe.  I just took some random symbols as a test.

  • c1borg

    Hi Tom great video again! yes caught the kids but the audio not quite as good as the previous vids.
    I have a question if you would be so kind, im getting great results with this but bad prediction (all BUY or SELL signals at the end) with any experiment I run. So its almost like the prediction tails off in the last 1/3 of the experiment, the most recent data (and the part we really want to be accurate). Any thoughts on this? I am using 1 year of daily data split 70/30, would you say this is ideal for a daily prediction?
    Lastly how do I join the forums?

  • http://www.neuralmarkettrends.com Tom

    @c1borg: I don't know why the audio got degraded in this one, i was using all the same settings and hardware.  Your question would be a great first post for the forums. If you'd be as kind to repost it there, maybe with more details and perhaps a post of the XML file, I'll give it a shot to troubleshoot it.

  • JohnS

    Tom,
    I've been trying to teach myself predictive modeling and regression analysis. I've been doing this to help me make better decisions on my stock pics and for a future application I'm planning on developing.
    I ran across your videos and notice the subject of these videos revolve around market data. I've enjoyed watching your videos and am looking forward to the last 3 videos talking about forecasting.
    I am headed to your site now to see what I need to do to be able to get access to the next videos.
    Thanks for doing these videos, they have helped me out a lot already.

  • http://www.neuralmarkettrends.com Tom

    @JohnS: The last three vids will be posted here and I decided not to lock them up.

  • JohnS

    Oh, that will be nice!
    Thanks for putting in the hard work and time to make these videos. I know it is a time consuming process having done some for work myslef and I just wanted to let you know I really appreciate it.
    Thanks
    John

  • http://www.neuralmarkettrends.com Tom

    @JohnS: Hopefully I'll be recording #8 tonight or Wednesday, it all depends on how the kids behave. =)

  • JohnS

    Sweet! I am looking forward to it!

  • David Aitken

    Tom – your tutorials are GREAT; thank you very much!

    Once I have built this one though, how would I go about saving the nicely weighted model and running real data through it to make predictions?

    For example I used your gold data from the first tutorials, training data for the weighting optimization, and now I want to run the real data into it to see how well it does.

    Pointers would be greatly appreciated!

    David

  • Vani

    Hello Tom,

    I have a quick question. I have 4 columns in my excel sheet, where 3 of them contain real values and the fourth one contains categorical values (1,2,3..). What is the data type that should be used for the categorical type data? Can i feed all of them together into Rapid Miner for cross validation?
    Your response will be greatly appreciated.

    Vani

  • Tom

    @Vani: You can try integer or nominal types, whatever works with your data better.

  • Vani

    Hello Tom,

    Thank you for your quick response! Is there any memeory limitation for RapidMiner? When I input 150,000 records, it gives me an error which says, “this process would need more memory than the maximum amount of memory avaialable”. I am trying to do a Cross-validation.

    Thank you!
    Vani

  • Tom

    @Vani: the only limitation is your PC hardware. Can you add more memory to your machine.

  • JonN

    Tom,
    Nice job with the tutorials, appreciate it.

    I wanted to ask, if you use the “Optimize Weights(Evolutionary)” operator, that would be more powerful than the “Optimize Selection(Evolutionary)” operation. Obviously, a weight of “0″ is like not selecting an attribute, but a weighted attribute retains more information for prediction.

    Have you found it to be the case that attribute weighting is better than attribute selection when working on technical analysis inputs for predicting market movements?
    Thanks,
    JonN

  • Maninred

    Thank you for your videotutorials.
    That´s great stuff.

    Is it possible to get a example spreadsheet for this tutorial?

    Greetings from germany.

  • Maninred

    I´m sorry for that -> the hard drive crash.
    I have it by my own two times until now.
    A horrible event.

    Thank you for the quick answer and again for the great work.

  • Thomas Ott

    Sorry , I lost some data files in a hard drive crash a while ago and this was a part of that crash.

  • Tonio Barmadosa

    I understand that evolutionary weighting is something similar to principal component analysis, right? It basically reduces your input features to the most important ones.

    My question is why would you apply evolutionary weighting on your data set, if the neural net would automatically assign weights for each input feature anyway?

    ps. I might not understand it correctly, since I’m kind of new to the topic.