Fraud Analytics in RapidMiner

This is a great video presentation on Fraud Analytics use case with RapidMiner. See my notes below.

Some key concepts

  • More complex model, the lower the training error but higher test error.
  • Simple models are better, try explaining them to children.
  • Data Scientists understand the technical aspect, need to communicate results with analysts.
  • Sell results to businesses. Tie $ to the results.
  • Speak same language with business. Map performance metrics to business related figures.
  • AUC and recall doesn't necessarily mean $ to the business, show how.
  • A/B testing method widely used in Marketing. Also consider a "do nothing" model and compare with implementing data science solution.
  • Don't fear sharing best practices and ideas with similar businesses.
  • Fraud model follows traditional validation method. 80% Training and 20% as Holdout.
  • Both training/holdout sets taken across same time period.
  • Handy trick, use sum of transactions as example weights. (this is cool)
  • Apply $ value to your true/false positive/negatives.
  • Compare with Default model (no model).
  • Generate a money plot!

Some questions

  • How does this relate to regression?
  • If a simple model not good enough, how do you sell a complex one like Deep Learning?
  • Is it better to have the Data Science team be embedded in the Business Unit or as a separate team?
  • How do you try to explain the uncertainty of prediction intervals to business stakeholders?
  • How do you account for seasonal drift?
  • The model will drift overtime, should the model be updated or retrained over time?
  • Do you build a model to optimize business results or is it a byproduct of the prediction?
Show Comments