I found this talk to be fascinating. I've been a big fan of LIME but never really understood the details of how it works under the hood. I understood that it works on an observation by observation basis but I never knew that it permutates data, tests against the black box model, and then builds a simple linear model to explain it.
Really cool. My notes are below the video.
- Input > black box > output; when don't understand the black box like neural nets
- Example, will the loan default?
- Typical classification problem
- Loan and applicant information relative to historical data
- Linear relationships are easy
- Nonlinear relationships via a Decision Tree can still be interpreted
- Big data creates more complexity and dimensions
- One way to overcome this: use feature importance
- Feature importance doesn't give us any understanding if it's a linear or nonlinear relationship
- Gets better with partial dependence plots
- Can't do partial dependence plots for neural nets
- You can create Bayesian Networks / shows dependencies of all variables including output variable and strength of relationship
- Bummer: Not as accurate as some other algorithms
- Can give you global understanding but not detailed explanation
- Accuracy vs Interpretablity tradeoff. Does it exist?
- Enter LIME! Local Interpretable Model-agnostic Explanations
- At a local level, it uses a linear model to explain the prediction
- Creates an observation, creates fakes data (permutation), then it calculates a similarity store between the fake and original data, then it takes your black box algo (neural nets?), tries different combinations of predictors
- Takes those features with similarity scores, fits a simple model to it to define weights and scores to explain it
- Without know what the model picks up on if it's really signal or noise. You need LIME to verify!
- Can apply to NLP/Text models
- Why is it important? Trust / Predict / Improve
- LIME helps feature engineering by none ML practitioners
- LIME can help comply with GDPR
- Understanding our models can help prevent vulnerable people