I watched this great video of G5 explaining how they use H20-3, AutoML, and Driverless AI to build an NLP model and put it in production. Really cool. It uses the AWS stack and AWS Lambda. My summary notes below:
G5 started with zero in ML and in 3 months built an ML pipeline
G5 is a leader in marketing optimization for real estate marketing companies
They find leads for their customers
Owned/Paid/Earned media are breadcrumbs that site visitors leave
Clicks are not the most important interaction, it’s a call (90% of the time)
How to classify caller intent?
Build a training set from unstructured call data
Started with 110,000 unlabeled data set
Hired people to listen to them and hand score them
Problem: Every one scores things a bit different
Built a questionnaire to find the similar workers that would score the data the same way
Every day took a sample and reviewed them for consistency
Experimented with H2O-3 for testing
Took training set and rain it through H2o-3 and built a Word2Vec model
Used AutoML to understand the parameters Word2Vec model
Ended up with 500 features and enriched with metadata (day of the week, length of call, etc)
Took that processed training set and put it through Driverless AI
Driverless AI came up with a model with a high 95% accuracy model, beating the 89% benchmark.
Driverless AI made it simple to put the model in production
Results from Driverless AI feature interaction is making G5 rethink dropping the Word2Vec model and go completely in Driverless
DevOps needs to make sure the customers can use the results
A phone call comes in, it gets transcribed in AWS Lambda, then vectorizes with the same Lambda model that does the training. This is done so you can get the same feature set every time (for model scoring)
H20-3 makes transitions between R and Python easy
This model saves 3 minutes per call vs human listening, at 1 million calls a month that is 50,000 hours saved
Best part, it reduces the scoring time by 99% vs competitors
Questions and Answers
Do you need to retrain all or part of your manual labels periodically to tackle model shift? Yes, hand scoring continues and retraining is done and compared with model in production to see if shift occurs
How to maintain the models and how often to u refresh? Right now it’s a monthly cadence of review and update
I’m in Mountain View this week for our annual Sales Kick-Off meeting and will be staying for H2O World this coming Monday and Tuesday. If our registration numbers are an indication, this H20 World will be our largest ever! As the ‘hip kids’ say, it’s going to be lit!
H20 World Schedule
The first day of H2O World is all about training. In London we had 500 people show up for training on the first day and almost twice that on the second day. I fully expect that number to be larger for San Francisco. There’s going to be breakout training for AutoML, Driverless AI, and much more. I’ll be stationed at the Driverless AI booth, so come and say hi!
I’ll be stationed at the Driverless AI booth, so come say hi!
The next day is all about presentations. I’m itching to see Tanya Berger-Wolf’s presentation “AI and Humans Combatting Extinction Together.” I’m also excited to see the “Women and Inclusion in Tech Panel” on Day 2. Then there’s Dmitry’s “Lessons Learned” presentation on Kaggle Airbus Ship Detection Challenge.
There are some people I want to finally meet at H2O World, like Rueben Diaz from Vision Banco and Leland Wilkenson. Rueben did a Driverless AI deployment for Vision Banco in Paraguay and I want to hear how he did on on IBM’s Power 9 box. I also want to finally meet Leland in person. I usually hear him on the phone during company meetings. Leland is the leader of Driverless AI’s AutoViz development and has a deep knowledge of visualizations and is the author of ‘The Grammar of Graphics.’
There’s always so many presentations, ideas, awesome coffee talk, that two days are never enough. All in all, I expect this to be a fun but intense few days. There’s always such an amazing vibe when you go to H2O world, the ideas, the problem solving. It feels like changing the world for the better is possible.
Full H2O World schedule and speaker list here. Hope to see you there!
Dmitry Larko, Kaggle Grandmaster, and Senior Data Scientist at H2O.ai goes into depth on how to apply feature engineering in general and in Driverless AI. This video is over a year old and the version of Driverless AI shown is in beta form. The current version is much more developed today.
This is by far one of the best videos I’ve seen on the topic of feature engineering, not because I work for H2O.ai, but because it approaches the concepts in an easy to understand manner. Plus Dmitry does an awesome job of helping watchers understand with great examples.
The question and answer part is also very good, especially the discussion on overfitting. My notes from the video are below.
Feature engineering is extremely important in model building
“Coming up with features is difficult, time-consuming, requires expert knowledge. “Applied machine learning” is basically feature engineering” – Andrew Ng
Common Machine Learning workflow (see image below)
What is feature engineering? Example uses Polar coordinate conversions for linear classifications
Creating a target variable is NOT feature engineering
Removing duplicates/Missing values/Scaling/Normalization/Feature Selection IS NOT feature engineering
Feature Selection should be done AFTER feature engineering