10 Experts on the Biggest Roadblocks to Bringing ML Models to Production

In its infancy, Machine Learning was hailed as a silver bullet, something that could solve your problems and automate tasks with high-quality output with less effort required than ever. It's been predicted that 87% of AI projects will never make it into production. Why? We asked ML experts what they believe to be the main reasons that ML projects fail.

TL;DR:

• A disconnect between the science and real-world application of the ML solutions and business expectations.
• Unreasonable expectations from business hierarchies on both the outcome and cost of many ML projects.
• Aiming to develop MLOps without due diligence and research into some of the challenges that could be faced along the way.
• To make an ML project successful, we need to find the right problem, use the right data, and develop the right method. Many ML projects do not meet these three requirements.
• It’s not easy for companies to find enough data to run useful and high-quality ML models.

Lavi NigamData Scientist, Gartner“The biggest reason for project failure is ML processes maturity. There are too many moving and distinct pieces in ML/DL workflows and they are not mostly tied intrinsically by a single tool/framework. Enterprises and open source projects are trying to bridge this gap and I feel in the next 3 years we will see good progress in this area. Open source end-to-end MLOps are very important since they will help with wider adoption just like Tensorflow did for deep learning. Another big issue which increases project failure is that currently data science is evolving and has not reached ‘enterprise ready consensus’ with regards to set practices for different domains.”Sparkle R.Associate Director Data Science, Johnson & Johnson“ML projects still fail because of the disconnect between the science and real-world application of the ML solutions and business expectations. We are now in a society that prefers to use the word ML to demonstrate prestige despite having no clear impact. While there are many other reasons that can contribute to a ML project’s failure, the major factors include: a lack of high-quality data, poorly designed research questions, overly optimistic business expectations and a disconnect between the developers, product owners, data scientist and the ML-based system end users.”Remo StorniSenior Software Engineer, Apple “The biggest reason for ML project failures are unreasonable expectations about what is possible and unreasonable optimism about the project. People have trouble expecting the unexpected. There are a number of ML projects that fail and often these are problems that ML can't easily solve properly right now like conversational AI. There is a much bigger number of problems where ML falls short of expectations or where the project hits unexpected data or engineering problems.”Ravi DalalSenior Computer Vision Engineer, Walmart“I think this happens because people don’t do enough prep work before starting the project. It doesn’t make sense to apply a deep learning model where a logistic regression can do the trick. So, HOMEWORK before starting the project is very critical for the success of any ML project”.Piero MolinoStaff ML / NLP Research Scientist, Stanford University"Because of the uncertainty baked in many parts of the machine learning development process. It is very difficult to assess beforehand if a machine learning project will succeed without analyzing the data, training initial models, evaluating them and then iterating. Product teams still don't know if there's enough signal in the data to begin with, how much data would they need, how expensive annotating it would be, and in many cases they don't have great ways to monitor and evaluate models. All these uncertainty sources, together with wrong expectations, may lead to failure of ML projects."Manmeet SinghMachine Learning Lead, Apple“There are a variety of reasons ML projects fail. Sometimes ML projects are initiated without synergy on expectations, objectives, and success criteria of the project between the business and ML teams. There could be other reasons ranging from lack of expected data, expertise, limitations in the technology itself for certain domains. This is a long-term investment & hence clear strategy and leadership support are necessary for success.”Jason GauciSoftware Engineering Manager, Facebook “Many projects fail because people don't trade off the benefit of success with the consequence of failure. Imagine a smart garage door opener that could open your garage door in anticipation of your arrival. If it opens when you want, it saves a button press. But if it opens when you aren't around, someone can rob your house. Because of this extreme tradeoff, the model needs to be impossibly accurate because it's competing with a button press that is almost 100% accurate”Zhiyong (Sean) XieDirector, AI, Pfizer “Most people just tried to find nails with an existing hammer. To make an ML project successful, we need to find the right problem, use the right data, and develop the right method. Many ML projects do not meet these three requirements. There is also miscommunication between the ML scientists and domain experts. It is not easy to find enough data to train the model. New methods need to be developed based only on the problem and available data.”Indu KhatriMachine Learning Lead, HSBC“Most ML projects fail because of a lack of understanding among the Business Executives about how ML models apply to their business. This usually leads to ML teams not solving the right problem for the business and underwhelming results. I have heard about projects that are stuck in research because of unclear problem statements.”Yaman KumarPhD Computer Science, University of Buffalo “Projects failing in ML, in my opinion, is chiefly due to three reasons: under-specification, over-expectation and clean toy datasets. While most of our datasets are so clean that the models trained on them hardly work in the real conditions, the ones that do are marred by the hype surrounding AI and ML. Underspecification normally comes from a lack of maturity in the field.”