Galaxy Summit 2022 was held this week(03/29), with this year’s topic being ‘The Future Enterprise’. It’s difficult to talk about the future of anything without the subject of AI coming up, so naturally we were happy to get involved.
InfuseAI CEO, CL Kao, hosted the the AI Deployment Challenges panel with three guests, each bringing a unique perspective to the discussion:
- Fabiana Clemente — Founder and CDO at YData
- Kai Yang — VP of Product at Landing AI
- Angus Kong — Director of Engineering at Fazz Financial Group
One thing was clear after the panel — it’s all about the data.
CL opened by emphasizing that there has been a tremendous growth in the number of AI and machine learning tools, the “modern data stack”, and ecosystem of related tools. However, reliably deploying models in production, and keeping them up-to-date is a challenge — which is why it’s the topic of this year’s panel.
To start the discussion CL asked the panelists for their failure stories of AI in production.
Realistic expectations
Kai was first to offer insight into model deployment failure stating that “models [never work] right out of the lab”. Therefore, it’s essential to set the expectations of the client. The client should understand that AI is not a magic solution that is guaranteed to work on the first deployment.
The other important consideration is the data. Kai recounted an issue they had with a client who provided sample image data for a visual inspection project that did not represent production data. Together with unrealistic expectations on the potential performance of the model, this resulted in a difficult situation for the AI team, and an unsatisfied client.
“We deployed the first model in shadow mode… then we used the right MLOps platform… to collect data, fine tune the model, and used MLOps to facilitate the performance monitoring… [it] turned out to be a win-win for my team, and my customers” — Kai Yang, Landing.ai
The best method to ensure the client is happy, Kai explained, is to first deploy a model in shadow-mode. That is, let the model make predictions, but not decisions. The AI team can then monitor the model performance and perform fine tuning with MLOps tools, before putting it into production.
Incomplete data leads to bias
The subject of data resonated with the rest of the panel, and data became the hot topic for the rest of the discussion.
The quality of data was a key point for Fabiana , who mentioned noisy data, incomplete data, and incorrect variability, as being issues that can lead to issues in the model.
Making assumptions about data can also be dangerous. A common assumption, according to Fabiana, is that “big volume [of data], equals quality”, which leads to the misconception that there is enough data for a specific use-case. To build a model from such data, Fabiana explained, could result in bias, and ultimately a failed model deployment.
Fabiana also raised some questions that data science teams should be asking themselves when considering the quality of their data:
- How can data science teams show that the data is not good enough?
- How can data science teams set expectations on the results that can be obtained from data?
- How can data science teams iterate on data to make it more suitable?
Attack the model
Bias is a serious issue for data scientists, “it’s not easy to spot”, said Angus, “you need to be creative to find bias in your model”. Angus explained that while at Google they discovered bias while creating Gmail’s Smart Reply feature — the model would incorrectly infer male or female pronouns based on certain keywords contained in the email subject.
Angus said they formed a team that would essentially “attack the model” to figure out all possible biases that may have been introduced from the data.
CL agreed that if the data is already biased, then a huge amount of work is required to deal with it. The discussion then lead onto Andrew Ng, and his concept of data-centric AI — the idea that data quality is the key, instead of iterating through models.
Bringing data to the forefront, CL then asked the panelists — is data-centric a new concept?
Data is the focus of iteration
“We’ve all heard “garbage in, garbage out, It’s common knowledge” said Fabiana. To be data-centric means that “data is the central object and the focus of iteration”.
“You are working under the assumption that your data can be improved, and can be de-noised… [through] labeling…better data management, profiling the data … so you can avoid inconsistencies and the introduction of biases” — Fabiana Clemente, YData.ai
Naturally, as a co-founder at YData, Fabiana is passionate about data quality. It’s data being at the core of your focus that defines a data-centric approach.
Data-centric isn’t new, but the tools and best practices are
Kai agreed that the concept of data-centric AI is not new. Even so, he explained, one of the most famous computer vision datasets, ImageNet, likely contains “100,000 labeling issues”. An eye opening figure, for sure. “Even knowing that fact” Kai continued, “ both academia and industry are still using ImageNet as the benchmarking standard”. Kai rephrased to emphasize his statement — “We know there’s a problem, but we still use it”.
“The high-level concept of data-centric AI is reasonably [well known], but the tools and best practices… are relatively new” — Kai Yang, Landing.ai
Data is the key
“People are paying way more attention to… the model itself, and ignore how critical the data is.” said Kai, referring to a project his team did researching the main focus of AI papers for the past 5 years — “only 1% of the papers in the AI community talk about data”. Model-related technology was the focus of the others, he explained, supporting his team’s findings that data is not the primary focus in most AI projects.
In terms of his customer’s projects, Kai went on, “99% of the time, the problem lies in the data, not with the model”,
Angus offered a different perspective, that while the concept of data-centric AI might not be new, certain industries are only now starting to pay attention to it. “We already have powerful enough [modelling] tools…” said Angus, “now we [need to] look at where we got the garbage” referring to data quality.
Angus ended his answer with a brief but poignant remark that machine learning tools are becoming a commodity, with CL adding that the commoditization of models means that data is now the key.
Synthetic ’til you make it
The panel ended with questions between the panelists, and Kai started with a question for Fabiana about how synthetic data can be used in domains with very small initial datasets.
Fabiana split her answer between two scenarios. The first was for solving cold-start problems in projects with small ground-truth datasets. “Using the right techniques” Fabiana stated, “we can augment very small datasets”. She explained that this is a trade-off, however, until the extent of production data can be increased.
In scenarios with no data at all, “It might get trickier”. Fabianna explained that smart-synthetic data needs a small ground-truth dataset to learn from. In the cases without any data we begin to “touch on the realm of simulation, or rule based approaches”.
Fabianna explained that these approaches are not optimal, but as long as you have business knowledge, it could be enough to create a useful dataset to use as a starting point.
Synthetic data can be used for accelerating a project. An approach that Fabiana states has been used at CERN, and also in Covid-19 research, while production data was gathered.
Control the customer experience
CL then asked Angus, as a customer-facing business, about the things they consider when adopting an AI stack.
Angus pointed out that for customer-facing applications, they prefer to build ‘in-house’. That way, they can control the customer experience. For infrastructure, though, and things that are considered “non-core”, Angus stated they would consider external products.
For “empowering my data science team” Angus said, “I’m happy to introduce external resources”.
Lastly, answering CL’s last question on the subject of data iteration frequency, Kai explained that it should be taken on a case-by-case basis. For medical applications with little new data on a regular basis, the data does not need to be updated so often, maybe monthly. However, for product recommendation engines, data might be updated every few days.
Advice corner
CL asked the panelists for some advice before wrapping up.
Angus — “It’s a continuous battle to maintain your machine learning solutions”
Kai offered three pieces of advice in relation to expectations, investment and tooling, and project scope:
- “Set up the right expectation with your boss, or client, is the first critical step. [If] you fail this, the project will likely fail”
- “Invest in.. MLOps to support the evolution of AI projects”
- “Start small, and improve on it”
Fabiana — “Work on your foundations — infrastructure and data are very important to the success of your project”
Thanks to all of the panelists who took part, and to the viewers who joined to watch!
Make sure to follow the panelists and their companies on Twitter:
Recent Comments