If you’re using machine learning and AI as part of your business, you need a tool that will give you visibility into the models that are in production: How is their performance? What data are they getting? Are they behaving as expected? Is there bias? Is there data drift?
Clearly, you can’t do machine learning without a tool to monitor your models. We all know it’s a must-have tool, but until recently, most organizations had to build it themselves. It’s true that companies the size of Uber can build a solution like Michelangelo. But for most companies, building a monitoring platform can quickly transition into something kludgy and complex. In the article Understanding ML monitoring debt, we wrote about how monitoring needs have a tendency to scale at warp speed and you’re likely to find that your home-grown limited solution is simply not good enough.
This article will help you with some of the key advantages of using a best-of-breed model observability platform like Superwise, versus building it yourself.
| Let’s compare | Build | Buy |
|---|---|---|
| Time to value | 1 – 2 years for MVP. | 1 day |
| Required effort | 3 – 5 data scientists and machine learning engineers to build MVP for 2 years. | 1 engineer to integrate with Superwise. |
| Total cost of ownership | 30% of DS and MLE time to maintain and adjust a limited solution and react to ongoing business issues through troubleshooting. | Easily expands for new use cases and accommodates maintenance, upgrades, patches, and industry best practices. |
| Standardization | None. Different DS and MLE teams can use different tools, metrics, or practices to measure drift, performance, and model quality. |
Built-in. Multiple teams can work on different ML stacks and use one standard method for measurements and monitoring. |
| One source of truth | Different roles use diverse dashboards and measurements for the same use case: DS, MLE, business analyst. | Different roles get alerts and notifications on different channels but all from the same source of truth. |
Time to value
The common approach to traditional software is: if there’s an off-the-shelf solution that answers your needs, don’t waste time having your developers build one and get into technical debt. After all, building is not just about creating the tool. It involves personnel requirements, maintenance, opportunity cost, and time to value—not to mention quality assurance, patch fixes, platform migrations, and more. Face it, you want your team to be busy using their expertise to advance your company’s core business.
Required effort
As data scientists and engineers, we love to create technology that solves problems. It’s very tempting to say ‘hey let’s do it ourselves and it’ll have exactly what we want’, especially in a startup environment. If your solution supports diverse scenarios and use cases, you’ll need to customize each one. And that means a lot of extra work. When you use ML for many different use cases, you need a single tool that can handle all the scenarios—present and future—and doesn’t need to be tweaked or customized for each one. Is it really practical to invest hours of your best experts’ time to design and build a solution if one already exists and has been proven in the market? It’s worth seeking out a vendor that has already solved the problem, perfected their solution, and rounded up all the best practices in the area of monitoring.
TCO
A tool that can monitor your machine learning models’ behavior is a system like any other that you develop. It needs to be maintained and upgraded to offer visibility for new features, additional use cases, and fresh technology. As time passes, the TCO of a monitoring tool will begin to grow, requiring more maintenance, additional expertise, and time for troubleshooting. Ask yourself if this will be the best investment of your resources.
Standardization
Will your monitoring work when there are multiple teams depending on the same tool? Everyone has different needs for how to track, what to track, and how to visualize the data. If you find the right tool ready-made, you’ll be starting off with one single source of truth that meets everyone’s needs. It’s critical to have a dedicated tool that can handle all the monitoring needs of all the teams involved to ensure they are synchronized and work with standardized measurements.
One source of truth
MLOps is not just about putting the right tools in place. It’s about establishing one common language and standard processes: when to retrain, how to rollout a new version to production, how to define SLA on model issues, and more. To make this happen, you need to first initiate a central method to collect, measure, and monitor all the relevant pieces of information.
Just a few short years ago, there simply was no option to buy ready-made tools that could monitor your AI models in production. We didn’t think about whether it was worth the cost of buying them or if it was the right thing to do. We simply went and built it. Happily, today, there are so many amazing things we can take off the shelf and you should not have to sacrifice the features you need.
At Superwise, we spent the last two years building a monitoring solution that is adaptable, super-customizable, expandable – and always growing. It can handle what you need for now and the future, without you having to invest time and effort to build, troubleshoot, and maintain your own monitoring system.
This blog has been republished by AIIA. To view the original article, please click HERE.
Recent Comments