EVOLUTION OF MODELOPS: NOW A MORE ADVANCED ARTIFICIAL INTELLIGENCE

All about Agile, Ansible, DevOps, Docker, EXIN, Git, ICT, Jenkins, Kubernetes, Puppet, Selenium, Python, etc

ModelOps is a set of automated practices and tools that help deploy, manage, monitor, and improve models in production. The approach is designed to be model-centric, which means everything is instrumented around the model, from deployment to governance to inference and monitoring to scale.

Far and wide, investment in artificial intelligence and machine learning are drastically increasing and new data science projects are underway to build predictive and analytical models for various purposes. However, while companies plan to scale up sophisticated Artificial Intelligence solutions in a reasonable time, the harsh reality is that the adoption of these solutions is often stalled because companies generally focus more on development than on the operationalization of the models. On that note, ModelOps comes to the rescue bringing advancements in AI.

 

ModelOps Tools

Since the ModelOps approach brings all the players together, several emerging start-ups, as well as enterprise companies, offer ModelOps solutions to orchestrate these components collectively in an end-to-end fully automated model life cycle. Let us have a look at the figure below showing how by managing a platform enterprises can govern and scale any AI initiatives.

Powerful platforms like ModelOp center typically integrate with development platforms, IT systems, and enterprise applications so that businesses can leverage and extend ongoing investments in AI and IT. In this way, data scientists can work at scale using the tools they know best.

 

ModelOps-

• is primarily focused on the governance and life cycle management of AI and decision models (including machine learning, knowledge graphs, rules, optimization, linguistic and agent-based models). Core capabilities include the management of model development environments, model repository, champion-challenger testing, model rollout/rollback, and CI/CD (continuous implementation/continuous delivery) integration
• enables the retuning, retraining, or rebuilding of AI models, providing an uninterrupted flow between the development, operationalization, and maintenance of models within AI-based systems
• provides business domain experts autonomy to assess the quality (interpret the outcomes and validate KPIs) of AI models in production and facilitates the ability to promote or demote AI models for inferencing without a full dependency on data scientists or ML engineers.

 

A More Advanced AI

AI answers Distress and Help-calls

Emergency relief services are flooded with distress and help calls in the event of any emergency. Managing such a huge number of calls is time-consuming and expensive when done manually. The chances of critical information being lost or unobserved are also a possibility. In such cases, AI can work as a 24/7 dispatcher. AI systems and voice assistants can analyze massive amounts of calls, determine what type of incident occurred and verify the location. They can not only interact with callers naturally and process those calls, but can also instantly transcribe and translate languages. AI systems can analyze the tone of voice for urgency, filtering redundant or less urgent calls and prioritizing them based on the emergency.

 

Predictive Analytics for Proactive Disaster Management

Machine learning and other data science approaches are not limited to assisting the on-ground relief teams or assisting only after the actual emergency. Machine learning approaches such as predictive analytics can also analyze past events to identify and extract patterns and populations vulnerable to natural calamities. A large number of supervised and unsupervised learning approaches are used to identify at-risk areas and improve predictions of future events. For instance, clustering algorithms can classify disaster data based on severity. They can identify and segregate climatic patterns which may cause local storms with the cloud conditions which may lead to a widespread cyclone.

Predictive machine learning models can also help officials distribute supplies to where people are going, rather than where they were by analyzing real-time behavior and movement of people.

In addition, predictive analytics techniques can also provide insight for understanding the economic and human impact of natural calamities. Artificial neural networks take in information such as region, country, and natural disaster type to predict the potential monetary impact of natural disasters.

Recent advances in cloud technologies and numerous open-source tools have enabled predictive analytics with almost no initial infrastructure investment. So agencies with limited resources can also build systems based on data science and develop more sophisticated models to analyze disasters.

As with every progressing technology, AI will also build on its existing capabilities. It has the potential to eliminate outages before they are detected and give disaster response leaders an informed, clearer picture of the disaster area, ultimately saving lives.

Unlocking the Value of AI in Business Applications with ModelOps

AI is fast becoming critical to business and IT applications and operations. Organizations have been investing in artificial intelligence capabilities for years to stay competitive, are hiring the best data scientist teams and are investing more and more in artificial intelligence and machine learning systems. However, implementing AI / ML models is not easy and the risk of failure is just around the corner. A solid methodology is needed to reduce this risk and enable companies to succeed.

xxxxxxxxxxxxxxxxxxxxx

What is ModelOps and what problems does it solve

AI executives have been working to get more models in business for years now. The first hurdle was getting data scientists hired and tools for rapid model creation. That problem has been solved. The next hurdle is getting those models into production in a timely, compliant manner. Companies have a backlog of models that are sitting idle and degrading — contributing no value/revenue to the business. We refer to this as model debt. Or they are working to get their first AI models into production and finding that it is much more complicated and taking much longer than they anticipated. And most important, they are realizing that models, especially ML models intrinsically bring a lot of risk inside the entire organization. Board level understand that if they want to use AI at scale, they must have some process standardization and automation, with strong governance.

Infographic elaborated by the author

Coming from the operations, and being involved in many digital transformation programs, it is not surprising seeing why ModelOps is the cornerstone of every AI initiative. There is a huge difference between the mindset/requirements/collaboration across teams in the Labs vs in Production. And it is not a first in the enterprise world: something very similar happened with software: software developers’ code was not going into production in a timely and effective way. DevOps helped us with software improving collaboration across IT teams, accelerate deploy cycles, and deliver better experiences with modern software development methodology. In a similar way, ModelOps is helping enterprises to operationalize models (AI, machine learning, traditional models). And here it is even more complicated as models drift over time and carry a huge amount of board-level risk across enterprises. Data Scientists build models with multiple tools and languages, and then in most cases, the models never see the light in production, or if they do it takes a long time — sometimes too much and the model is no more useful at that point. If they make it into production, very often models run without proper monitoring, controls and overall governance, so they do not always perform as they should and they may expose the entire company to multiple kinds of risk: compliance, reputational, etc.… And this is an enterprise-level risk.

That said, we can define ModelOps as a combination of culture, policies, practices and tools that is key to an organization’s ability to operationalize AI, ML and analytic models at scale, evolving and improving model governance and performance at a faster pace than existing/homegrown methods. This speed enables organizations to employ AI decisioning to better serve their customers, secure their businesses, and compete more effectively in their market.

Shadow AI: the key issue that organizations are going to have to contend with within the coming years

An important aspect that is often underestimated in the early stages is that ModelOp and MLOps are distinct and separate from each other. While MLOps is for Data Scientists, ModelOps is a focus primarily for CIOs. Indeed, models in production must be monitored and governed 24x7 — and regulations are coming and not only for the Financial Services Industry. And this is something that should be handled by the CIO organization. The risk is to have another situation as we had for Shadow IT — we can call it Shadow AI: each BU is putting models in production without standardization across the enterprise, and we have a wild west of models. Even the simple question: “how many models in production are there?” becomes a hard one to answer, not to talk about having visibility into the state and status of each model in production, and not to mention questions related to compliance and risk management.

If we think of Shadow IT, it was not necessarily bad, as it spiked innovation. Certainly, the CIO organization had to control it, not really eliminate it. The problem we’ve been seeing a lot, and I mention it in my recent articles, is that organizations are still treating models as some asset at the BU level, that belong to the BU and Data Scientists even in production and not as Enterprise assets that should be managed centrally, like many other shared services managed by the IT organization. This is a big mindset shift that is required. And the starting point is to understand that ModelOps is necessarily separated and distinct from Data Science. Labs and Production should be like Church and State. Data Scientists should not be asked to double down as Operational resources too, as they have neither the bandwidth nor the skillset and nor the interest of managing 24x7 complex model life cycles that ensure a proper operationalization.

CIOs should step up with the help of the Chief Risk Officers and implement ModelOps.

Potential risks for companies that operate without the right ModelOps capabilities in place

In a recent report by Forrester, it has been declared “Your AI Transformation is doomed without ModelOps”. ModelOps is of course the core capability that enterprises need to deploy, monitor, and govern AI/ML Models. Companies that won’t decide to adopt ModelOps risking having a hard time keeping up with their competition and certainly won’t exceed it or lead.

Good governance absolutely requires rigorous policies, practices and tools to enforce controls and track steps and changes for auditability, risk management and compliance. The risks of doing ModelOps badly are unreliable business decisioning (that impacts revenue), regulatory and compliance penalties and potential breaches that lead to huge financial and brand costs.

And this is not only true for highly regulated industries such as Financial Services.

• A manufacturer that uses AI to drive product design decisions, finds that a machine-learning model made errors, leading to a serious customer safety issue. To defend against liability, the company is challenged to prove that its AI model was properly developed and maintained.
• The HR department in a large retailer uses a third-party software tool to assess employees and make compensation and career advancement decisions. The tool incorporates capabilities that allow a “citizen data scientist” — in this case, an HR manager with no data science or risk management background — to create and deploy AI models to assess employees and determine salaries, bonuses and promotions. The model is deployed and operated without any involvement from the risk management or compliance teams. An employee passed over for promotion claims that the AI model suffers from bias, sues for unfair labor practices and challenges the company to prove that the model is fair.

These are just a few of the real-world challenges that enterprises of all types may soon face as AI becomes more widely and routinely employed. Indeed, Gartner predicts that 50% of AI use cases will be assessed for risk by 2024, and 15% of application leaders will face board-level investigations into AI failures by 2022. Clearly, it’s well past time to pay attention to AI compliance.

How organizations should get started to do ModelOps

To Analytics and Data Science teams: involve your IT department and Risk department early on, as they are there to take this operational aspect off your plate, and it will save costs and avoid exposing the entire company to unwanted risk.

Leverage a ModelOps platform to automate and standardize the process of designing model life cycle of production models. Most of them have lego pieces that will reduce the time to production, such as out-of-the-box workflows, controls, monitoring and remediation pre-designed processes.

Photo by Slidebean on Unsplash

Another important aspect as you select a ModelOps platform is to respect the needs of Data Scientists and of the business users: DS uses in average 3 model creation tools/languages and this number will only grow and tools will phase in and out constantly. Models run on-prem, in the cloud, and serve multiple disperse apps. This “multi-multi” situation requires an agnostic ModelOps Platform.

Final thoughts

Model operations are a must-have capability to operationalize Al at scale. It comprises tools, technologies, and practices to enable organizations to deploy, monitor, and govern AI/ML models and other analytical models in production applications. ModelOps is about more than moving bits. Deploying models doesn’t end with provisioning infrastructure and copying code. Machine learning models are unique in that they must be constantly monitored while in production and regularly retrained, requiring the collaboration of a host of stakeholders from data scientists to ops pros.

Organizations should start to understand why ModelOps is important, the capabilities they need, and how to develop or acquire them.

References

ModelOp, “State of ModelOps 2021” report. April 15, 2021

Gartner,” Innovation Insight for ModelOps”, Farhan Choudhary, Shubhangi Vashisth, Arun Chandrasekaran, Eric Brethenoux, 6 August 2020

Forrester,” Introducing ModelOps to Operationalize AI”, Kjell Carlsson, Ph.d., and Mike Gualtieri, August 13, 2020.

Market Guide for AI Trust, Risk and Security Management. September 2021, Avivah Litan, Farhan Choudhary, Jeremy D’Hoinne

MODELOPS VS MLOPS: HERE IS WHAT YOU NEED TO KNOW

The major differences between ModelOps and MLOps

One area marked by confusion today is understanding the differences between ModelOps vs. MLOps. ModelOps is the missing link for today’s approach, connecting together existing data management solutions and model training tools to the value delivered via business applications. By incorporating ModelOps into your AI pipeline, you’ll move past last-mile challenges with operationalizing AI and begin to see the return on your investments in the form of reduced costs, increased revenues, and better risk management.

Recently, ModelOps has emerged as the critical link to addressing last-mile delivery challenges for AI deployments. ModelOps is a superset of MLOps, which refers to the processes involved to operationalize and manage AI models in use in production systems. ModelOps tools provide all the capabilities of MLOps, but also provide two important additions:

1. ModelOps tools allow you to operationalize all AI models, whereas MLOps tools focus primarily on machine learning models.

2. While MLOps tools allow collaboration amongst various teams and stakeholders involved in building AI-enabled applications (data science teams, machine learning engineers, software developers), ModelOps tools provide dashboards, reporting, and information for business leaders. This provides teams with transparency and autonomy to work in a collaborative manner for AI at scale.

Because all information is governed, tracked, and auditable, ModelOps tools provide transparency into AI usage across an enterprise. Not only is this essential for monitoring model performance, drift detection, and retraining for AI models, but it enables insight into AI health. Teams can better manage and plan for infrastructure costs, while also maintaining control over access to sensitive business data through governance and role-based access control. By automating the logging and tracking of this information, data science teams, machine learning engineers, and software development teams can focus on building and maintaining systems, while business and IT leaders can easily access reporting metrics for ongoing monitoring.

ModelOps will be one key to unlocking value with AI for the enterprise. If you look at all the other parts of the AI pipeline – data management, data wrangling, model training, model deployment and management, and business applications, ModelOps is the connective tissue. It links the disparate pieces of the pipeline to deliver value through business applications. By providing a shared tool to track and manage AI assets across all management stakeholders, an organization can:

• Reduce risks associated with “shadow” solutions built outside the purview of the IT department
• Reduce redundancies leading to better allocation of resources and increased reuse of models

MLOps helps data scientists with rapid experimentation and deployment of ML models during the data science process.   It is a feature of mature and maturing data science platforms like Amazon Sagemaker, Domino Data Lab, and DataRobot. ModelOps is enterprise operations and governance for all AI and analytic models in production that ensures independent validation and accountability of all models in production that enable business-impacting decisions no matter how those models are created.  ModelOps platforms like ModelOp Center automate all aspects of model operations, regardless of the type of model, how developed, or where the model is run. MLOps tools and features are used for developing machine learning (ML) models.  It includes the actual coding of the ML model, testing, training, validation, and retraining.  Data Scientists are responsible for the model development, working closely with the DataOps and Data Analytics teams to identify the proper data and data sets for the model.  The Data Scientists are typically aligned with a line of business and remain focused on the goals of that particular business unit or a specific project. ModelOps platforms and capabilities are used to ensure reliable and optimal outcomes for any and all models in production.  It includes managing all aspects of models in production, such as inventorying models that are in production, ensuring production models are providing reliable decision-making, and adhering to all regulatory, compliance, and risk requirements and controls.  CIOs and IT Operations, working with lines of business, are responsible for establishing and implementing a ModelOps platform that meets the needs of the enterprise.

 

The Value of MLOps and ModelOps

MLOps and ModelOps are complementary solutions, not competitive ones. ModelOps solutions can’t build models, and MLOps can’t govern and manage production models throughout their lifecycle across the enterprise. Some MLOps solutions offer limited management capabilities, but the limitations tend to become evident when enterprises begin to scale AI efforts and uniformly enforce risk and compliance controls.  Additionally, the “tried and true” practice of having checks and balances between development and production operations applies to every model that is developed and put in production. ModelOps platforms automate the risk, regulatory, and operational aspects of models and ensure that models can be audited and evaluated for technical conformance, business value, and business and operational risk.  By combining these enterprise capabilities with the efficiency of MLOps tools, enterprises can exploit the investment in their MLOps tools and build a foundational platform for accelerating, scaling, and governing AI across the enterprise.