What Happens to MLOps in the GenAI age?

Why MLOps?

A few years ago, our industry abounded with stories of failed machine learning (ML) projects. We all knew of a high-profile solution that someone tried to build that never quite made it all the way to production. Articles told us how “80% of ML projects fail” or that organisations were not seeing the value they expected from ML.

The challenge was that while organisations had worked hard to recruit many talented individuals with a good grasp of advanced algorithms and mathematics, they had struggled to take their knowledge and shape it into useful products and services. They had failed to operationalise the wizardry we had come to know as data science.

This is where machine learning operations (MLOps) came in. MLOps has no standard definition. For me, it means a collection of practices, techniques and capabilities that take ML solutions from idea to production, and then run them successfully post-production. This encapsulates model training, experiment tracking, model registries, pipelines, orchestration, post-deployment model monitoring, infrastructure and platforms and a whole host of other components that are required to make these solutions an operational asset and not just an intellectual exercise.

I have written about MLOps extensively, and there are a lot of fun technical details, but I think the core principles are actually quite simple. What is important to make MLOps a success in your organisation is that you consider the important dimensions of the problem, which I like to condense down into the ‘four Ps’:

People: How do you not only get the best people, but upskill all of your people to understand the core components of any machine learning solution? How do you create a passion in the organisation for creating smart products? How do you upskill those with subject matter expertise in technical areas?

Product: How do you develop a product mindset? How do you change the focus from POCs to production? How will you manage the products you build post-deployment?

Process: What should your operating model look like? How will you fund the ML development lifecycle? How can you streamline necessary but potentially cumbersome processes like governance? How do you ensure compliance but still encourage innovation and move at pace?

Patterns: How will you build for reuse? How will you take the learnings from your implementations and hold them up as exemplars for the rest of your organisation?

If you remember these core concepts and work to answer these questions then I believe you will be in a good place to build a strong operational capability for your ML and AI solutions.

As we know, things are changing very rapidly, so what else do we need to think about for MLOps given the rise of capabilities like generative AI (GenAI)?

What does GenAI mean for MLOps?

Everyone has GenAI on their minds, and organisations are definitely experimenting with it. But how many are actually going into production with it? Given the aforementioned high rates of failure of ML projects, the failure rate is also likely to be high for a new capability like GenAI. Why might it be hard to go into production for these solutions, especially given everything we have learned through the MLOps journey for more traditional ML?

There are many challenges around deploying GenAI. I’ve listed some here for you to consider.

Generative solutions are naturally less deterministic: It can be harder to estimate behaviour and define clear risk envelopes for your solution. This means that the value proposition for these projects has to be very strong and allow for fluctuation in output. Operationally, this also means that more care has to be taken to define and identify what will be out-of-tolerance behaviour in different settings.

Evaluation and monitoring are difficult: This relates to the previous point on non-determinism but is wider. The techniques and tooling around evaluation and monitoring are evolving rapidly. We can leverage a lot from natural language processing (NLP) but ground truth can be hard to define.

Cost estimation is a dark art: Even paying for third-party ‘AI-as-a-service’ solutions can still be expensive. Cost profiles are quite uncertain for the operations of these solutions, even as the costs of API calls come down. If you host, fine-tune, and serve open-source (or your own) models, the compute costs can be high.

Prompt engineering and PromptOps are new capabilities: Working with generative AI means working with natural language prompts in many settings. The creation of effective prompt templates, standardisation and curation are interesting challenges.

New deployment patterns and an evolving stack: Architecture functions and teams looking to adopt the latest best practices have a moving target to hit. The pace of innovation means that even the APIs of packages that are becoming standard in the stack are constantly changing, so solutions must be continually adapted. The architecture patterns are also evolving at pace, so our teams need to be willing to move fast to prove these designs and implement them.

Security: Generative models now provide a new attack surface for malicious actors. High-profile cases of the ‘jailbreaking’ of LLM-based services like OpenAI’s ChatGPT or Google’s Bard have shown that the right prompts can make the models behave erratically or circumvent any guardrails placed on them entirely.

There have also been attacks that access the data behind the application front-ends, meaning some sensitive data is potentially up for grabs by those with the right skills. This poses some serious questions about organisational readiness for the deployment of GenAI and should be carefully considered.

All in all, these points do raise some new considerations for our operational practice. However, it is important to note that the capabilities we need to build for MLOps in the world of GenAI is additive. We don’t need to throw away all of the work we have done on platforms, processes, people and products to this point. In fact, it is the foundation on which we will build the MLOps functions of the future. If you stick to these principles, start tackling the challenges outlined above and stay focussed on creating compelling value propositions, then you should see that your MLOps journey can continue at pace and deliver the impact you want.

This article was featured in the second Edition of our Driven by Data Magazine. You can download the magazine and read more articles like this by clicking here.