Why 85% of ML Models Never Reach Production (And How ZenML Fixes That)

The production gap in machine learning is one of the most persistent problems in the industry, and the numbers remain damning in 2026. Research consistently shows that 85% of ML models never make it to production, and approximately 45% of ML projects fail specifically due to poor monitoring and retraining pipelines. The root cause is almost never the model itself — it’s the infrastructure around it. Teams build a model in a Jupyter notebook, spend months trying to productionize it using SageMaker, Vertex AI, or a custom Kubeflow cluster, and then discover that any infrastructure change requires rewriting their entire training logic. The research-to-production handoff becomes a six-month project every single time.

ZenML addresses this with a fundamentally different architecture: your pipeline code stays entirely cloud-agnostic, and the “stack” — the collection of infrastructure components like orchestrator, artifact store, and experiment tracker — is a swappable configuration. When your team decides to migrate from AWS SageMaker to Google Vertex AI, you change the stack, not the pipeline. When you need to reproduce a training run from eight months ago, ZenML’s artifact tracking gives you deterministic access to the exact data, code, and configuration used. The result: teams using ZenML report cutting their research-to-production timeline from months to days, because the deployment machinery is already built into the framework.

What Is ZenML? The Stack-Based MLOps Framework Explained

ZenML is an open-source, Apache 2.0-licensed MLOps framework built around a single core abstraction: the stack. A stack is a named collection of infrastructure components — an orchestrator (where pipelines run), an artifact store (where data and models are saved), and optionally an experiment tracker, model deployer, and alerter. The stack abstraction is what makes ZenML’s anti-lock-in promise concrete: your @pipeline-decorated Python functions remain identical whether you’re running locally or on Kubernetes. Only the active stack changes.

ZenML has accumulated 5.2k+ GitHub stars as of 2026, reflecting strong developer adoption in a market where the global MLOps space is valued at $4.39 billion and growing at nearly 46% CAGR toward an estimated $90 billion by 2035. Unlike heavier platforms like Kubeflow (which requires a dedicated Kubernetes cluster and platform team), ZenML is designed for the majority case: a small-to-medium ML team that wants production-grade pipelines without the operational overhead. It supports 50+ integrations including MLflow, Weights & Biases, Great Expectations, Seldon, BentoML, and every major cloud provider. The Apache 2.0 license means zero vendor lock-in concerns — you own your pipelines, your data, and your infrastructure choices.

The framework positions itself as a portability layer over orchestrators, not an orchestrator itself. ZenML can use Airflow, Kubeflow Pipelines, Prefect, or plain local execution as the orchestrator — you decide based on your team’s existing tooling.

ZenML Core Concepts: Steps, Pipelines, Stacks, and Artifacts

ZenML’s programming model is built on four interlocking concepts that every practitioner needs to understand before writing a single line of code. Steps are Python functions decorated with @step — they represent a single unit of work in your ML workflow, such as data ingestion, preprocessing, model training, or evaluation. Pipelines are Python functions decorated with @pipeline that wire steps together by calling them in sequence. Stacks define the infrastructure where pipelines run. And artifacts are the typed data outputs that flow between steps, automatically versioned and tracked by ZenML’s artifact store.

Steps are the core unit of reuse. Each @step function has typed inputs and outputs, which ZenML uses to automatically serialize and deserialize data via materializers. This means if you return a pd.DataFrame from one step, the next step receives a pd.DataFrame — ZenML handles storage transparently.

from zenml import step, pipeline
from sklearn.datasets import load_iris
import pandas as pd

@step
def load_data() -> pd.DataFrame:
    iris = load_iris(as_frame=True)
    return iris.frame

@step
def train_model(data: pd.DataFrame) -> dict:
    from sklearn.linear_model import LogisticRegression
    X = data.drop("target", axis=1)
    y = data["target"]
    model = LogisticRegression(max_iter=200)
    model.fit(X, y)
    accuracy = model.score(X, y)
    return {"accuracy": accuracy}

@pipeline
def iris_pipeline():
    data = load_data()
    train_model(data)

Stacks are where the cloud-agnostic magic lives. The active stack is set via zenml stack set my-stack, and from that point forward, running iris_pipeline() uses whatever orchestrator and artifact store that stack defines — no code changes required.

Artifacts are versioned automatically. Every output of every step is stored with a version number, metadata, and lineage information. You can query any artifact from any past pipeline run, compare versions across experiments, and reproduce results exactly — solving the reproducibility crisis that plagues notebook-driven workflows.

Getting Started with ZenML: Installation and Your First Pipeline

Getting ZenML running locally takes under five minutes, making it one of the more developer-friendly MLOps frameworks available in 2026. The installation is a standard pip install, and the default stack uses local orchestration and local file storage — no cloud credentials or Docker setup required for your first pipeline.

pip install zenml
zenml init          # Initialize ZenML in your project directory
zenml up            # Start the ZenML server (optional, for dashboard access)

After init, your project directory contains a .zen folder that marks it as a ZenML repository. The default stack is automatically created — it uses the local orchestrator and stores artifacts in ~/.config/zenml/local_stores/. You can inspect your stack at any time:

zenml stack list
zenml stack describe

To run the pipeline you defined in the previous section:

if __name__ == "__main__":
    iris_pipeline()

Running this produces a structured output showing each step executing, with artifact URIs and run IDs printed to the console. The ZenML dashboard (accessible at http://localhost:8080 after zenml up) shows the full DAG, artifact lineage, and step-level metadata.

Step caching is enabled by default. If you run the same pipeline twice with identical inputs and code, ZenML skips steps whose outputs haven’t changed, retrieving cached artifacts instead. For expensive training steps or LLM API calls, this translates to real cost savings — teams report cutting iteration costs by 40-60% on data-heavy pipelines simply by letting ZenML cache intermediate results. Disable caching on a per-step basis with @step(enable_cache=False) when you need fresh outputs.

Handling Custom Data Types with Materializers

ZenML’s built-in materializers handle pd.DataFrame, NumPy arrays, PyTorch tensors, sklearn models, and more. For custom types, you define a materializer by subclassing BaseMaterializer:

from zenml.materializers.base_materializer import BaseMaterializer
import joblib

class SklearnModelMaterializer(BaseMaterializer):
    ASSOCIATED_TYPES = (LogisticRegression,)
    
    def load(self, data_type):
        return joblib.load(self.uri + "/model.joblib")
    
    def save(self, model):
        joblib.dump(model, self.uri + "/model.joblib")

Custom materializers are registered globally or per-step, giving you precise control over serialization for any data type your pipeline produces.

Building Production Stacks: Orchestrators, Artifact Stores, and Trackers

A production ZenML stack typically combines three components: an orchestrator that schedules and runs pipeline DAGs at scale, an artifact store backed by cloud object storage for durability and sharing across team members, and an experiment tracker that captures hyperparameters, metrics, and model versions. ZenML ships with integrations for all of these out of the box, and swapping components requires only CLI commands — not code changes. This is what makes ZenML’s anti-lock-in guarantee practical rather than theoretical: in 2026, teams use a local stack for development (zero config, runs in seconds) and a cloud stack for production (SageMaker + S3 + MLflow, or Vertex AI + GCS + W&B), with no pipeline code changes between environments. The 63% of organizations that report high integration complexity across ML systems typically suffer from infrastructure coupling — ZenML’s stack model breaks that coupling by design. Each stack component is independently versioned, registered, and replaceable. You can upgrade your artifact store, switch orchestrators, or add a model deployer without touching any @step or @pipeline code.

Orchestrators: Local → Airflow → Kubernetes

The key point: the same @pipeline function runs on all of these. Switching orchestrators is a stack operation, not a code change. This is ZenML’s foundational design decision.

Building an AWS Stack

# Install AWS integration
pip install "zenml[aws]"

# Register components
zenml artifact-store register s3_store \
    --flavor=s3 \
    --path=s3://my-bucket/zenml-artifacts

zenml orchestrator register sagemaker_orchestrator \
    --flavor=sagemaker \
    --execution_role_arn=arn:aws:iam::123456789:role/SageMakerRole

zenml experiment-tracker register mlflow_tracker \
    --flavor=mlflow \
    --tracking_uri=https://my-mlflow-server.com

# Assemble and activate the stack
zenml stack register production_aws \
    -o sagemaker_orchestrator \
    -a s3_store \
    -e mlflow_tracker

zenml stack set production_aws

Now iris_pipeline() runs on SageMaker with artifacts in S3 and metrics in MLflow — with zero changes to the pipeline code. To switch to GCP, register GCP components and set a GCP stack. Your pipeline code is untouched.

Artifact Store Best Practices

Configure artifact stores with versioning enabled on your cloud buckets. ZenML’s artifact lineage graph becomes invaluable when debugging production issues: you can trace any model prediction back through the exact training data version, preprocessing parameters, and code commit that produced it. Enable S3 versioning or GCS object versioning to ensure artifact immutability.

ZenML vs. Kubeflow vs. MLflow: Which MLOps Tool Do You Actually Need?

Choosing between ZenML, Kubeflow Pipelines, and MLflow is one of the most common decisions ML teams face in 2026, and the right answer depends heavily on your team’s existing infrastructure, operational maturity, and growth trajectory. ZenML positions itself as a portability layer that can use Kubeflow or MLflow internally, which changes the comparison significantly. Understanding where each tool excels prevents costly architecture mistakes that teams typically discover only after six months of implementation.

Choose ZenML when: Your team doesn’t have a dedicated platform team. You want to start local and graduate to cloud. You need to switch cloud providers or orchestrators without rewriting pipelines. You’re building both classical ML and LLM workflows in the same organization.

Choose Kubeflow when: You have a dedicated K8s platform team. Your organization is fully committed to Kubernetes and isn’t switching. You need the specific features of Kubeflow’s UI (pipeline visualizations, hyperparameter tuning via Katib).

Choose MLflow when: You only need experiment tracking and a model registry, not full pipeline orchestration. You’re adding observability to an existing pipeline system and don’t need to change the orchestration layer.

The pragmatic answer for most teams: Use ZenML with MLflow as the experiment tracker. You get ZenML’s pipeline portability and ZenML’s artifact tracking combined with MLflow’s mature model registry and UI. This combination covers 90% of production MLOps requirements.

Advanced ZenML Features: Caching, Model Control Plane, and LLMOps

ZenML’s advanced feature set in 2026 goes well beyond basic pipeline orchestration, addressing the full lifecycle of ML models in production and extending into LLMOps for teams building AI applications with large language models. These capabilities distinguish ZenML from simpler orchestration tools and represent the framework’s evolution from a pipeline runner into a complete MLOps platform. Three features in particular drive the most business value: step caching (which directly reduces cloud compute and API costs), the Model Control Plane (which gives teams a single registry with full lineage across the entire model lifecycle), and LLMOps support (which extends the same reproducibility guarantees from classical ML to fine-tuning and evaluation of large language models). Teams that adopt ZenML’s caching and MCP together typically cut their retraining iteration cycle from days to hours — not by making training faster, but by skipping work that doesn’t need to be repeated and surfacing the exact lineage of every model artifact that reaches production.

Step Caching for Cost and Speed

ZenML’s caching system computes a cache key for each step based on the step code, input artifacts, and step configuration. If the cache key matches a previous run, the step is skipped and its outputs are retrieved from the artifact store. For a typical training pipeline with expensive data preprocessing, this means:

• First run: 45 minutes (preprocessing + training + evaluation)
• Second run (same data, different model hyperparameters): 12 minutes (preprocessing cached, training re-runs)
• Third run (same hyperparameters): ~30 seconds (all steps cached)

For LLM pipelines where individual steps may cost $5-50 in API calls, caching transforms the economics of iteration.

Model Control Plane

ZenML’s Model Control Plane (MCP) — introduced in 2024 and significantly expanded in 2026 — is a model registry that understands the full lineage of every model version: which pipeline produced it, which artifact versions it consumed, which metrics it achieved, and which deployments are currently serving it. Access it via the @step(model=Model(name="iris_classifier", version="production")) decorator or the Python SDK:

from zenml.model.model import Model

@step
def register_model(accuracy: float) -> None:
    model = Model(name="iris_classifier")
    model.log_metadata({"accuracy": accuracy})
    if accuracy > 0.95:
        model.set_stage("production", force=True)

LLMOps and Agentic AI Pipelines

ZenML’s 2025-2026 releases added first-class LLMOps support. You can now build pipelines that fine-tune models, evaluate them against benchmarks, and deploy them — using the same @step/@pipeline decorator pattern as classical ML. The artifact tracking system handles prompt templates, evaluation datasets, and model weights with the same lineage guarantees.

@step
def evaluate_llm(model_id: str, eval_dataset: pd.DataFrame) -> dict:
    # Run evals against your fine-tuned model
    results = run_evals(model_id, eval_dataset)
    return {"pass_rate": results.pass_rate, "avg_latency": results.avg_latency}

@pipeline
def llm_fine_tuning_pipeline():
    dataset = prepare_eval_dataset()
    model_id = fine_tune_model(dataset)
    metrics = evaluate_llm(model_id, dataset)
    gate_deployment(metrics)

ZenML also integrates with agentic frameworks — you can run Crew AI or LangGraph agent workflows as ZenML steps, giving you reproducibility and monitoring for agent pipelines that would otherwise be opaque.

Deploying ZenML to Cloud (AWS, GCP, Azure) Without Lock-In

Cloud deployment with ZenML follows a consistent pattern regardless of provider: register cloud-specific stack components, assemble them into a named stack, and activate it — and teams report completing full cloud migrations in days rather than the months typical of tightly coupled MLOps systems. This is ZenML’s most important practical guarantee, and it works in production. The 63% of organizations that report high integration complexity across ML systems have used ZenML’s stack swap to migrate between cloud providers without touching pipeline code. The mechanics are always the same: pip install "zenml[<provider>]" for provider-specific integrations, register the components via the ZenML CLI, assemble them into a named stack, and zenml stack set <name>. From that point forward, every pipeline run targets that infrastructure with zero code changes. The three major cloud providers — AWS (SageMaker + S3), GCP (Vertex AI + GCS), and Azure (Azure ML + Azure Blob) — each have mature ZenML integrations with full step containerization support. Because ZenML wraps each step in a Docker container for cloud execution, your exact Python environment is reproducible across runs and across providers, eliminating the environment drift that causes most cloud deployment failures.

AWS Deployment Pattern

For AWS, the recommended production stack combines SageMaker Pipelines (orchestrator), S3 (artifact store), and MLflow on EC2 or AWS MLflow (experiment tracker). Add an ECR container registry for step containerization:

pip install "zenml[aws,mlflow,sklearn]"

zenml container-registry register ecr_registry \
    --flavor=aws \
    --uri=123456789.dkr.ecr.us-east-1.amazonaws.com

zenml stack register aws_production \
    -o sagemaker_orchestrator \
    -a s3_artifact_store \
    -e mlflow_tracker \
    -c ecr_registry

zenml stack set aws_production
zenml stack up  # Provisions infrastructure using the stack's deployment spec

GCP Deployment Pattern

For GCP, the production stack uses Vertex AI Pipelines (orchestrator) and Google Cloud Storage (artifact store):

pip install "zenml[gcp,mlflow]"

zenml orchestrator register vertex_orchestrator \
    --flavor=vertex \
    --project=my-gcp-project \
    --location=us-central1

zenml artifact-store register gcs_store \
    --flavor=gcp \
    --path=gs://my-bucket/zenml

zenml stack register gcp_production \
    -o vertex_orchestrator \
    -a gcs_store \
    -e mlflow_tracker

zenml stack set gcp_production

To migrate the same pipeline from the AWS stack to the GCP stack: zenml stack set gcp_production, then re-run. No code changes. This is the lock-in escape hatch that ZenML exists to provide.

CI/CD Integration

ZenML pipelines integrate naturally with GitHub Actions or GitLab CI. The recommended pattern triggers a pipeline run on every merge to main:

# .github/workflows/train.yml
- name: Run training pipeline
  run: |
    zenml stack set ${{ env.ZENML_STACK }}
    python pipelines/train.py

The ZENML_STACK environment variable switches between staging and production stacks without touching the pipeline code.

ZenML Pro vs. Open Source: When to Upgrade

ZenML Pro is the managed cloud offering that adds team collaboration, RBAC, a hosted dashboard, and managed infrastructure provisioning on top of the open-source core. The open-source version is genuinely production-ready for single teams and small organizations — ZenML does not use open-core tactics that cripple the OSS offering. The upgrade decision comes down to team size and operational overhead tolerance.

Stay on open source if: You have one ML team. You’re comfortable hosting and maintaining the ZenML server. You don’t need cross-team artifact sharing with fine-grained access control.

Upgrade to Pro if: Multiple ML teams need shared artifact stores and model registries with RBAC. You want Anthropic or Slack-based SLA-backed support. You’d rather pay $X/month than maintain the ZenML server infrastructure.

The open-source ZenML server can be self-hosted on any Kubernetes cluster or as a Docker container. Most teams start with the OSS server and graduate to Pro once the team grows beyond 5-10 ML engineers.

Production Best Practices and Common Pitfalls to Avoid

Reaching production with ZenML requires getting the framework right from the start: teams that adopt ZenML early in a project reach production 3-5x faster than those who retrofit MLOps tooling onto an existing notebook-driven workflow. This speed advantage is not accidental — it is the direct result of enforcing the right patterns from day one rather than trying to reconstruct reproducibility and lineage after the fact. Given that 85% of ML models never reach production and 45% fail specifically due to poor monitoring and retraining infrastructure, the common thread in successful ZenML deployments is disciplined adherence to a few non-negotiable patterns: deterministic steps, typed outputs, versioned pipeline runs correlated with git commits, and data validation before every retraining run. These are not ZenML-specific best practices — they are MLOps fundamentals that ZenML’s architecture makes easier to enforce consistently across a growing team. The pitfalls below are the specific ways teams break these fundamentals when using ZenML, drawn from patterns observed in production deployments across fintech, healthcare, and e-commerce. Addressing them early prevents the silent failures and debugging marathons that consume engineering time in less structured pipelines.

Design for Reproducibility from Day One

Every step output should be fully deterministic given its inputs. Avoid side effects in steps — don’t write to databases, send API calls with side effects, or depend on global mutable state. ZenML’s caching assumes determinism; if a step has side effects, disable caching explicitly with @step(enable_cache=False).

Common pitfall: Using datetime.now() inside a step without disabling cache. ZenML caches the output on the first run, and subsequent runs return the stale timestamp instead of the current time.

Version Your Pipelines Alongside Your Models

ZenML pipeline runs are tagged with the git commit SHA if you configure it. Use this: pipeline.run(run_name=f"train-{git_sha}-{timestamp}"). This gives you a direct link between model artifacts and the exact code that produced them — essential for debugging production issues six months after training.

Use Typed Step Outputs

Always annotate step outputs with Python type hints. ZenML uses these to select the correct materializer and to enable type checking at the framework level. Untyped steps fall back to pickle serialization, which breaks cross-Python-version compatibility and defeats artifact portability.

# Bad: ZenML falls back to pickle
@step
def train():
    return model

# Good: ZenML selects the sklearn materializer
@step  
def train() -> sklearn.base.BaseEstimator:
    return model

Monitor Data Drift in Production Pipelines

Configure a data validator component (Great Expectations or Evidently) in your production stack. Run validation as the first step of every retraining pipeline — if data drift exceeds thresholds, fail early rather than training on bad data and deploying a degraded model.

Set Up Alerters for Pipeline Failures

ZenML’s alerter component integrates with Slack. Register a Slack alerter in your production stack so pipeline failures trigger immediate notifications rather than silently failing:

zenml alerter register slack_alerter \
    --flavor=slack \
    --slack_token=$SLACK_BOT_TOKEN \
    --default_slack_channel_id=C0XXXXXXX

Common Pitfalls Summary

FAQ

Q: Can I use ZenML if I’m not on Kubernetes? Yes. ZenML’s default stack uses local execution with no Docker or Kubernetes required. You can graduate to cloud orchestrators later by registering new stack components — the pipeline code never changes. Many teams run ZenML on a single VM with the local or Airflow orchestrator indefinitely.

Q: Does ZenML replace MLflow or Weights & Biases? No — ZenML integrates with them. ZenML is a pipeline framework; MLflow and Weights & Biases are experiment trackers. You add MLflow or W&B as the experiment tracker component in your ZenML stack, and ZenML automatically logs runs to your tracking server. You get the best of both: ZenML’s portability plus your preferred tracking UI.

Q: How does ZenML handle large datasets that don’t fit in memory? ZenML passes artifact URIs between steps, not the data itself. Each step receives a reference to the artifact in the artifact store and materializes (loads) only the data it needs. For very large datasets, implement a custom materializer that loads data lazily or in chunks. ZenML doesn’t buffer step outputs through the orchestrator — data lives in S3/GCS, not in the pipeline executor’s memory.

Q: Is ZenML production-ready for enterprise use? Yes. ZenML is Apache 2.0 licensed, supports RBAC (in Pro), integrates with existing enterprise tooling (Airflow, SageMaker, Vertex AI), and is used in production by companies across fintech, healthcare, and e-commerce. The open-source server can be deployed on-premise with no data leaving your infrastructure.

Q: How long does it take to migrate an existing ML project to ZenML? For a typical training script, adding ZenML decorators and restructuring code into steps takes 1-3 days. The bigger investment is configuring production stacks (1-2 weeks) and adding proper artifact tracking and monitoring (1-2 weeks). Most teams reach a working production pipeline within a month. The migration pays back within 2-3 retraining cycles through caching savings and reduced debugging time.