On a recent project I was in charge of researching and checking on MLOps tools / solutions, more specifically how to wrap our models post-training on APIs.
We were using GCP (Google Cloud Platform) for our data tools, ingestion and datalake, so nothing more obvious than checking on Vertex AI.
We didn’t have anything for the serving models at the time, but we were already heavily using k8s (GKE), as our whole data stack, including Airflow, dbt and others, all being deployed / running on k8s.
We had our monitoring setup with Prometheus/Grafana on Kubernetes, and it was our main alert system with Prometheus Alerts.
As most of our resources, CI/CD and infra were already in place, we started to research for k8s alternatives to solve just the serving side of the models, so serving them as APIs to systems and end users.
And we found Seldon.io, or Seldon-core as our perfect alternative. Seldon-core is an open-source K8s native solution to deploy, serve and monitor ml-models.
It wraps your model (on any framework of your choice) into a microservices deployment within K8s, with features like A/B testing, canary deployments, easy integration with gateways as Ambassador and more.
Another factor was that Vertex AI was still in beta at the time, and so we compared some aspects between both solutions:
- Avoiding vendor lock-in:
Vertex AIties the serving layer, pipelines, and often your training workflow to GCP. So the whole ml-development cycle is locked-in.- With seldon we could run on any
K8scluster — GKE, EKS, AKS, on-prem, bare metal — so our deployment would be fully portable.
- Already-invested Kubernetes-native stack:
- We were using
K8sfor our whole data tool stack. Seldon fitted naturally into that GitOps/K8s-native workflow — same CI/CD, same manifests-as-code, same observability stack (Prometheus/Grafana). - We would need to start a parallel operational model / way of working exclusive to
Vertex AI(IAM, Cloud Console, gcloud CLI, maybe Cloud Build and others) alongside ourK8stooling and infra deployments with helm and terraform.
- We were using
- Cost control at scale:
Vertex AI’s managed endpoints carry a premium for the managed layer, and predictable high-volume inference workloads can get expensive compared to running your own optimally-sizedK8snodes (including spot/preemptible instances) with Seldon. For lean, high-throughput serving, self-managed often wins on unit economics — at the cost of your team owning ops.
- Fine-grained control over serving internals:
- Seldon gives full control over the inference graph — custom pre/post-processors, ensembling, A/B testing, shadow deployments, canarying, outlier/drift detectors (Alibi Detect integration) — all as composable, inspectable
K8sresources. Vertex AIoffers some of this (traffic splitting, model monitoring) but as a fixed managed feature set; you can’t reach into the internals.
- Seldon gives full control over the inference graph — custom pre/post-processors, ensembling, A/B testing, shadow deployments, canarying, outlier/drift detectors (Alibi Detect integration) — all as composable, inspectable
- Multi-framework, heterogeneous model support:
- Seldon was built to serve arbitrary frameworks, which is convenient if your models are heterogeneous and you want one consistent deployment abstraction.
Vertex AIsupports many frameworks too, but custom/unusual model formats sometimes need more prebuilt-container wrangling.
- No data residency / compliance ambiguity:
- Some regulated industries (finance, healthcare, government) require full control over where inference runs and what telemetry leaves the cluster.
- Self-hosting via Seldon on your own infrastructure (even in GCP, via GKE) avoids questions about what a managed service does with your model artifacts and request/response data.
PS: I think most of these concerns raised above still apply to Vertex AI in comparison with a K8s native solution as Seldon, like the complete lock-in within GCP, lack of framework freedom and lack of K8s controls over scalability and fine resource adjustment.
After such considerations we started some small POCs to deploy the Seldon-core stack, and some models.
I’ll be demonstrating a simple example, by using seldon-core v1, the same version we used on our test. Version v2 has been out for some time now, and I’ll revisit that version later on, deploying the same demo.
Preparing the podman / minikube environment (ARM-MacOS users)
This setup is mainly for Arm - Mac users, as the seldon-core-controller v1.19 does not have an image on ARM architecture. If you have a linux VM, or you already have your own cluster setup for remote access, feel free to skip this section. I didn’t want to bring qemu for emulation, and the remote setup is easy, and you could have a linux vm as well with podman and minikube on your mac, but as I had this mini-pc with only linux setup it was ideal to try a minikube remote installation. Just one notice as well, minikube is not meant to be a production cluster, so do not repeat this setup instead of having a proper cluster configuration and installation, this is only for test purposes.
I’m going to use minikube for this demo, but I’ll have to deploy it on a linux box, my little arch linux pc.
I have tried to use my own arm mac m2-pro laptop, but had some issues with the seldon-controller, on the latest version there’s no support for arm on the seldon-core-controller.
I have a podman rootless environment on this laptop, and the setup is on a previous post.
As it’s a remote box / pc, I’ll have to configure minikube to expose the single-node cluster to my network, and here’s the command I’ve used to create my minikube single-node cluster:
minikube start --container-runtime=containerd --memory=4G --driver=podman --listen-address=0.0.0.0 --apiserver-ips=192.168.1.74
Ps: 192.168.1.74 is the internal IP of the eno interface (cable), so change the IP to yours.
Explaining some of the extra arguments:
-
--container-runtime: container run-time for the cluster. To be closer to current kubernetes clusters, I do usually usecontainerd, but feel free to change it. -
--listen-address:127.0.0.1listens only on localhost,0.0.0.0wildcard makes it listen on all available addresses/interfaces, including localhost. -
--apiserver-ips: will add this ip to thekube-apiservercertificates, making that we can make our calls remotely from another cluster through the external IP.
Let’s setup our kubectl remote connectivity now.
Let’s extract the config from our context on minikube with the following command:
k config view
apiVersion: v1
clusters:
- cluster:
certificate-authority: /home/fsouza/.minikube/ca.crt
extensions:
- extension:
last-update: Sun, 19 Jul 2026 15:16:34 CEST
provider: minikube.sigs.k8s.io
version: v1.38.1
name: cluster_info
server: https://127.0.0.1:33307
name: minikube
contexts:
- context:
cluster: minikube
extensions:
- extension:
last-update: Sun, 19 Jul 2026 15:16:34 CEST
provider: minikube.sigs.k8s.io
version: v1.38.1
name: context_info
namespace: seldon-models
user: minikube
name: minikube
current-context: minikube
kind: Config
users:
- name: minikube
user:
client-certificate: /home/fsouza/.minikube/profiles/minikube/client.crt
client-key: /home/fsouza/.minikube/profiles/minikube/client.key
This is the entry on the host cluster itself, so we have to adapt this to our kubectl client context with it’s external IPs, and we have to copy all 3 certificates to our machine for configuring the context properly.
Our terraform/opentofu provider will be based on this context / config too.
The cert files we need to copy are the following: /home/fsouza/.minikube/ca.crt, /home/fsouza/.minikube/profiles/minikube/client.crt and /home/fsouza/.minikube/profiles/minikube/client.key.
Choose whatever the method you want to copy the files over to your local machine.
I’ve just use a simple scp command to copy the files over:
scp fsouza@fsouza1-wired:/home/fsouza/.minikube/profiles/minikube/client.crt \
fsouza@fsouza1-wired:/home/fsouza/.minikube/profiles/minikube/client.key \
fsouza@fsouza1-wired:/home/fsouza/.minikube/ca.crt \
/Users/fsouza/.minikube-fsouza1
In my case I’ve copied all cert files to the path /home/fsouza/.minikube-fsouza1/.
Let’s then adapt / create a new server entry / context entry on our ~/.kube/config:
apiVersion: v1
clusters:
- cluster:
certificate-authority: /Users/fsouza/.minikube-fsouza1/ca.crt
extensions:
- extension:
provider: minikube.sigs.k8s.io
version: v1.38.1
name: cluster_info
server: https://192.168.1.74:33307
name: minikube-fsouza1
contexts:
- context:
cluster: minikube-fsouza1
extensions:
- extension:
provider: minikube.sigs.k8s.io
version: v1.38.1
name: context_info
namespace: seldon-models
user: minikube-fsouza1
name: minikube-fsouza1
users:
- name: minikube-fsouza1
user:
client-certificate: /Users/fsouza/.minikube-fsouza1/client.crt
client-key: /Users/fsouza/.minikube-fsouza1/client.key
I’ve adapted here all paths to the certs copied locally, and the IP to the cluster.
And finally if we run our cluster-info command:
k cluster-info
Kubernetes control plane is running at https://192.168.1.74:33307
CoreDNS is running at https://192.168.1.74:33307/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
k get po -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system coredns-7d764666f9-x92t4 1/1 Running 1 (2d2h ago) 2d8h 10.244.0.3 minikube <none> <none>
kube-system etcd-minikube 1/1 Running 1 (2d2h ago) 2d8h 192.168.49.2 minikube <none> <none>
kube-system kindnet-vjvv5 1/1 Running 1 (2d2h ago) 2d8h 192.168.49.2 minikube <none> <none>
kube-system kube-apiserver-minikube 1/1 Running 1 (2d2h ago) 2d8h 192.168.49.2 minikube <none> <none>
kube-system kube-controller-manager-minikube 1/1 Running 1 (2d2h ago) 2d8h 192.168.49.2 minikube <none> <none>
kube-system kube-proxy-s5zwc 1/1 Running 1 (2d2h ago) 2d8h 192.168.49.2 minikube <none> <none>
kube-system kube-scheduler-minikube 1/1 Running 1 (2d2h ago) 2d8h 192.168.49.2 minikube <none> <none>
kube-system metrics-server-9d74bb658-b6rxh 1/1 Running 1 (2d2h ago) 2d8h 10.244.0.2 minikube <none> <none>
kube-system storage-provisioner 1/1 Running 2 (2d2h ago) 2d8h 192.168.49.2 minikube <none> <none>
Seldon Core / Controllers / CRDs installation
Just a graph view of the kubernetes resouces deployed:
seldon-demo-model"] subgraph kd["Deployment - seldon-demo-model-classifier*"] subgraph pod["Pod - seldon-demo-model-classifier*"] cse["seldon-container-engine"] cc["classifier"] end end sd ---> kd end subgraph ss["seldon-system namespace"] cd["Seldon Controller
Deployment - seldon-controller-manager"] end cd ---> |Injects| cse end
Ps: There are more objects deployed as part of this demo, like Services, ReplicaSets and others.
Let’s deploy the necessary components and the model now to our K8s cluster.
We have to first install the seldon-controller and CRDs needed for us to be deploying the model.
Seldon has a helm chart for deploying these resources, so it makes it very easy to deploy the whole set of CRDs and controllers.
Let’s add the repo to helm:
helm repo add seldonio https://storage.googleapis.com/seldon-charts
helm repo update
And install the operator / controller and CRDs:
kubectl create ns seldon-system
helm install seldon-core-operator seldonio/seldon-core-operator --namespace seldon-system
Check the installation reference to install a specific version.
Seldon Model - Joblib package, Image Build and Deployment
Clone the repo to follow the rest of the guide:
From now on we will need to use the resources on our github repo, better to clone it to follow through.
After the whole controller and CRDs installation it’s time to serve the model through seldon.
Steps we need to cover now:
- Generate the joblib binary.
- Build the image.
- Deploy the model.
So for the first step go to the directory SeldonMockClassifier/.
Let’s generate a new .venv python environment in order to run the training / joblib generation.
uv sync --locked
Be sure you’re using the python venv created:
source ./.venv/bin/activate
PS: I’m using uv, so check the astral guide for the installation: https://docs.astral.sh/uv/getting-started/installation/.
Navigate now to the directory SeldonMockClassifier/training-and-model-package.
Let’s run the python training and joblib generator:
python training-and-model-package/generate_model.py
Or if you didn’t source your new python venv, you can run with uv as well:
uv run python training-and-model-package/generate_model.py
This should generate a new joblib package file under the same directory: SeldonMockClassifier/training-and-model-package/model.joblib.
This package is referred in Dockerfile, so if you generate the file in a different location, copy it to this same location or change the Dockerfile.
...
COPY --chown=nonroot:nonroot --chmod=744 MyModel.py ./training-and-model-package/model.joblib .
...
With the model.joblib file generated we can now build our image.
One easy way is to use the container engine within minikube to build the image, and then the image is fully available:
minikube image build . -t seldon-mock-model:v1.0.0
Let’s check if the image has been built:
[fsouza@fsouza1 ~]$ minikube image ls
registry.k8s.io/pause:3.10.1
registry.k8s.io/kube-scheduler:v1.35.1
registry.k8s.io/kube-proxy:v1.35.1
registry.k8s.io/kube-controller-manager:v1.35.1
registry.k8s.io/kube-apiserver:v1.35.1
registry.k8s.io/etcd:3.6.6-0
registry.k8s.io/coredns/coredns:v1.13.1
gcr.io/k8s-minikube/storage-provisioner:v5
docker.io/seldonio/seldon-core-operator:1.19.0
docker.io/seldonio/seldon-core-executor:1.19.0
docker.io/library/seldon-mock-model:v1.0.0
docker.io/kindest/kindnetd:v20260213-ea8e5717
docker.io/kindest/kindnetd:v20250512-df8de77b
I see our image with the internal URI docker.io/library/seldon-mock-model:v1.0.0.
For our deployment, I have a opentofu / terraform module and resources to do the deployment.
Let’s now go to the directory infra/.
There we have two directories: deploy-models and module-seldon-model:
deploy-models- tf resources to deploy models tok8s.module-seldon-model- tf module with the actual deployment logic / resources.
We make use of the module to abstract the whole logic and resources needed to deploy the models, but provider configuration is on the deploy side, plus of course the model deployment.
Go to the directory infra/deploy-models.
Let’s init tofu or terraform (download all providers):
tofu init
Or:
terraform init
All providers functionality should be the same between opentofu and terraform.
One thing you will have to adapt to your system is the context and config path location for k8s in the infra/deploy-models/meta-kubernetes.tf:
locals {
# Change the k8s config
config_path = "~/.kube/config"
config_context = "minikube-fsouza1"
}
config_path: it’s the config with all your context and connections configurations,config_context: context that should be used to run all commands, so apply / deploy all commands here.
Change it to your k8S setup.
Feel free to change all resources names.
Check the plan before trying to apply for any inconsistencies:
tofu plan
terraform plan
To apply and deploy or module:
tofu apply -auto-approve
terraform apply -auto-approve
The module itself is using a provider called gavinbunney/kubectl, check it ouit the module at the directory infra/module-seldon-model.
It’s very useful when creating CRD resources, so non-regular api k8s resources, the ones you will find at the hashicorp/kubernetes (but not sure if they have similar nowadays).
So the resource in the module is called kubectl_manifest, you can push any arbitrary yaml resource through it, but it tracks changes to the internals, and generates in the end applies/patches and creates depending on the situation.
The caveat in using it, is that you have to yamlencode some of your vars pushed as input, and put then on the correct indent within the yaml, which in some cases can be a tedious back-and-forth task.
I recommend having the yaml ready, and then replace minimal parts of it, final values / inputs only.
In our case we have a graph resource, that it’s a complex concept within seldon, and it depends on the model, so it has to yamlencode our input.
Check the module README.md for more details.
We should be able now to see our deployment, and we can port-forward it for testing:
Switch your context to the new seldon-models namespace or include -n seldon-models in the commands below.
kubectl get all -n seldon-models
NAME READY STATUS RESTARTS AGE
pod/seldon-demo-model-classifier-0-classifier-59584b6b4c-gg249 2/2 Running 0 11h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/seldon-demo-model-classifier ClusterIP 10.109.220.66 <none> 8000/TCP,5001/TCP 5d19h
service/seldon-demo-model-classifier-classifier ClusterIP 10.100.225.180 <none> 9000/TCP,9500/TCP 5d19h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/seldon-demo-model-classifier-0-classifier 1/1 1 1 5d19h
NAME DESIRED CURRENT READY AGE
replicaset.apps/seldon-demo-model-classifier-0-classifier-56fd74ddcd 0 0 0 5d19h
replicaset.apps/seldon-demo-model-classifier-0-classifier-59584b6b4c 1 1 1 12h
replicaset.apps/seldon-demo-model-classifier-0-classifier-655d5dd86f 0 0 0 5d19h
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
horizontalpodautoscaler.autoscaling/seldon-demo-model-classifier-0-classifier Deployment/seldon-demo-model-classifier-0-classifier cpu: 0%/70%, memory: 27%/70% 1 3 1 5d19h
kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
seldon-demo-model-classifier ClusterIP 10.109.220.66 <none> 8000/TCP,5001/TCP 5d19h
seldon-demo-model-classifier-classifier ClusterIP 10.100.225.180 <none> 9000/TCP,9500/TCP 5d19h
Let’s port-forward the seldon-demo-model-classifier-classifier service:
kubectl port-forward services/seldon-demo-model-classifier-classifier 9000:9000
Choose another local port if 9000 is busy, so change [local_port]:[remote_port].
kubectl port-forward services/seldon-demo-model-classifier-classifier 9000:9000
Forwarding from 127.0.0.1:9000 -> 9000
Forwarding from [::1]:9000 -> 9000
Let’s make a call to our model:
curl -v http://localhost:9000/api/v1.0/predictions \
-H 'Content-Type: application/json' \ -d '{"data": {"names": ["sepal_length", "sepal_width", "petal_length", "petal_width"], "ndarray": [[0.1, 1.5, 5.4, 7.2]]}}'
* Host localhost:9000 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
* Trying [::1]:9000...
* Connected to localhost (::1) port 9000
> POST /api/v1.0/predictions HTTP/1.1
> Host: localhost:9000
> User-Agent: curl/8.7.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 118
>
* upload completely sent off: 118 bytes
< HTTP/1.1 200 OK
< Server: gunicorn
< Date: Sat, 25 Jul 2026 09:30:01 GMT
< Connection: close
< Content-Type: application/json
< Content-Length: 210
< Access-Control-Allow-Origin: *
<
{"data":{"names":["t:0","t:1","t:2","t:3"],"ndarray":[[2.0,3.706771757685523e-12,2.0320340153976877e-10,0.9999999997930897]]},"meta":{"requestPath":{"classifier":"docker.io/library/seldon-mock-model:v1.0.0"}}}
* Closing connection
{"data":{"names":["t:0","t:1","t:2","t:3"],"ndarray":[[2.0,3.706771757685523e-12,2.0320340153976877e-10,0.9999999997930897]]},"meta":{"requestPath":{"classifier":"docker.io/library/seldon-mock-model:v1.0.0"}}}
Conclusion
Seldon is a wonderful wrapper for serving models, but it comes with its complexity and moving parts.
If you’re invested in control, centralizing the whole setup, and having a platform team to manage and evolve it, it’s great.
It’s great as well if you’re invested in k8s, and you have already a setup for other parts of the stack for monitoring, batch and others.
The integration between batch can be made very tight, for example with jobs triggered by Airflow within k8s making use of the models.
If you want your users to have more autonomy, Vertex AI might be more suitable, with less operational burden, but with more vendor-lock-in process / technology and less flexible.
Some operational work has to be done even for Vertex AI though, like creating LBs and the whole set of logical objects behind them (NEGs, UrlMaps and so on).