Skip to main content
Version: Next

General Technical Review

  • Project: HAMi
  • Project Version: v2.8.0
  • Website: https://project-hami.io
  • Date Updated: 2026-04-13
  • Template Version: v1.0
  • Description: HAMi is a Kubernetes middleware for heterogeneous AI device sharing, isolation, and scheduling across GPUs/NPUs and other accelerators.

Day 0 - Planning Phase​

Scope​

  • Roadmap and scope process: Scope is defined through issues/PRs and maintainer review, with governance maintained in the HAMi community repo. The roadmap is maintained in GitHub issues (for example, per-version tracking such as https://github.com/Project-HAMi/HAMi/issues/1615). After each version is released, the next weekly community meeting discusses roadmap goals for upcoming releases in detail, and each goal is assigned an assignee on the issue. Community users can reply on those issues to add or request features.

  • Target personas:

    • Platform engineers: Deploy HAMi as middleware on Kubernetes 1.23+ clusters with NVIDIA driver 440+ support for straightforward integration into existing platform products. GPU management and isolation are a common product differentiator that HAMi supports end to end.
    • MLOps teams: Use GPU virtualization to reduce job wait times; rely on topology-aware scheduling and device-type selection to place training and inference workloads efficiently.
    • Individual AI workload owners: Often operate a small GPU footprint; use sharing and multiplexing to improve utilization without changing application code.
    • IT operations teams: Raise cluster utilization through GPU sharing; consolidate heterogeneous accelerators with integrated monitoring in a single cluster to lower maintenance cost.

  • Primary use case: Share and isolate accelerator resources (especially GPU memory/core) for multiple workloads on Kubernetes without application code changes.

  • Additional supported use cases: Topology-aware scheduling, multiple scheduling policies (binpack/spread), dynamic MIG support for NVIDIA, and heterogeneous device support (e.g., NVIDIA, Cambricon, Hygon, Iluvatar, MetaX, Enflame, Ascend integration references).

  • Unsupported/non-goal use cases: HAMi is not a model serving framework, training framework, or general-purpose cluster autoscaler. It focuses on accelerator virtualization/scheduling.

  • Organizations that benefit: Public cloud, private cloud, enterprise AI platforms, telecom, finance, education, manufacturing, and internet companies operating shared accelerator clusters.

  • End-user research: Formal, survey-style user research is still limited. The clearest public evidence of production adoption is CNCF end-user case studies, including multiple community-submitted stories that feature HAMi; the HAMi-filtered case study list is the up-to-date catalog on cncf.io. Representative named end users include:

Usability​

  • Interaction model: Operators and workload owners interact with HAMi through Helm, Kubernetes Pod specs, component Services, and chart-driven scheduler/webhook settings.

    • Lifecycle: Install, upgrade, and uninstall with Helm (see Installation below and the repository README quick start).
    • Workloads: Request shared accelerators with Pod requests/limits plus HAMi annotations for slices, device selection, and scheduling policy.
    • Observability: Inspect cluster-wide device and scheduling overview via the HAMi scheduler metrics HTTP endpoint; query the device-plugin vGPU monitor interface for live NVIDIA per-pod usage (listen ports are Helm-configurable; defaults are documented in the README and the configure page).

  • UX/UI: The default experience is Kubernetes-native: kubectl, the Pod/Node APIs, and YAML/Helm values. For graphical administration and monitoring, deploy the optional HAMi WebUI; Grafana-oriented examples are documented separately.

  • Production integration: HAMi is in-cluster middleware: teams must already run a supported Kubernetes cluster. In production it extends the control plane with a mutating admission webhook, scheduler extender integration (default kube-scheduler or the chart-managed scheduler), device plugins, and CDI-capable runtimes where applicable. Additional batch/AI scheduler ecosystems are documented or tracked as follows:

Design​

Design principles​

The following principles guide HAMi's architecture and feature trade-offs (see also the design overview):

  1. Kubernetes-native control paths
    • Accelerators are requested, scheduled, and allocated through standard Kubernetes surfaces: Pod specs, the scheduler extender protocol, the device plugin gRPC API, and admission webhooks.
    • HAMi avoids introducing proprietary control planes; cluster state remains in the Kubernetes API and node/device plugin reporting paths operators already trust.
  2. No application code changes
    • Workloads express sharing and isolation with resource requests, limits, and annotations. No rewrites of training, inference, or HPC code to "speak HAMi."
    • This keeps adoption friction low for platform teams onboarding existing AI/ML jobs.
  3. Pluggable heterogeneous devices
    • Device-specific logic lives in device plugins and in-container control libraries (for example HAMi-core for NVIDIA, analogous components for other vendors).
    • On the scheduling side, HAMi exposes a device API; each device implementation can extend this API to add device-specific scheduling optimizations.
    • New hardware families integrate by extending these layers rather than forking the whole stack.
  4. Policy-driven scheduling
    • Placement uses filter/score semantics on the scheduler extender so administrators can steer binpack, spread, topology awareness, and vendor-specific constraints consistently.
    • Decisions are recorded on Pods (via annotations), which enables scenario-specific scheduling behavior for different workload classes.
  5. Hard isolation on shared accelerators
    • Where the stack supports it, memory and compute caps are enforced inside containers, not only at admission time, so tenants cannot exceed declared slices on shared devices.
  6. Clear separation of responsibilities
    • Mutating admission validates and normalizes requests; the extender selects nodes; device plugins allocate and mount; in-container libraries enforce limits.
    • Each stage has a narrow contract, and intermediate state is written to Pod/Node annotations, which helps operators localize issues quickly.

Best practices​

Engineering and community practices that keep HAMi maintainable and safe to run in production:

  1. Stay aligned with Kubernetes semantics
    • Follow upstream scheduler extender, device plugin, and admission webhook contracts; prefer deprecating non-standard behavior with clear release notes rather than silent divergence.
  2. Automate verification
    • Gate changes with CI (lint, unit tests, chart packaging, and workflows that exercise realistic scheduling and device paths) to catch regressions before merge.
  3. Test in realistic environments
    • Validate features against representative accelerator hardware, drivers, and runtime combinations; document supported matrices and known limitations per vendor.
  4. Community-driven prioritization
    • Use public issues, roadmap discussions, and community meetings so roadmap items reflect operator pain and contributor capacity (see Roadmap and scope process under Scope).
  5. Documentation and operability first
    • Ship user-facing docs, examples, and Helm values commentary alongside features so platforms can install, tune, and observe HAMi without reading the implementation.

Architecture requirements​

Core components include MutatingWebhook, Scheduler Extender, Device Plugins, and in-container control libraries.

Environment differences​

  • PoC/dev/test/prod: the core deployment model is the same (GPU-enabled Kubernetes with HAMi); production environments typically apply stricter node labeling, webhook TLS hardening, scheduler/device policy tuning, and monitoring integration.

Service dependencies in cluster​

Kubernetes API server, kube-scheduler/Volcano scheduling integration, kubelet device plugin APIs, and (optionally) cert-manager for webhook certificates.

IAM model​

Uses Kubernetes RBAC for component service accounts and least-required API operations for webhook patch jobs and scheduler operations.

Sovereignty/compliance​

Data plane is cluster-local; no mandatory external telemetry service. Regional/organizational compliance posture depends on deployment choices.

HA requirements​

Scheduler supports leader election and configurable replicas.

  • Workload side: HAMi v2.5 addressed running-task crashes during reinstall. This is not a guarantee for upgrades or transient control-plane faults; stop or reschedule GPU workloads before upgrading.
  • Scheduling side: since HAMi v2.8, multi-replica scheduler deployment with leader election is supported to provide high availability for scheduling decisions.

Resource requirements (CPU/memory/network)​

Configurable per component via Helm values. The chart leaves resources unset by default, so production clusters should set explicit requests/limits. The following estimates are practical planning baselines for HAMi v2.8.0 on Kubernetes 1.23+, with NVIDIA sharing enabled and normal scheduling churn.

  • Assumptions for estimates: 1 scheduler replica (kube-scheduler + HAMi extender), 1 device-plugin DaemonSet pod per GPU node (device-plugin + vgpu-monitor), Prometheus scraping every 15-30s, and no unusual pod-creation spikes.
Component / scopeCPU (recommended request to typical peak)Memory (recommended request to typical peak)Network (control/observability plane)
Scheduler pod (per replica)300m-800m512Mi-2Gi0.2-1 Mbps typical, 2-5 Mbps burst during heavy pod churn
Device-plugin pod (per GPU node)150m-600m256Mi-1Gi0.05-0.2 Mbps per node (telemetry and control signaling)
Metrics scraping overhead (per target, 15s interval)N/AN/A50-300 KiB/scrape, about 0.03-0.16 Mbps
Cluster total (N GPU nodes)(0.3 to 0.8) + N * (0.15 to 0.6) cores(0.5 to 2) + N * (0.25 to 1) GiBApproximately linear with N based on per-node telemetry plus scheduler control traffic
Example footprint (N=20 GPU nodes)3.3-12.8 cores5.5-22 GiB1.2-5 Mbps typical aggregate control/metrics traffic, with higher short bursts during large scheduling events

Storage requirements​

HAMi has no mandatory external database. Storage use is mainly image layers, logs/metrics buffers, and hostPath-mounted runtime paths used by scheduler/device-plugin components. The following are practical sizing estimates for production planning.

  • Assumptions for estimates: default image retention policy, standard Kubernetes/container runtime logging, and mixed training/inference workloads with normal pod churn.
Component / scopeEstimated storage footprintNotes
Scheduler pod (per replica)0.2-0.8 GiB node-local ephemeral storageMostly image layers, logs, and temporary runtime files
Device-plugin + monitor pod (per GPU node)0.3-1.2 GiB node-local ephemeral storageIncludes plugin/monitor image layers, logs, and runtime temp/cache paths
Host runtime paths used by HAMi on GPU nodes0.2-1 GiB recommended free space on top of pod ephemeral usageCovers /var/lib/kubelet/device-plugins, /usr/local/vgpu, /var/run/cdi, and transient files under /tmp
Cluster total (N GPU nodes, 1 scheduler replica)approximately (0.2 to 0.8) + N * (0.5 to 2.2) GiBCombines scheduler ephemeral + per-GPU-node pod + host runtime reserve
Example footprint (N=20 GPU nodes)approximately 10.2-44.8 GiB cluster-wide reserved capacityPlanning envelope for stable operations and upgrade headroom

API design​

  • Uses Kubernetes native APIs (Pods, Nodes, annotations, admission webhooks, device plugin gRPC).
  • Defaults and optional configurations are exposed in Helm chart values and config docs.
  • No custom CRD is required for core workflows.
  • Device-slice resource names are exposed as Kubernetes resources, so users can request and limit slices directly in resources.limits / resources.requests.
  • GPU scheduling policies and advanced behaviors are expressed through Pod annotations.
  • HAMi runtime behavior can be configured through scheduler and device-plugin ConfigMaps, with Helm values as the primary entry point.

Release process​

Semantic versioning, tagged releases, release branches, automated image/chart/release-note workflows, with documented manual verification steps.

Installation​

# 1) Label GPU nodes for HAMi management
kubectl label nodes <gpu-node-name> gpu=on

# 2) Add HAMi Helm repository
helm repo add hami-charts https://project-hami.github.io/HAMi/
helm repo update

# 3) Install HAMi into kube-system
helm install hami hami-charts/hami -n kube-system

# 4) (Optional) Customize values during install
# helm install hami hami-charts/hami -n kube-system -f values.yaml

# 5) Verify core components are running
kubectl get pods -n kube-system | grep -E "hami-scheduler|hami-device-plugin"

Security​

See separate document

Day 1 - Installation and Deployment Phase​

Project Installation and Configuration​

Project Enablement and Rollback​

  • Enable/disable in live cluster: Deploy/upgrade/uninstall via Helm. Enabling does not require control-plane downtime; workloads may experience scheduling behavior changes when webhook/scheduler integration changes.
  • Behavior changes when enabled:
    • Pod mutation for recognized device requests.
    • Scheduling for device-sharing workloads routed through HAMi logic and policy.
    • Device plugin allocation/mount/env handling for runtime enforcement.
  • Testing enable/disable: Deployment and uninstall paths are exercised through e2e workflows; PRs are expected to pass required e2e checks before merge.
  • Resource cleanup: Helm uninstall removes chart-managed resources; no core CRDs are introduced by default path.

Rollout, Upgrade and Rollback Planning​

  • Compatibility cadence: Kubernetes and related dependencies are tracked through CI and automated dependency updates (including Dependabot). Version evolution is governed by the documented release process and release workflows.
  • Rollback procedures: Use helm rollback for in-place rollback, or uninstall and redeploy with a previous chart version when a full re-provisioning path is preferred.
  • How rollout/rollback can fail:
    • Rolling back images without rolling back the matching chart/configuration revision.
    • Failing to roll back customized values and policy ConfigMaps together with workloads/images.
    • Performing multi-version jump rollbacks (for example two or more minor versions) without staged validation.
  • Impact to running workloads: HAMi v2.5 addressed running-task crashes during reinstall, but upgrades and rollbacks can still affect active workloads; stop or reschedule GPU workloads before upgrading. New scheduling/allocation decisions may also change.
  • Rollback indicators: Primary indicators include admission/scheduling errors, abnormal pending-pod growth, device plugin registration issues, and policy mismatch symptoms. Operationally, the project recommends staying on recent supported releases whenever possible.
  • Upgrade path testing: Unit/e2e coverage exists; explicit long-chain upgrade->downgrade->upgrade matrices are still evolving and should be expanded for incubation.
  • Deprecation communication: Deprecations are communicated in documentation and community meetings, with explicit in-doc annotations. A typical transition window keeps old and new APIs available for one subsequent release, followed by old API removal after the next release cycle.
  • Alpha/beta capabilities: Exposed by configuration flags and values; users opt in through chart values and documented settings.
CNCFHAMi is a CNCF Incubating project