Troubleshooting

This page collects common errors you may encounter when running wasmCloud workloads and the resolutions for each. Entries include the literal error message, the underlying cause, and the fix.

If you hit an error that isn't listed here, please open an issue on github.com/wasmCloud/wasmCloud or ask in the wasmCloud Slack so we can document it.

Workload deployment

Missing host interface implementation in the linker

You see an error similar to the following when a WorkloadDeployment fails to start:

component imports instance `wasi:config/store@0.2.0-rc.1`, but a matching implementation was not found in the linker

Cause: A wasmCloud host advertises the set of host plugins it provides (for example, wasi:keyvalue, wasi:blobstore, wasi:config, wasi:logging, wasmcloud:messaging), but a workload must declare which of those interfaces its components import. The runtime resolves and links interfaces declaratively from the workload spec; if the import isn't declared, the linker has no matching implementation to wire in and the component fails to instantiate.

Fix: Add the missing interface under spec.hostInterfaces (in a Workload) or spec.template.spec.hostInterfaces (in a WorkloadDeployment). For the error above, the relevant entry is:

yaml

hostInterfaces:
  - namespace: wasi
    package: config
    version: "0.2.0-rc.1"
    interfaces:
      - store
    config:
      example.greeting: "hello from wasi:config"
      example.environment: "dev"

Each entry requires namespace, package, and interfaces; version and config are optional. The same pattern applies to other host plugins:

yaml

hostInterfaces:
  - namespace: wasi
    package: keyvalue
    interfaces: [store, atomics, batch]
  - namespace: wasi
    package: blobstore
    interfaces: [blobstore]
    config:
      buckets: my-bucket
  - namespace: wasmcloud
    package: messaging
    interfaces: [consumer, handler]
    config:
      subscriptions: 'orders.received'

See also: Host Interfaces for the full field reference (including the name field for multi-backend binding) and the HostInterface API reference. If you're hitting this error in wash dev rather than on Kubernetes, see Debugging Components: Unknown import errors.

Outbound HTTP call from a component is blocked

A component that imports wasi:http/outgoing-handler and is bound to the corresponding host interface still cannot reach an external URL — the call returns an error to the component, even though the network path appears clear.

Cause: The host enforces a per-component allowlist of outbound destinations via localResources.allowedHosts, and the policy fails closed: any outbound HTTP request whose host doesn't match an entry is blocked at the wasmCloud layer before the request leaves the process — independently of any Kubernetes NetworkPolicy. A component with no allowedHosts list at all is denied every outbound call, so this is the expected first-run symptom for any component that makes outbound HTTP, not just a sign of a misconfigured list.

Fix: Add the host to allowedHosts for the component:

yaml

components:
  - name: http-component
    image: ghcr.io/wasmcloud/components/http-hello-world-rust:0.1.0
    localResources:
      allowedHosts:
        - api.example.com
        - storage.googleapis.com
        - "*.s3.amazonaws.com"

Entries are matched against the request's host (no scheme, no path) using case-insensitive comparison. A leading wildcard like *.example.com matches any subdomain but not the bare example.com — list both if you need both.

If allowedHosts is empty or the field is omitted, all outbound HTTP requests are denied — the policy fails closed. To allow unrestricted outbound HTTP, set allowedHosts: ["*"] explicitly. (In local development, wash dev substitutes the allow-all policy when the field is omitted from .wash/config.yaml, which is why a component can work in the dev loop and then fail on Kubernetes.)

* grants HTTP egress only: it opens no wasi:sockets name lookup. A component that opens a raw socket to a hostname needs an allowedIpNameLookups entry, or an allowedHosts entry naming that host, even under allowedHosts: ["*"]. Since 2.11.0 this applies in the dev loop too, which enforces the socket policy with no opt-out.

See also: Workload Security — Restricting outbound HTTP with allowedHosts. If outbound HTTP is blocked at the cluster level rather than the wasmCloud level, see the same page for NetworkPolicy guidance.

Since 2.10.0 the denial reaches the guest as a handleable http-request-denied error on both WASI 0.2 and 0.3; earlier releases trapped 0.2 guests, so the symptom was an instance fault rather than an error value.

A raw socket is denied with nothing in the host log

A component using wasi:sockets gets access-denied on connect or bind, and the host logs say nothing about it at any level.

Cause: Since 2.11.0 the socket address policy runs in enforce mode by default. For a connect or bind the refusal path is silent: only count mode logs what it would refuse, and it logs at debug. So for this one decision, the mode that allows traffic is observable and the mode that blocks it is not.

How to confirm it: two ways, neither of which is grepping the log.

1. Read the metric. The host increments socket_policy.denied with a reason label — not_permitted, bind_not_permitted, host_loopback_not_permitted, host_owned_port, blocked_range, or no_capacity. This needs an OpenTelemetry exporter, meaning OTEL_EXPORTER_OTLP_ENDPOINT set on the host. Note that the meters are only installed by wash host: under wash dev and in embedded hosts, refusals are enforced but not counted.
2. Reproduce under --socket-egress count. The connection is then allowed and a debug line names the address and reason. Run with RUST_LOG=wash_runtime=debug.

A denied name lookup is logged, and names the fix

This silence applies only to connects and binds. A refused wasi:sockets name lookup reaches the guest as permanent-resolver-failure and the host logs it at warn, naming what to add. The wildcard case even has its own message:

lookup of api.internal denied: allowedHosts has *, which permits connections but opens no name lookup. Add api.internal, or *, to allowedIpNameLookups

Lookup denials are not governed by --socket-egress, so reproducing under count changes nothing for them, and they have no socket_policy.denied reason. If the symptom is a failed lookup rather than a failed connect, read the host log first.

Fix: Add the destination to the component's allowedHosts, and, for a connection made by hostname, grant the lookup — either an allowedIpNameLookups entry or an allowedHosts entry naming that host. Remember that allowedHosts: ["*"] permits connections but opens no lookup. A quota exhaustion reports out-of-memory rather than access-denied, deliberately, so you are not hunting for a policy that is not the cause.

See also: Raw-socket egress policy and connection quotas and Name grants on raw sockets.

Workload is refused for setting a backend url, root, or prefix

A workload that deployed cleanly on 2.10.x is refused at deploy on 2.11.0, with a message like:

it sets root, which belong to the operator; a manifest that set them could point itself at another backend or widen its own grant. Ask the operator to declare them under host.plugins

Cause: Through 2.10.x the multiplexed keyvalue, blobstore, messaging, and postgres plugins claimed no configuration keys, so the default workloadConfig: deny refused nothing on them and a manifest could name its own backend address. Since 2.11.0 those address keys belong to the operator. This fires even on a chart install that declares no plugins block, because deny is the default either way.

Fix: Move the address from the workload manifest to the host group's plugins entry, keeping backend on the workload. Declare it under bindings.<label>.config rather than the entry's own config when workloads must not share a store. To find every affected workload before committing, set workloadConfig: warn on the plugin entry: it permits them and logs exactly what deny would refuse.

See also: Operator-owned backend addresses for the full migration.

A hostPath volume is refused

A workload with a hostPath volume fails to start, and the host warns that it refused a hostPath volume outside the host paths this host permits, or names a matched mount or reserved path.

Cause: Since 2.11.0 a hostPath volume is permitted only inside a directory the host allows, and no directories are allowed by default. Unlike a socket refusal, this one is logged per refusal with the canonical path and also reaches the Workload's Sync condition, so kubectl describe workload <name> shows it.

Fix: List the parent directory under runtime.hostGroups[].hostPaths.allowed (host flag --allowed-host-path). If the path is inside an allowed directory and still refused, it is hitting a reserved path: the host reserves its own credential, CA, cache, and config paths, and refuses kernel filesystems (/proc, /sys, /dev) and the scratch root.

See also: Host filesystem access.

Workload image fails to pull from a private registry

A WorkloadDeployment never reaches its desired replica count, and either the underlying Artifact fails to resolve or the host records a registry authentication error when fetching the component image. The pull URL points to a private registry (private GHCR, ECR, ACR, a self-hosted registry, etc.).

Cause: wasmCloud references component images by registry URL. When the image is private, the resource that references it needs registry credentials in the form of a Kubernetes docker-registry Secret, pointed to via the imagePullSecret field. If the field is omitted, or the Secret doesn't grant pull access to the image, the pull fails.

Fix:

1. Create a docker-registry Secret in the workload's namespace:

   shell

   kubectl create secret docker-registry ghcr-secret \
     --namespace default \
     --docker-server=ghcr.io \
     --docker-username=<github-username> \
     --docker-password=<github-pat-with-read-packages>

2. Reference the Secret from the resource that fetches the image. On an Artifact (recommended for automatic rollout on new image versions):

   yaml

   apiVersion: runtime.wasmcloud.dev/v1alpha1
   kind: Artifact
   metadata:
     name: my-component
     namespace: default
   spec:
     image: ghcr.io/my-org/my-component:0.1.0
     imagePullSecret:
       name: ghcr-secret

   Or directly on a WorkloadComponent (or WorkloadService) when not using an Artifact:

   yaml

   components:
     - name: my-component
       image: ghcr.io/my-org/my-component:0.1.0
       imagePullSecret:
         name: ghcr-secret

3. The Secret must live in the same namespace as the resource that references it.

See also: ArtifactSpec and the CRDs guide. For mirroring the wasmCloud chart's own images to a private registry (air-gapped installs), see Private Registries and Air-Gapped Deployments.

Every image is pulled again, or pullPolicy: Never fails, after upgrading to 2.11.0

After upgrading, hosts re-download images they had cached, disk usage roughly doubles for a while, and workloads using pullPolicy: Never fail to start.

Cause: 2.11.0 changed the OCI disk cache layout. Entries are now scoped by the resolved registry user, the HTTP or HTTPS transport, and the CA snapshot used to fetch the artifact, and each entry stores its digest and component bytes together in one file. Older entries do not carry that scope and are ignored, so every image is fetched once more. Never returns a cached artifact only inside the same scope, so it fails until the image has been pulled once under the credentials, transport, and trust the host will actually use.

Fix: Let the hosts re-pull once. On Kubernetes the washlet removes the old directories on its first control-loop turn, with no age check, and sweeps again every five minutes after. For workloads pinned to Never, pull once with IfNotPresent under the credentials they will use, then switch back.

Related behavior worth knowing:

• A password rotation does not invalidate the cache. Scoping is by registry user, not password. One pull under a non-Never policy revalidates each entry's manifest with the new password without downloading it again, after which Never accepts it. A different username, anonymous access, or a new bearer token is a new scope and re-fetches — a bearer token names no user, so each token is its own scope.
• IfNotPresent now revalidates with the registry even for a digest-pinned reference, so that a reused disk entry has to pass the caller's current authorization and TLS policy rather than being treated as fresh authorization by virtue of being on disk. Expect slightly more registry traffic than 2.10.x for digest-pinned workloads. Never is the policy that stays offline.
• Keep cache.key with the cache. Entry names are keyed with a secret the cache creates once in its root. Deleting it, or copying a cache without it, orphans every entry.
• A re-pushed tag can take up to five seconds to be seen under IfNotPresent and Never, which reuse a completed identical pull for that long. Always resolves the registry afresh once any prior pull has finished.
• wash dev and wash oci never sweep their cache, so remove the old <reference>_<hash> directories under the wash cache directory by hand.

Requests to a workload return 400 with no application log line

A request reaches the host but gets a 400 before any component runs, and nothing appears in the workload's logs.

Cause: Since 2.10.0 the host normalizes the request scheme and authority before dispatch. A request with a missing or malformed Host header is rejected at the ingress with 400. Guests behind a TLS listener also see https as the scheme, where earlier releases passed http through.

Fix: Check the client or proxy that sets the Host header. If a component branches on the scheme it receives, note that a TLS-terminated request now reports https.

P3 component traps with waitable cannot be used synchronously while added to a waitable set

A WASI P3 component (one that exports the async WASI 0.3 interfaces, such as wasi:http/handler@0.3.0) worked before an upgrade to 2.10.0 and now fails on the first request that waits on I/O, such as reading the request body. HTTP callers get a 500, or a truncated response if the trap fires after the response starts. The host logs:

wasm trap: waitable cannot be used synchronously while added to a waitable set
failed to invoke component: pooled instance dropped the request

The second line appears only for components that use an instance pool (poolSize).

Cause: Since 2.10.0 the host runs on Wasmtime 48, which traps a synchronous stream or future cancel, or a subtask.cancel, on a waitable that is still in a waitable set (the group of handles an async component waits on together). The component model spec requires this (component-model#647, wasmtime#13708). The async runtime in wit-bindgen 0.58 and earlier does exactly that when its inter-task-wakeup feature is enabled; wit-bindgen 0.59.0 fixed it (wit-bindgen#1638). A common affected stack is wasip3 0.7.x with http-compat plus a direct wit-bindgen = "0.57.1" dependency with inter-task-wakeup. WASI P2 components, P3 components on wit-bindgen 0.59 or later, and P3 components without inter-task-wakeup are not affected.

To check a component, don't rely on the wit-bindgen version in its producers metadata (wasm-tools metadata show). A component can report wit-bindgen 0.60 while embedding the 0.57.1 runtime through wasip3 0.7. Check the source project instead:

shell

cargo tree --target all -e features -i wit-bindgen

The component is affected if this lists wit-bindgen 0.58 or earlier with the feature inter-task-wakeup. If the lockfile carries more than one wit-bindgen, name the version (-i wit-bindgen@0.57.1).

Fix: Rebuild on wasip3 0.9.0 and wit-bindgen 0.62.0. Keep a direct wit-bindgen dependency at the same version wasip3 uses, so Cargo builds one runtime with those features:

toml

wasip3 = { version = "0.9.0", features = ["http-compat"] }
wit-bindgen = { version = "0.62.0", default-features = false, features = [
    "async",
    "async-spawn",
    "inter-task-wakeup",
] }

Coming from wit-bindgen 0.57, rename wit_bindgen::spawn to spawn_local. A rebuilt component also runs on 2.6.0 through 2.9.x (Wasmtime 47), so you can redeploy it before upgrading. For a published component, use a release rebuilt this way: for example, ghcr.io/wasmcloud/components/oci-registry:0.1.0 is affected, and 0.1.1 is the same credential-free source rebuilt on wit-bindgen 0.62. Stay on the 0.1.x line for the local-ingress example, which serves the registry over plain HTTP with no credentials; 0.2.0 and later require HTTP Basic authentication.

Since 2.11.0 the host diagnoses this itself. Alongside the fault it logs, once per workload deploy:

guest trapped using a waitable synchronously while it was in a waitable set. This is the known signature of wit-bindgen's async runtime at 0.58 or older with inter-task-wakeup (usually wasip3 0.7.x with http-compat), which Wasmtime 48 and later rejects. If this component, or a component linked into it, uses that runtime, rebuild it on wasip3 0.9 or wit-bindgen 0.59 or newer

The hint is keyed on the trap type rather than its message text, so a linked call that flattens a callee's trap into a plain string does not mislead you into blaming the caller.

A call between sibling components returns HTTP 500 with an empty body

A multi-component workload returns a 500 with no body and no application log line, on a request path where one component passes a resource handle it obtained from a sibling to another sibling. The host logs either a trap reading borrow handles still remain at the end of the call or a type mismatch on the linked call.

Cause: Two related bugs in how the dynamic linker passed resource handles between components sharing a store, both fixed in 2.11.0. A borrowed handle lifted into the host was never released back, so the caller trapped when its call scope closed — that is the borrow handles still remain signature. Separately, a handle nested inside a record, tuple, list, option, variant, or result fell through to a path that re-marshalled it, so the callee received something that no longer matched the type it declared; that one surfaces as a type mismatch rather than the borrow trap. The nested case is the broader of the two, since nesting a handle in a record is far more idiomatic WIT than a bare borrow<T> parameter. Both affected the WASI 0.2 and WASI 0.3 call paths. A middleware or decorator chain is the usual topology: an auth component borrowing a session resource produced by a session component and handing it to a handler.

Fix: Upgrade to 2.11.0. The bug was present from 2.5.0 through 2.10.3 and is not fixed in any 2.10.x patch release. No component change is needed.

Scheduling

Workload stays unscheduled — no host matches hostSelector

A WorkloadDeployment reports zero ready replicas indefinitely. The image pulls cleanly, there are no runtime errors, and kubectl describe workloaddeployment shows the workload as pending without recent placement events — the operator has simply found no host to put it on.

Cause: hostSelector is a label selector matched against Host metadata labels. If no Host object carries labels matching the selector, the workload has nowhere to land. The most common cause is a typo or mismatch between the workload's selector (commonly hostgroup: default) and the labels on the host pool defined in Helm values (runtime.hostGroups[]).

Fix: Compare the selector to your host labels.

shell

# List host labels across namespaces
kubectl get hosts -A --show-labels

# Inspect the selector on the workload
kubectl get workloaddeployment <name> -n <namespace> \
  -o jsonpath='{.spec.template.spec.hostSelector}'

Either update the workload's selector to match an existing host group:

yaml

spec:
  template:
    spec:
      hostSelector:
        hostgroup: my-team

…or add a matching host group in Helm values. The host group's name becomes the value of the hostgroup label on Host resources it produces:

yaml

runtime:
  hostGroups:
    - name: my-team
      replicas: 3

If a host group exists but the workload still doesn't schedule, also check allowSharedHosts — see the next entry.

See also: runtime.hostGroups for host group configuration, WorkloadSpec for the selector field.

Workload stays unscheduled with allowSharedHosts: false

After enabling operator.allowSharedHosts: false, workloads that previously scheduled successfully now stay unscheduled, even though hostSelector still matches host labels in another namespace.

Cause: As of wasmCloud 2.1, allowSharedHosts: false enforces namespace-level scheduling isolation. A workload is only placed onto hosts whose Host.environment matches the workload's own namespace (or a value explicitly set via WorkloadDeployment.spec.template.spec.environment). Hosts in other namespaces are skipped regardless of label matches. This is the intended security boundary, but it can surprise teams who relied on cross-namespace host sharing before the upgrade.

Fix: Choose the model that matches your isolation requirements.

• Run a host pool in the workload's namespace. Add a host group to Helm values that the operator creates in the workload's namespace:

  yaml

  runtime:
    hostGroups:
      - name: team-a
        namespace: team-a
        replicas: 2

• Opt in to a specific shared namespace by setting environment on the WorkloadDeployment, pointing to a namespace where a host pool is intentionally shared:

  yaml

  spec:
    template:
      spec:
        environment: shared-infra
        hostSelector:
          hostgroup: shared

  Cross-namespace placement requires that sharing is allowed for the target — if allowSharedHosts: false is set globally, the workload's namespace must match the host's environment for placement to succeed.

• Re-enable host sharing if your isolation requirements allow it, returning to a cluster-wide host pool model:

  yaml

  operator:
    allowSharedHosts: true

See also: wasmCloud 2.1.0 release notes for the full namespace-scoped scheduling model.

Hosts and namespaces

Host objects don't appear in the expected namespace

After upgrading to wasmCloud 2.1, kubectl get hosts in your workload's namespace returns no results, or Host objects you expected to find are missing.

Cause: As of wasmCloud 2.1, the Host CRD is namespace-scoped rather than cluster-scoped. The runtime-operator creates every Host object in its own namespace — not in the namespace where the underlying host pod actually runs. Each Host records the pod's location in the new Host.environment field (a top-level field on the Host resource, not nested under spec), populated automatically from the downward API for in-cluster hosts (or supplied via --environment for external ones).

Fix: Look in the operator's namespace, or list across all namespaces:

shell

# List every host you have permission to see
kubectl get hosts -A

# List hosts in the operator's namespace
kubectl get hosts -n <operator-namespace>

The ENVIRONMENT column in the default kubectl get hosts output shows each host's environment. To filter by environment, use a jsonpath query against the top-level environment field:

shell

kubectl get hosts -A -o jsonpath='{range .items[?(@.environment=="team-a")]}{.metadata.name}{"\n"}{end}'

If you were relying on ClusterRole bindings to grant host visibility, note that user-facing host roles in 2.1 are generated as namespaced Roles. A namespace admin can now grant host visibility within their own namespace without cluster-admin involvement.

See also: Migration to v2 for upgrade notes, and the Host API reference.

Host health

Host pods restart under load with exit code 137, not OOMKilled

A host pod restarts with exit code 137 and no OOMKilled event after a period of heavy inbound HTTP, with liveness probe timeouts and a pinned CPU core beforehand.

Cause: On releases before 2.9.0, a host that ran out of file descriptors could spin on a failing accept loop, flood its logs, and miss its liveness probe until the kubelet killed it.

Fix: Upgrade to 2.9.0 or later. The host raises its own descriptor limit at startup, backs off on repeated accept failures, and bounds its listener with maxHttpIngressConnections.

Workloads stop starting with a max-process-memory error

A Workload reports Error with a message naming max-process-memory, such as:

this host is past max-process-memory (…) and accepts no new workloads until it falls back below 95% of it

Guests on the same host may also see memory growth fail.

Cause: Since 2.11.0 the host bounds its whole process footprint, defaulting to 90% of the cgroup memory limit. Past the limit it refuses guest memory growth and new workload starts, which is deliberately a visible failure rather than waiting for the kernel to OOM-kill the pod. Refusals lift once the footprint falls back under 95% of the limit, after a full minute below it without growth, or after five consecutive failed footprint reads.

Fix: Raise the pod's resources.limits.memory, reduce what the host is being asked to hold (replicas, poolSize, or workloads per host), or raise --max-process-memory explicitly. --max-process-memory off disables it. Watch the process_memory.footprint, process_memory.high_water, and process_memory.limit gauges and the process_memory.refused counter, which is kept separate from the guest budget's guest_memory.refused so you can tell the two limits apart.

Note that this limit is enforced even when guestMemoryMode is count — count governs the guest budget only, so if you are running in count to avoid refusals, this one still applies. See host process memory.

A host pod is Running but not Ready

The pod passes liveness but stays out of its Service endpoints, or flaps in and out under load.

Cause: Since 2.9.0 host pods serve HTTP probes on their own port (default 8081), and the failing probe's body names the condition, visible in kubectl describe pod:

Condition	Probe	Meaning
starting	readiness	Normal transition while the host comes up
draining	readiness	Normal transition after SIGTERM
http_ingress_saturated	readiness	The ingress connection ceiling is reached
stalled	liveness	The command loop has stalled; the host needs a restart
http_ingress_stopped	liveness	HTTP ingress stopped permanently; the host needs a restart

Fix: For http_ingress_saturated, raise maxHttpIngressConnections or add replicas. For starting that never clears, check NATS reachability against runtime.natsConnectTimeoutSeconds.

kubectl get hosts is empty while host pods are Running

Host resources disappear (taking their assigned Workloads with them) while the host pods themselves are healthy.

Cause: On releases before 2.9.0, an operator that lost its own NATS connection or restarted after downtime could judge every host unresponsive at once and delete the Host resources.

Fix: Upgrade to 2.9.0 or later, and check operator NATS connectivity. Since 2.9.0, hosts that fall silent together stay Unknown: a Host is removed only when the operator has been hearing the rest of the fleet continuously for the unreachable window (60 seconds by default).