Automatic Instrumentation As A Kubernetes Virtual Device

At Keyval, we are on a mission to instrument any application, anywhere, at any scale. To achieve this, we need to make instrumentation as easy and accessible as possible. A key part of this is to not require any code changes by extending containers. In this blog post, we will compare different ways to extend containers, and why we chose to use Kubernetes Device Plugins at Odigos.

Why extend containers?

Adding files and environment variables to containers after they are deployed is a common practice. For example, you may want to add a configuration file, a secret, or in our case, automatic instrumentation. Kubernetes provides several ways to extend containers:

In the upcoming sections, we will test these methods and see how they compare.

Minimal YAML clutter

Kubernetes YAMLs are already very verbose. Adding more YAML to extend containers only makes it even more difficult to read and maintain. This also makes it harder to roll back the added changes if something goes wrong. Our first reason for choosing Device Plugins is that they require minimal YAML changes. For example, using a ConfigMap to add a configuration file and environment variables to a container requires the following YAML:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
    containers:
    - name: my-container
      image: my-container-image
+     volumeMounts:
+     - name: python-instrumentation
+       mountPath: /instrumentation
+     env:
+       - name: PYTHONPATH
+         value: /instrumentation
+       - name: OTEL_RESOURCE_ATTRIBUTES
+         value: service.name=myservice
+   volumes:
+   - name: python-instrumentation
+       configMap:
+       name: python-instrumentation

This is a lot of YAML for a simple task. Device Plugin, allows us to achieve the same result with the following (much shorter) YAML:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
    containers:
    - name: my-container
      image: my-container-image
+   resources:
+     limits:
+       instrumentation.odigos.io/python: 1

Init containers with shared volume are not shown, but they require even more YAML changes than ConfigMaps and Secrets.

Working with security contexts

Kubernetes allows to specify security contexts for containers. This allows to run containers as a specific user, group, or with specific capabilities. For example, you may want to run a container as a non-root user to improve security. This is not possible when using init containers with shared volumes, as the shared volume is mounted as root. However, this is possible when using device plugins, as the device is mounted as the user specified in the security context. For this reason, operators like Vault Agent Injector require adding many annotations to work with security contexts.

Scheduler friendly

Automatic instrumentation may not be available on every node. For example, Odigos uses eBPF to auto instrument Go applications. Requiring Linux kernel with eBPF support is a reasonable requirement, but it may not be available on every node. This is not a problem when using device plugins, as the scheduler will only schedule pods that require the device to nodes that have the device. When using init containers or ConfigMaps there is no way to specify this requirement, and the pod may be scheduled to a node that does not support automatic instrumentation.

Conclusion

ConfigMaps and Secrets are great, they are simple to use and work well for straightforward use cases. However, they are not a good fit for more complex cases like automatic instrumentation. Init containers with shared volumes are a bit more flexible, but they require more YAML changes and do not work well with security contexts. Device Plugins are the best option for automatic instrumentation, as they require minimal YAML changes, work well with security contexts, and are scheduler friendly. In the future, device plugins will get even better, when KEP-3063 is implemented.