1. Explain the difference between a Pod's restart policy (Always, OnFailure, Never) and how it interacts with the kubelet's crash-loop backoff mechanism. When would you choose OnFailure over Always for a production workload?

The restart policy determines whether the kubelet restarts a container after it exits: Always restarts regardless of exit code, OnFailure only on non-zero exit, and Never prevents restarts. The kubelet implements exponential backoff (10s, 20s, 40s up to 5min) to avoid hammering failed containers. OnFailure is appropriate for batch jobs or sidecar containers that should only restart on actual errors, not successful completion, whereas Always suits long-running services like web servers that should recover from transient failures automatically.

2. Design a Pod spec for a senior Platform Engineer that demonstrates proper resource requests, limits, and QoS class placement. What happens when a Node runs out of memory and your Pod has QoS class Burstable?

A well-designed Pod includes CPU and memory requests (for scheduling and QoS calculation) plus limits (hard caps enforced by cgroups). Requests equal limits creates Guaranteed QoS (highest priority eviction protection), requests less than limits creates Burstable (medium priority). When a Node has memory pressure, the kubelet evicts Pods in reverse priority order: Best Effort first, then Burstable by memory usage ratio, then Guaranteed if necessary. A Burstable Pod will be among the first evicted if it exceeds its request, making it unsuitable for critical workloads without proper requests tuned to expected usage.

3. A Pod is stuck in Pending state for 30 minutes. Walk through the debugging steps a Platform Engineer should take using kubectl and what Pod conditions or events might reveal the root cause.

First, run kubectl describe pod to check the Status, Conditions (PodScheduled, Initialized, Ready), and Events fields for scheduling failures or admission controller rejections. Check node capacity and taints with kubectl get nodes and kubectl describe node, then verify resource requests don't exceed available capacity. Examine init containers (InitializedCondition) as they must complete before the main container starts; a stuck init container blocks the whole Pod. Finally, check admission webhooks in cluster and whether image pull is failing silently by inspecting the ContainersReady condition and looking for ImagePullBackOff or similar status.

4. Explain init containers and their role in Pod initialization. In what scenarios would a Platform Engineer use init containers instead of readiness probes, and what are the trade-offs?

Init containers run sequentially before app containers start; they must complete successfully (exit code 0) or the Pod fails. They are ideal for setup tasks like downloading configuration, waiting for dependencies, or validating environment state. Unlike readiness probes which check ongoing health, init containers guarantee a one-time setup phase and prevent the app container from starting in a bad state. The trade-off is that init containers extend startup time and failures block the entire Pod; readiness probes are more flexible for detecting transient issues during steady-state operation.

5. A Pod has multiple containers. Container A logs successfully but Container B never produces logs despite the container being in Running state. What are the possible causes and how would you troubleshoot?

Container B might be redirecting stdout/stderr to files instead of the container's standard streams; check kubectl logs with specific container name and file paths in the container filesystem using kubectl exec. The container might be crashing immediately after startup but with a restart policy that restarts it too quickly for you to catch; check container status with kubectl describe pod and look for CrashLoopBackOff. Verify that the container's entrypoint and command are actually outputting logs and not silencing stderr. Also check if logs are being collected by a sidecar logging container or if the app writes to a non-standard descriptor; in some legacy apps, this requires binding /proc/1/fd/1 to understand log flow.

6. What are ephemeral containers and when would a Platform Engineer use them in production? What are the security and operational implications?

Ephemeral containers (alpha/beta feature depending on Kubernetes version) are temporary containers attached to a running Pod for debugging without stopping the Pod or restarting it. Use them when you need to inspect a distroless or minimal image that has no shell or debugging tools, avoiding the need to rebuild images with debug sidecar containers. Security implications include that they inherit the Pod's service account and security context, so they can access sensitive mounts and network namespaces; operationally, they're useful in production only if cluster RBAC strictly limits who can create them and if audit logging captures their usage.

7. Describe the Pod lifecycle phases (Pending, Running, Succeeded, Failed, Unknown) and explain under what conditions a Pod transitions between each phase. When would you see Unknown phase in production?

Pending means the Pod is waiting for resources or scheduling, Running means all containers are started, Succeeded/Failed indicate the Pod completed (usually for batch jobs), and Unknown means communication with the kubelet was lost. Phase transitions occur when the scheduler assigns a node (Pending to Running), containers exit (Running to Succeeded/Failed), or when init containers fail (stays in Pending). Unknown phase appears when the Node goes offline or the kubelet becomes unresponsive; the control plane cannot confirm container status and waits for the node to rejoin or a pod eviction timeout before cleaning up the Pod.

8. Design a liveness and readiness probe strategy for a slow-starting service that takes 60 seconds to become healthy. What probe parameters would you set and why, and what happens if the probes are misconfigured?

Set a readiness probe with initialDelaySeconds of 60 to avoid sending traffic before the service is ready, periodSeconds of 10 to check health frequently once running, and failureThreshold of 3 so temporary blips don't remove traffic. For liveness, use a longer initialDelaySeconds (120) and higher failureThreshold (5) to avoid killing a legitimately slow startup, but keep it lower than readiness to detect hangs. Misconfigured probes (too strict thresholds or short delays) cause unnecessary restarts and cascading failures, while too lenient probes allow unhealthy traffic and slow recovery. A common mistake is setting readiness probe identically to liveness, which can cause thrashing if a service temporarily can't handle health checks.

9. Explain how Pod priority and preemption work in Kubernetes. How would you design a PriorityClass strategy for a mixed workload cluster with critical services and batch jobs?

PriorityClass assigns numeric priority to Pods; when a high-priority Pod cannot be scheduled, the scheduler preempts (evicts) lower-priority Pods to make room. Design a three-tier strategy: critical system services get highest priority (e.g., 1000000), production workloads get medium (e.g., 100000), and batch jobs get low (e.g., 0). Use Pod Disruption Budgets in conjunction to protect even low-priority workloads from unnecessary preemption during voluntary disruptions like drains. Set preemption policy to PreemptLowerPriority for critical services and never for best-effort batch; this prevents batch jobs from evicting production services while still allowing batch to use spare capacity.

10. A Pod requires access to host-level networking (hostNetwork: true) and host process ID namespace (hostPID: true). Explain the security implications and under what conditions a Platform Engineer should allow this in production.

hostNetwork exposes the Pod to the host's network stack, allowing sniffing of all node traffic and bypass of NetworkPolicies; hostPID allows the Pod to see and signal all host processes. These should only be granted to system-level Pods like CNI plugins, node-exporter, or system monitoring agents that require low-level host access. Use Pod Security Standards (restricted/baseline) and RBAC to limit who can create Pods with these flags; combine with securityContext runAsNonRoot and readOnlyRootFilesystem to minimize blast radius. Document the business justification and regularly audit which Pods use these flags, as they significantly increase the blast radius of container compromises.

11. Explain static Pods and their use cases. How do they differ from managed Pods, and what are the operational risks of relying on them in a production cluster?

Static Pods are defined by files on the Node filesystem and created directly by the kubelet without involving the API server; they're commonly used for system components like kubelet, etcd, and kube-apiserver in kubeadm clusters. They differ from managed Pods in that they have no corresponding object in the API server, so kubectl cannot fully manage them, and they cannot use services or be scheduled by the scheduler. The operational risk is that static Pod definitions can drift from the cluster's desired state, making them invisible to monitoring and policy enforcement; kubeadm mitigates this by storing definitions in a well-known location and including them in cluster upgrades, but manual static Pod management is error-prone and should be avoided.

12. A workload requires guaranteed inter-pod communication with sub-millisecond latency within a single node. Design a Pod spec using appropriate namespaces and network plugins. What are the performance and security trade-offs?

Use localhost or loopback communication by placing Pods in the same Pod with multiple containers, or use hostNetwork with careful firewalling, or deploy a high-performance CNI like Cilium in eBPF mode with native routing. Multi-container Pods in the same Pod share a network namespace and can communicate via localhost on different ports with virtually zero latency, but they share resource limits and failure fate. hostNetwork achieves similar latency but exposes all containers to host networking and makes container isolation nearly impossible, increasing security blast radius. For security, prefer multi-container Pods with strict NetworkPolicies and securityContext isolation; use performance profiling to verify the latency requirement is real before accepting these trade-offs.

13. Explain how Pod disruption budgets (PDBs) interact with voluntary and involuntary disruptions. How would you design a PDB strategy for a stateful workload like a database cluster running on Kubernetes, and what are the gotchas when using maxUnavailable vs minAvailable?

PDBs protect Pods during voluntary disruptions (node drains, cluster upgrades) but not involuntary ones (node failures, out-of-memory kills). For a stateful database cluster, you'd typically use minAvailable set to a quorum number (e.g., 2 out of 3 replicas) to ensure majority availability during maintenance. The gotcha is that PDBs only delay eviction—they don't prevent it if the disruption is severe enough, and misconfiguring minAvailable too high can block cluster upgrades entirely. For non-critical workloads, maxUnavailable offers more flexibility by allowing a percentage or count of Pods to be disrupted simultaneously. Always test PDB behavior with actual drain operations in staging to verify your assumptions.

14. A Pod uses a local ephemeral volume (emptyDir) but the Node runs out of disk space. Explain what happens to the Pod, how the kubelet detects this, and what mitigation strategies you would implement at the platform level.

When Node disk space is exhausted, the kubelet triggers a disk-pressure condition and begins evicting Pods in order of QoS class and resource usage; emptyDir Pods without requests/limits are vulnerable. The kubelet monitors disk usage through the eviction manager, which uses thresholds like imagefs.available and nodefs.available to determine eviction necessity. At the platform level, implement strict storage quotas on nodes, use local persistent volumes with proper capacity planning, configure appropriate eviction thresholds in kubelet config, and monitor disk pressure metrics via Prometheus. Additionally, set resource requests and limits on all Pods to improve QoS class placement, ensuring critical workloads have Guaranteed QoS and are evicted last.

15. Design a Pod networking strategy for a multi-tenant platform where tenant A and tenant B must not communicate at the network layer, but both must reach a shared monitoring sidecar. How would you use network policies, namespaces, and Pod labels to enforce this?

Isolate each tenant into separate namespaces (tenant-a, tenant-b) and deploy the shared monitoring sidecar in a dedicated namespace (monitoring). Use NetworkPolicy resources to deny all ingress traffic by default in each tenant namespace, then explicitly allow egress only to the monitoring namespace using label selectors and namespace selectors. The monitoring sidecar should accept connections from both tenant namespaces via a permissive NetworkPolicy. Label Pods in each namespace distinctly (tenant: a, tenant: b) and reference these labels in NetworkPolicy rules to maintain isolation while allowing the specific communication path. This approach ensures strict Layer 3/4 isolation while permitting the required monitoring data collection.

16. A Pod has a sidecar container that performs log rotation and cleanup on a shared volume. The main application container writes logs continuously. Design a container restart policy and probe strategy that ensures the sidecar doesn't disrupt the main container if it crashes, and both containers stay healthy.

Use a Pod spec with restartPolicy: Always to ensure both containers restart independently if they fail, but define a liveness probe only on the main application container, not the sidecar, to prevent cascading restarts. Configure the sidecar with a long terminationGracePeriodSeconds (e.g., 60 seconds) so it can complete cleanup operations before the Pod is terminated. Set readiness probes only on the main container to control traffic routing; the sidecar doesn't need one since it's not serving requests. If the sidecar crashes repeatedly, the kubelet enters crash-loop backoff and restarts with exponential delays, but the main container continues running if its liveness probe passes. Monitor sidecar restart counts separately via metrics to detect persistent issues without affecting application availability.

17. Explain how Kubernetes handles graceful Pod termination (SIGTERM vs SIGKILL). Design a shutdown sequence for a Pod that must drain in-flight requests, close database connections, and persist state before exiting. What kubelet parameters would you tune?

When a Pod is terminated, kubelet sends SIGTERM to all containers and waits for the period defined by terminationGracePeriodSeconds (default 30s) before forcibly killing processes with SIGKILL. For a stateful application, set terminationGracePeriodSeconds to a value that allows your shutdown handlers to complete (e.g., 120 seconds for a database), configure preStop hooks to signal graceful shutdown logic, and implement signal handlers in your application code to flush buffers and close connections on SIGTERM. The kubelet parameter pod-eviction-timeout controls how long a kubelet waits for Pods to gracefully terminate before forcibly evicting them during a drain. Monitor actual shutdown duration in staging to calibrate the grace period accurately, and always include a preStop hook that initiates graceful shutdown independently of application-level signal handling for defense in depth.

18. A Pod is assigned to a Node via nodeSelector, but the selector becomes unsatisfiable after a node label is removed during maintenance. Explain what happens to the Pod and design a platform-level strategy to prevent this scenario in production.

The Pod remains Running on its current Node because nodeSelector only affects scheduling, not enforcement; Kubernetes does not evict Pods whose selectors become unsatisfiable post-scheduling. To prevent this, use Pod Disruption Budgets combined with cluster auto-scaler policies, and implement node affinity with preferredDuringSchedulingIgnoredDuringExecution for non-critical workloads instead of hard requirements. At the platform level, establish label management standards that prevent accidental removal (e.g., immutable labels via admission webhooks or RBAC), use node pools or taints instead of selectors for workload isolation, and implement audit logging to catch label mutations. For critical workloads requiring guaranteed node placement, combine nodeSelector with topology spread constraints and priority classes to handle node churn gracefully.

19. Explain the difference between Pod security standards (PSS) and Pod security policies (PSP). If you had to choose one for a senior platform role managing a large cluster, which would you implement and why?

Pod security policies are cluster-wide admission controllers that enforce security standards at Pod creation time but are deprecated as of Kubernetes 1.25 and removed in 1.28; Pod security standards are simpler label-based policies applied at the namespace level with three profiles (restricted, baseline, privileged) that are easier to manage and understand. For a senior platform role, implement Pod security standards because they offer better maintainability, namespace-level granularity, and alignment with Kubernetes' future direction; you can still enforce restrictive defaults via namespace labels. Use the restricted profile for multi-tenant workloads and baseline for legacy applications, then handle exceptions explicitly via documentation and monitoring rather than complex PSP rules. Combine PSS with a validating webhook for custom security policies that require logic beyond standard profiles, ensuring both compliance and operational flexibility.

20. A Pod requests 1 CPU and 512 MiB memory, but during peak load it spends 90% of time in CPU throttling despite having CPU limit set to 2 CPUs. Explain why this occurs and what metrics you would examine to diagnose the issue.

CPU throttling occurs because the kubelet uses cpu.stat.nr_throttled from cgroups to enforce the CPU limit; if your Pod tries to exceed its limit, the kernel pauses the process, regardless of whether the Node has idle CPU available. The discrepancy between requested (1 CPU) and limited (2 CPUs) means the Pod is only guaranteed 1 CPU, so during overload the scheduler may overcommit the node. Examine metrics like container_cpu_usage_seconds_total, container_cpu_throttling_seconds_total, and node CPU pressure via kube_node_status_pressure to understand throttling frequency and system-wide contention. Increase the CPU request to match actual peak usage (e.g., 1.8 CPUs if throttling indicates consistent demand), review node density and workload distribution, and consider enabling CPU Manager for guaranteed-QoS Pods if sub-millisecond latency is critical.

21. Design a Pod spec that uses a downward API to expose resource limits and node information to the container environment. Explain what information is available through downward API and what is intentionally excluded.

Use the downward API via env and envFrom to expose Pod metadata like namespace, name, UID, labels, and resource limits by referencing fieldRef and resourceFieldRef in the Pod spec; for example, set MEMORY_LIMIT from resources.limits.memory and POD_NAMESPACE from metadata.namespace. The downward API provides Pod-level metadata and resource definitions but intentionally excludes node information like Node name, IP, or resource capacity—use nodeRef or status.hostIP if you need Node details. For sensitive data, avoid exposing secrets through downward API; use mounted secret volumes instead for encryption at rest and access control. This pattern is useful for applications that need to tune themselves based on resource limits (e.g., garbage collector heap sizing in Java), and is commonly used in init containers to validate resource configuration before the main application starts.

22. A Pod is in CrashLoopBackOff state. The container logs show it exited with code 127 (command not found), but you verified the binary exists in the image. What are the possible root causes and how would you systematically debug this?

Exit code 127 typically indicates a missing interpreter or library dependency, not a missing binary; possible causes include a missing shell (if using sh -c in your entrypoint), missing shared libraries (libc, libssl), or incorrect image architecture (AMD64 vs ARM64 mismatch). Debug systematically by running kubectl debug pod to spawn a debug container in the Pod's namespace and verify the binary exists and is executable, then use ldd (on Linux binaries) or otool (on macOS binaries) to check dependencies. Inspect the Dockerfile to confirm the base image includes all runtime dependencies, and verify the entrypoint and command fields in the Pod spec don't introduce shell syntax that requires a shell binary. Use kubectl describe pod to check image pull errors, and review image scan results if using a container registry that performs vulnerability scanning, as some tools may strip or modify binaries during optimization.

23. Design a Pod mutation webhook that automatically injects resource requests, limits, and security context defaults for all Pods in a cluster. What validation should occur after mutation, and what are the failure modes if the webhook becomes unavailable?

Implement a MutatingWebhookConfiguration that targets all Pods and injects reasonable defaults (e.g., 100m CPU request, 128 MiB memory limit, runAsNonRoot: true, readOnlyRootFilesystem: true) based on label-based workload classification. After mutation, apply a ValidatingWebhookConfiguration to ensure injected values meet organization policies (e.g., limits >= requests, no privileged containers), creating a defense-in-depth approach. Configure the webhook with failurePolicy: Ignore to allow Pod creation if the webhook is unavailable, ensuring cluster availability during webhook outages; use sideEffects: None to guarantee idempotency and allow retries. Monitor webhook latency and error rates via Prometheus metrics, implement timeout values (e.g., 5 seconds), and deploy the webhook in a highly available configuration (multiple replicas, Pod disruption budget). Test mutation logic thoroughly in a staging cluster before rolling to production, as buggy mutations can render entire workloads unschedulable or insecure.

24. Explain how the kubelet's device plugin framework enables a Pod to request and use GPUs or other hardware devices. What lifecycle events does a Pod go through when a device is requested, and what happens if a device becomes unavailable after allocation?

The device plugin framework allows vendors (NVIDIA, AMD, etc.) to expose custom devices as extended resources (e.g., nvidia.com/gpu) that Pods can request; the kubelet communicates with device plugins via gRPC to discover devices, allocate them during scheduling, and monitor their health. When a Pod requests a GPU, the scheduler places it on a Node with available GPU capacity, then the device plugin's Allocate method is called before the container starts, allowing the plugin to perform initialization (e.g., setting environment variables, mounting device files). If a device becomes unavailable after allocation (hardware failure, reset), the device plugin signals unhealthy via ListAndWatch, causing the kubelet to mark the Node as NotReady and evict affected Pods with grace period. For critical GPU workloads, implement Pod disruption budgets and pod-to-device affinity via node selectors to control placement, and monitor device health metrics (temperature, errors, power) to proactively replace failing hardware before eviction occurs.

25. Explain how Pod affinity and anti-affinity rules interact with Pod topology spread constraints. When would you use podAntiAffinity with requiredDuringSchedulingIgnoredDuringExecution versus preferredDuringSchedulingIgnoredDuringExecution for a stateless service, and what are the cluster-wide scheduling implications?

Pod affinity rules enforce co-location or separation at scheduling time, while topology spread constraints distribute Pods across failure domains for high availability. Use requiredDuringScheduling for strict separation in production (e.g., database replicas on different nodes) and preferredDuringScheduling for soft constraints when flexibility is acceptable. Mixing both with tight topology spreads can cause unschedulable Pods if the cluster topology doesn't match constraints, so a senior engineer must validate cluster shape before deployment.

26. A Pod running a long-lived gRPC service receives a SIGTERM during a rolling update, but in-flight requests are dropped immediately instead of waiting for completion. Design a Pod shutdown sequence using preStop hooks and explain how you would coordinate graceful draining with the load balancer health checks. What kubelet and application-level changes are required?

Implement a preStop hook that signals the application to stop accepting new connections and drain existing requests within the terminationGracePeriodSeconds window (typically 30-60 seconds). Coordinate with the load balancer by removing the Pod from the endpoint slice before SIGTERM is sent; Kubernetes does this automatically via the EndpointSlice controller's readiness gate. On the application side, implement proper signal handling to complete in-flight gRPC streams before exiting, and set readiness probes to fail immediately when shutdown begins, preventing new traffic ingress.

27. Explain the difference between container lifecycle hooks (preStart, postStart, preStop) and how they interact with probes and the Pod startup sequence. Under what conditions would postStart hook failure cause the entire Pod to fail to reach Running state, and how would you detect this in a monitoring pipeline?

PreStart and postStart hooks execute synchronously during container startup before the container is marked Running; if either fails, the kubelet restarts the container based on the restart policy. PostStart hook failure does not prevent the container from being marked Running by default, but the failure is logged as a warning in Pod events; liveness probes can then detect a degraded state if the hook impact corrupts application state. Monitor for PostStart hook execution failures by querying Pod events with kubectl describe pod and alerting on RestartCountIncreased events that correlate with hook failures in logs.

28. Design a multi-container Pod where Container A (application) and Container B (sidecar exporter) share a mounted volume for metrics. Container B crashes frequently due to out-of-memory issues. How would you configure restartPolicy, probes, and resource limits to prevent Container B crashes from terminating the entire Pod, and what are the trade-offs?

Set restartPolicy to Always so Container B restarts independently without affecting Container A. Use separate resource limits for each container; Container B should have strict memory limits with appropriate requests to trigger pod eviction before the node's memory is exhausted. Configure a liveness probe for Container B that checks the exporter's health endpoint, allowing fast detection of crashes, but do not use hostNetwork or privileged mode. The trade-off is that Container B outages will result in incomplete metrics export until recovery, so the monitoring system must handle missing scrapes gracefully.

29. A Pod uses hostNetwork: true to bind to a privileged port (80) for a reverse proxy. Explain the security implications, network namespace behavior, and how you would audit and restrict this capability in a multi-tenant cluster using Pod Security Standards and network policies.

Pods with hostNetwork: true share the Node's network namespace, bypassing the Pod network isolation layer and allowing direct access to host network interfaces and routing tables. This is a critical security risk in multi-tenant environments because the Pod can sniff traffic from other Pods and the host. Restrict hostNetwork via Pod Security Standards by enforcing the 'Restricted' baseline, which disallows hostNetwork, and supplement with audit logging via policy enforcement tools. For the reverse proxy use case, use a Kubernetes Service with type LoadBalancer or Ingress instead of hostNetwork, which provides port binding through the service mesh without host namespace access.

30. Explain how the kubelet's memory manager policy (static policy) works and when you would configure containerized Pods with guaranteed QoS class alongside NUMA topology pinning. What happens to a Pod's memory performance if the NUMA affinity is violated after node topology changes?

The kubelet's memory manager with 'static' policy pins memory allocations to specific NUMA nodes for guaranteed QoS Pods, reducing latency in NUMA-aware systems by keeping memory local to CPUs. Configure guaranteed QoS (requests equal limits) with topology manager in single-numa-node policy to force Pod placement respecting NUMA boundaries at scheduling time. If NUMA topology changes after scheduling (e.g., node hardware reconfiguration), the kubelet does not automatically remigrate memory; the Pod continues running with degraded NUMA locality until rescheduled, so this is a rare but critical edge case handled by manual pod eviction.

31. A Pod is configured with cpu: 500m request and cpu: 1000m limit. During a traffic spike, the Pod hits the CPU limit and enters throttling for 5 minutes, but CPU metrics show only 600m actual usage during this period. Explain why throttling occurs despite sub-limit usage and what metrics would clarify whether the issue is CFS scheduler contention or application-level blocking.

CPU limits are enforced by the CFS bandwidth control mechanism, which allocates a time slice per scheduling period; if a process exhausts its slice before the period ends, it is throttled regardless of overall node CPU availability. The discrepancy between 600m usage and throttling indicates that the Pod's thread scheduler is experiencing high contention or the measurement is sampling a lower peak than the actual spike. Examine kubelet metrics like container_cpu_cfs_throttled_seconds_total and container_cpu_cfs_periods_total to detect CFS throttling, and compare against application profiling metrics (e.g., CPU flame graphs) to distinguish kernel-level throttling from application blocking or lock contention.

32. Design a Pod admission control strategy that prevents resource request requests without corresponding limits, enforces minimum CPU and memory allocations per Pod, and allows exemptions for specific namespaces (e.g., system-critical components). How would you implement this using ValidatingAdmissionPolicies or a mutating webhook, and what are the operational risks?

Use ValidatingAdmissionPolicies (CEL-based, available in Kubernetes 1.26+) or a custom mutating webhook to intercept Pod creates/updates and validate that every container has both requests and limits, with minimum thresholds enforced by CEL expressions or code logic. Implement exemption logic using namespace labels (e.g., allow-exempt: 'true') or service account selectors to permit system components like kubelet-managed Pods. Operational risks include false rejections of legitimate workloads if thresholds are too high, increased API latency from webhook calls during cluster scaling, and webhook failure modes (fail-closed vs fail-open) that can block cluster operations; mitigate by setting appropriate timeouts and logging all rejections.

33. A Pod uses a ConfigMap with immutable: true flag for security-critical configuration. How does immutability prevent accidental changes, and what happens if you attempt to update the ConfigMap or the Pod references it after it becomes immutable? Design a configuration update strategy that maintains immutability.

Immutable ConfigMaps prevent any updates to the data field after the immutable flag is set, enforcing that configuration changes require creating a new ConfigMap with a unique name and updating the Pod spec to reference it. Attempting to update an immutable ConfigMap results in an API server error; the Pod continues using the old ConfigMap until manually re-created. Implement a strategy where each configuration version is a separate immutable ConfigMap (e.g., app-config-v1, app-config-v2), and use GitOps or a Helm chart to manage rolling updates by incrementing the ConfigMap name and rolling out new Pod revisions via Deployment, ensuring no in-flight configuration changes corrupt the system.

34. A DaemonSet Pod must run on every node including control plane nodes, but recent security policies have tainted the control plane with NoSchedule. Explain how taints and tolerations work in this scenario and design a toleration strategy that allows the DaemonSet to run everywhere while respecting other workload constraints.

Control plane nodes typically have taints like node-role.kubernetes.io/control-plane:NoSchedule to prevent general workloads; DaemonSets automatically add tolerations for this taint to bypass the restriction and run on all nodes. If you've applied custom taints to control planes, add explicit toleration in the DaemonSet spec that matches the taint (key, operator, and effect) to allow scheduling. Ensure DaemonSet tolerations are specific and documented, and use labels and nodeAffinity in other Pods to avoid accidentally tolerating the control plane taint, which would break isolation between system and user workloads.

35. A Pod's readiness probe uses exec to run a shell command that checks a database connection. The probe times out after 5 seconds. Explain potential causes (container environment, shell availability, command blocking) and how you would refactor this for reliability at scale, including the trade-off between exec, http, and tcpSocket probes.

Exec probes require the container to have a shell (sh, bash) and the probe command must complete within the timeoutSeconds window; timeouts occur when the database connection is slow, the shell is missing, or the command is blocked by file locks or resource contention. Refactor to use httpGet probes if the application exposes a health endpoint, as they are faster and more reliable than spawning a shell; if database connectivity must be verified, use a lightweight health check HTTP endpoint that does not block on full transaction latency. TcpSocket probes are fastest but least reliable for actual health assessment. At scale, exec probes generate kubelet load and slow pod startup, so prefer http or tcp and implement application-level health checks that respond quickly without expensive I/O.

36. Explain the relationship between Pod termination grace period (terminationGracePeriodSeconds) and the container stop timeout enforced by the container runtime. If a Pod's grace period is 30 seconds but the container runtime enforces a 10-second stop timeout, what happens and how would you diagnose container forceful termination issues?

The Pod's terminationGracePeriodSeconds defines how long Kubernetes waits for the Pod to terminate gracefully; the container runtime (Docker, containerd) has its own stop timeout. If the runtime timeout is shorter than the Pod grace period, the container is forcefully killed (SIGKILL) before the grace period expires, truncating graceful shutdown. Diagnose this by checking container logs for abrupt termination (no shutdown log lines), checking kubelet logs for 'timeout waiting for pod', and examining the container runtime configuration (e.g., containerd config.toml for timeout_stop_container). Align the container runtime timeout with the Pod's terminationGracePeriodSeconds, typically by increasing the runtime timeout or reducing the grace period to match reality.