FAQ

Question: Every Pod, Job, Deployment, and any other type that runs a container must have the resources attribute set, otherwise the deploy will fail with a similar error:

Pods "master-685f855ff-gg9sr" is forbidden: failed quota: default-w2qv7: must specify limits.cpu,limits.memory,requests.cpu,requests.memory:Deployment does not have minimum availability.

Answer: Pod, Job, Deployment is missing the following resource section with appropriate values:

resources:
  requests:
    cpu: 1
    memory: 512Mi
  limits:
    cpu: 1
    memory: 512Mi

Question: Deployment returns message simmilar to:

Error creating: pods "app-cronjob-28837272-7k594" is forbidden: exceeded quota: default-kcq58, requested: limits.memory=8Gi, used: limits.memory=41872Mi, limited: limits.memory=45000Mi

Answer: This error means the Pod you’re trying to create requests more memory (limits.memory=8Gi) than the available quota allows. Here, the namespace has a memory limit of 45,000 Mi (about 43.95 Gi)defined in the resource quota object (default-kcq58, see kubectl get resourcequota -n [your-namespace] default-kcq58), and your Pod creation would exceed this because the quota is nearly fully used (used: limits.memory=41872Mi).

Options for Resolving:

• Reduce the Memory Request of the Pod:

  Adjust the limits.memory in the Pod spec to request less memory, ideally fitting within the remaining quota (about 3128 Mi, or around 3 Gi).

• Request an Increase in Quota (if you have control over the cluster settings):

  Contact us at k8s@cerit-sc.cz to increase the memory quota for the namespace or create explicit project namespace with higher quotas then the personal namespace..

Question: No GPU found, nvidia-smi returns command not found.

Answer: The deployment is missing request for GPU like:

resources:
  limits:
    nvidia.com/gpu: 1

or

resources:   
  limits:
    cerit.io/gpu-mem: 1

Question: Deployment returns message similar to:

CreateContainerConfigError (container has runAsNonRoot and image will run as root (pod: "mongo-db-846b7bfc7-qrlqt_namespace-ns(5d3538ab-7493-41ab-bd94-a4256c236f6f)", container: mongo-db-test))

Answer: The deployment is missing the securityContext section and the container image (in this case mongo) does not contain numeric USER. To fix this, just extend the deployment definition like this:

image: cerit.io/nextflowproxy:v1.2
imagePullPolicy: Always
securityContext:
  runAsUser: 1000
  runAsGroup: 1000
resources:
  limits:
    cpu: 4
    memory: 8192Mi

The runAsUser and runAsGroup lines are important.

See full security context settings here.

Question: Helm deployment returns error code 413

Answer: HTML code 413 means entity too larger. Helm stores whole deployment (including all the local files in the chart no matter if they have .yaml suffix) and values into a Secret object. Limit of the Secret size is about 1.5MB. Verify, if there is no big file in the whole chart.

Question: How can I fix the following type of error:

Error creating: pods "gmx-18dafd65-49d5-4263-8da2-e7d574b69930-nrtb9" is forbidden: violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "gmx-18dafd65-49d5-4263-8da2-e7d574b69930" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "gmx-18dafd65-49d5-4263-8da2-e7d574b69930" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "gmx-18dafd65-49d5-4263-8da2-e7d574b69930" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "gmx-18dafd65-49d5-4263-8da2-e7d574b69930" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

Answer: The error message is due to the Pod not meeting the requirements of the restricted PodSecurity standard in Kubernetes. To fix this, you need to add a securityContext to the Pod and container specification. Here’s how you can address each issue:

• allowPrivilegeEscalation: Set this to false.
• Capabilities: Drop all capabilities to meet the restricted policy.
• runAsNonRoot: Ensure the Pod or container is set to run as a non-root user.
• seccompProfile: Set the seccompProfile.type to "RuntimeDefault" or "Localhost".

Example Pod spec:

apiVersion: v1
kind: Pod
metadata:
  name: gmx-pod
spec:
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: gmx-container
    image: your-image
    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - "ALL"

Explanation of the Changes:

• allowPrivilegeEscalation: false: Prevents the container from gaining additional privileges.
• capabilities.drop: ["ALL"]: Drops all Linux capabilities, which is required under the restricted policy.
• runAsNonRoot: true: Ensures the container doesn’t run as the root user.
• seccompProfile.type: "RuntimeDefault": Enforces the default seccomp profile for additional security.

Applying the Updated Spec:

Replace your-image with the appropriate container image name, and apply the updated configuration. This should resolve the error and allow the Pod to pass the restricted PodSecurity admission.

Question: I am getting an “OpenSSL version mismatch. Built against 30000020, you have 30400010” error.
Answer: This error typically occurs in a Conda environment when the system’s OpenSSL version differs from the one used by Conda. The issue occurs because Conda libraries take precedence over system libraries, but certain system binaries (such as ssh or git which is calling ssh) require the system’s OpenSSL version to function correctly.

As a temporary workaround, you can unset LD_LIBRARY_PATH environment variable before running the problematic command. For example:

LD_LIBRARY_PATH='' ssh user@host

This forces the system to use its default OpenSSL version instead of the one provided by Conda.