The Spark Operator aims to make specifying and running Spark applications as easy and idiomatic as running other workloads on Kubernetes. We have deployed cluster-wide Spark Operator that defines kinds
SparkApplication, full documentation on kinds’ structure is available here. The official and detailed user guide is available here.
You have to set
default, otherwise your spark application will fail on permission issues. The Spark example feature
hostPath as volume source but this won’t work in the cluster since host mounts are forbidden. Please use
persistentVolumeClaim instead (examples) and don’t forget to create PVC.
Spark UI is automatically created for your applications on HTTPS address
and is also visible in Rancher UI, follow the steps to see its final form. When you copy the address into the browser, omit the last character group
(/|$)(.*) (a Rancher related issue).
When you access the path, you should be presented with dashboard similar to
Custom SparkUI Ingress
If you want to provide custom annotations to Ingress (e.g. use different certificate issuer, we use
letsencrypt) or custom TLS secret you have to include following code snippet in spark application YAML.
sparkUIOptions: ingressAnnotations: [annotation_key]: [annotation_value] ... ingressTLS: (optional section) - secretName: [secretname_in_app_namespace] hosts: - [host]
We merge the items provided by you with our configuration. If you set the same annotation as we do, your value is propagated.