[WIP] Describe Spark submit in relation with client-mode (+ hadoop and dependencies)#25
[WIP] Describe Spark submit in relation with client-mode (+ hadoop and dependencies)#25echarles wants to merge 3 commits into
Conversation
|
Thanks Eric. cc/ @apache-spark-on-k8s/contributors |
squito
left a comment
squito
left a comment
There was a problem hiding this comment.
I'm very new to the k8s integration, still getting up to speed, just some comments from a first read, perhaps not relevant for the target audience here.
| ``` | ||
|
|
||
| ```scala | ||
| // Via cluster mode, we do not have full config as the hadoop conf dir is not mounted by configmap propagation but only available in the Spark Context. |
|
|
||
| It is important to acknowledge this as during the creation of the manager and schedulers described in earlier point, some definitions are created and some communication with the Kubernetes cluster (the REST API) are instanciated. | ||
|
|
||
| In the `InCluster` case, as pre-requisite to the next steps, we ask the user to define a property `spark.kubernetes.driver.pod.name` with the value being the exact name of the Pod where he is. |
There was a problem hiding this comment.
"where they are" -- we can be gender neutral
|
|
||
| ### Submit in client mode from a client with restricted network access | ||
|
|
||
| This is not possible. When the executor is invoked. |
There was a problem hiding this comment.
this is a little vague -- can you explain what restrictions don't work? obviously, if there is no communication out from the client, nothing is going to work. But one might expect that you only need some very limited communication from the client. Eg., only client to driver pod, but not client to executors.
|
|
||
| # Dependencies | ||
|
|
||
| For `cluster-mode`, we have the configuration orchestrator and its Steps which ensure that when `--jars` or `-Dspark.jars` are defined (same reaseonaring applies for `--files`): |
There was a problem hiding this comment.
this comment really goes from all cluster managers with spark, just came to my mind as I was reading this and something you might want to consider.
one thing which has tripped up a ton of users is understanding when jars become available and how you override various things. As an extreme example, you can't use "--jars" to replace the code of spark's Executor, as the executor itself has already started by that point. (Nobody would want to do that, but there are plenty of cases where you might want to replace some dependency, and its tricky because sometimes they are loaded before they you get to downloading and loading "--jars".)
3 participants
DISCLAIMER: The state of this documents is pre-alpha and [WIP] (Work in Progress). It contains some [TDV] (To Be Validated) and [TBC] (To Be Checked) and is not intended to be merged in the Spark documentation as it is.
The goal of this document is to review the current submission process in cluster mode and introduce a way to also fully support the client mode thtat is mandatory for the exploraty projects (notebooks...). We want to ensure that the way the Spark Driver and Executors lifecycle is correctly understood to take correct decisions to let evolve the architecture. The Shuffle Service and Resource Staging Server are not impacted by the previous considerations, so we don't cover them.