Add documentation for Zeppelin with Spark on Kubernetes#21
Conversation
|
Is the transparent background a potential problem? Presumably fine against a white background, but could that change? |
|
I can't pull the netlify link up - does anybody else have that issue? |
|
The doc looks good - I am wondering if we should include this while it is experimental. Or somehow tag this doc as experimental. @foxish what do you think? |
|
I like the idea of marking as experimental and getting it out. It would help us garner feedback. If someone can verify the working of the tutorial in its current state, we can go ahead. |
|
@echarles, would you be open to demo-ing this at next week's SIG meeting? It would help a lot of us understand where this effort is at. |
felixcheung
left a comment
felixcheung
left a comment
There was a problem hiding this comment.
I think it's cool to document this.
Although it could be hard to maintain if we reference ongoing PRs
| > At the time being, the needed code is not integrated in the `master` branches of `apache-zeppelin` nor the `apache-spark-on-k8s/spark` repositories. | ||
| > You are welcome to already ty it out and send any feedback and question. | ||
|
|
||
| Firs things firs, you have to choose the following modes in which you will run Zeppelin with Spark on Kubernetes: |
| For now, to be able to test these combinations, you need to build specific branches (see hereafter) or to use third-party Helm charts or Docker images. The needed branches and related PR are listed here: | ||
|
|
||
| 1. Spark-k8s driven branch: In-cluster client mode [see pull request #456](https://github.com/apache-spark-on-k8s/spark/pull/456) | ||
| 2. Apache Zeppeoin driven branch: Add support to run Spark interpreter on a Kubernetes cluster [see pull request #2637](https://github.com/apache/zeppelin/pull/2637) |
There was a problem hiding this comment.
Just wanted to point where this branch resides... I have remove that to avoid confusion.
|
|
||
|  | ||
|
|
||
| Build a new Zepplin based on [#456 In-cluster client mode](https://github.com/apache-spark-on-k8s/spark/pull/456). |
|
|
||
|  | ||
|
|
||
| Build a new Zepplin based on [#2637 Spark interpreter on a Kubernetes](https://github.com/apache/zeppelin/pull/2637). |
There was a problem hiding this comment.
this one doesn't seem to be updated...?
|
|
||
| Firs things firs, you have to choose the following modes in which you will run Zeppelin with Spark on Kubernetes: | ||
|
|
||
| + The `Kubernetes modes`: Can be `in-cluster` (within a Pod) or `out-cluster` (from outside the Kubernetes cluster). |
There was a problem hiding this comment.
what are the proper terminology in k8s world? is "out-cluster" the right term?
There was a problem hiding this comment.
I had the same question and from the already used/seen in-cluster, I have deduced 'out-cluster`. Happy to change to any other more official terminology.
…he spark-interpreter-spark branch
|
@felixcheung Thx a lot for your reviews (just pushed the fixes). @foxish Happy to demo this during the next SIG meeting (22 Nov). IMHO it is not bad to publish early docs if it the needed steps are clear (no release, need to build branches...) to get early-adopters feedbacks as much as possible. |
|
|
||
| Build a new Spark and their associated docker images based on [#2637 Spark interpreter on a Kubernetes](https://github.com/apache/zeppelin/pull/2637). | ||
|
|
||
| Once done, any vanilla Apache Zeppelin deployed in a Kubernetes Pod (your can use a Helm chart for this) will work out-of-the box with the following interpreter settings: |
There was a problem hiding this comment.
does this helm chart for this (use a different image for a newer Zeppelin though)
https://github.com/kubernetes/charts/blob/master/stable/spark/templates/spark-zeppelin-deployment.yaml
shall we link it?
There was a problem hiding this comment.
I have added a section at the end "how to test" and linked to the chart.
|
I am still OK with documentation, as long as it's clearly marked experimental |
|
@erikerlandson It is now documented as |
| 2. `in-cluster` with `spark-cluster` mode. | ||
| 3. `out-cluster` with `spark-cluster` mode. | ||
|
|
||
| For now, to be able to test these combinations, you need to build specific branches (see hereafter) or to use third-party Helm charts or Docker images. The needed branches and related PR are listed here: |
There was a problem hiding this comment.
As discussed in the meeting today, we want to ensure that these branches merge before we can publish documentation.
|
Should I close this one? Doesn't seem like it will be merged and we will move soon to apache repo. |
4 participants
First draft documentation to further discuss and prepare the WIP for Zeppelin with Spark on Kubernetes.