Skip to content

Commit 57a3bce

Browse files
thesuperzapperthesuperzapper
committed
docs: cleanup README
Signed-off-by: Mathew Wicks <[email protected]>
1 parent a1bc9af commit 57a3bce

35 files changed

+2574
-1828
lines changed

‎charts/airflow/CHANGELOG.md‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -105,11 +105,11 @@ TBD
105105
106106
> 🟨 __NOTES__ 🟨
107107
>
108-
> - You can now use Secrets and ConfigMaps to define your `airflow.{users,connections,pools,variables}`, see the docs:
109-
> - [How to create airflow users?](https://github.com/airflow-helm/charts/tree/main/charts/airflow#how-to-create-airflow-users)
110-
> - [How to create airflow connections?](https://github.com/airflow-helm/charts/tree/main/charts/airflow#how-to-create-airflow-connections)
111-
> - [How to create airflow variables?](https://github.com/airflow-helm/charts/tree/main/charts/airflow#how-to-create-airflow-variables)
112-
> - [How to create airflow pools?](https://github.com/airflow-helm/charts/tree/main/charts/airflow#how-to-create-airflow-pools)
108+
> - You may now use Secrets and ConfigMaps to define your `airflow.{users,connections,pools,variables}`:
109+
> - [How to manage airflow users?](docs/faq/security/airflow-users.md)
110+
> - [How to manage airflow connections?](docs/faq/dags/airflow-connections.md)
111+
> - [How to manage airflow variables?](docs/faq/dags/airflow-variables.md)
112+
> - [How to manage airflow pools?](docs/faq/dags/airflow-pools.md)
113113
114114
### Added
115115
- allow referencing Secrets/ConfigMaps in `airflow.{users,connections,pools,variables}` ([#281](https://github.com/airflow-helm/charts/pull/281))
@@ -262,7 +262,7 @@ TBD
262262
- native support for [Airflow 2.0's HA scheduler](https://airflow.apache.org/docs/apache-airflow/stable/scheduler.html#running-more-than-one-scheduler), see the new `scheduler.replicas` value
263263
- significantly improved git-sync system by moving to [kubernetes/git-sync](https://github.com/kubernetes/git-sync)
264264
- significantly improved pip installs by moving to an init-container
265-
- added a [guide for integrating airflow with your "Microsoft AD" or "OAUTH"](README.md#how-to-authenticate-airflow-users-with-ldapoauth)
265+
- added docs for [How to integrate airflow with LDAP or OAUTH?](docs/faq/security/ldap-oauth.md)
266266
- general cleanup of almost every helm file
267267
- significant docs/README rewrite
268268

‎charts/airflow/CONTRIBUTING.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,7 @@ Most non-patch changes will require documentation updates.
7272

7373
If you __ADD a value__:
7474
- ensure the value has a descriptive docstring in `values.yaml`
75-
- ensure the value is listed under `Values Reference` in [README.md](README.md#values-reference)
75+
- ensure the value is listed under `Helm Values` in [README.md](README.md#helm-values)
7676
- Note, only directly include the value if it's a top-level value like `airflow.level_1`, otherwise only include `airflow.level_1.*`
7777

7878
If you __bump the version__:

‎charts/airflow/README.md‎

Lines changed: 135 additions & 1820 deletions
Large diffs are not rendered by default.

‎charts/airflow/UPGRADE.md‎

Lines changed: 0 additions & 1 deletion
This file was deleted.

‎charts/airflow/docs/faq/configuration/airflow-configs.md‎

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
[🔗 Return to `Table of Contents` for more FAQ topics 🔗](https://github.com/airflow-helm/charts/tree/main/charts/airflow#frequently-asked-questions)
2+
3+
> Note, this page was written for the [`User-Community Airflow Helm Chart`](https://github.com/airflow-helm/charts/tree/main/charts/airflow)
4+
5+
# How to set airflow configs?
6+
7+
## airflow.cfg
8+
9+
While we don't expose the `airflow.cfg` file directly, you may use [environment variables](https://airflow.apache.org/docs/stable/howto/set-config.html) to set Airflow configs.
10+
11+
The `airflow.config` value makes this easier, each key-value is mounted as an environment variable on each Pod:
12+
13+
```yaml
14+
airflow:
15+
config:
16+
## security
17+
AIRFLOW__WEBSERVER__EXPOSE_CONFIG: "False"
18+
19+
## dags
20+
AIRFLOW__CORE__LOAD_EXAMPLES: "False"
21+
AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: "30"
22+
23+
## email
24+
AIRFLOW__EMAIL__EMAIL_BACKEND: "airflow.utils.email.send_email_smtp"
25+
AIRFLOW__SMTP__SMTP_HOST: "smtpmail.example.com"
26+
AIRFLOW__SMTP__SMTP_MAIL_FROM: "[email protected]"
27+
AIRFLOW__SMTP__SMTP_PORT: "25"
28+
AIRFLOW__SMTP__SMTP_SSL: "False"
29+
AIRFLOW__SMTP__SMTP_STARTTLS: "False"
30+
31+
## domain used in airflow emails
32+
AIRFLOW__WEBSERVER__BASE_URL: "http://airflow.example.com"
33+
34+
## ether environment variables
35+
HTTP_PROXY: "http://proxy.example.com:8080"
36+
```
37+
38+
> 🟦 __Tip__ 🟦
39+
>
40+
> To store sensitive configs in Kubernetes secrets, you may use the `airflow.extraEnv` value.
41+
>
42+
> For example, to set `AIRFLOW__CORE__FERNET_KEY` from a Secret called `airflow-fernet-key` containing a key called `value`:
43+
>
44+
> ```yaml
45+
> airflow:
46+
> extraEnv:
47+
> - name: AIRFLOW__CORE__FERNET_KEY
48+
> valueFrom:
49+
> secretKeyRef:
50+
> name: airflow-fernet-key
51+
> key: value
52+
> ```
53+
54+
## webserver_config.py
55+
56+
We expose the `web.webserverConfig.*` values to define your Flask-AppBuilder `webserver_config.py` file.
57+
58+
For example, a minimal `webserver_config.py` file that uses [`AUTH_DB`](https://flask-appbuilder.readthedocs.io/en/latest/security.html#authentication-database):
59+
60+
```yaml
61+
web:
62+
webserverConfig:
63+
## the full content of the `webserver_config.py` file, as a string
64+
stringOverride: |
65+
from airflow import configuration as conf
66+
from flask_appbuilder.security.manager import AUTH_DB
67+
68+
# the SQLAlchemy connection string
69+
SQLALCHEMY_DATABASE_URI = conf.get('core', 'SQL_ALCHEMY_CONN')
70+
71+
# use embedded DB for auth
72+
AUTH_TYPE = AUTH_DB
73+
74+
## the name of an existing Secret containing a `webserver_config.py` key
75+
## NOTE: if set, takes precedence over `web.webserverConfig.stringOverride`
76+
#existingSecret: "my-airflow-webserver-config"
77+
```
78+
79+
> 🟦 __Tip__ 🟦
80+
>
81+
> We also provide more detailed docs on [how to integrate airflow with LDAP or OAUTH](../security/ldap-oauth.md).
82+
83+
## airflow_local_settings.py
84+
85+
We expose the `airflow.localSettings.*` values to define your `airflow_local_settings.py` file.
86+
87+
For example, an `airflow_local_settings.py` file that sets a [cluster policy](https://airflow.apache.org/docs/apache-airflow/stable/concepts/cluster-policies.html) to reject dags with no tags:
88+
89+
```yaml
90+
airflow:
91+
localSettings:
92+
## the full content of the `airflow_local_settings.py` file, as a string
93+
stringOverride: |
94+
from airflow.models import DAG
95+
from airflow.exceptions import AirflowClusterPolicyViolation
96+
97+
def dag_policy(dag: DAG):
98+
"""Ensure that DAG has at least one tag"""
99+
if not dag.tags:
100+
raise AirflowClusterPolicyViolation(
101+
f"DAG {dag.dag_id} has no tags. At least one tag required. File path: {dag.fileloc}"
102+
)
103+
104+
## the name of an existing Secret containing a `airflow_local_settings.py` key
105+
## NOTE: if set, takes precedence over `airflow.localSettings.stringOverride`
106+
#existingSecret: "my-airflow-local-settings"
107+
```
108+
109+
For example, an `airflow_local_settings.py` file that sets the default KubernetesExecutor container image:
110+
111+
```yaml
112+
airflow:
113+
localSettings:
114+
## the full content of the `airflow_local_settings.py` file, as a string
115+
stringOverride: |
116+
# use a custom `xcom_sidecar` image for KubernetesPodOperator()
117+
from airflow.kubernetes.pod_generator import PodDefaults
118+
PodDefaults.SIDECAR_CONTAINER.image = "gcr.io/PROJECT-ID/custom-sidecar-image"
119+
```

‎charts/airflow/docs/faq/configuration/airflow-plugins.md‎

Lines changed: 170 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,170 @@
1+
[🔗 Return to `Table of Contents` for more FAQ topics 🔗](https://github.com/airflow-helm/charts/tree/main/charts/airflow#frequently-asked-questions)
2+
3+
> Note, this page was written for the [`User-Community Airflow Helm Chart`](https://github.com/airflow-helm/charts/tree/main/charts/airflow)
4+
5+
# How to load airflow plugins?
6+
7+
There are multiple ways to load [airflow plugins](https://airflow.apache.org/docs/apache-airflow/stable/plugins.html) when using the chart.
8+
9+
## Option 1 - embedded into container image (recommended)
10+
11+
This chart uses the official [apache/airflow](https://hub.docker.com/r/apache/airflow) images, you may extend the airflow container image with your airflow plugins.
12+
13+
For example, here is a Dockerfile that extends `airflow:2.1.4-python3.8` with custom plugins:
14+
15+
```dockerfile
16+
FROM apache/airflow:2.1.4-python3.8
17+
18+
# plugin files can be copied under `/home/airflow/plugins`
19+
# (where `./plugins` is relative to the docker build context)
20+
COPY plugins/* /home/airflow/plugins/
21+
22+
# plugins exposed as python packages can be installed with pip
23+
RUN pip install --no-cache-dir \
24+
example==1.0.0
25+
```
26+
27+
After building and tagging your Dockerfile as `MY_REPO:MY_TAG`, you may use it with the chart by specifying `airflow.image.*`:
28+
29+
```yaml
30+
airflow:
31+
image:
32+
repository: MY_REPO
33+
tag: MY_TAG
34+
35+
## WARNING: even if set to "Always" do not reuse tag names, as containers only pull the latest image when restarting
36+
pullPolicy: IfNotPresent
37+
```
38+
39+
## Option 2 - git-sync dags repo
40+
41+
> 🟥 __Warning__ 🟥
42+
>
43+
> With "Option 2", you must manually restart the webserver and scheduler pods for plugin changes to take effect.
44+
45+
If you are using git-sync to [load your DAG definitions](../dags/load-dag-definitions.md), you may also include your plugins in this repo.
46+
47+
For example, if your DAG git repo includes plugins under `./PATH/TO/PLUGINS`:
48+
49+
```yaml
50+
airflow:
51+
configs:
52+
## NOTE: there is an extra `/repo/` in the path
53+
AIRFLOW__CORE__PLUGINS_FOLDER: /opt/airflow/dags/repo/PATH/TO/PLUGINS
54+
55+
dags:
56+
## NOTE: this is the default value
57+
#path: /opt/airflow/dags
58+
59+
gitSync:
60+
enabled: true
61+
repo: "[email protected]:USERNAME/REPOSITORY.git"
62+
branch: "master"
63+
revision: "HEAD"
64+
syncWait: 60
65+
sshSecret: "airflow-ssh-git-secret"
66+
sshSecretKey: "id_rsa"
67+
68+
# "known_hosts" verification can be disabled by setting to ""
69+
sshKnownHosts: |-
70+
github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==
71+
```
72+
73+
## Option 3 - persistent volume
74+
75+
> 🟥 __Warning__ 🟥
76+
>
77+
> With "Option 3", you must manually restart the webserver and scheduler pods for plugin changes to take effect.
78+
79+
You may load airflow plugins that are stored in a Kubernetes [Persistent Volume](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) by using the `airflow.extraVolumeMounts` and `airflow.extraVolumes` values.
80+
81+
For example, to mount a PersistentVolumeClaim called `airflow-plugins` that contains airflow plugin files at its root:
82+
83+
```yaml
84+
airflow:
85+
configs:
86+
## NOTE: this is the default value
87+
#AIRFLOW__CORE__PLUGINS_FOLDER: /opt/airflow/plugins
88+
89+
extraVolumeMounts:
90+
- name: airflow-plugins
91+
mountPath: /opt/airflow/plugins
92+
## NOTE: if plugin files are not at the root of the volume, you may set a subPath
93+
#subPath: "path/to/plugins"
94+
readOnly: true
95+
96+
extraVolumes:
97+
- name: airflow-plugins
98+
persistentVolumeClaim:
99+
claimName: airflow-plugins
100+
```
101+
102+
## Option 4 - ConfigMaps or Secrets
103+
104+
> 🟥 __Warning__ 🟥
105+
>
106+
> With "Option 4", you must manually restart the webserver and scheduler pods for plugin changes to take effect.
107+
108+
You may load airflow plugins that are sored in Kubernetes Secrets or ConfigMaps by using the `airflow.extraVolumeMounts` and `airflow.extraVolumes` values.
109+
110+
For example, to mount airflow plugin files from a ConfigMap called `airflow-plugins`:
111+
112+
```yaml
113+
workers:
114+
configs:
115+
## NOTE: this is the default value
116+
#AIRFLOW__CORE__PLUGINS_FOLDER: /opt/airflow/plugins
117+
118+
extraVolumeMounts:
119+
- name: airflow-plugins
120+
mountPath: /opt/airflow/plugins
121+
readOnly: true
122+
123+
extraVolumes:
124+
- name: airflow-plugins
125+
configMap:
126+
name: airflow-plugins
127+
```
128+
129+
> 🟦 __Tip__ 🟦
130+
>
131+
> Your `airflow-plugins` ConfigMap might look something like this.
132+
>
133+
> ```yaml
134+
> apiVersion: v1
135+
> kind: ConfigMap
136+
> metadata:
137+
> name: airflow-plugins
138+
> data:
139+
> my_airflow_plugin.py: |
140+
> from airflow.plugins_manager import AirflowPlugin
141+
>
142+
> class MyAirflowPlugin(AirflowPlugin):
143+
> name = "my_airflow_plugin"
144+
> ...
145+
> ```
146+
147+
> 🟦 __Tip__ 🟦
148+
>
149+
> You may include the ConfigMap as an [extra manifest](../kubernetes/extra-manifests.md) of the chart using the `extraManifests` value.
150+
>
151+
> ```yaml
152+
> extraManifests:
153+
> - |
154+
> apiVersion: v1
155+
> kind: ConfigMap
156+
> metadata:
157+
> name: airflow-plugins
158+
> labels:
159+
> app: {{ include "airflow.labels.app" . }}
160+
> chart: {{ include "airflow.labels.chart" . }}
161+
> release: {{ .Release.Name }}
162+
> heritage: {{ .Release.Service }}
163+
> data:
164+
> my_airflow_plugin.py: |
165+
> from airflow.plugins_manager import AirflowPlugin
166+
>
167+
> class MyAirflowPlugin(AirflowPlugin):
168+
> name = "my_airflow_plugin"
169+
> ...
170+
> ```

‎charts/airflow/docs/faq/configuration/airflow-version.md‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
[🔗 Return to `Table of Contents` for more FAQ topics 🔗](https://github.com/airflow-helm/charts/tree/main/charts/airflow#frequently-asked-questions)
2+
3+
> Note, this page was written for the [`User-Community Airflow Helm Chart`](https://github.com/airflow-helm/charts/tree/main/charts/airflow)
4+
5+
# How to set the airflow version?
6+
7+
> 🟦 __Tip__ 🟦
8+
>
9+
> There is a default version (`airflow.image.tag`) of airflow shipped with each version of the chart, see the default [values.yaml](../../../values.yaml) for the current one.
10+
11+
> 🟦 __Tip__ 🟦
12+
>
13+
> Many versions of airflow versions are supported by the chart, please see the [Airflow Version Support](../../..#airflow-version-support) matrix.
14+
15+
## Airflow 2.X
16+
17+
For example, to use airflow `2.1.4`, with python `3.7`:
18+
19+
```yaml
20+
airflow:
21+
image:
22+
repository: apache/airflow
23+
tag: 2.1.4-python3.7
24+
```
25+
26+
## Airflow 1.10
27+
28+
> 🟥 __Warning__ 🟥
29+
>
30+
> To use an `airflow.image.tag` with Airflow `1.10+`, you must set `airflow.legacyCommands` to `true`.
31+
32+
For example, to use airflow `1.10.15`, with python `3.8`:
33+
34+
```yaml
35+
airflow:
36+
# WARNING: this must be "true" for airflow 1.10
37+
legacyCommands: true
38+
39+
image:
40+
repository: apache/airflow
41+
tag: 1.10.15-python3.8
42+
```
43+
44+
## Building a Custom Image
45+
46+
Airflow provides documentation on [building custom docker images](https://airflow.apache.org/docs/docker-stack/build.html), you may follow this process to create a custom image.
47+
48+
For example, after building and tagging your Dockerfile as `MY_REPO:MY_TAG`, you may use it with the chart by specifying `airflow.image.*`:
49+
50+
```yaml
51+
airflow:
52+
# WARNING: this must be "true" for airflow 1.10
53+
#legacyCommands: true
54+
55+
image:
56+
repository: MY_REPO
57+
tag: MY_TAG
58+
59+
## WARNING: even if set to "Always" do not reuse tag names, as containers only pull the latest image when restarting
60+
pullPolicy: IfNotPresent
61+
```

0 commit comments

Comments
 (0)