Skip to content

Commit d3f716e

Browse files
Z1Wupan3793
Z1Wu
authored and
pan3793
committed
[KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine
### Why are the changes needed? - There are some SQLs need to access hbase such as SQL read from Hive-Hbase Storage Handler tables. And for proxy-user basd SparkSQLEngine, it is unable to renew delegation token by itself. It would be convenient if kyuubi can renew hbase delegation token. - Currently kyuubi server only supports updating hdfs and hms delegation token. ### How was this patch tested? #### unit test Add a unit test which fetching delegation token from a hbase cluster created by in docker containter. #### manual integrate test with hbase deps compile with `-Dhbase.deps.scope=compile` ``` ./build/dist --tgz --spark-provided --flink-provided --hive-provided --name spark3.5 -Pspark3.5 -Dhbase.deps.scope=compile ``` <img width="489" height="35" alt="image" src="https://github.com/user-attachments/assets/c47303b9-1c6e-44a8-801f-de45ed732be1" /> run kyuubi server with hbase related conf ``` conf hbase.zookeeper.quorum xxxx hbase.security.authentication kerberos hbase.master.kerberos.principal xxx hbase.regionserver.kerberos.principal xxx ``` request hbase token <img width="1632" height="278" alt="image" src="https://github.com/user-attachments/assets/e73b9534-5c3e-47f4-b1b4-df4ffef900a5" /> run sample sql <img width="1898" height="723" alt="image" src="https://github.com/user-attachments/assets/b7aa53a4-dd13-47fc-9218-31e01f43056e" /> #### manual integrate test without hbase deps compile without `-Dhbase.deps.scope=compile` ``` ./build/dist --tgz --spark-provided --flink-provided --hive-provided --name spark3.5 -Pspark3.5 ``` <img width="496" height="48" alt="image" src="https://github.com/user-attachments/assets/3d07b0ad-fa81-4aa9-aa98-436908afe721" /> only fetch HMS token and hdfs token <img width="1468" height="58" alt="image" src="https://github.com/user-attachments/assets/4665f6f8-99df-4671-8d44-b64ffc8ac98d" /> run sample query <img width="1345" height="571" alt="image" src="https://github.com/user-attachments/assets/c33e6d34-d3d6-4429-9612-347fcee832c3" /> ### Was this patch authored or co-authored using generative AI tooling? No Closes #7182 from Z1Wu/feat/hbase_delegation_token_support. Closes #7156 3c61b4f [wuziyi] [fix] format e58288e [wuziyi] [fix] close connection 9c8e7bf [z1wu] Update kyuubi-server/src/test/scala/org/apache/kyuubi/WithSecuredHBaseContainer.scala 10d7dbb [wuziyi] [fix] log format 95c7700 [wuziyi] fix format aa32571 [wuziyi] fix NPE 781b502 [wuziyi] format 426d300 [wuziyi] exclude unnecessary dependency 92cb102 [wuziyi] update a58dff7 [wuziyi] update aad4c2e [wuziyi] update 9244dac [wuziyi] [fix] spi order e15f855 [wuziyi] fix 8ba55cb [wuziyi] switch to hbase-shaded-client-byo-hadoop client jar (byo stands for `bring your own`) 6a65ecf [wuziyi] update (revert to unshaded hbase client for test) a86eb2a [wuziyi] update 1769c5e [wuziyi] update 7373260 [wuziyi] [feat] add hbase delegation token manager to support long running proxy spark applications which need to access secure hbase cluster Lead-authored-by: wuziyi <[email protected]> Co-authored-by: wuziyi <[email protected]> Co-authored-by: z1wu <[email protected]> Signed-off-by: Cheng Pan <[email protected]>
1 parent 7605900 commit d3f716e

File tree

7 files changed

+614
-0
lines changed

7 files changed

+614
-0
lines changed

‎build/Dockerfile.HBase‎

Lines changed: 319 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,319 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
18+
# This Dockerfile is for building a HBase Cluster used in testing
19+
20+
# Usage:
21+
# Run the docker command below
22+
# docker build \
23+
# --build-arg APACHE_MIRROR="https://archive.apache.org/dist" \
24+
# --build-arg HBASE_VERSION="2.5.12" \
25+
# --build-arg ZK_VERSION="3.8.4" \
26+
# --file build/Dockerfile.HBase \
27+
# --tag nekyuubi/kyuubi-hbase-cluster:<tag> \
28+
# .
29+
# Options:
30+
# -f, --file this docker file
31+
# -t, --tag the target repo and tag name
32+
# more options can be found with -h, --help
33+
34+
FROM eclipse-temurin:8-jdk-focal
35+
36+
ENV DEBIAN_FRONTEND=noninteractive
37+
38+
ARG HBASE_VERSION=2.5.12
39+
ARG APACHE_MIRROR=https://archive.apache.org/dist
40+
ARG ZK_VERSION=3.8.1
41+
42+
ENV HBASE_HOME=/opt/hbase
43+
ENV ZK_HOME=/opt/zookeeper
44+
ENV JAVA_HOME=/opt/java/openjdk
45+
ENV PATH=$PATH:$HBASE_HOME/bin
46+
47+
RUN apt-get update && apt-get install -y \
48+
krb5-kdc \
49+
krb5-admin-server \
50+
wget \
51+
busybox \
52+
&& mkdir /opt/busybox && \
53+
busybox --install /opt/busybox
54+
55+
# download hbase
56+
RUN wget -q ${APACHE_MIRROR}/hbase/${HBASE_VERSION}/hbase-${HBASE_VERSION}-bin.tar.gz && \
57+
tar -xzf hbase-${HBASE_VERSION}-bin.tar.gz -C /opt && \
58+
mv /opt/hbase-${HBASE_VERSION} ${HBASE_HOME} && \
59+
rm hbase-${HBASE_VERSION}-bin.tar.gz
60+
61+
# download zookeeper
62+
RUN wget -q ${APACHE_MIRROR}/zookeeper/zookeeper-${ZK_VERSION}/apache-zookeeper-${ZK_VERSION}-bin.tar.gz && \
63+
tar -xzf apache-zookeeper-${ZK_VERSION}-bin.tar.gz -C /opt && \
64+
mv /opt/apache-zookeeper-${ZK_VERSION}-bin ${ZK_HOME} && \
65+
rm apache-zookeeper-${ZK_VERSION}-bin.tar.gz
66+
67+
# add config files
68+
COPY <<"EOF" /etc/krb5.conf
69+
70+
[libdefaults]
71+
default_realm = TEST.ORG
72+
dns_lookup_realm = false
73+
dns_lookup_kdc = false
74+
75+
[realms]
76+
TEST.ORG = {
77+
kdc = localhost:88
78+
}
79+
80+
EOF
81+
82+
COPY <<"EOF" ${HBASE_HOME}/conf/hbase-site.xml
83+
84+
<configuration>
85+
<property>
86+
<name>hbase.cluster.distributed</name>
87+
<value>true</value>
88+
</property>
89+
<property>
90+
<name>hbase.rootdir</name>
91+
<value>file:///tmp/hbase-data</value>
92+
</property>
93+
<property>
94+
<name>hbase.security.authentication</name>
95+
<value>kerberos</value>
96+
</property>
97+
<property>
98+
<name>hbase.master.kerberos.principal</name>
99+
<value>hbase/[email protected]</value>
100+
</property>
101+
<property>
102+
<name>hbase.master.keytab.file</name>
103+
<value>/opt/hbase/conf/hbase.keytab</value>
104+
</property>
105+
<property>
106+
<name>hbase.regionserver.kerberos.principal</name>
107+
<value>hbase/[email protected]</value>
108+
</property>
109+
<property>
110+
<name>hbase.regionserver.keytab.file</name>
111+
<value>/opt/hbase/conf/hbase.keytab</value>
112+
</property>
113+
<property>
114+
<name>hbase.security.authorization</name>
115+
<value>true</value>
116+
</property>
117+
<property>
118+
<name>hbase.master.hostname</name>
119+
<value>localhost</value>
120+
</property>
121+
<property>
122+
<name>hbase.regionserver.hostname</name>
123+
<value>localhost</value>
124+
</property>
125+
<property>
126+
<name>hbase.unsafe.regionserver.hostname</name>
127+
<value>localhost</value>
128+
</property>
129+
<property>
130+
<name>hbase.regionserver.ipc.address</name>
131+
<value>0.0.0.0</value>
132+
</property>
133+
<property>
134+
<name>hbase.master.ipc.address</name>
135+
<value>0.0.0.0</value>
136+
</property>
137+
<property>
138+
<name>hbase.zookeeper.quorum</name>
139+
<value>localhost</value>
140+
</property>
141+
<property>
142+
<name>hbase.zookeeper.property.clientPort</name>
143+
<value>2181</value>
144+
</property>
145+
<property>
146+
<name>hbase.zookeeper.client.keytab.file</name>
147+
<value>/opt/hbase/conf/hbase.keytab</value>
148+
</property>
149+
<property>
150+
<name>hbase.zookeeper.client.kerberos.principal</name>
151+
<value>hbase/[email protected]</value>
152+
</property>
153+
<property>
154+
<name>hbase.unsafe.stream.capability.enforce</name>
155+
<value>false</value>
156+
</property>
157+
<property>
158+
<name>hbase.coprocessor.region.classes</name>
159+
<value>org.apache.hadoop.hbase.security.token.TokenProvider,org.apache.hadoop.hbase.security.access.AccessController</value>
160+
</property>
161+
<property>
162+
<name>hbase.coprocessor.master.classes</name>
163+
<value>org.apache.hadoop.hbase.security.access.AccessController</value>
164+
</property>
165+
</configuration>
166+
167+
EOF
168+
169+
COPY <<"EOF" ${HBASE_HOME}/conf/core-site.xml
170+
171+
<configuration>
172+
<property>
173+
<name>hadoop.security.authentication</name>
174+
<value>kerberos</value>
175+
</property>
176+
<property>
177+
<name>hadoop.security.authorization</name>
178+
<value>true</value>
179+
</property>
180+
<property>
181+
<name>hadoop.proxyuser.hbase.hosts</name>
182+
<value>*</value>
183+
</property>
184+
<property>
185+
<name>hadoop.proxyuser.hbase.groups</name>
186+
<value>*</value>
187+
</property>
188+
</configuration>
189+
190+
EOF
191+
192+
COPY <<"EOF" ${ZK_HOME}/conf/zk-jaas.conf
193+
Server {
194+
com.sun.security.auth.module.Krb5LoginModule required
195+
useKeyTab=true
196+
storeKey=true
197+
useTicketCache=false
198+
keyTab="/opt/zookeeper/conf/zookeeper.keytab"
199+
principal="zookeeper/[email protected]";
200+
};
201+
202+
Client {
203+
com.sun.security.auth.module.Krb5LoginModule required
204+
useTicketCache=true;
205+
};
206+
EOF
207+
208+
COPY <<"EOF" ${ZK_HOME}/conf/zoo.cfg
209+
tickTime=2000
210+
dataDir=/opt/zookeeper/data
211+
clientPort=2181
212+
# Kerberos settings
213+
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
214+
kerberos.keytab.file=/opt/zookeeper/conf/zk.keytab
215+
kerberos.principal=zookeeper/[email protected]
216+
requireClientAuthScheme=sasl
217+
EOF
218+
219+
COPY <<"EOF" /entrypoint.sh
220+
#!/bin/bash
221+
set -e
222+
223+
# Function: Wait for a service to be available on specified ports
224+
# Parameters:
225+
# $1 - Service name
226+
# $2 - Timeout in seconds, default 60 seconds
227+
# $3 - Check interval in seconds, default 1 second
228+
# Remaining parameters - List of ports to check
229+
wait_for_service() {
230+
local service_name="$1"
231+
local timeout="${2:-60}"
232+
local interval="${3:-1}"
233+
shift 3 # Remove the first three parameters, remaining are ports
234+
235+
local end_time=$((SECONDS + timeout))
236+
local ports_available=false
237+
echo "Starting to wait for $service_name service to be available on ports $@..."
238+
239+
while [ $SECONDS -lt $end_time ]; do
240+
ports_available=true
241+
242+
# Check all provided ports
243+
for port in "$@"; do
244+
if ! /opt/busybox/nc localhost "$port" -e true; then
245+
ports_available=false
246+
break
247+
fi
248+
done
249+
250+
if $ports_available; then
251+
echo "$service_name service is now available"
252+
return 0
253+
else
254+
echo "$service_name service is not available, retrying in $interval seconds..."
255+
sleep $interval
256+
fi
257+
done
258+
259+
echo "Timeout reached. $service_name service is still not available."
260+
exit 1
261+
}
262+
263+
export REALM="TEST.ORG"
264+
export KADMIN_PASS="admin"
265+
export HBASE_CONF_DIR="${HBASE_HOME}/conf"
266+
export ZK_CONF_DIR="${ZK_HOME}/conf"
267+
268+
echo "Creating KDC database..."
269+
kdb5_util create -r ${REALM} -s -P ${KADMIN_PASS}
270+
271+
echo "Creating principals and keytabs..."
272+
kadmin.local -q "addprinc -randkey hbase/localhost@${REALM}"
273+
kadmin.local -q "ktadd -k ${HBASE_CONF_DIR}/hbase.keytab hbase/localhost@${REALM}"
274+
275+
kadmin.local -q "addprinc -randkey zookeeper/localhost@${REALM}"
276+
kadmin.local -q "ktadd -k ${ZK_CONF_DIR}/zookeeper.keytab zookeeper/localhost@${REALM}"
277+
278+
mkdir -p ${HBASE_HOME}/logs/
279+
mkdir -p ${ZK_HOME}/logs/
280+
281+
echo "Principals created. Starting supervisor..."
282+
# start kdc service
283+
echo "Start kdc ..."
284+
/etc/init.d/krb5-kdc start
285+
wait_for_service "kdc" 15 1 88
286+
287+
# start zookeeper server
288+
export JVMFLAGS="-Djava.security.auth.login.config=/opt/zookeeper/conf/zk-jaas.conf"
289+
/opt/zookeeper/bin/zkServer.sh start
290+
unset JVMFLAGS
291+
wait_for_service "zookeeper" 15 1 2181
292+
293+
kinit -kt ${HBASE_CONF_DIR}/hbase.keytab hbase/localhost@${REALM}
294+
295+
# start hbase master server
296+
nohup /opt/hbase/bin/hbase master start > "/opt/hbase/logs/master.log" 2>&1 < /dev/null &
297+
wait_for_service "master" 15 1 16000
298+
299+
# start hbase region server
300+
nohup /opt/hbase/bin/hbase regionserver start > "/opt/hbase/logs/regionserver.log" 2>&1 < /dev/null &
301+
wait_for_service "master" 15 1 16020
302+
303+
tail -F --follow=name --retry -v /opt/zookeeper/logs/* /opt/hbase/logs/*
304+
EOF
305+
306+
RUN chmod +x /entrypoint.sh
307+
308+
# HBase Master UI
309+
EXPOSE 16010
310+
# Kerberos
311+
EXPOSE 88/udp
312+
# zookeeper
313+
EXPOSE 2181
314+
# HBase Master
315+
EXPOSE 16000
316+
# HBase RegionServer
317+
EXPOSE 16020
318+
319+
ENTRYPOINT ["/entrypoint.sh"]

‎kyuubi-server/pom.xml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -123,6 +123,11 @@
123123
<version>${kyuubi-relocated.version}</version>
124124
</dependency>
125125

126+
<dependency>
127+
<groupId>org.apache.hbase</groupId>
128+
<artifactId>hbase-shaded-client-byo-hadoop</artifactId>
129+
</dependency>
130+
126131
<dependency>
127132
<groupId>org.antlr</groupId>
128133
<artifactId>antlr4-runtime</artifactId>

‎kyuubi-server/src/main/resources/META-INF/services/org.apache.kyuubi.credentials.HadoopDelegationTokenProvider‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,5 +15,6 @@
1515
# limitations under the License.
1616
#
1717

18+
org.apache.kyuubi.credentials.HBaseDelegationTokenProvider
1819
org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider
1920
org.apache.kyuubi.credentials.HiveDelegationTokenProvider

0 commit comments

Comments
 (0)