-
Notifications
You must be signed in to change notification settings - Fork 607
refactor: upgrade HBase and replace custom hbase-shaded-endpint #3021
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
vaijosh
wants to merge
5
commits into
apache:master
Choose a base branch
from
vaijosh:Hbase-265-upgrade
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
c4eac44
[Improve] Upgrade HBase version and replace custom hbase-shaded-endpo…
vaijosh 4d1bfa1
refactor: upgrade HBase and replace custom hbase-shaded-endpint
vaijosh 21f9774
refactor: upgrade HBase to 2.6.5 and prune transitives (#3021)
vaijosh bc64ce7
Merge branch 'apache:master' into Hbase-265-upgrade
vaijosh 52b1d1c
refactor: upgrade HBase and replace custom hbase-shaded-endpint#3021
vaijosh File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,109 @@ | ||
| # Licensed to the Apache Software Foundation (ASF) under one or more | ||
| # contributor license agreements. See the NOTICE file distributed with | ||
| # this work for additional information regarding copyright ownership. | ||
| # The ASF licenses this file to You under the Apache License, Version 2.0 | ||
| # (the "License"); you may not use this file except in compliance with | ||
| # the License. You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
| # | ||
| # Standalone HBase 2.6.5 image for HugeGraph HBase backend testing. | ||
| # Exposes ZooKeeper on 2181 with znode /hbase (matching hugegraph.properties defaults). | ||
|
|
||
| FROM eclipse-temurin:11-jre-jammy | ||
|
|
||
| ARG HBASE_VERSION=2.6.5 | ||
| ARG HBASE_PRIMARY_URL=https://archive.apache.org/dist/hbase/${HBASE_VERSION}/hbase-${HBASE_VERSION}-bin.tar.gz | ||
| ARG HBASE_FALLBACK_URL=https://downloads.apache.org/hbase/${HBASE_VERSION}/hbase-${HBASE_VERSION}-bin.tar.gz | ||
| ARG ALLOW_UNVERIFIED_DOWNLOAD=false | ||
|
|
||
| ENV HBASE_VERSION=${HBASE_VERSION} | ||
| ENV HBASE_HOME=/opt/hbase | ||
| ENV PATH=${HBASE_HOME}/bin:${PATH} | ||
|
|
||
| RUN apt-get update && apt-get install -y --no-install-recommends \ | ||
| curl \ | ||
| netcat-openbsd \ | ||
| && rm -rf /var/lib/apt/lists/* | ||
|
|
||
| RUN set -eux; \ | ||
| download_ok=0; \ | ||
| downloaded_url=""; \ | ||
| for url in "${HBASE_PRIMARY_URL}" "${HBASE_FALLBACK_URL}"; do \ | ||
| [ -n "${url}" ] || continue; \ | ||
| echo "Downloading HBase from ${url}"; \ | ||
| if curl -fL --retry 8 --retry-delay 5 --retry-all-errors \ | ||
| --connect-timeout 30 --max-time 1800 "${url}" -o /tmp/hbase.tar.gz; then \ | ||
| download_ok=1; \ | ||
| downloaded_url="${url}"; \ | ||
| break; \ | ||
| fi; \ | ||
| echo "Download failed for ${url}, trying next source..."; \ | ||
| rm -f /tmp/hbase.tar.gz; \ | ||
| done; \ | ||
| if [ "${download_ok}" -ne 1 ]; then \ | ||
| echo "Unable to download HBase ${HBASE_VERSION} tarball from configured sources"; \ | ||
| exit 1; \ | ||
| fi; \ | ||
| checksum_ok=0; \ | ||
| for checksum_url in "${downloaded_url}.sha512" "${HBASE_PRIMARY_URL}.sha512" "${HBASE_FALLBACK_URL}.sha512"; do \ | ||
| [ -n "${checksum_url}" ] || continue; \ | ||
| if curl -fL --retry 5 --retry-delay 3 --retry-all-errors \ | ||
| --connect-timeout 30 "${checksum_url}" -o /tmp/hbase.tar.gz.sha512 2>/dev/null; then \ | ||
| checksum_ok=1; \ | ||
| break; \ | ||
| fi; \ | ||
| done; \ | ||
| if [ "${checksum_ok}" -eq 1 ]; then \ | ||
| echo "Verifying SHA512 checksum..."; \ | ||
| expected_hash=$(grep -Eio '[a-f0-9]{128}' /tmp/hbase.tar.gz.sha512 | head -n 1 | tr 'A-F' 'a-f' || true); \ | ||
| if [ -z "$expected_hash" ]; then \ | ||
| expected_hash=$(awk '{for (i = 1; i <= NF; i++) if (length($i) >= 8 && $i ~ /^[0-9A-Fa-f]+$/) hash = hash $i} END {if (length(hash) >= 128) print tolower(substr(hash, 1, 128))}' /tmp/hbase.tar.gz.sha512 || true); \ | ||
| fi; \ | ||
| if [ -n "$expected_hash" ]; then \ | ||
| actual_hash=$(sha512sum /tmp/hbase.tar.gz | awk '{print $1}'); \ | ||
| if [ "$expected_hash" = "$actual_hash" ]; then \ | ||
| echo "SHA512 verified OK"; \ | ||
| else \ | ||
| echo "ERROR: SHA512 mismatch"; \ | ||
| echo " Expected: $expected_hash"; \ | ||
| echo " Actual: $actual_hash"; \ | ||
| exit 1; \ | ||
| fi; \ | ||
| else \ | ||
| if [ "${ALLOW_UNVERIFIED_DOWNLOAD}" = "true" ]; then \ | ||
| echo "WARNING: Could not parse SHA512 file (ALLOW_UNVERIFIED_DOWNLOAD=true)"; \ | ||
| else \ | ||
| echo "ERROR: Could not parse SHA512 file"; \ | ||
| exit 1; \ | ||
| fi; \ | ||
| fi; \ | ||
| else \ | ||
| if [ "${ALLOW_UNVERIFIED_DOWNLOAD}" = "true" ]; then \ | ||
| echo "WARNING: Could not download SHA512 file (ALLOW_UNVERIFIED_DOWNLOAD=true)"; \ | ||
| else \ | ||
| echo "ERROR: Could not download SHA512 file"; \ | ||
| exit 1; \ | ||
| fi; \ | ||
| fi; \ | ||
| tar -xzf /tmp/hbase.tar.gz -C /opt; \ | ||
| mv /opt/hbase-${HBASE_VERSION} ${HBASE_HOME}; \ | ||
| rm -f /tmp/hbase.tar.gz /tmp/hbase.tar.gz.sha512 | ||
|
|
||
| # hbase-site.xml: standalone mode, znode=/hbase (matches hugegraph.properties) | ||
| COPY hbase-site.xml ${HBASE_HOME}/conf/hbase-site.xml | ||
|
|
||
| # Entrypoint: start HBase then tail the master log | ||
| COPY entrypoint.sh /entrypoint.sh | ||
| RUN chmod +x /entrypoint.sh | ||
|
|
||
| EXPOSE 2181 16000 16010 16020 16030 | ||
|
|
||
| ENTRYPOINT ["/entrypoint.sh"] | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,72 @@ | ||
| # Licensed to the Apache Software Foundation (ASF) under one or more | ||
| # contributor license agreements. See the NOTICE file distributed with | ||
| # this work for additional information regarding copyright ownership. | ||
| # The ASF licenses this file to You under the Apache License, Version 2.0 | ||
| # (the "License"); you may not use this file except in compliance with | ||
| # the License. You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
| # | ||
| # HBase standalone for local HugeGraph HBase backend development & testing. | ||
| # | ||
| # Usage: | ||
| # Start: docker compose -f docker/hbase/docker-compose.hbase.yml up -d | ||
| # Wait: docker compose -f docker/hbase/docker-compose.hbase.yml logs -f hbase | ||
| # Verify: nc -z localhost 2181 && echo "ZooKeeper OK" | ||
| # Test: mvn test -pl hugegraph-server/hugegraph-test -am -P core-test,hbase | ||
| # Stop: docker compose -f docker/hbase/docker-compose.hbase.yml down -v | ||
| # | ||
| # HugeGraph test config (hugegraph-server/hugegraph-test/src/main/resources/hugegraph.properties): | ||
| # hbase.hosts=localhost | ||
| # hbase.port=2181 | ||
| # hbase.znode_parent=/hbase | ||
| # | ||
| # Ports exposed to host: | ||
| # 2181 - ZooKeeper (HBase embedded) | ||
| # 16000 - HBase Master RPC | ||
| # 16010 - HBase Master Web UI -> http://localhost:16010 | ||
| # 16020 - HBase RegionServer RPC | ||
| # 16030 - HBase RegionServer Web UI -> http://localhost:16030 | ||
|
|
||
| services: | ||
| hbase: | ||
| build: | ||
| context: . | ||
| dockerfile: Dockerfile | ||
| args: | ||
| HBASE_VERSION: "2.6.5" | ||
| # Optional overrides for flaky networks/corporate mirrors. | ||
| HBASE_PRIMARY_URL: "${HBASE_PRIMARY_URL:-https://downloads.apache.org/hbase/2.6.5/hbase-2.6.5-bin.tar.gz}" | ||
| HBASE_FALLBACK_URL: "${HBASE_FALLBACK_URL:-https://archive.apache.org/dist/hbase/2.6.5/hbase-2.6.5-bin.tar.gz}" | ||
| image: hugegraph/hbase:2.6.5 | ||
| container_name: hg-hbase-test | ||
| hostname: hbase | ||
| ports: | ||
| - "2181:2181" # ZooKeeper (matches hbase.port in hugegraph.properties) | ||
| - "16000:16000" # Master RPC | ||
| - "16010:16010" # Master Web UI -> http://localhost:16010 | ||
| - "16020:16020" # RegionServer RPC | ||
| - "16030:16030" # RegionServer Web UI -> http://localhost:16030 | ||
| volumes: | ||
| - hbase-data:/tmp/hbase | ||
| - hbase-zk-data:/tmp/zookeeper | ||
| healthcheck: | ||
| # nc -z confirms ZooKeeper is accepting connections | ||
| test: ["CMD", "nc", "-z", "localhost", "2181"] | ||
| interval: 10s | ||
| timeout: 10s | ||
| retries: 15 | ||
| start_period: 90s | ||
| restart: unless-stopped | ||
|
|
||
| volumes: | ||
| hbase-data: | ||
| hbase-zk-data: | ||
|
|
||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| #!/bin/bash | ||
| # | ||
| # Licensed to the Apache Software Foundation (ASF) under one or more | ||
| # contributor license agreements. See the NOTICE file distributed with | ||
| # this work for additional information regarding copyright ownership. | ||
| # The ASF licenses this file to You under the Apache License, Version 2.0 | ||
| # (the "License"); you may not use this file except in compliance with | ||
| # the License. You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
| # | ||
| set -e | ||
|
|
||
| echo "Starting HBase ${HBASE_VERSION} standalone..." | ||
|
|
||
| # Start services explicitly to avoid SSH-based helper assumptions in containers | ||
| ${HBASE_HOME}/bin/hbase-daemon.sh start zookeeper | ||
| ${HBASE_HOME}/bin/hbase-daemon.sh start master | ||
| ${HBASE_HOME}/bin/hbase-daemon.sh start regionserver | ||
|
|
||
| echo "HBase started. Waiting for ZooKeeper on port 2181..." | ||
| zk_attempts=0 | ||
| until nc -z localhost 2181; do | ||
| zk_attempts=$((zk_attempts + 1)) | ||
| if [ "$zk_attempts" -ge 120 ]; then | ||
| echo "Timed out waiting for ZooKeeper on 2181" | ||
| ${HBASE_HOME}/bin/hbase-daemon.sh status || true | ||
| exit 1 | ||
| fi | ||
| sleep 1 | ||
| done | ||
| echo "ZooKeeper is ready." | ||
|
|
||
| echo "Waiting for HBase Master..." | ||
| master_attempts=0 | ||
| until echo "status 'simple'" | ${HBASE_HOME}/bin/hbase shell -n 2>/dev/null | grep -E -q "([1-9][0-9]*[[:space:]]+live[[:space:]]+servers|[1-9][0-9]*[[:space:]]+servers|servers:[[:space:]]*[1-9])"; do | ||
| master_attempts=$((master_attempts + 1)) | ||
| if [ "$master_attempts" -ge 180 ]; then | ||
| echo "Timed out waiting for HBase master/regionserver readiness" | ||
| ${HBASE_HOME}/bin/hbase-daemon.sh status || true | ||
| tail -n 80 ${HBASE_HOME}/logs/hbase-*.out ${HBASE_HOME}/logs/hbase-*.log \ | ||
| 2>/dev/null || true | ||
| exit 1 | ||
| fi | ||
| sleep 3 | ||
| done | ||
| echo "HBase is ready. Master + RegionServer online." | ||
|
|
||
| # Tail all daemon logs so `docker logs` includes startup/runtime issues | ||
| shopt -s nullglob | ||
| log_files=("${HBASE_HOME}/logs"/hbase-*.out "${HBASE_HOME}/logs"/hbase-*.log) | ||
| if [ ${#log_files[@]} -gt 0 ]; then | ||
| exec tail -F "${log_files[@]}" | ||
| fi | ||
| exec tail -f /dev/null | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,89 @@ | ||
| <?xml version="1.0"?> | ||
| <!-- | ||
| Licensed to the Apache Software Foundation (ASF) under one or more | ||
| contributor license agreements. See the NOTICE file distributed with | ||
| this work for additional information regarding copyright ownership. | ||
| The ASF licenses this file to You under the Apache License, Version 2.0 | ||
| (the "License"); you may not use this file except in compliance with | ||
| the License. You may obtain a copy of the License at | ||
|
|
||
| http://www.apache.org/licenses/LICENSE-2.0 | ||
|
|
||
| Unless required by applicable law or agreed to in writing, software | ||
| distributed under the License is distributed on an "AS IS" BASIS, | ||
| WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| See the License for the specific language governing permissions and | ||
| limitations under the License. | ||
| --> | ||
| <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> | ||
| <configuration> | ||
| <!-- Standalone mode: use local filesystem --> | ||
| <property> | ||
| <name>hbase.rootdir</name> | ||
| <value>file:///tmp/hbase</value> | ||
| </property> | ||
| <!-- Embedded ZooKeeper data directory --> | ||
| <property> | ||
| <name>hbase.zookeeper.property.dataDir</name> | ||
| <value>/tmp/zookeeper</value> | ||
| </property> | ||
| <!-- Keep ZooKeeper quorum explicit for containerized single-node setup --> | ||
| <property> | ||
| <name>hbase.zookeeper.quorum</name> | ||
| <value>localhost</value> | ||
| </property> | ||
| <property> | ||
| <name>hbase.zookeeper.property.clientPort</name> | ||
| <value>2181</value> | ||
| </property> | ||
| <!-- Allow unlimited ZooKeeper client connections --> | ||
| <property> | ||
| <name>hbase.zookeeper.property.maxClientCnxns</name> | ||
| <value>0</value> | ||
| </property> | ||
| <!-- Match hugegraph.properties: hbase.znode_parent=/hbase --> | ||
| <property> | ||
| <name>zookeeper.znode.parent</name> | ||
| <value>/hbase</value> | ||
| </property> | ||
| <!-- Longer tick + session for slow CI/test environments --> | ||
| <property> | ||
| <name>hbase.zookeeper.property.tickTime</name> | ||
| <value>6000</value> | ||
| </property> | ||
| <property> | ||
| <name>zookeeper.session.timeout</name> | ||
| <value>180000</value> | ||
| </property> | ||
| <!-- Pseudo-distributed mode so master + regionserver run as distinct daemons --> | ||
| <property> | ||
| <name>hbase.cluster.distributed</name> | ||
| <value>true</value> | ||
| </property> | ||
| <!-- Local FS in single-node Docker may not support async WAL hflush --> | ||
| <property> | ||
| <name>hbase.wal.provider</name> | ||
| <value>filesystem</value> | ||
| </property> | ||
| <property> | ||
| <name>hbase.unsafe.stream.capability.enforce</name> | ||
| <value>false</value> | ||
| </property> | ||
| <property> | ||
| <name>hbase.master.hostname</name> | ||
| <value>localhost</value> | ||
| </property> | ||
| <property> | ||
| <name>hbase.regionserver.hostname</name> | ||
| <value>localhost</value> | ||
| </property> | ||
| <property> | ||
| <name>hbase.master.ipc.address</name> | ||
| <value>0.0.0.0</value> | ||
| </property> | ||
| <property> | ||
| <name>hbase.regionserver.ipc.address</name> | ||
| <value>0.0.0.0</value> | ||
| </property> | ||
| </configuration> | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.