Skip to content

Add more options to df_engine docker build, slim down features for some packages #1540

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
JakeDern wants to merge 9 commits into
base: main
Choose a base branch
from
+24 −9
Open

Add more options to df_engine docker build, slim down features for some packages #1540

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
8e43370
c
JakeDern Dec 6, 2025
20d07eb
Add more options
JakeDern Dec 6, 2025
b57ac8d
misc fixes
JakeDern Dec 6, 2025
217fe54
update dockerfile
JakeDern Dec 6, 2025
fc4b967
revert yaml change
JakeDern Dec 6, 2025
2875e26
Fix up features
JakeDern Dec 7, 2025
bb931ca
Move fs feature from workspace to otap crate
JakeDern Dec 7, 2025
de50e47
Move azure features to otap crate from workspace
JakeDern Dec 7, 2025
c6171b6
Default RUSTFLAGS in dockerfile
JakeDern Dec 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions rust/otap-dataflow/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ miette = { version="7.6.0", features = ["fancy"] }
mimalloc-rust = "0.2.1"
nix = { version = "0.30.1", features = ["process", "signal"] }
num_enum = "0.7"
object_store = "0.12.3"
object_store = {version = "0.12.3", default-features = false}
once_cell = "1.20.2"
opentelemetry = "0.31.0"
opentelemetry-proto = { version = "0.31", default-features = false, features = ["gen-tonic-messages", "logs"]} #TODO - use it from submodule instead of crate(?)
Expand Down Expand Up @@ -150,10 +150,10 @@ zip = "=4.2.0"
byte-unit = "5.2.0"

# Azure Monnitor Exporter
azure_identity = "0.30.0"
azure_core = {version = "0.30.1", default-features = false }
azure_identity = {version = "0.30.0", default-features = false }
flate2 = "1.1.5"
reqwest = "0.12.24"
azure_core = "0.30.1"
reqwest = { version = "0.12.24", default-features = false }
time = "0.3.44"
wiremock = "0.6.5"

Expand Down
15 changes: 15 additions & 0 deletions rust/otap-dataflow/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,27 @@ COPY --from=otel-arrow /proto/opentelemetry-proto /build/proto/opentelemetry-pro

COPY . /build/rust/dataflow/.

# RUSTFLAGS can be used to pass argument to rustc e.g.
# --build-arg RUSTFLAGS="-C target-feature=-gfni"
ARG RUSTFLAGS="-C target-cpu=native"
ENV RUSTFLAGS=${RUSTFLAGS}

# These are used to enable features and are passed to carg's --features argument
# e.g. --build-arg FEATURES="azure,quiver-persistence"
ARG FEATURES
ENV FEATURES=${FEATURES}

# Build dataflow engine
RUN ./cross-arch-build.sh

# The runtime image
FROM docker.io/alpine:3.22@sha256:4b7ce07002c69e8f3d704a9c5d6fd3053be500b7f1c69fc0d80990c2ad8dd412
LABEL maintainer="The OpenTelemetry Authors"

# Passing --build-arg BUILD=1 adds additional debugging features to the image.
ARG DEBUG="0"
RUN if [ "$DEBUG" = "1" ]; then apk add --no-cache gdb; fi

RUN addgroup dataflow \
&& adduser \
--ingroup dataflow \
Expand Down
6 changes: 3 additions & 3 deletions rust/otap-dataflow/crates/otap/Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ futures-timer.workspace = true
http.workspace = true
humantime-serde.workspace = true
log.workspace = true
object_store.workspace = true
object_store = {workspace = true, features = ["fs"]}
parquet.workspace = true
thiserror = { workspace = true }
serde_with = { workspace = true }
Expand Down Expand Up @@ -70,8 +70,8 @@ geneva-uploader = { version = "0.3.0", optional = true }
# Azure Monitor Exporter dependencies
azure_identity = { workspace = true, optional = true }
flate2 = { workspace = true, optional = true }
reqwest = { workspace = true, optional = true }
azure_core = { workspace = true, optional = true }
reqwest = { workspace = true, optional = true, features = ["rustls-tls"] }
azure_core = { workspace = true, optional = true, features = ["reqwest"] }

# OpenTelemetry proto with tonic feature enabled
opentelemetry-proto = { workspace = true, optional = true }
Expand Down
4 changes: 2 additions & 2 deletions rust/otap-dataflow/cross-arch-build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,5 +16,5 @@ else
fi

rustup target add "${RUST_TARGET}"
cargo build --release --target="${RUST_TARGET}"
cp "target/${RUST_TARGET}/release/df_engine" .
cargo build --release --features "$FEATURES" --target="${RUST_TARGET}"
cp "target/${RUST_TARGET}/release/df_engine" .
Loading