[Bug]: Deployments failing without clear cause / error

[suptejas]

Error Message and Logs

Our deployments have been in-deterministically failing with no obvious cause. Here's a few deployment logs on our end that just failed, and would work perfectly with a redeploy.

2025-Nov-05 10:19:19.363823
Found a suitable build server (Builder).
2025-Nov-05 10:19:19.370090
Starting deployment of dimensionhq/ai:master to Webhooks - 2.
2025-Nov-05 10:19:19.728111
Preparing container with helper image: ghcr.io/coollabsio/coolify-helper:1.0.11
2025-Nov-05 10:19:19.905615
[CMD]: docker stop --time=30 uw4kosoc84ooc808ww4080ww
2025-Nov-05 10:19:19.905615
Flag --time has been deprecated, use --timeout instead
2025-Nov-05 10:19:19.907955
Error response from daemon: No such container: uw4kosoc84ooc808ww4080ww
2025-Nov-05 10:19:20.143670
[CMD]: docker rm -f uw4kosoc84ooc808ww4080ww
2025-Nov-05 10:19:20.143670
Error response from daemon: No such container: uw4kosoc84ooc808ww4080ww
2025-Nov-05 10:19:20.365835
[CMD]: docker run -d --name uw4kosoc84ooc808ww4080ww  --rm -v /root/.docker/config.json:/root/.docker/config.json:ro -v /var/run/docker.sock:/var/run/docker.sock ghcr.io/coollabsio/coolify-helper:1.0.11
2025-Nov-05 10:19:20.365835
0b60882c666a468473fc10c4a2350c55587bb7cd4cb62e200bba037838f453e8
2025-Nov-05 10:19:22.053986
[CMD]: docker exec uw4kosoc84ooc808ww4080ww bash -c 'GIT_SSH_COMMAND="ssh -o ConnectTimeout=30 -p 22 -o Port=22 -o LogLevel=ERROR -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" git ls-remote [https://x-access-token:<REDACTED>@github.com/dimensionhq/ai.git](https://x-access-token:%3CREDACTED%3E@github.com/dimensionhq/ai.git) refs/heads/master'
2025-Nov-05 10:19:22.053986
eb20a0771f925bf19364784030b4df74a55441f5	refs/heads/master
2025-Nov-05 10:19:22.188928
----------------------------------------
2025-Nov-05 10:19:22.194547
Importing dimensionhq/ai:master (commit sha eb20a0771f925bf19364784030b4df74a55441f5) to /artifacts/uw4kosoc84ooc808ww4080ww.
2025-Nov-05 10:19:22.451630
[CMD]: docker exec uw4kosoc84ooc808ww4080ww bash -c 'git clone --depth=1 --recurse-submodules --shallow-submodules -b 'master' '[https://x-access-token:<REDACTED>@github.com/dimensionhq/ai.git'](https://x-access-token:%3CREDACTED%3E@github.com/dimensionhq/ai.git') '/artifacts/uw4kosoc84ooc808ww4080ww' && cd '/artifacts/uw4kosoc84ooc808ww4080ww' && if [ -f .gitmodules ]; then git submodule sync && GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" git submodule update --init --recursive --depth=1; fi && cd '/artifacts/uw4kosoc84ooc808ww4080ww' && GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null" git lfs pull'
2025-Nov-05 10:19:22.451630
Cloning into '/artifacts/uw4kosoc84ooc808ww4080ww'...
2025-Nov-05 10:19:24.006780
[CMD]: docker exec uw4kosoc84ooc808ww4080ww bash -c 'cd /artifacts/uw4kosoc84ooc808ww4080ww && git log -1 eb20a0771f925bf19364784030b4df74a55441f5 --pretty=%B'
2025-Nov-05 10:19:24.006780
Merge branch 'master' of https://github.com/dimensionhq/ai
2025-Nov-05 10:19:24.769603
Image not found (realmsoftwareinc/webhooks-prod:eb20a0771f925bf19364784030b4df74a55441f5). Building new image.
2025-Nov-05 10:19:26.046158
[CMD]: docker exec uw4kosoc84ooc808ww4080ww bash -c 'cat /artifacts/uw4kosoc84ooc808ww4080ww/apps/webhooks/Dockerfile'
2025-Nov-05 10:19:26.046158
# Build dependencies and virtualenv in a dedicated stage
2025-Nov-05 10:19:26.046158
FROM python:3.11-slim AS builder
2025-Nov-05 10:19:26.046158
2025-Nov-05 10:19:26.046158
WORKDIR /app
2025-Nov-05 10:19:26.046158
2025-Nov-05 10:19:26.046158
# Install build tooling and uv (no recommends to keep layer slim)
2025-Nov-05 10:19:26.046158
RUN apt-get update && apt-get install -y --no-install-recommends \
2025-Nov-05 10:19:26.046158
gcc \
2025-Nov-05 10:19:26.046158
g++ \
2025-Nov-05 10:19:26.046158
curl \
2025-Nov-05 10:19:26.046158
&& rm -rf /var/lib/apt/lists/*
2025-Nov-05 10:19:26.046158
2025-Nov-05 10:19:26.046158
RUN pip install --no-cache-dir uv
2025-Nov-05 10:19:26.046158
2025-Nov-05 10:19:26.046158
# Copy workspace files needed for dependency resolution
2025-Nov-05 10:19:26.046158
COPY pyproject.toml uv.lock ./
2025-Nov-05 10:19:26.046158
COPY packages/shared /app/packages/shared
2025-Nov-05 10:19:26.046158
COPY apps/webhooks /app/apps/webhooks
2025-Nov-05 10:19:26.046158
2025-Nov-05 10:19:26.046158
# Install Python dependencies into .venv
2025-Nov-05 10:19:26.046158
RUN uv sync --frozen --package webhooks
2025-Nov-05 10:19:26.046158
2025-Nov-05 10:19:26.046158
# Clean up build artefacts to keep the copied layer trim
2025-Nov-05 10:19:26.046158
RUN rm -rf /root/.cache /root/.uv-cache
2025-Nov-05 10:19:26.046158
2025-Nov-05 10:19:26.046158
# Final runtime image
2025-Nov-05 10:19:26.046158
FROM python:3.11-slim
2025-Nov-05 10:19:26.046158
2025-Nov-05 10:19:26.046158
WORKDIR /app
2025-Nov-05 10:19:26.046158
2025-Nov-05 10:19:26.046158
# Install curl (for health check) and Infisical CLI with minimal extras
2025-Nov-05 10:19:26.046158
RUN apt-get update && apt-get install -y --no-install-recommends curl \
2025-Nov-05 10:19:26.046158
&& curl -1sLf 'https://artifacts-cli.infisical.com/setup.deb.sh' | bash \
2025-Nov-05 10:19:26.046158
&& apt-get install -y --no-install-recommends infisical \
2025-Nov-05 10:19:26.046158
&& rm -rf /var/lib/apt/lists/*
2025-Nov-05 10:19:26.046158
2025-Nov-05 10:19:26.046158
# Copy in the prebuilt virtualenv and application code
2025-Nov-05 10:19:26.046158
COPY --from=builder /app/.venv /app/.venv
2025-Nov-05 10:19:26.046158
COPY --from=builder /app/packages/shared /app/packages/shared
2025-Nov-05 10:19:26.046158
COPY --from=builder /app/apps/webhooks /app/apps/webhooks
2025-Nov-05 10:19:26.046158
COPY pyproject.toml uv.lock ./
2025-Nov-05 10:19:26.046158
2025-Nov-05 10:19:26.046158
# Expose port 8000
2025-Nov-05 10:19:26.046158
EXPOSE 8000
2025-Nov-05 10:19:26.046158
2025-Nov-05 10:19:26.046158
# Set environment variables
2025-Nov-05 10:19:26.046158
ENV PYTHONPATH=/app
2025-Nov-05 10:19:26.046158
ENV PYTHONUNBUFFERED=1
2025-Nov-05 10:19:26.046158
ENV PATH="/app/.venv/bin:$PATH"
2025-Nov-05 10:19:26.046158
2025-Nov-05 10:19:26.046158
# Health check
2025-Nov-05 10:19:26.046158
HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \
2025-Nov-05 10:19:26.046158
CMD curl -f http://localhost:8000/health || exit 1
2025-Nov-05 10:19:26.046158
2025-Nov-05 10:19:26.046158
# Run the application with Infisical and Gunicorn
2025-Nov-05 10:19:26.046158
# Note: Set INFISICAL_TOKEN and INFISICAL_PROJECT_ID environment variables when running the container
2025-Nov-05 10:19:26.046158
# Example: docker run --env INFISICAL_TOKEN=$INFISICAL_TOKEN --env INFISICAL_PROJECT_ID=your-project-id your-image
2025-Nov-05 10:19:26.046158
CMD ["sh", "-c", "infisical run --projectId $INFISICAL_PROJECT_ID --command 'cd /app/apps/webhooks/webhooks && /app/.venv/bin/gunicorn main:app -c gunicorn.conf.py'"]
2025-Nov-05 10:19:26.382688
Creating build-time .env file in /artifacts (outside Docker context).
2025-Nov-05 10:19:26.930010
[CMD]: docker exec uw4kosoc84ooc808ww4080ww bash -c 'cat /artifacts/build-time.env'
2025-Nov-05 10:19:26.930010
SOURCE_COMMIT='eb20a0771f925bf19364784030b4df74a55441f5'
2025-Nov-05 10:19:26.930010
COOLIFY_URL=''
2025-Nov-05 10:19:26.930010
COOLIFY_FQDN=''
2025-Nov-05 10:19:26.930010
COOLIFY_BRANCH='master'
2025-Nov-05 10:19:26.930010
COOLIFY_RESOURCE_UUID='b0808cwg8c4ko4kk84kkcks0'
2025-Nov-05 10:19:26.930010
COOLIFY_CONTAINER_NAME='b0808cwg8c4ko4kk84kkcks0-101917583267'
2025-Nov-05 10:19:26.930010
INFISICAL_PROJECT_ID="65a15447c63fc2a03460007d"
2025-Nov-05 10:19:26.930010
INFISICAL_TOKEN="st.885a8fa6-7c1c-47bb-8bfb-715555b46050.b9107cb09ef5069f89ac2720ab9c424a.1bd120fcdff9cbbab6c6c0e1f7678875"
2025-Nov-05 10:19:27.264299
[CMD]: docker exec uw4kosoc84ooc808ww4080ww bash -c 'cat /artifacts/uw4kosoc84ooc808ww4080ww/apps/webhooks/Dockerfile'
2025-Nov-05 10:19:27.264299
# Build dependencies and virtualenv in a dedicated stage
2025-Nov-05 10:19:27.264299
FROM python:3.11-slim AS builder
2025-Nov-05 10:19:27.264299
2025-Nov-05 10:19:27.264299
WORKDIR /app
2025-Nov-05 10:19:27.264299
2025-Nov-05 10:19:27.264299
# Install build tooling and uv (no recommends to keep layer slim)
2025-Nov-05 10:19:27.264299
RUN apt-get update && apt-get install -y --no-install-recommends \
2025-Nov-05 10:19:27.264299
gcc \
2025-Nov-05 10:19:27.264299
g++ \
2025-Nov-05 10:19:27.264299
curl \
2025-Nov-05 10:19:27.264299
&& rm -rf /var/lib/apt/lists/*
2025-Nov-05 10:19:27.264299
2025-Nov-05 10:19:27.264299
RUN pip install --no-cache-dir uv
2025-Nov-05 10:19:27.264299
2025-Nov-05 10:19:27.264299
# Copy workspace files needed for dependency resolution
2025-Nov-05 10:19:27.264299
COPY pyproject.toml uv.lock ./
2025-Nov-05 10:19:27.264299
COPY packages/shared /app/packages/shared
2025-Nov-05 10:19:27.264299
COPY apps/webhooks /app/apps/webhooks
2025-Nov-05 10:19:27.264299
2025-Nov-05 10:19:27.264299
# Install Python dependencies into .venv
2025-Nov-05 10:19:27.264299
RUN uv sync --frozen --package webhooks
2025-Nov-05 10:19:27.264299
2025-Nov-05 10:19:27.264299
# Clean up build artefacts to keep the copied layer trim
2025-Nov-05 10:19:27.264299
RUN rm -rf /root/.cache /root/.uv-cache
2025-Nov-05 10:19:27.264299
2025-Nov-05 10:19:27.264299
# Final runtime image
2025-Nov-05 10:19:27.264299
FROM python:3.11-slim
2025-Nov-05 10:19:27.264299
2025-Nov-05 10:19:27.264299
WORKDIR /app
2025-Nov-05 10:19:27.264299
2025-Nov-05 10:19:27.264299
# Install curl (for health check) and Infisical CLI with minimal extras
2025-Nov-05 10:19:27.264299
RUN apt-get update && apt-get install -y --no-install-recommends curl \
2025-Nov-05 10:19:27.264299
&& curl -1sLf 'https://artifacts-cli.infisical.com/setup.deb.sh' | bash \
2025-Nov-05 10:19:27.264299
&& apt-get install -y --no-install-recommends infisical \
2025-Nov-05 10:19:27.264299
&& rm -rf /var/lib/apt/lists/*
2025-Nov-05 10:19:27.264299
2025-Nov-05 10:19:27.264299
# Copy in the prebuilt virtualenv and application code
2025-Nov-05 10:19:27.264299
COPY --from=builder /app/.venv /app/.venv
2025-Nov-05 10:19:27.264299
COPY --from=builder /app/packages/shared /app/packages/shared
2025-Nov-05 10:19:27.264299
COPY --from=builder /app/apps/webhooks /app/apps/webhooks
2025-Nov-05 10:19:27.264299
COPY pyproject.toml uv.lock ./
2025-Nov-05 10:19:27.264299
2025-Nov-05 10:19:27.264299
# Expose port 8000
2025-Nov-05 10:19:27.264299
EXPOSE 8000
2025-Nov-05 10:19:27.264299
2025-Nov-05 10:19:27.264299
# Set environment variables
2025-Nov-05 10:19:27.264299
ENV PYTHONPATH=/app
2025-Nov-05 10:19:27.264299
ENV PYTHONUNBUFFERED=1
2025-Nov-05 10:19:27.264299
ENV PATH="/app/.venv/bin:$PATH"
2025-Nov-05 10:19:27.264299
2025-Nov-05 10:19:27.264299
# Health check
2025-Nov-05 10:19:27.264299
HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \
2025-Nov-05 10:19:27.264299
CMD curl -f http://localhost:8000/health || exit 1
2025-Nov-05 10:19:27.264299
2025-Nov-05 10:19:27.264299
# Run the application with Infisical and Gunicorn
2025-Nov-05 10:19:27.264299
# Note: Set INFISICAL_TOKEN and INFISICAL_PROJECT_ID environment variables when running the container
2025-Nov-05 10:19:27.264299
# Example: docker run --env INFISICAL_TOKEN=$INFISICAL_TOKEN --env INFISICAL_PROJECT_ID=your-project-id your-image
2025-Nov-05 10:19:27.264299
CMD ["sh", "-c", "infisical run --projectId $INFISICAL_PROJECT_ID --command 'cd /app/apps/webhooks/webhooks && /app/.venv/bin/gunicorn main:app -c gunicorn.conf.py'"]
2025-Nov-05 10:19:27.306071
Final Dockerfile:
2025-Nov-05 10:19:27.928385
[CMD]: docker exec uw4kosoc84ooc808ww4080ww bash -c 'cat /artifacts/uw4kosoc84ooc808ww4080ww/apps/webhooks/Dockerfile'
2025-Nov-05 10:19:27.928385
# Build dependencies and virtualenv in a dedicated stage
2025-Nov-05 10:19:27.928385
ARG COOLIFY_BUILD_SECRETS_HASH=4387256370ab25c71c6c7f93ce46e2a426fafb4ba429afe60f75c57a2cc75894
2025-Nov-05 10:19:27.928385
ARG COOLIFY_CONTAINER_NAME=b0808cwg8c4ko4kk84kkcks0-101917583267
2025-Nov-05 10:19:27.928385
ARG COOLIFY_RESOURCE_UUID=b0808cwg8c4ko4kk84kkcks0
2025-Nov-05 10:19:27.928385
ARG COOLIFY_BRANCH=master
2025-Nov-05 10:19:27.928385
ARG SOURCE_COMMIT=eb20a0771f925bf19364784030b4df74a55441f5
2025-Nov-05 10:19:27.928385
ARG INFISICAL_TOKEN=st.885a8fa6-7c1c-47bb-8bfb-715555b46050.b9107cb09ef5069f89ac2720ab9c424a.1bd120fcdff9cbbab6c6c0e1f7678875
2025-Nov-05 10:19:27.928385
ARG INFISICAL_PROJECT_ID=65a15447c63fc2a03460007d
2025-Nov-05 10:19:27.928385
FROM python:3.11-slim AS builder
2025-Nov-05 10:19:27.928385
2025-Nov-05 10:19:27.928385
WORKDIR /app
2025-Nov-05 10:19:27.928385
2025-Nov-05 10:19:27.928385
# Install build tooling and uv (no recommends to keep layer slim)
2025-Nov-05 10:19:27.928385
RUN apt-get update && apt-get install -y --no-install-recommends \
2025-Nov-05 10:19:27.928385
gcc \
2025-Nov-05 10:19:27.928385
g++ \
2025-Nov-05 10:19:27.928385
curl \
2025-Nov-05 10:19:27.928385
&& rm -rf /var/lib/apt/lists/*
2025-Nov-05 10:19:27.928385
2025-Nov-05 10:19:27.928385
RUN pip install --no-cache-dir uv
2025-Nov-05 10:19:27.928385
2025-Nov-05 10:19:27.928385
# Copy workspace files needed for dependency resolution
2025-Nov-05 10:19:27.928385
COPY pyproject.toml uv.lock ./
2025-Nov-05 10:19:27.928385
COPY packages/shared /app/packages/shared
2025-Nov-05 10:19:27.928385
COPY apps/webhooks /app/apps/webhooks
2025-Nov-05 10:19:27.928385
2025-Nov-05 10:19:27.928385
# Install Python dependencies into .venv
2025-Nov-05 10:19:27.928385
RUN uv sync --frozen --package webhooks
2025-Nov-05 10:19:27.928385
2025-Nov-05 10:19:27.928385
# Clean up build artefacts to keep the copied layer trim
2025-Nov-05 10:19:27.928385
RUN rm -rf /root/.cache /root/.uv-cache
2025-Nov-05 10:19:27.928385
2025-Nov-05 10:19:27.928385
# Final runtime image
2025-Nov-05 10:19:27.928385
FROM python:3.11-slim
2025-Nov-05 10:19:27.928385
2025-Nov-05 10:19:27.928385
WORKDIR /app
2025-Nov-05 10:19:27.928385
2025-Nov-05 10:19:27.928385
# Install curl (for health check) and Infisical CLI with minimal extras
2025-Nov-05 10:19:27.928385
RUN apt-get update && apt-get install -y --no-install-recommends curl \
2025-Nov-05 10:19:27.928385
&& curl -1sLf 'https://artifacts-cli.infisical.com/setup.deb.sh' | bash \
2025-Nov-05 10:19:27.928385
&& apt-get install -y --no-install-recommends infisical \
2025-Nov-05 10:19:27.928385
&& rm -rf /var/lib/apt/lists/*
2025-Nov-05 10:19:27.928385
2025-Nov-05 10:19:27.928385
# Copy in the prebuilt virtualenv and application code
2025-Nov-05 10:19:27.928385
COPY --from=builder /app/.venv /app/.venv
2025-Nov-05 10:19:27.928385
COPY --from=builder /app/packages/shared /app/packages/shared
2025-Nov-05 10:19:27.928385
COPY --from=builder /app/apps/webhooks /app/apps/webhooks
2025-Nov-05 10:19:27.928385
COPY pyproject.toml uv.lock ./
2025-Nov-05 10:19:27.928385
2025-Nov-05 10:19:27.928385
# Expose port 8000
2025-Nov-05 10:19:27.928385
EXPOSE 8000
2025-Nov-05 10:19:27.928385
2025-Nov-05 10:19:27.928385
# Set environment variables
2025-Nov-05 10:19:27.928385
ENV PYTHONPATH=/app
2025-Nov-05 10:19:27.928385
ENV PYTHONUNBUFFERED=1
2025-Nov-05 10:19:27.928385
ENV PATH="/app/.venv/bin:$PATH"
2025-Nov-05 10:19:27.928385
2025-Nov-05 10:19:27.928385
# Health check
2025-Nov-05 10:19:27.928385
HEALTHCHECK --interval=30s --timeout=30s --start-period=5s --retries=3 \
2025-Nov-05 10:19:27.928385
CMD curl -f http://localhost:8000/health || exit 1
2025-Nov-05 10:19:27.928385
2025-Nov-05 10:19:27.928385
# Run the application with Infisical and Gunicorn
2025-Nov-05 10:19:27.928385
# Note: Set INFISICAL_TOKEN and INFISICAL_PROJECT_ID environment variables when running the container
2025-Nov-05 10:19:27.928385
# Example: docker run --env INFISICAL_TOKEN=$INFISICAL_TOKEN --env INFISICAL_PROJECT_ID=your-project-id your-image
2025-Nov-05 10:19:27.928385
CMD ["sh", "-c", "infisical run --projectId $INFISICAL_PROJECT_ID --command 'cd /app/apps/webhooks/webhooks && /app/.venv/bin/gunicorn main:app -c gunicorn.conf.py'"]
2025-Nov-05 10:19:27.942703
----------------------------------------
2025-Nov-05 10:19:27.949099
Building docker image started.
2025-Nov-05 10:19:27.957779
To check the current progress, click on Show Debug Logs.
2025-Nov-05 10:19:28.573409
[CMD]: docker exec uw4kosoc84ooc808ww4080ww bash -c 'cat /artifacts/build.sh'
2025-Nov-05 10:19:28.573409
cd /artifacts/uw4kosoc84ooc808ww4080ww && set -a && source /artifacts/build-time.env && set +a && docker build   --network host -f /artifacts/uw4kosoc84ooc808ww4080ww/apps/webhooks/Dockerfile --build-arg SOURCE_COMMIT --build-arg COOLIFY_URL --build-arg COOLIFY_FQDN --build-arg COOLIFY_BRANCH --build-arg COOLIFY_RESOURCE_UUID --build-arg COOLIFY_CONTAINER_NAME --build-arg INFISICAL_PROJECT_ID --build-arg INFISICAL_TOKEN --build-arg COOLIFY_BUILD_SECRETS_HASH=636980d03d37af4b459b987881db690134f9b90ab8d67d21c1fc8e99e0f83d41 --build-arg 'SOURCE_COMMIT' --build-arg 'COOLIFY_URL' --build-arg 'COOLIFY_FQDN' --build-arg 'COOLIFY_BRANCH' --build-arg 'COOLIFY_RESOURCE_UUID' --build-arg 'COOLIFY_CONTAINER_NAME' --progress plain -t realmsoftwareinc/webhooks-prod:eb20a0771f925bf19364784030b4df74a55441f5 /artifacts/uw4kosoc84ooc808ww4080ww
2025-Nov-05 10:19:29.156678
[CMD]: docker exec uw4kosoc84ooc808ww4080ww bash -c 'bash /artifacts/build.sh'
2025-Nov-05 10:19:29.156678
#0 building with "default" instance using docker driver
2025-Nov-05 10:19:29.156678
2025-Nov-05 10:19:29.156678
#1 [internal] load build definition from Dockerfile
2025-Nov-05 10:19:29.156678
#1 transferring dockerfile: 2.52kB done
2025-Nov-05 10:19:29.156678
#1 DONE 0.0s
2025-Nov-05 10:19:29.156678
2025-Nov-05 10:19:29.156678
#2 [auth] library/python:pull token for registry-1.docker.io
2025-Nov-05 10:19:29.156678
#2 DONE 0.0s
2025-Nov-05 10:19:29.156678
2025-Nov-05 10:19:29.156678
#3 [internal] load metadata for docker.io/library/python:3.11-slim
2025-Nov-05 10:19:29.156678
#3 DONE 0.1s
2025-Nov-05 10:19:29.262918
#4 [builder 1/9] FROM docker.io/library/python:3.11-slim@sha256:fa9b525a0be0c5ae5e6f2209f4be6fdc5a15a36fed0222144d98ac0d08f876d4
2025-Nov-05 10:19:29.262918
#4 DONE 0.0s
2025-Nov-05 10:19:29.262918
2025-Nov-05 10:19:29.262918
#5 [internal] load .dockerignore
2025-Nov-05 10:19:29.262918
#5 transferring context: 2B done
2025-Nov-05 10:19:29.262918
#5 DONE 0.0s
2025-Nov-05 10:19:29.262918
2025-Nov-05 10:19:29.262918
#6 [internal] load build context
2025-Nov-05 10:19:29.262918
#6 transferring context: 1.44MB 0.1s done
2025-Nov-05 10:19:29.510044
#6 DONE 0.1s
2025-Nov-05 10:19:29.510044
2025-Nov-05 10:19:29.510044
#7 [builder 6/9] COPY packages/shared /app/packages/shared
2025-Nov-05 10:19:29.510044
#7 CACHED
2025-Nov-05 10:19:29.510044
2025-Nov-05 10:19:29.510044
#8 [builder 2/9] WORKDIR /app
2025-Nov-05 10:19:29.510044
#8 CACHED
2025-Nov-05 10:19:29.510044
2025-Nov-05 10:19:29.510044
#9 [builder 4/9] RUN pip install --no-cache-dir uv
2025-Nov-05 10:19:29.510044
#9 CACHED
2025-Nov-05 10:19:29.510044
2025-Nov-05 10:19:29.510044
#10 [builder 3/9] RUN apt-get update && apt-get install -y --no-install-recommends   gcc   g++   curl   && rm -rf /var/lib/apt/lists/*
2025-Nov-05 10:19:29.510044
#10 CACHED
2025-Nov-05 10:19:29.510044
2025-Nov-05 10:19:29.510044
#11 [builder 5/9] COPY pyproject.toml uv.lock ./
2025-Nov-05 10:19:29.510044
#11 CACHED
2025-Nov-05 10:19:29.510044
2025-Nov-05 10:19:29.510044
#12 [builder 6/9] COPY packages/shared /app/packages/shared
2025-Nov-05 10:19:29.510044
#12 CACHED
2025-Nov-05 10:19:29.510044
2025-Nov-05 10:19:29.510044
#13 [builder 7/9] COPY apps/webhooks /app/apps/webhooks
2025-Nov-05 10:19:29.510044
#13 DONE 0.1s
2025-Nov-05 10:19:29.510044
2025-Nov-05 10:19:29.510044
#14 [builder 8/9] RUN uv sync --frozen --package webhooks
2025-Nov-05 10:19:29.594969
#14 0.238 Using CPython 3.11.14 interpreter at: /usr/local/bin/python3
2025-Nov-05 10:19:29.594969
#14 0.238 Creating virtual environment at: .venv
2025-Nov-05 10:19:29.749236
#14 0.305    Building webhooks @ file:///app/apps/webhooks
2025-Nov-05 10:19:29.749236
#14 0.309    Building shared @ file:///app/packages/shared
2025-Nov-05 10:19:29.749236
#14 0.392 Downloading aiohttp (1.7MiB)
2025-Nov-05 10:19:29.911749
#14 0.392 Downloading psycopg-binary (4.2MiB)
2025-Nov-05 10:19:29.911749
#14 0.393 Downloading pynacl (1.3MiB)
2025-Nov-05 10:19:29.911749
#14 0.394 Downloading sqlalchemy (3.2MiB)
2025-Nov-05 10:19:29.911749
#14 0.396 Downloading asyncpg (3.0MiB)
2025-Nov-05 10:19:29.911749
#14 0.397 Downloading pydantic-core (2.0MiB)
2025-Nov-05 10:19:29.911749
#14 0.398 Downloading cryptography (4.3MiB)
2025-Nov-05 10:19:29.911749
#14 0.399 Downloading virtualenv (5.7MiB)
2025-Nov-05 10:19:29.911749
#14 0.400 Downloading ruff (12.8MiB)
2025-Nov-05 10:19:29.911749
#14 0.400 Downloading uvloop (3.8MiB)
2025-Nov-05 10:19:29.911749
#14 0.400 Downloading pygments (1.2MiB)
2025-Nov-05 10:19:29.911749
#14 0.400 Downloading grpcio (6.2MiB)
2025-Nov-05 10:19:30.555268
#14 1.199    Building nats-py==2.11.0
2025-Nov-05 10:19:31.339158
#14 1.983  Downloading pydantic-core
2025-Nov-05 10:19:31.491550
#14 2.034    Building tonyg-rfc3339==0.1
2025-Nov-05 10:19:31.491550
#14 2.133    Building nkeys==0.2.1
2025-Nov-05 10:19:31.598344
#14 2.147  Downloading pynacl
2025-Nov-05 10:19:31.602095
#14 2.241  Downloading aiohttp
2025-Nov-05 10:19:31.763871
#14 2.407  Downloading asyncpg
2025-Nov-05 10:19:31.999697
#14 2.643  Downloading psycopg-binary
2025-Nov-05 10:19:32.105619
#14 2.749  Downloading virtualenv
2025-Nov-05 10:19:32.252014
#14 2.755  Downloading ruff
2025-Nov-05 10:19:32.252014
#14 2.818  Downloading uvloop
2025-Nov-05 10:19:32.252014
#14 2.890  Downloading grpcio
2025-Nov-05 10:19:32.489971
#14 2.929  Downloading sqlalchemy
2025-Nov-05 10:19:32.489971
#14 2.943  Downloading cryptography
2025-Nov-05 10:19:32.489971
#14 2.983  Downloading pygments
2025-Nov-05 10:19:35.169371
#14 5.813       Built nkeys==0.2.1
2025-Nov-05 10:19:35.410078
#14 6.052       Built nats-py==2.11.0
2025-Nov-05 10:19:35.576134
#14 6.084       Built shared @ file:///app/packages/shared
2025-Nov-05 10:19:35.576134
#14 6.218       Built webhooks @ file:///app/apps/webhooks
2025-Nov-05 10:19:35.576134
#14 6.218       Built tonyg-rfc3339==0.1
2025-Nov-05 10:19:35.729658
#14 6.221 Prepared 115 packages in 5.94s
2025-Nov-05 10:19:35.793532
#14 6.435 Installed 115 packages in 212ms
2025-Nov-05 10:19:35.793532
#14 6.435  + aiofiles==25.1.0
2025-Nov-05 10:19:35.793532
#14 6.435  + aiogoogle==5.17.0
2025-Nov-05 10:19:35.793532
#14 6.435  + aiohappyeyeballs==2.6.1
2025-Nov-05 10:19:35.793532
#14 6.435  + aiohttp==3.13.0
2025-Nov-05 10:19:35.793532
#14 6.435  + aiosignal==1.4.0
2025-Nov-05 10:19:35.793532
#14 6.435  + annotated-types==0.7.0
2025-Nov-05 10:19:35.793532
#14 6.435  + anyio==4.11.0
2025-Nov-05 10:19:35.793532
#14 6.435  + apscheduler==3.11.0
2025-Nov-05 10:19:35.793532
#14 6.435  + async-timeout==5.0.1
2025-Nov-05 10:19:35.793532
#14 6.435  + asyncpg==0.30.0
2025-Nov-05 10:19:35.793532
#14 6.435  + attrs==25.4.0
2025-Nov-05 10:19:35.793532
#14 6.435  + cachetools==6.2.1
2025-Nov-05 10:19:35.793532
#14 6.435  + certifi==2025.10.5
2025-Nov-05 10:19:35.793532
#14 6.435  + cffi==2.0.0
2025-Nov-05 10:19:35.793532
#14 6.435  + cfgv==3.4.0
2025-Nov-05 10:19:35.793532
#14 6.435  + charset-normalizer==3.4.4
2025-Nov-05 10:19:35.793532
#14 6.435  + click==8.3.0
2025-Nov-05 10:19:35.793532
#14 6.435  + cryptography==46.0.2
2025-Nov-05 10:19:35.793532
#14 6.435  + distlib==0.4.0
2025-Nov-05 10:19:35.793532
#14 6.435  + distro==1.9.0
2025-Nov-05 10:19:35.793532
#14 6.435  + dnspython==2.8.0
2025-Nov-05 10:19:35.793532
#14 6.435  + email-validator==2.3.0
2025-Nov-05 10:19:35.793532
#14 6.435  + fastapi==0.119.0
2025-Nov-05 10:19:35.793532
#14 6.435  + fastapi-cli==0.0.13
2025-Nov-05 10:19:35.793532
#14 6.435  + fastapi-cloud-cli==0.3.1
2025-Nov-05 10:19:35.793532
#14 6.435  + filelock==3.20.0
2025-Nov-05 10:19:35.793532
#14 6.435  + frozenlist==1.8.0
2025-Nov-05 10:19:35.793532
#14 6.435  + google-api-core==2.26.0
2025-Nov-05 10:19:35.793532
#14 6.435  + google-auth==2.41.1
2025-Nov-05 10:19:35.793532
#14 6.435  + google-cloud-kms==3.6.0
2025-Nov-05 10:19:35.793532
#14 6.435  + google-cloud-pubsub==2.31.1
2025-Nov-05 10:19:35.793532
#14 6.435  + googleapis-common-protos==1.70.0
2025-Nov-05 10:19:35.793532
#14 6.435  + greenlet
2025-Nov-05 10:19:35.950300
==3.2.4
2025-Nov-05 10:19:35.950300
#14 6.435  + grpc-google-iam-v1==0.14.2
2025-Nov-05 10:19:35.950300
#14 6.435  + grpcio==1.75.1
2025-Nov-05 10:19:35.950300
#14 6.435  + grpcio-status==1.75.1
2025-Nov-05 10:19:35.950300
#14 6.435  + gunicorn==23.0.0
2025-Nov-05 10:19:35.950300
#14 6.435  + h11==0.16.0
2025-Nov-05 10:19:35.950300
#14 6.435  + httpcore==1.0.9
2025-Nov-05 10:19:35.950300
#14 6.435  + httptools==0.7.1
2025-Nov-05 10:19:35.950300
#14 6.435  + httpx==0.28.1
2025-Nov-05 10:19:35.950300
#14 6.435  + identify==2.6.15
2025-Nov-05 10:19:35.950300
#14 6.435  + idna==3.11
2025-Nov-05 10:19:35.950300
#14 6.439  + importlib-metadata==8.7.0
2025-Nov-05 10:19:35.950300
#14 6.439  + iniconfig==2.1.0
2025-Nov-05 10:19:35.950300
#14 6.439  + jinja2==3.1.6
2025-Nov-05 10:19:35.950300
#14 6.439  + jiter==0.11.0
2025-Nov-05 10:19:35.950300
#14 6.439  + markdown-it-py==4.0.0
2025-Nov-05 10:19:35.950300
#14 6.439  + markupsafe==3.0.3
2025-Nov-05 10:19:35.950300
#14 6.439  + mdurl==0.1.2
2025-Nov-05 10:19:35.950300
#14 6.440  + multidict==6.7.0
2025-Nov-05 10:19:35.950300
#14 6.440  + nats-py==2.11.0
2025-Nov-05 10:19:35.950300
#14 6.440  + nkeys==0.2.1
2025-Nov-05 10:19:35.950300
#14 6.440  + nodeenv==1.9.1
2025-Nov-05 10:19:35.950300
#14 6.440  + openai==1.109.1
2025-Nov-05 10:19:35.950300
#14 6.440  + opentelemetry-api==1.37.0
2025-Nov-05 10:19:35.950300
#14 6.440  + opentelemetry-sdk==1.37.0
2025-Nov-05 10:19:35.950300
#14 6.440  + opentelemetry-semantic-conventions==0.58b0
2025-Nov-05 10:19:35.950300
#14 6.440  + packaging==25.0
2025-Nov-05 10:19:35.950300
#14 6.440  + platformdirs==4.5.0
2025-Nov-05 10:19:35.950300
#14 6.440  + pluggy==1.6.0
2025-Nov-05 10:19:35.950300
#14 6.440  + pre-commit==4.3.0
2025-Nov-05 10:19:35.950300
#14 6.440  + propcache==0.4.1
2025-Nov-05 10:19:35.950300
#14 6.441  + proto-plus==1.26.1
2025-Nov-05 10:19:35.950300
#14 6.441  + protobuf==6.32.1
2025-Nov-05 10:19:35.950300
#14 6.441  + psycopg==3.2.10
2025-Nov-05 10:19:35.950300
#14 6.441  + psycopg-binary==3.2.10
2025-Nov-05 10:19:35.950300
#14 6.441  + psycopg-pool==3.2.6
2025-Nov-05 10:19:35.950300
#14 6.441  + pyasn1==0.6.1
2025-Nov-05 10:19:35.950300
#14 6.441  + pyasn1-modules==0.4.2
2025-Nov-05 10:19:35.950300
#14 6.441  + pybase64==1.4.2
2025-Nov-05 10:19:35.950300
#14 6.441  + pycparser==2.23
2025-Nov-05 10:19:35.950300
#14 6.442  + pydantic==2.12.1
2025-Nov-05 10:19:35.950300
#14 6.442  + pydantic-core==2.41.3
2025-Nov-05 10:19:35.950300
#14 6.442  + pydantic-settings==2.11.0
2025-Nov-05 10:19:35.950300
#14 6.442  + pygments==2.19.2
2025-Nov-05 10:19:35.950300
#14 6.442  + pynacl==1.6.0
2025-Nov-05 10:19:35.950300
#14 6.442  + pytest==8.4.2
2025-Nov-05 10:19:35.950300
#14 6.442  + python-dateutil==2.9.0.post0
2025-Nov-05 10:19:35.950300
#14 6.442  + python-dotenv==1.1.1
2025-Nov-05 10:19:35.950300
#14 6.442  + python-multipart==0.0.20
2025-Nov-05 10:19:35.950300
#14 6.442  + pyyaml==6.0.3
2025-Nov-05 10:19:35.950300
#14 6.442  + redis==6.4.0
2025-Nov-05 10:19:35.950300
#14 6.442  + requests==2.32.5
2025-Nov-05 10:19:35.950300
#14 6.442  + rich==14.2.0
2025-Nov-05 10:19:35.950300
#14 6.442  + rich-toolkit==0.15.1
2025-Nov-05 10:19:35.950300
#14 6.442  + rignore==0.7.0
2025-Nov-05 10:19:35.950300
#14 6.442  + rsa==4.9.1
2025-Nov-05 10:19:35.950300
#14 6.442  + ruff==0.14.0
2025-Nov-05 10:19:35.950300
#14 6.442  + sentry-sdk==2.41.0
2025-Nov-05 10:19:35.950300
#14 6.442  + shared==0.1.0 (from file:///app/packages/shared)
2025-Nov-05 10:19:35.950300
#14 6.443  + shellingham==1.5.4
2025-Nov-05 10:19:35.950300
#14 6.443  + six==1.17.0
2025-Nov-05 10:19:35.950300
#14 6.443  + slack-sdk==3.37.0
2025-Nov-05 10:19:35.950300
#14 6.443  + sniffio==1.3.1
2025-Nov-05 10:19:35.950300
#14 6.443  + sqlalchemy==2.0.44
2025-Nov-05 10:19:35.950300
#14 6.443  + starlette==0.48.0
2025-Nov-05 10:19:35.950300
#14 6.443  + structlog==25.4.0
2025-Nov-05 10:19:35.950300
#14 6.443  + tonyg-rfc3339==0.1
2025-Nov-05 10:19:35.950300
#14 6.443  + tqdm==4.67.1
2025-Nov-05 10:19:35.950300
#14 6.443  + turbopuffer==1.3.1
2025-Nov-05 10:19:35.950300
#14 6.443  + typer==0.19.2
2025-Nov-05 10:19:35.950300
#14 6.443  + typing-extensions==4.15.0
2025-Nov-05 10:19:35.950300
#14 6.443  + typing-inspection==0.4.2
2025-Nov-05 10:19:35.950300
#14 6.443  + tzlocal==5.3.1
2025-Nov-05 10:19:35.950300
#14 6.443  + urllib3==2.5.0
2025-Nov-05 10:19:35.950300
#14 6.443  + uvicorn==0.37.0
2025-Nov-05 10:19:35.950300
#14 6.443  + uvloop==0.21.0
2025-Nov-05 10:19:35.950300
#14 6.443  + virtualenv==20.35.3
2025-Nov-05 10:19:35.950300
#14 6.443  + watchfiles==1.1.0
2025-Nov-05 10:19:35.950300
#14 6.443  + webhooks==0.1.0 (from file:///app/apps/webhooks)
2025-Nov-05 10:19:35.950300
#14 6.443  + websockets==15.0.1
2025-Nov-05 10:19:35.950300
#14 6.443  + yarl==1.22.0
2025-Nov-05 10:19:35.950300
#14 6.444  + zep-cloud==3.5.0
2025-Nov-05 10:19:35.950300
#14 6.444  + zipp==3.23.0
2025-Nov-05 10:19:37.586385
#14 DONE 8.2s
2025-Nov-05 10:19:37.746014
#15 [builder 9/9] RUN rm -rf /root/.cache /root/.uv-cache
2025-Nov-05 10:19:44.770014
#15 DONE 7.2s
2025-Nov-05 10:19:50.259887
#16 [stage-1 3/7] RUN apt-get update && apt-get install -y --no-install-recommends curl   && curl -1sLf 'https://artifacts-cli.infisical.com/setup.deb.sh' | bash   && apt-get install -y --no-install-recommends infisical   && rm -rf /var/lib/apt/lists/*
2025-Nov-05 10:19:50.412590
#16 CACHED
2025-Nov-05 10:19:50.412590
2025-Nov-05 10:19:50.412590
#17 [stage-1 4/7] COPY --from=builder /app/.venv /app/.venv
2025-Nov-05 10:19:54.030020
#17 DONE 3.8s
2025-Nov-05 10:19:54.139159
#18 [stage-1 5/7] COPY --from=builder /app/packages/shared /app/packages/shared
2025-Nov-05 10:19:54.139159
#18 DONE 0.1s
2025-Nov-05 10:19:54.250116
#19 [stage-1 6/7] COPY --from=builder /app/apps/webhooks /app/apps/webhooks
2025-Nov-05 10:19:54.250116
#19 DONE 0.1s
2025-Nov-05 10:19:54.405723
#20 [stage-1 7/7] COPY pyproject.toml uv.lock ./
2025-Nov-05 10:19:54.455673
#20 DONE 0.2s
2025-Nov-05 10:19:54.645685
#21 exporting to image
2025-Nov-05 10:19:54.645685
#21 exporting layers
2025-Nov-05 10:19:56.973033
#21 exporting layers 2.5s done
2025-Nov-05 10:19:57.087921
#21 writing image sha256:d284e7f248539ffb86adae9e166d03cd29425ac74c5bbe5691a0aa3219dfe2eb done
2025-Nov-05 10:19:57.087921
#21 naming to docker.io/realmsoftwareinc/webhooks-prod:eb20a0771f925bf19364784030b4df74a55441f5 done
2025-Nov-05 10:19:57.087921
#21 DONE 2.5s
2025-Nov-05 10:19:57.140786
Building docker image completed.
2025-Nov-05 10:19:57.156075
Creating .env file with runtime variables for build phase.
2025-Nov-05 10:19:57.691747
Oops something is not okay, are you okay? 😢
2025-Nov-05 10:19:57.701848
Deployment failed. Removing the new version of your application.
2025-Nov-05 10:19:57.952779
Gracefully shutting down build container: uw4kosoc84ooc808ww4080ww
2025-Nov-05 10:19:58.199636
[CMD]: docker stop --time=30 uw4kosoc84ooc808ww4080ww
2025-Nov-05 10:19:58.199636
Flag --time has been deprecated, use --timeout instead
2025-Nov-05 10:19:58.603460
uw4kosoc84ooc808ww4080ww
2025-Nov-05 10:19:58.857150
[CMD]: docker rm -f uw4kosoc84ooc808ww4080ww
2025-Nov-05 10:19:58.857150
Error response from daemon: No such container: uw4kosoc84ooc808ww4080ww

Or, another build that failed for no apparent reason, this time on the builder:

2025-Nov-04 21:11:16.151285
Found a suitable build server (Builder).
2025-Nov-04 21:11:16.162132
Starting deployment of dimensionhq/ai:master to Webhooks - 2.
2025-Nov-04 21:11:16.500066
Preparing container with helper image: ghcr.io/coollabsio/coolify-helper:1.0.11
2025-Nov-04 21:11:16.703351
[CMD]: docker stop --time=30 b0o844sgkwkow48oc4gc8cgk
2025-Nov-04 21:11:16.703351
Flag --time has been deprecated, use --timeout instead
2025-Nov-04 21:11:16.705523
Error response from daemon: No such container: b0o844sgkwkow48oc4gc8cgk
2025-Nov-04 21:11:16.890495
[CMD]: docker rm -f b0o844sgkwkow48oc4gc8cgk
2025-Nov-04 21:11:16.890495
Error response from daemon: No such container: b0o844sgkwkow48oc4gc8cgk
2025-Nov-04 21:11:17.130715
[CMD]: docker run -d --name b0o844sgkwkow48oc4gc8cgk  --rm -v /root/.docker/config.json:/root/.docker/config.json:ro -v /var/run/docker.sock:/var/run/docker.sock ghcr.io/coollabsio/coolify-helper:1.0.11
2025-Nov-04 21:11:17.130715
702832ce4fbd3a6abf2976103a23fec9c9d5642de8d7fbbe920ed7645cc74ea9
2025-Nov-04 21:11:18.542726
Oops something is not okay, are you okay? 😢
2025-Nov-04 21:11:18.550276
Deployment failed. Removing the new version of your application.
2025-Nov-04 21:11:18.736879
Gracefully shutting down build container: b0o844sgkwkow48oc4gc8cgk
2025-Nov-04 21:11:18.923578
[CMD]: docker stop --time=30 b0o844sgkwkow48oc4gc8cgk
2025-Nov-04 21:11:18.923578
Flag --time has been deprecated, use --timeout instead
2025-Nov-04 21:11:19.088910
b0o844sgkwkow48oc4gc8cgk
2025-Nov-04 21:11:19.297149
[CMD]: docker rm -f b0o844sgkwkow48oc4gc8cgk
2025-Nov-04 21:11:19.297149
Error response from daemon: No such container: b0o844sgkwkow48oc4gc8cgk

Final example:

2025-Nov-05 02:32:59.297151
Found a suitable build server (Builder).
2025-Nov-05 02:32:59.306247
Starting deployment of dimensionhq/gmail-mcp-server:master to MCP - 2.
2025-Nov-05 02:32:59.721620
Preparing container with helper image: ghcr.io/coollabsio/coolify-helper:1.0.11
2025-Nov-05 02:33:02.138241
----------------------------------------
2025-Nov-05 02:33:02.147524
Importing dimensionhq/gmail-mcp-server:master (commit sha 0062e28d4ef88173db8c0a9e33eca23b8b8c75d6) to /artifacts/i8k04s4scgkcok0wwwswoscs.
2025-Nov-05 02:33:04.576587
Image not found (realmsoftwareinc/gmail-mcp-server-production:0062e28d4ef88173db8c0a9e33eca23b8b8c75d6). Building new image.
2025-Nov-05 02:33:07.614573
----------------------------------------
2025-Nov-05 02:33:07.624301
Building docker image started.
2025-Nov-05 02:33:07.633300
To check the current progress, click on Show Debug Logs.
2025-Nov-05 02:33:51.372999
Building docker image completed.
2025-Nov-05 02:33:52.638262
----------------------------------------
2025-Nov-05 02:33:52.649146
Pushing image to docker registry (realmsoftwareinc/gmail-mcp-server-production:0062e28d4ef88173db8c0a9e33eca23b8b8c75d6).
2025-Nov-05 02:34:14.362435
Tagging and pushing image with latest tag.
2025-Nov-05 02:34:17.144804
Found a suitable build server (Builder).
2025-Nov-05 02:34:17.155960
Starting deployment of dimensionhq/gmail-mcp-server:master to MCP - 2.
2025-Nov-05 02:34:17.526919
Preparing container with helper image: ghcr.io/coollabsio/coolify-helper:1.0.11
2025-Nov-05 02:34:18.291348
----------------------------------------
2025-Nov-05 02:34:18.300366
Rolling update started.
2025-Nov-05 02:34:34.775583
----------------------------------------
2025-Nov-05 02:34:34.792699
Rolling update started.
2025-Nov-05 02:34:38.307388
New container started.
2025-Nov-05 02:34:38.318536
New container started.
2025-Nov-05 02:34:38.328167
Custom healthcheck found in Dockerfile.
2025-Nov-05 02:34:38.337170
Waiting for healthcheck to pass on the new container.
2025-Nov-05 02:34:38.346321
Waiting for the start period (5 seconds) before starting healthcheck.
2025-Nov-05 02:34:43.892990
Attempt 1 of 3 | Healthcheck status: "healthy"
2025-Nov-05 02:34:43.910443
Healthcheck logs:   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2025-Nov-05 02:34:43.910443
Dload  Upload   Total   Spent    Left  Speed
2025-Nov-05 02:34:43.910443
0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100    64  100    64    0     0   4002      0 --:--:-- --:--:-- --:--:--  4266
2025-Nov-05 02:34:43.910443
{"status":"ok","transport":"streamable-http","mode":"stateless"} | Return code: 0
2025-Nov-05 02:34:43.929915
New container is healthy.
2025-Nov-05 02:34:43.945834
Removing old containers.
2025-Nov-05 02:34:46.188552
Rolling update completed.
2025-Nov-05 02:34:46.260320
Gracefully shutting down build container: i8k04s4scgkcok0wwwswoscs
2025-Nov-05 02:35:14.238820
Oops something is not okay, are you okay? 😢
2025-Nov-05 02:35:14.250226
Error: No such object: k00cg8gccos8s888cgo4w804-023024867976
2025-Nov-05 02:35:14.261788
Deployment failed. Removing the new version of your application.
2025-Nov-05 02:35:14.553878
Gracefully shutting down build container: i8k04s4scgkcok0wwwswoscs

Steps to Reproduce

I'm not certain how to reproduce this issue, but I would appreciate guidance on how to look deeper into the logs or inside Coolify to see what actually went wrong.

We setup a Builder instance with 6 vCPU & 16 GB of ram, with a maximum of 2 concurrent deployments. This issue occurs across several different resources and several different Docker images.

Example Repository URL

No response

Coolify Version

v4.0.0-beta.441

Are you using Coolify Cloud?

No (self-hosted)

Operating System and Version (self-hosted)

Ubuntu 24.04

Additional Information

No response

[coderabbitai]

A summary of the changes CodeRabbit can apply:

• Update app/Jobs/ApplicationDeploymentJob.php::failed() to always append full exception details (exception type, code, message, file, line) and the first ~10 stack frames to the deployment logs (not just when getMessage() is non-empty), and adjust cleanup/error handling so cleanup failures don’t hide the real deployment error.

• Add two new RCA documents (docs/bug-analysis/deployment-failures-analysis.md and docs/bug-analysis/deployment-failures-race-condition.md) that document intermittent post-build deployment failures, identify a race condition in container cleanup and inadequate exception logging in ApplicationDeploymentJob::failed(), provide log patterns/evidence, and serve as the basis for implementing improved error handling and container lifecycle fixes.

• ✅ Create PR with these edits
• 📋 Get copyable edits

[algora-pbc]

💎 $50 bounty • Tejas Ravishankar

Steps to solve:

1. Start working: Comment /attempt #7113 with your implementation plan
2. Submit work: Create a pull request including /claim #7113 in the PR body to claim the bounty
3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

❗ Important guidelines:

• To claim a bounty, you need to provide a short demo video of your changes in your pull request
• If anything is unclear, ask for clarification before starting as this will help avoid potential rework
• Low quality AI PRs will not receive review and will be closed
• Do not ask to be assigned unless you've contributed before

Thank you for contributing to coollabsio/coolify!

[algora-pbc]

💎 $80 bounty • Tejas Ravishankar

Steps to solve:

1. Start working: Comment /attempt #7113 with your implementation plan
2. Submit work: Create a pull request including /claim #7113 in the PR body to claim the bounty
3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

❗ Important guidelines:

• To claim a bounty, you need to provide a short demo video of your changes in your pull request
• If anything is unclear, ask for clarification before starting as this will help avoid potential rework
• Low quality AI PRs will not receive review and will be closed
• Do not ask to be assigned unless you've contributed before

Thank you for contributing to coollabsio/coolify!

[ShadowArcanist]

Hey @suptejas

1. How is the failing deployment were triggered ?
2. Are you using a private repository with Github App?
3. Does all deployments fail like this or some deployments only?
4. Which build pack you are using?

We have another issue with same behavior (#6648) and I am currently testing it.

[suptejas]

1. Via Webhooks
2. Yes, private repository with Coolify GitHub app
3. I'm not fully certain yet - but this issue occurs across several different Dockerfiles and projects on Coolify, but redeploying tends to fix it usually (until it breaks again a few deployments after!)
4. Dockerfile

[ShadowArcanist]

1. Via Webhooks

   2. Yes, private repository with Coolify GitHub app

   3. I'm not fully certain yet - but this issue occurs across several different Dockerfiles and projects on Coolify, but redeploying tends to fix it usually (until it breaks again a few deployments after!)

   4. Dockerfile

Thanks, are you using Preview Deployments feature?

[suptejas]

Nope, I don't think so (but I'm not fully certain - happy to check if you could let me know where to look)

[ShadowArcanist]

Nope, I don't think so (but I'm not fully certain - happy to check if you could let me know where to look)

You can check it on advanced settings:

and on deployments page you can see Preview deployments by this tag:

I created a new repo and tried deploying over 25 times and it failed without any error 3 times so far. This issue is extremely hard to reproduce T-T

[ShadowArcanist]

/attempt #7113