Skip to content

chore: dev from dump improvement #1131

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
chedieck wants to merge 6 commits into
base: master
Choose a base branch
from
Open

chore: dev from dump improvement #1131

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
c5493f8
fix: don't sed dump
chedieck Jun 1, 2026
ea376a0
chore: improve testing from dump
chedieck Jun 1, 2026
424bfd1
chore: improve testing from dump 2
chedieck Jun 1, 2026
e4d2616
fix: dev-from-dump setup (named volume, prisma deploy, pv progress)
chedieck Jun 2, 2026
c05b008
feat: SKIP_CACHE_REBUILD to prevent I/O saturation with prod data
chedieck Jun 2, 2026
85a2ba2
chore: ignore .forever
chedieck Jun 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .env.from-dump
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,2 +1,4 @@
IGNORE_SEEDING=true
DONT_EXECUTE_TRIGGERS=true
FROM_DUMP=true
SKIP_CACHE_REBUILD=true
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,3 +39,4 @@ prisma-local/seeds/productionTxs.csv
paybutton-config.json

dump.sql*
.forever
Comment thread
Fabcien marked this conversation as resolved.
12 changes: 12 additions & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,18 @@ dev-from-dump:
$(touch_local_env)
docker compose -f docker-compose-from-dump.yml --env-file .env --env-file .env.local --env-file .env.from-dump up --build -d

stop-dev-from-dump:
docker compose -f docker-compose-from-dump.yml --env-file .env --env-file .env.local --env-file .env.from-dump down

reset-dev-from-dump:
make stop-dev-from-dump && make dev-from-dump

reset-dev-from-dump-keep-db:
docker compose -f docker-compose-from-dump.yml --env-file .env --env-file .env.local --env-file .env.from-dump up --build -d --force-recreate --no-deps paybutton

nuke-dev-from-dump:
docker compose -f docker-compose-from-dump.yml --env-file .env --env-file .env.local --env-file .env.from-dump down -v

logs-dev:
docker logs -f paybutton-dev

Expand Down
4 changes: 4 additions & 0 deletions docker-compose-from-dump.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ services:
- ./scripts/db/init-supertokens-db.sql:/home/mysql/raw_entrypoint/3-init-supertokens-db.sql
- ./scripts/db/prisma-shadow-db-fix.sql:/home/mysql/raw_entrypoint/4-prisma-shadow-db-fix.sql
- ./dump.sql:/home/mysql/raw_entrypoint/5-dump.sql
- paybutton-dump-db-data:/var/lib/mysql

users-service:
container_name: paybutton-users-service
Expand Down Expand Up @@ -69,3 +70,6 @@ services:
- ./redis:/data/redis:z
- ./scripts:/data/scripts
command: sh -c "sed -i 's/\\r//g' ./scripts/redis-start.sh && sh ./scripts/redis-start.sh"

volumes:
paybutton-dump-db-data:
19 changes: 12 additions & 7 deletions redis/paymentCache.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,9 +31,11 @@ export async function * getUserUncachedAddresses (userId: string): AsyncGenerato
}

export const getPaymentList = async (userId: string): Promise<Payment[]> => {
const uncachedAddressStream = getUserUncachedAddresses(userId)
for await (const address of uncachedAddressStream) {
void await CacheSet.addressCreation(address)
if (process.env.SKIP_CACHE_REBUILD === undefined) {
const uncachedAddressStream = getUserUncachedAddresses(userId)
for await (const address of uncachedAddressStream) {
void await CacheSet.addressCreation(address)
}
}
return await getCachedPaymentsForUser(userId)
Comment on lines 33 to 40
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot Jun 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | 🏗️ Heavy lift

Don't turn first-run cache misses into empty payment history.

With SKIP_CACHE_REBUILD=true in .env.from-dump and a fresh Redis container in docker-compose-from-dump.yml, the first dev-from-dump boot has no payment cache yet. These guards now skip both the eager rebuild and the per-address lazy init, so getPaymentList / getPaymentStream can return zero payments for real production data instead of falling back to MariaDB.

Keep the bulk warmup disabled if needed, but uncached addresses still need a lazy DB-backed path.

Also applies to: 309-323

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@redis/paymentCache.ts` around lines 33 - 40, The current guard around cache
warmup in getPaymentList prevents the per-address lazy initialization when
SKIP_CACHE_REBUILD is set, causing a first-run cache miss to return an empty
payment list; change the logic so SKIP_CACHE_REBUILD only disables the
bulk/warmup path but still iterates getUserUncachedAddresses and calls
CacheSet.addressCreation for each address (so the DB-backed lazy path runs),
updating the same behavior in the corresponding getPaymentStream area (the code
referencing getUserUncachedAddresses, CacheSet.addressCreation,
getCachedPaymentsForUser/getPaymentStream) to ensure uncached addresses fallback
to MariaDB rather than producing an empty result.

}
Expand Down Expand Up @@ -304,6 +306,7 @@ export const clearPaymentCacheForAddress = async (addressString: string): Promis
}

export const initPaymentCache = async (address: Address): Promise<boolean> => {
if (process.env.SKIP_CACHE_REBUILD !== undefined) return false
const cachedKeys = await getCachedWeekKeysForAddress(address.address)
if (cachedKeys.length === 0) {
await CacheSet.addressCreation(address)
Expand All @@ -313,10 +316,12 @@ export const initPaymentCache = async (address: Address): Promise<boolean> => {
}

export async function * getPaymentStream (userId: string): AsyncGenerator<Payment> {
const uncachedAddressStream = getUserUncachedAddresses(userId)
for await (const address of uncachedAddressStream) {
console.log('[CACHE]: Creating cache for address', address.address)
await CacheSet.addressCreation(address)
if (process.env.SKIP_CACHE_REBUILD === undefined) {
const uncachedAddressStream = getUserUncachedAddresses(userId)
for await (const address of uncachedAddressStream) {
console.log('[CACHE]: Creating cache for address', address.address)
await CacheSet.addressCreation(address)
}
}
const userButtonIds: string[] = (await fetchPaybuttonArrayByUserId(userId))
.map(p => p.id)
Expand Down
18 changes: 16 additions & 2 deletions scripts/db-entrypoint.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,25 @@ sed_vars () {
cd /home/mysql/raw_entrypoint || exit

for file in *.sql; do
sed_vars "$file"
if [[ "$file" == *dump* ]]; then
echo "Linking dump file: $file"
ln -s /home/mysql/raw_entrypoint/"$file" /home/mysql/entrypoint/"$file"
else
echo "Copying file $file after seding vars"
sed_vars "$file"
fi
done

cd ../entrypoint/ || exit

for file in *.sql; do
mariadb -u root -p"$MYSQL_ROOT_PASSWORD" < "$file"
if [[ "$file" == *dump* ]]; then
filesize=$(stat -Lc%s "$file" 2>/dev/null || stat -Lf%z "$file" 2>/dev/null)
echo "Importing $file ($(numfmt --to=iec $filesize)) ..."
pv "$file" | mariadb -u root -p"$MYSQL_ROOT_PASSWORD"
else
echo "Importing $file ..."
mariadb -u root -p"$MYSQL_ROOT_PASSWORD" < "$file"
fi
echo "Done importing $file"
done
2 changes: 1 addition & 1 deletion scripts/db/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ FROM mariadb:latest
RUN mkhomedir_helper mysql

RUN apt-get update || echo && \
Copy link
Copy Markdown
Collaborator

@Fabcien Fabcien Jun 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I understand the use of || echo here, if the goal is to avoid failures then || true seems more appropriated

apt-get install -y gettext || echo
apt-get install -y gettext pv || echo
Comment on lines 5 to +6
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot Jun 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Fail the image build when gettext/pv installation fails.

The || echo clauses swallow apt-get failures, so this image can build without envsubst or pv and then fail much later inside scripts/db-entrypoint.sh. That makes the dump workflow flaky and much harder to diagnose.

Suggested fix 
-RUN apt-get update || echo && \
-      apt-get install -y gettext pv || echo
+RUN apt-get update && \
+      apt-get install -y --no-install-recommends gettext pv && \
+      rm -rf /var/lib/apt/lists/*
🧰 Tools
🪛 Trivy (0.69.3)

[error] 5-6: 'apt-get' missing '--no-install-recommends'

'--no-install-recommends' flag is missed: 'apt-get update || echo && apt-get install -y gettext pv || echo'

Rule: DS-0029

Learn more

(IaC/Dockerfile)

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/db/Dockerfile` around lines 5 - 6, The Dockerfile's RUN line uses
"apt-get update || echo" and "apt-get install -y gettext pv || echo" which hides
failures and lets the image build without envsubst/pv causing runtime errors in
scripts/db-entrypoint.sh; remove the "|| echo" fallbacks and make the RUN step
fail on error (so apt-get update and apt-get install return non-zero on
failure), ensuring missing packages like gettext (envsubst) and pv cause the
build to fail early and visibly.


RUN mkdir -p /home/mysql/raw_entrypoint && \
mkdir -p /home/mysql/entrypoint
Expand Down
14 changes: 8 additions & 6 deletions scripts/paybutton-server-start.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,17 +17,19 @@ logtime=$(date +%Y-%m-%d@%H:%M)
[ -e logs/jobs.log ] && mv logs/jobs.log logs/history/jobs_"$logtime".log
[ -e logs/ws-server.log ] && mv logs/ws-server.log logs/history/ws-server_"$logtime".log

if [ "$ENVIRONMENT" = "production" ]; then
if [ "$FROM_DUMP" = "true" ] || [ "$ENVIRONMENT" = "production" ]; then
yarn prisma migrate deploy || exit 1
yarn prisma generate || exit 1
pm2 start yarn --time --interpreter ash --name jobs --output logs/jobs.log --error logs/jobs.log -- initJobs || exit 1
pm2 start yarn --time --interpreter ash --name WSServer --output logs/ws-server.log --error logs/ws-server.log -- initWSServer || exit 1
pm2 start yarn --time --interpreter ash --name next --output logs/next.log --error logs/next.log -- prod || exit 1
else
yarn prisma migrate dev || exit 1
yarn prisma db seed || exit 1
pm2 start yarn --time --interpreter ash --name jobs --output logs/jobs.log --error logs/jobs.log -- initJobs || exit 1
pm2 start yarn --time --interpreter ash --name WSServer --output logs/ws-server.log --error logs/ws-server.log -- initWSServer || exit 1
fi

pm2 start yarn --time --interpreter ash --name jobs --output logs/jobs.log --error logs/jobs.log -- initJobs || exit 1
pm2 start yarn --time --interpreter ash --name WSServer --output logs/ws-server.log --error logs/ws-server.log -- initWSServer || exit 1
if [ "$ENVIRONMENT" = "production" ]; then
pm2 start yarn --time --interpreter ash --name next --output logs/next.log --error logs/next.log -- prod || exit 1
else
pm2 start yarn --time --interpreter ash --name next --output logs/next.log --error logs/next.log -- dev || exit 1
fi
pm2 logs next
Loading