Memory limit exceeded during ghost install process

[danieldinter]

Summary

My Ghost blog is running on a virtual machine provided by an internet hosting service. They limit single processes to 1.5 GB memory usage. Processes exceeding 1.5 GB memory are automatically killed.

When I try to upgrade Ghost from v5.86.2 to the recent version (i.e. v5.100.1) using the Ghost CLI, the process is killed due to memory usage during the step that says
Installing dependencies > [5/5] Building fresh packages...

Steps to Reproduce & Log File

I switched away from trying to upgrade my blog (as it fails) to setting up a new blog on a green field on the same machine. That makes testing easier.

1. Run the install command (notice the "Killed" at the end):

$ ghost install --no-stack --no-setup-linux-user --no-setup-systemd --no-setup-nginx --no-setup-mysql --no-start --no-enable

Love open source? We’re hiring JavaScript Engineers to work on Ghost full-time.
https://careers.ghost.org

✔ Checking system Node.js version - found v18.20.4
✔ Checking current folder permissions
✔ Checking memory availability
✔ Checking free space
✔ Checking for latest Ghost version
✔ Setting up install directory
☲ Downloading and installing Ghost v5.100.1 > Installing dependencies > [5/5] Building fresh packages...Killed

2. Switch to current version directory: $ cd versions/5.100.1/
3. Try to build dependencies manually (also gets killed):

$ yarn install
yarn install v1.22.22
[1/5] Validating package.json...
warning ghost@5.100.1: The engine "cli" appears to be invalid.
[2/5] Resolving packages...
warning Resolution field "moment@2.24.0" is incompatible with requested version "moment@^2.29.1"
warning Resolution field "moment@2.24.0" is incompatible with requested version "moment@^2.29.4"
warning Resolution field "moment@2.24.0" is incompatible with requested version "moment@2.29.1"
warning Resolution field "moment@2.24.0" is incompatible with requested version "moment@^2.27.0"
warning Resolution field "moment@2.24.0" is incompatible with requested version "moment@2.29.4"
warning Resolution field "moment@2.24.0" is incompatible with requested version "moment@2.29.4"
warning Resolution field "moment-timezone@0.5.45" is incompatible with requested version "moment-timezone@0.5.34"
warning Resolution field "jackspeak@2.1.1" is incompatible with requested version "jackspeak@^3.1.2"
warning Resolution field "@tryghost/errors@1.3.5" is incompatible with requested version "@tryghost/errors@1.3.6"
warning Resolution field "@tryghost/logging@2.4.18" is incompatible with requested version "@tryghost/logging@2.4.19"
warning Resolution field "moment@2.24.0" is incompatible with requested version "moment@2.29.1"
warning Resolution field "moment@2.24.0" is incompatible with requested version "moment@2.29.3"
warning Resolution field "@tryghost/errors@1.3.5" is incompatible with requested version "@tryghost/errors@^1.3.6"
warning Resolution field "@tryghost/errors@1.3.5" is incompatible with requested version "@tryghost/errors@^1.3.6"
warning Resolution field "@tryghost/errors@1.3.5" is incompatible with requested version "@tryghost/errors@^1.3.6"
warning Resolution field "moment@2.24.0" is incompatible with requested version "moment@2.27.0"
warning Resolution field "@tryghost/errors@1.3.5" is incompatible with requested version "@tryghost/errors@1.3.1"
warning Resolution field "@tryghost/logging@2.4.18" is incompatible with requested version "@tryghost/logging@2.4.10"
warning Resolution field "moment@2.24.0" is incompatible with requested version "moment@^2.29.1"
[3/5] Fetching packages...
warning lru.min@1.1.1: The engine "bun" appears to be invalid.
warning lru.min@1.1.1: The engine "deno" appears to be invalid.
[4/5] Linking dependencies...
warning " > bookshelf@1.2.0" has incorrect peer dependency "knex@>=0.15.0 <0.22.0".
[5/5] Building fresh packages...
[-/6] ⠄ waiting...
[-/6] ⠄ waiting...
[-/6] ⠄ waiting...
Killed⠄ re2

The messages showing before the kill were:

[5/5] Building fresh packages...
[-/6] ⠂ waiting...
[-/6] ⠂ waiting...
[-/6] ⠂ waiting...
[4/6] ⠄ re2
[6/6] ⠄ @sentry/profiling-node

As it became apparent that memory could be a problem, I tried to set the NODE_OPTIONS parameter --max-old-space-size=1024:

1. I tried setting it in ~/.npmrc: node-options=--max-old-space-size=1024
2. I tried setting an environment variable:

$ export NODE_OPTIONS="--max-old-space-size=1024"
$ ghost install …

However, the parameter is not passed by node to yarn. The builds failed again. Although yarn lists the parameter:

$ yarn config list
yarn config v1.22.22
info yarn config
{
 ...
}
info npm config
{
  'node-options': '--max-old-space-size=1024',
  ...
}
Done in 0.07s.

The only way that works is using cross-env:

$ npm i -g cross-env
$ cross-env NODE_OPTIONS=--max-old-space-size=1024 yarn install

That way, the yarn install command finishes regularly. However, I cannot continue the Ghost CLI install process from this point, as it tells me the directory is non-empty:

$ ghost install --no-stack --no-setup-linux-user --no-setup-systemd --no-setup-nginx --no-setup-mysql --no-start --no-enable
Love open source? We’re hiring JavaScript Engineers to work on Ghost full-time.
https://careers.ghost.org

A SystemError occurred.

Message: Current directory is not empty, Ghost cannot be installed here.

Debug Information:
    OS: CentOS Linux, v7
    Node Version: v18.20.4
    Ghost-CLI Version: 1.26.1
    Environment: production
    Command: 'ghost install --no-stack --no-setup-linux-user --no-setup-systemd --no-setup-nginx --no-setup-mysql --no-start --no-enable'

Try running ghost doctor to check your system for known issues.

You can always refer to https://ghost.org/docs/ghost-cli/ for troubleshooting.

Solutions

If I may I like to propose two solutions:

1. Support cross-env to pass node parameters to yarn.
2. Allow manual build of dependencies, e.g. by adding a switch to Ghost CLI that ignores non-empty directories.

Technical details

This is automatically output by Ghost-CLI if an error occurs, please copy & paste:

• OS: CentOS Linux 7 (Core)
• Node Version: v18.20.4
• Ghost-CLI Version: 1.26.1
• Environment: production
• Command: install

Bug submission checklist

Please fill out this checklist to acknowledge that you followed the requirements to submit a bug report.

• Tried to find help in the forum & docs
• Checked for existing issues
• Attached log file
• Provided technical details incl. operating system

[acburdine]

so - adding cross-env wouldn't fix the issue here I think, but I do think we can pass the NODE_OPTIONS env var through to the underlying yarn install command.

[acburdine]

@danieldinter if you have the ability - would you be able to test the change in #1900? I wasn't able to replicate the behavior you're seeing locally, but the change in that PR should allow you to set NODE_OPTIONS='--max-old-space-size=1024' and have it be respected by yarn install.

[danieldinter]

@acburdine thanks for your effort. I tested the PR and the process still gets killed. Looks like the parameter is not passed. What I did:

$ git clone https://github.com/TryGhost/Ghost-CLI.git
$ cd Ghost-CLI/
$ git fetch origin pull/1900/head:memoryfix
$ git checkout memoryfix
$ yarn install

$ yarn link
yarn link v1.22.22
warning There's already a linked binary called "ghost" in your global Yarn bin. Could not link this package's "ghost" bin entry.
success Registered "ghost-cli".
info You can now run `yarn link "ghost-cli"` in the projects where you want to use this package and it will be used instead.
Done in 0.14s.

$ cd ~/ghost-new/
$ yarn link "ghost-cli"
yarn link v1.22.22
success Using linked package for "ghost-cli".
Done in 0.10s.

$ rm -R content/ node_modules/ versions/ yarn.lock

$ export NODE_OPTIONS="--max-old-space-size=1024"
$ echo $NODE_OPTIONS
--max-old-space-size=1024

$ ghost install --no-stack --no-setup-linux-user --no-setup-systemd --no-setup-nginx --no-setup-mysql --no-start --no-enable

Love open source? We’re hiring JavaScript Engineers to work on Ghost full-time.
https://careers.ghost.org

✔ Checking system Node.js version - found v18.20.4
✔ Checking current folder permissions
✔ Checking memory availability
✔ Checking free space
✔ Checking for latest Ghost version
✔ Setting up install directory
☱ Downloading and installing Ghost v5.100.1 > Installing dependencies > [5/5] Building fresh packages...Killed

[vikaspotluri123]

@dani I see a fetch but I don't see a checkout - can you confirm if you're running on main or this PR?

[danieldinter]

@vikaspotluri123 yes, indeed, I did a checkout:

$ git fetch origin pull/1900/head:memoryfix
$ git checkout memoryfix

Sorry for the confusion. I also see the changes made in Ghost-CLI/lib/tasks/yarn-install.js on the file system.

[danieldinter]

A problem might be that I first yarn linked the package and then ran rm because the first creates a symlink in node_modules/ which has then been deleted. I tested again:

$ cd ~/ghost-new/
$ yarn link "ghost-cli"
yarn link v1.22.22
success Using linked package for "ghost-cli".
Done in 0.12s.
$ ghost install --no-stack --no-setup-linux-user --no-setup-systemd --no-setup-nginx --no-setup-mysql --no-start --no-enable

Love open source? We’re hiring JavaScript Engineers to work on Ghost full-time.
https://careers.ghost.org

A SystemError occurred.

Message: Current directory is not empty, Ghost cannot be installed here.

Debug Information:
    OS: CentOS Linux, v7
    Node Version: v18.20.4
    Ghost-CLI Version: 1.26.1
    Environment: production
    Command: 'ghost install --no-stack --no-setup-linux-user --no-setup-systemd --no-setup-nginx --no-setup-mysql --no-start --no-enable'

Try running ghost doctor to check your system for known issues.

You can always refer to https://ghost.org/docs/ghost-cli/ for troubleshooting.
$ ls -la node_modules/
.. ghost-cli -> ../../.config/yarn/link/ghost-cli

I tried to install it globally:

$ npm uninstall -g ghost-cli
removed 596 packages in 6s
$ yarn global add file:/home/…/Ghost-CLI/

Then ran the ghost install command but again it was killed.

[danieldinter]

I edited the file Ghost-CLI/lib/tasks/yarn-install.js to be:

            // #1898: pass through NODE_OPTIONS so max_old_space_size can be configured
            // in memory-constrained environments
            console.log("process.env.NODE_OPTIONS: " + process.env.NODE_OPTIONS);
            if (process.env.NODE_OPTIONS) {
                env.NODE_OPTIONS = process.env.NODE_OPTIONS;
                console.log("env.NODE_OPTIONS: " + env.NODE_OPTIONS);
            }

Running yarn global add file:/home/…/Ghost-CLI/ and ghost install ... leads to the output

✔ Checking for latest Ghost version
✔ Setting up install directory
☱ Downloading and installing Ghost v5.100.1 > Downloadingprocess.env.NODE_OPTIONS: --max-old-space-size=1024
env.NODE_OPTIONS: --max-old-space-size=1024
☱ Downloading and installing Ghost v5.100.1 > Installing dependencies > [5/5] Building fresh packages...Killed

So the variable is set but it is ignored by yarn. The same behaviour occurs when I manually run yarn install on the command line - it ignores NODE_OPTIONS. Usage of cross-env fixes this.

[acburdine]

hmm - thanks for the additional tests/examples @danieldinter

I'll look more into what cross-env is doing that makes it work correctly with NODE_OPTIONS. If I can figure out what they're doing under the hood to make the node options accepted by yarn then I'll replicate that in the CLI 👍🏻

[acburdine]

so I looked a bit further and we're also using cross-spawn under the hood (which cross-env is also using), though likely a rather old version of it.

I'll try to replicate the conditions you're experiencing to see if I can get a memory killed error, though I'm having a hard time finding a working copy of a centos7 iso that boots in a vm on my mac 😅

[github-actions]

Our bot has automatically marked this issue as stale because there has not been any activity here in some time. The issue will be closed soon if there are no further updates, however we ask that you do not post comments to keep the issue open if you are not actively working on a PR. We keep the issue list minimal so we can keep focus on the most pressing issues. Closed issues can always be reopened if a new contributor is found. Thank you for understanding 🙂

[github-actions]

Our bot has automatically marked this issue as stale because there has not been any activity here in some time. The issue will be closed soon if there are no further updates, however we ask that you do not post comments to keep the issue open if you are not actively working on a PR. We keep the issue list minimal so we can keep focus on the most pressing issues. Closed issues can always be reopened if a new contributor is found. Thank you for understanding 🙂