Redundant files inflate zip size

[lafiosca]

First, I'd like to thank you for creating this tool. I found it linked from this StackOverflow question. I'm working on several lerna/yarn/typescript monorepos which include Lambda functions. I was looking for simple way to build an individual Lambda with only the dependencies it needed (avoiding the inherent circular symlink problem lerna creates), and your tool is great for that use case.

I was able to use it for one of my projects, but I ran into trouble with a more complex project. It seems when the zip file is created, some dependency paths can be archived multiple times redundantly. I first noticed this when manually unzipping a file and being prompted about paths being overwritten:

$ unzip ../lambda.zip
Archive:  ../lambda.zip
  inflating: build/app.js
  inflating: node_modules/aws-serverless-express/CHANGELOG.md
  inflating: node_modules/express/History.md
replace node_modules/express/History.md? [y]es, [n]o, [A]ll, [N]one, [r]ename: y
  inflating: node_modules/express/History.md
replace node_modules/aws-serverless-express/CHANGELOG.md? [y]es, [n]o, [A]ll, [N]one, [r]ename: y
  inflating: node_modules/aws-serverless-express/CHANGELOG.md
  inflating: node_modules/morgan/HISTORY.md
  inflating: node_modules/cors/CONTRIBUTING.md
  inflating: node_modules/boom/CHANGELOG.md
replace node_modules/express/History.md? [y]es, [n]o, [A]ll, [N]one, [r]ename:

I didn't think too much about it because it still seemed to work. But then I ran into this error when trying to deploy my most complex Lambda:

Unzipped size must be smaller than 262144000 bytes (Service: AWSLambdaInternal; Status Code: 400; Error Code: InvalidParameterValueException

262MB would be pretty large for a Lambda, I thought. I locally rebuilt the file and indeed the zip was 272MB:

-rw-r--r--  1 joe staff 272660200 Sep 11 13:56 lambda.zip

However when I unzipped it and allowed all redundant paths to be overwritten, the resulting directory was only 25MB:

$ unzip -o ../lambda.zip
Archive:  ../lambda.zip
  inflating: build/app.js
  inflating: node_modules/aws-serverless-express/CHANGELOG.md
  inflating: node_modules/express/History.md
  inflating: node_modules/express/History.md
  inflating: node_modules/aws-serverless-express/CHANGELOG.md
  inflating: node_modules/morgan/HISTORY.md
[...]
warning:  deferred symlink (node_modules/sequelize/node_modules/.bin/uuid) failed:
          invalid placeholder file
warning:  deferred symlink (node_modules/sequelize/node_modules/.bin/uuid) failed:
          invalid placeholder file

$ du -hs .
25M	.

Would it be difficult to add some type of de-duping to the tool so that it does not include the same path more than once in the zip file? (For now I'm going to try to have my build process automatically unzip and rezip the contents before deploying.)

As a side note, this should probably be a separate issue: at the end of that output you can see that the zip file included some of the bin symlinks which were pointing to the project's root node_modules/.bin and became invalid. I don't think this will impact my use case.

Thank you again!

[lafiosca]

I will look into the code and see if I can write up a PR for this. I feel like maybe it could be straightforward since it seems to apply to the main logic (unrelated to the zip support). The list of relative module paths generated by the tool includes duplicates. For example, attached is the output of:

../../node_modules/.bin/slice-node-modules -e build/index.js --exclude aws-sdk | sort | uniq -c | sort -n

dupes.txt

[lafiosca]

It appears this only occurs when using -e, not when using -p. I think the existing findNodeModules logic is deduping, but when it gets mapped/concatted in handleEntry, the same packages can be listed multiple times. I hope to submit a PR today.

[lafiosca]

I looked into the code, and rather than try to mess with the trickier part of the logic, I think it would be sufficient to just dedupe the final result array of paths before zipping. I tried that and found that my zip was generated with a much more manageable size of about 6MB.

However, while attempting that simple equality de-dupe, I found that there is another consideration here:

      1 ../../node_modules/send
      1 ../../node_modules/send/node_modules/http-errors
      1 ../../node_modules/send/node_modules/ms

The second and third items listed here are still redundant to the first item because it recursively zips. I think I will have to make the dedupe logic a little more intelligent to work around this case.