Need to bundle specific gRPC binary with Lambda function using serverless webpack

[Matthew Pfluger]

• Serverless-Webpack Version you're using: 5.0.0
• Webpack version you're using: 3.11.0
• Serverless Framework Version you're using: 1.26.1
• Operating System: win-64

I'm trying to use the serverless framework to build a Lambda function that pulls messages from Google PubSub. When using the PubSub NodeJS package, one of the dependencies is gRPC which requires that a binary file be downloaded during the install process. Since I'm on a windows laptop for dev, the gRPC install script downloads node-v48-win32-x64-unknown. However, Lambda runs on Linux, and it expects build node-v48-linux-x64-glibc. Per this comment, it is possible to force npm to rebuild the gRPC package to include the Linxu binary.

My problem is that I can't figure out how to include that in the serverless deployment. When serverless webpack bundles everything up, it includes gRPC with the windows version. When this whole package is successfully deployed to AWS, the Lambda function throws the following error:

Unable to import module 'handler': Error
Expected directory: node-v48-linux-x64-glibc
Found: [node-v48-win32-x64-unknown]
This problem can often be fixed by running "npm rebuild" on the current system
Original error: Cannot find module '/var/task/node_modules/grpc/src/node/extension_binary/node-v48-linux-x64-glibc/grpc_node.node'
at Object. (/var/task/node_modules/grpc/src/grpc_extension.js:53:17)...

So my question is: how can I insert the package rebuild command as part of the serverless-webpack process so the correct version of the gRPC binary gets sent to AWS? It's as if the serverless-webpack is doing a full npm install, and I'd like to insert after that is done to rebuild the gRPC module. I understand that this is a complicated situation and question, and I'm happy to provide any other information.

Serverless.yaml

service: myService

provider:
  name: aws
  runtime: nodejs6.10
  stage: dev
  region: us-east-1

plugins:
  # Use serverless-webpack plugin to transpile ES6/ES7
  - serverless-webpack
  - serverless-offline
  
custom:
  webpack:
    includeModules: true # enable auto-packing of external modules

functions:
  pullData:
    handler: handler.pullData
    events:
      - http:
          path: pullData
          method: post
          cors: true

webpack.config.js

const path = require('path');
const slsw = require("serverless-webpack");
const nodeExternals = require("webpack-node-externals");

module.exports = {
  entry: './handler.js',
  target: 'node',
  externals: [nodeExternals()],
  module: {
    loaders: [{
      test: /\.js$/,
      loaders: ['babel-loader'],
      include: __dirname,
      exclude: /node_modules/
    }]
  },
  output: {
    libraryTarget: 'commonjs',
    path: path.join(__dirname, '.webpack'),
    filename: 'handler.js'
  }
};

[Frank Schmid]

Hi @pflugs30 , thanks for asking.

Yes, that is indeed a special setup, especially that you need to package something that is foreign to the build architecture.
An idea would be to try to add a "prepare" script to your package.json that executes the build of the linux variant. According to https://docs.npmjs.com/misc/scripts prepare scripts are run when doing a npm install without arguments - i.e. it would run it when sls-webpack does its npm install.

[Frank Schmid]

After thinking again, I doubt that the script approach will work at all, because currently the webpack plugin does not allow for custom scripts to be executed while packaging the functions/service.
It will create a package.json that only contains the dependencies but does not add scripts there.

However, with the new packager implementation, it would be possible to implement that as a new feature. I'll create a separate task for that and link this issue.

[Matthew Pfluger]

Thanks for the response, @HyperBrain. If there's anything I can do to help or test, please let me know. It appears that I may be stuck on my project until this is done.

For those who may find there way here via Google, here are things I've tried that have NOT worked:

1. Manually copying the correct compiled binary into the package created by Serverless. Failed due to incorrect SHA checksum (I think).
2. Manually editing the package.json file of the gRPC node module to hard code the version I wanted. Didn't actually copy the desired file into the serverless package.
3. Adding the --target... parameters to the serverless package command call.
4. Looking for a way to customize the webpack config file to force a rebuild of the gRPC package during build.

If anyone has other ideas, I'm all ears! :-) Thanks.

[Frank Schmid]

@pflugs30 Working on a fix right now. I think I'll have a PR ready soon, so that you can test if it solves the problem.

[Frank Schmid]

PR is available. Use "serverless-webpack": "github:serverless-heaven/serverless-webpack#support-scripts" to test if it works.

[Matthew Pfluger]

Thank you very much, @HyperBrain. I am beginning to test it right now, and I will get back to you ASAP. Thank you for your incredibly prompt response.

[Matthew Pfluger]

Closed with the resolution of Issue #344.

[Frank Schmid]

Released with 5.1.0

[Matthew Pfluger]

Further comment for future Googlers (including myself)...

I now am using nvm for Windows to dynamically switch between versions of Node and NPM. I am also now designing for NodeJS v8.10 since AWS Lambdas now support it. As such, when I redeployed my Lambda function, I encountered the following error:

Unable to import module 'handler': Error
Expected directory: node-v57-linux-x64-glibc
Found: [node-v48-linux-x64-glibc, node-v57-win32-x64-unknown]
This problem can often be fixed by running "npm rebuild" on the current system
Original error: Cannot find module '/var/task/node_modules/grpc/src/node/extension_binary/node-v57-linux-x64-glibc/grpc_node.node'
at Object.<anonymous> (/var/task/node_modules/grpc/src/grpc_extension.js:53:17)
...

I realized I was still building the gprc for NodeJS v6.10. As such, I had to change the postinstall command within my package.json to one that would build the expected version, which is node-v57-linux-x64-glibc.

The corrected command in the package.json file is below:
"postinstall": "npm rebuild grpc --target=8.1.0 --target_arch=x64 --target_platform=linux --target_libc=glibc"

Here is the resulting section of the serverless.yaml file:

custom:
  webpack:
    packExternalModulesMaxBuffer: 1000000
    includeModules: true # enable auto-packing of external modules
    packagerOptions:
      scripts:
        - npm rebuild grpc --target=8.1.0 --target_arch=x64 --target_platform=linux --target_libc=glibc