Excessive Build Times with gatsby-source-shopify During 'source shopify variants' Stage

Question

Excessive Build Times with gatsby-source-shopify During 'source shopify variants' Stage

mh4rly opened this issue 8 months ago · comments

Preliminary Checks

This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/gatsbyjs/gatsby/issues
This issue is not a question, feature request, RFC, or anything other than a bug report directly related to Gatsby. Please post those things in GitHub Discussions: https://github.com/gatsbyjs/gatsby/discussions

Description

We are experiencing a significant issue with the gatsby-source-shopify plugin in our Gatsby application, specifically during the 'source shopify variants' stage of the build process. This problem is causing excessively long build times, which is severely impacting our development and deployment workflows.
When running our Gatsby build, the 'source shopify products' stage completes relatively quickly (around 34 to 47 seconds for 253 nodes). However, the process slows down during the 'source shopify variants' stage.
In a typical successful build, this stage takes approximately 1777.314 seconds (nearly 30 minutes) to process 645 nodes. This duration is excessively long compared to the rest of the build stages.
Attempts to Resolve:
We initially faced this issue with the following versions: Gatsby 4.25.7, gatsby-source-shopify 7.14.0, and gatsby-source-graphql 4.9.0.
To resolve this, we updated to the latest versions: Gatsby 5.13.1, gatsby-source-shopify 8.13.0, and gatsby-source-graphql 5.13.0. However, the issue persists even after updating.
Configuration Details:
Our configuration for the gatsby-source-shopify plugin in gatsby-config.js is as follows:

{
  resolve: "gatsby-source-shopify",
  options: {
    password: process.env.SHOPIFY_ADMIN_PASSWORD,
    storeUrl: process.env.GATSBY_SHOPIFY_STORE_URL,
    shopifyConnections: ["collections"],
    prioritize: true,
    downloadImages: false,
  },
}

Environment:

OS: macOS 14.2.1
CPU: (8) arm64 Apple M2
Node: 20.11.0
Yarn: 1.22.19
npm: 10.2.4
Browsers: Chrome 120.0.6099.234, Safari 17.2.1
This problem was not observed in earlier versions or builds until recently.
Relevant Netlify Logs
11:17:05 PM: success source shopify products - 44.167s - 253 nodes
11:46:43 PM: success source shopify variants - 1777.314s - 645 nodes
11:46:48 PM: success source shopify collections - 5.471s - 112 nodes
11:46:50 PM: success Checking for changed pages - 0.000s
11:46:50 PM: success source and transform nodes - 1829.728s
11:46:51 PM: success building schema - 0.993s

Reproduction Link

https://github.com/gatsbyjs/gatsby/tree/master/packages/gatsby-source-shopify

Steps to Reproduce

Install the gatsby-source-shopify plugin with the provided configuration.
Attempt to run the development server or build for production.

Expected Result

The server is expected to run quickly, completing the build process within a few minutes.

Actual Result

The build process takes an excessive amount of time, especially during the 'source shopify variants' stage.

Environment

System:OS: macOS 14.2.1
CPU: (8) arm64 Apple M2
Shell: 5.9 - /bin/zsh
Binaries:Node: 20.11.0 - ~/.nvm/versions/node/v20.11.0/bin/node
Yarn: 1.22.19 - ~/.nvm/versions/node/v16.13.2/bin/yarn
npm: 10.2.4 - ~/.nvm/versions/node/v20.11.0/bin/npm
Languages:Python: 3.10.12 - /Users/zach/.pyenv/shims/python
Browsers:Chrome: 120.0.6099.234
Safari: 17.2.1
npmPackages^5.13.1 => 5.13.1
    gatsby-plugin-algolia: ^1.0.3 => 1.0.3
    gatsby-plugin-eslint: ^4.0.2 => 4.0.4
    gatsby-plugin-google-gtag: ^5.13.0 => 5.13.0
    gatsby-plugin-google-tagmanager: ^5.13.0 => 5.13.0
    gatsby-plugin-image: ^3.13.0 => 3.13.0
    gatsby-plugin-manifest: ^5.13.0 => 5.13.0
    gatsby-plugin-netlify: ^5.1.1 => 5.1.1
    gatsby-plugin-nullish-coalescing-operator: ^1.0.1 => 1.0.1
    gatsby-plugin-offline: ^6.13.0 => 6.13.0
    gatsby-plugin-postcss: ^6.13.0 => 6.13.0
    gatsby-plugin-react-helmet: ^6.13.0 => 6.13.0
    gatsby-plugin-react-svg: ^3.1.0 => 3.3.0
    gatsby-plugin-sass: ^6.13.0 => 6.13.0
    gatsby-plugin-sharp: ^5.13.0 => 5.13.0
    gatsby-plugin-sitemap: ^6.13.0 => 6.13.0
    gatsby-source-filesystem: ^5.13.0 => 5.13.0
    gatsby-source-graphql: ^5.13.0 => 5.13.0
    gatsby-source-shopify: ^8.13.0 => 8.13.0
    gatsby-source-storyblok: ^7.0.4 => 7.0.4
    gatsby-transformer-remark: ^6.13.0 => 6.13.0
    gatsby-transformer-sharp: ^5.13.0 => 5.13.0

Config Flags

baseRelDir: true
Additional Information:
Netlify Build Logs
Successful Build (Slow):

6:17:05 AM EST: success source shopify products - 44.167s - 253 nodes
6:46:43 AM EST: success source shopify variants - 1777.314s - 645 nodes
6:46:48 AM EST: success source shopify collections - 5.471s - 112 nodes
Failed Build:
6:34:16 AM EST: success source shopify products - 34.202s - 253 nodes
6:51:24 AM EST: Failed during stage "building site": Command did not finish within the time limit