`allMarkdownRemark` graphql query in `createPages` intermittently hangs when `excerpt` or `timeToRead` fields are included
tbantle22 opened this issue · comments
Preliminary Checks
- This issue is not a duplicate. Before opening a new issue, please search existing issues: https://github.com/gatsbyjs/gatsby/issues
- This issue is not a question, feature request, RFC, or anything other than a bug report directly related to Gatsby. Please post those things in GitHub Discussions: https://github.com/gatsbyjs/gatsby/discussions
Description
We use gatsby for our blog and have about 650 markdown files. We use the gatsby-transformer-remark
plugin. This graphql query in createPages
often hangs when running gatsby build
or gatsby develop
. It never completes, errors, or gives any useful information about what went wrong. This is not a recent issue for us and has been around since at least Gatsby v4, maybe earlier. When it does complete, createPages
takes about 90s.
const result = await graphql(`
query CreatePagesQuery {
allMarkdownRemark(sort: { frontmatter: { date: DESC } }) {
edges {
node {
id
excerpt(pruneLength: 250)
timeToRead
fields {
slug
}
frontmatter {
title
author
authorHref
date
tags
}
}
}
}
}
`);
These are the gatsby build --verbose
logs where it hangs:
blog % yarn build
verbose set gatsby_log_level: "verbose"
verbose set gatsby_executing_command: "build"
verbose loading local command from: [path redacted]/web/node_modules/gatsby/dist/commands/build.js
verbose running command: build
verbose Running build in "production" environment
success compile gatsby files - 0.586s
success load gatsby config - 0.034s
verbose No adapter was found for the current environment. Skipping adapter initialization.
warn Plugin gatsby-plugin-hubspot is not compatible with your gatsby version 5.13.3 - It requires gatsby@^4.0.0-next
warn Plugin gatsby-plugin-hubspot is not compatible with your gatsby version 5.13.3 - It requires gatsby@^4.0.0-next
success load plugins - 3.531s
success onPreInit - 0.002s
success initialize cache - 0.032s
success copy gatsby files - 0.038s
success Compiling Gatsby Functions - 0.130s
success onPreBootstrap - 0.141s
verbose Creating 11 worker
success createSchemaCustomization - 0.012s
verbose Checking for deleted pages
verbose Deleted 0 pages
verbose Found 0 changed pages
success Checking for changed pages - 0.001s
success source and transform nodes - 4.462s
info Writing GraphQL type definitions to [path redacted]/web/packages/blog/.cache/schema.gql
success building schema - 0.140s
warn unable to find prism language 'yacc' for highlighting. applying generic code block
warn unable to find prism language 'c++' for highlighting. applying generic code block
warn unable to find prism language 'terraform' for highlighting. applying generic code block
warn unable to find prism language 'plan9_x86' for highlighting. applying generic code block
warn unable to find prism language 'make' for highlighting. applying generic code block
⠹ createPages
[ ] 0.000 s 0/1 0% Running gatsby-plugin-sharp.IMAGE_PROCESSING jobs
This step never moves past 0% or errors. The logs look similar for gatsby develop
, but without the last Running gatsby-plugin-sharp.IMAGE_PROCESSING jobs
line.
I recently discovered that removing the excerpt
and timeToRead
fields fixes the hanging problem and reduces createPages
to <1s. If I add a limit of up to 300 on allMarkdownRemark
, it also doesn't hang. This is reproducible using the gatsby cli on my local computer and in github actions.
blog % yarn build
verbose set gatsby_log_level: "verbose"
verbose set gatsby_executing_command: "build"
verbose loading local command from: [path redacted]/web/node_modules/gatsby/dist/commands/build.js
verbose running command: build
verbose Running build in "production" environment
success compile gatsby files - 0.574s
success load gatsby config - 0.025s
verbose No adapter was found for the current environment. Skipping adapter initialization.
warn Plugin gatsby-plugin-hubspot is not compatible with your gatsby version 5.13.3 - It requires gatsby@^4.0.0-next
warn Plugin gatsby-plugin-hubspot is not compatible with your gatsby version 5.13.3 - It requires gatsby@^4.0.0-next
success load plugins - 0.343s
success onPreInit - 0.003s
success initialize cache - 0.031s
success copy gatsby files - 0.038s
success Compiling Gatsby Functions - 0.123s
success onPreBootstrap - 0.131s
verbose Creating 11 worker
success createSchemaCustomization - 0.019s
verbose Checking for deleted pages
verbose Deleted 0 pages
verbose Found 0 changed pages
success Checking for changed pages - 0.001s
success source and transform nodes - 4.685s
info Writing GraphQL type definitions to [path redacted]/web/packages/blog/.cache/schema.gql
success building schema - 0.141s
success createPages - 0.176s
success createPagesStatefully - 0.031s
info Total nodes: 5429, SitePage nodes: 668 (use --verbose for breakdown)
verbose Number of node types: 8. Nodes per type: Directory: 65, File: 2386, ImageSharp: 1627, MarkdownRemark: 634, Site: 1, SiteBuildMetadata: 1, SitePage: 668, SitePlugin: 47
verbose Checking for deleted pages
verbose Deleted 0 pages
verbose Found 668 changed pages
success Checking for changed pages - 0.001s
[...]
Is there a better way to debug why this query is hanging when we include fields that require parsing our markdown files?
It is difficult to reproduce without our repository, which is private. But here are our gatsby config and package.json.
This is our package.json
:
{
"scripts": {
"build": "gatsby build --prefix-paths --verbose",
"develop": "gatsby develop",
},
"dependencies": {
"@dolthub/react-components": "^0.1.1",
"@dolthub/shared-components": "^0.1.0",
"@dolthub/tailwind-config": "^0.1.0",
"@react-icons/all-files": "^4.1.0",
"classnames": "^2.5.1",
"dotenv": "^16.4.1",
"gatsby": "^5.13.3",
"gatsby-plugin-canonical-urls": "^5.13.1",
"gatsby-plugin-feed": "^5.13.1",
"gatsby-plugin-google-gtag": "^5.13.1",
"gatsby-plugin-hubspot": "^2.0.0",
"gatsby-plugin-image": "^3.13.1",
"gatsby-plugin-manifest": "^5.13.1",
"gatsby-plugin-offline": "^6.13.1",
"gatsby-plugin-postcss": "^6.13.1",
"gatsby-plugin-sharp": "^5.13.1",
"gatsby-plugin-twitter": "^5.13.1",
"gatsby-remark-autolink-headers": "^6.13.1",
"gatsby-remark-copy-linked-files": "^6.13.1",
"gatsby-remark-images": "^7.13.1",
"gatsby-remark-prismjs": "^7.13.1",
"gatsby-remark-responsive-iframe": "^6.13.1",
"gatsby-source-filesystem": "^5.13.1",
"gatsby-transformer-remark": "^6.13.1",
"gatsby-transformer-sharp": "^5.13.1",
"graphql": "^16.8.1",
"js-search": "^2.0.1",
"prismjs": "^1.29.0",
"prop-types": "^15.8.1",
"react": "^18.2.0",
"react-dom": "^18.2.0",
"react-share": "^5.1.0"
},
"devDependencies": {
"@babel/core": "^7.23.9",
"@types/js-search": "^1.4.4",
"@types/prismjs": "^1.26.3",
"@typescript-eslint/eslint-plugin": "^6.21.0",
"@typescript-eslint/parser": "^6.18.0",
"babel-loader": "^9.1.3",
"eslint": "^8.49.0",
"eslint-plugin-jest": "^27.6.0",
"eslint-plugin-react": "^7.33.2",
"eslint-plugin-testing-library": "^6.2.0",
"postcss": "^8.4.32",
"prettier": "^3.2.5",
"stylelint": "^16.2.1",
"stylelint-config-recommended": "^14.0.0",
"typescript": "^5.3.3",
"webpack": "^5.89.0",
"yalc": "^1.0.0-pre.53"
}
}
And our gatsby.config
:
import dotenv from "dotenv";
import type { GatsbyConfig } from "gatsby";
dotenv.config({
path: `.env.${process.env.NODE_ENV}`,
});
const siteUrl = "url";
const title = "title";
const config: GatsbyConfig = {
pathPrefix: "/blog",
siteMetadata: {
title,
description: `redacted`,
author: `redacted`,
siteUrl,
},
graphqlTypegen: {
generateOnBuild: true,
typesOutputPath: `gatsby-types.d.ts`,
documentSearchPaths: [
`./src/**/*.{ts,tsx}`,
`../../node_modules/gatsby-*/**/*.js`,
"./gatsby-node.ts",
],
},
plugins: [
`gatsby-plugin-postcss`,
`gatsby-plugin-twitter`,
{
resolve: `gatsby-plugin-feed`,
options: {
query: `
{
site {
siteMetadata {
title
description
siteUrl
site_url: siteUrl
}
}
}
`,
feeds: [
{
serialize: ({ query: { site, allMarkdownRemark } }) =>
allMarkdownRemark.edges.map((edge) => {
return {
...edge.node.frontmatter,
description: edge.node.excerpt,
date: edge.node.frontmatter.date,
url: site.siteMetadata.siteUrl + edge.node.fields.slug,
guid: site.siteMetadata.siteUrl + edge.node.fields.slug,
custom_elements: [{ "content:encoded": edge.node.html }],
};
}),
query: `query FeedQuery {
allMarkdownRemark(sort: {frontmatter: {date: DESC}}) {
edges {
node {
excerpt
html
fields {
slug
}
frontmatter {
title
date
}
}
}
}
}`,
output: "/rss.xml",
title,
},
],
},
},
{
resolve: `gatsby-source-filesystem`,
options: {
name: `images`,
path: `${__dirname}/src/images`,
},
},
{
resolve: `gatsby-source-filesystem`,
options: {
name: `src`,
path: `${__dirname}/src/`,
},
},
{
resolve: `gatsby-transformer-remark`,
options: {
plugins: [
`gatsby-remark-copy-linked-files`,
`gatsby-remark-autolink-headers`,
`gatsby-remark-prismjs`,
{
resolve: `gatsby-remark-images`,
options: {
maxWidth: 856,
},
},
],
},
},
`gatsby-plugin-image`,
`gatsby-transformer-sharp`,
`gatsby-plugin-sharp`,
{
resolve: `gatsby-plugin-canonical-urls`,
options: {
siteUrl,
},
},
{
resolve: "gatsby-plugin-hubspot",
options: {
trackingCode: "redacted",
respectDNT: true,
productionOnly: true,
},
},
{
resolve: `gatsby-plugin-google-gtag`,
options: {
trackingIds: [ "redacted" ],
},
},
{
resolve: `gatsby-plugin-manifest`,
options: {
name: title,
icon: `src/images/favicon.png`,
short_name: title,
start_url: `/`,
background_color: `#182134`,
theme_color: `#182134`,
display: `minimal-ui`,
},
},
],
};
export default config;
And a simplified version of our gatsby-node
that still reproduces the issue:
import { createFilePath } from "gatsby-source-filesystem";
import path from "path";
const blogPostTemplate = path.resolve(`./src/templates/BlogPost.tsx`);
const blogListTemplate = path.resolve("./src/templates/BlogList.tsx");
export const onCreateNode = ({ node, getNode, actions }) => {
const { createNodeField } = actions;
if (node.internal.type === `MarkdownRemark`) {
const slug = createFilePath({ node, getNode, basePath: `pages` });
createNodeField({
node,
name: `slug`,
value: slug,
});
}
};
export const createPages = async ({ graphql, actions, reporter }) => {
const { createPage } = actions;
const result = await graphql(`
query CreatePagesQuery {
allMarkdownRemark(sort: { frontmatter: { date: DESC } }) {
edges {
node {
id
excerpt(pruneLength: 250)
timeToRead
fields {
slug
}
frontmatter {
title
author
authorHref
date
tags
}
}
}
}
}
`);
if (result.errors) {
reporter.panicOnBuild(`There was an error loading posts`, result.errors);
return;
}
if (!result.data) {
reporter.panicOnBuild(`No data found loading posts`);
return;
}
// Create blog post pages
const posts = result.data.allMarkdownRemark.edges;
posts.forEach(({ node }) => {
createPage({
path: node.fields.slug,
component: blogPostTemplate,
context: {
// Data passed to context is available
// in page queries as GraphQL variables.
slug: node.fields.slug,
},
});
});
// Create paginated blog list pages
const postsPerPage = 20;
const numPages = Math.ceil(posts.length / postsPerPage);
Array.from({ length: numPages }).forEach((_, i) => {
const firstPage = i === 0;
const currentPage = i + 1;
createPage({
path: firstPage ? "/" : `/page/${currentPage}`,
component: blogListTemplate,
context: {
limit: postsPerPage,
skip: i * postsPerPage,
numPages,
currentPage,
allBlogs: posts,
},
});
});
};
export const createSchemaCustomization = ({ actions }) => {
const { createTypes } = actions;
createTypes(`
type SitePage implements Node {
context: SitePageContext
}
type SitePageContextFields {
slug: String
}
type SitePageContextFrontmatter {
title: String
author: String
authorHref: String
date: Date
tags: String
}
type SitePageContextBlogNode {
id: String
excerpt: String
timeToRead: Int
fields: SitePageContextFields
frontmatter: SitePageContextFrontmatter
}
type SitePageContextBlogs {
node: SitePageContextBlogNode
}
type SitePageContext {
slug: String
limit: Int
skip: Int
numPages: Int
currentPage: Int
allBlogs: [SitePageContextBlogs]
}
`);
};
Reproduction Link
N/A
Steps to Reproduce
- Put
allMarkdownRemark
query increatePages
withexcerpt
and/ortimeToRead
fields - Run
gatsby build
orgatsby develop
- Hangs at
createPages
...
Expected Result
Build is successful or errors
Actual Result
Build hangs indefinitely
Environment
System:
OS: macOS 14.1
CPU: (12) arm64 Apple M3 Pro
Shell: 5.9 - /bin/zsh
Binaries:
Node: 20.11.1 - /private/var/folders/2f/47svy2p51k12pmjv4phs06k00000gn/T/xfs-804f9c02/node
Yarn: 4.0.2 - /private/var/folders/2f/47svy2p51k12pmjv4phs06k00000gn/T/xfs-804f9c02/yarn
npm: 10.2.4 - ~/.nvm/versions/node/v20.11.1/bin/npm
Languages:
Python: 3.9.2 - /Users/taylor/.pyenv/shims/python
Browsers:
Chrome: 121.0.6167.184
Safari: 17.1
Config Flags
None