What are all the numbers at the start when unpacking a stream?

[Kyle Mathews]

I'm testing msgpackr for a new service and I streamed a million objects into a file and then streamed it back for parsing. When logging out the unpacked data, I noticed that the first ~100 or so data entries were numbers e.g. 123. Only then did my data start coming in.

[Kris Zyp]

I can't be certain from here, but I am guessing that those are record ids, which is part of msgpackr record extension that identify the record structure that is being reused (123 is in the correct range for record ids, but wouldn't expect all the first entries to be record ids). With streams the record extension is enabled by default, so if you are streaming (and haven't set useRecords: false), the data will be encoded with it. And if you then read the data without streaming, using the unpack function alone, then it would be decoded incorrectly. I would certainly recommend using the records though, it is faster and more compact, and you can set the flag for the decoder/unpacker or use a stream to decode.

Anyway, if that's not the issue, and/or you want to show me some of the data, I can take a look.

[Kyle Mathews]

Hmm I was using streams for both encoding and decoding. My decoding code looks like the following. The first 74 were numbers before the first object came in.

const { UnpackrStream } = require(`msgpackr`)

const httpStream = got.stream(url)

receivingStream.on(`data`, (entry) => {
  counter++
  if (typeof entry !== `number`) {
    console.log(counter)
    process.exit()
  }
}) 
await pipeline(httpStream, receivingStream)

This is what the first part looks like of the encoded data (of 85 MBs that I encoded) that I was pulling in to decode:

{"type":"CONTROL_SEQUENCE","payload":{"id":"01G3HXP1B7CA1KFNVHRWNGDCBZ"}}
Ôr@�¤type§payload°CONTROL_SEQUENCEÔrA�¥event®START_SOURCINGÔrB�¤type§payload¬instructions«CREATE_NODEÔrC�§resolve¢id¤name§version­pluginOptions¨nodeAPIs«browserAPIs§ssrAPIs®pluginFilepath«packageJson¦parent¨children¨internalÙ�/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby/dist/internal-plugins/dev-404-pageÙ$d48d1c52-fe48-53cb-8f08-aa4b47dde5a5¬dev-404-page¥1.0.0ÔrD�§plugins��µcreatePagesStatefully��Ù�/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby/dist/internal-plugins/dev-404-pageÔrE�¤name«description§version¤main¦author§license¬dependencies¯devDependencies°peerDependencies³optionalDependecies²bundledDependecies¬dev-404-pageÙIInternal plugin to add a 404 page in development with helpful information¥1.0.0¨index.jsÙ%Kyle Mathews <[e-mail]>£MIT�����À�ÔrF�­contentDigest¤type¥owner§counterÙ c2a9538dd1008b867adc9ccde6e16808ªSitePlugin´internal-data-bridge�ÀB«CREATE_NODECÙ�/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby/dist/internal-plugins/load-babel-configÙ$1fc32581-893a-55e8-8927-bcd667e2b700±load-babel-config¥1.0.0D��®onPreBootstrap��Ù�/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby/dist/internal-plugins/load-babel-configE±load-babel-configÙ2Internal plugin that handles loading Babel configs¥1.0.0¨index.jsÙ%Kyle Mathews <[e-mail]>£MIT�����À�FÙ 2f3397a5fd5889daffcf7e96934da432ªSitePlugin´internal-data-bridge�ÀB«CREATE_NODECÙ�/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby/dist/internal-plugins/internal-data-bridgeÙ$a5079d69-ba80-53dc-82f9-0f440bd5448c´internal-data-bridge¥1.0.0D��«sourceNodes¯createResolvers��Ù�/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby/dist/internal-plugins/internal-data-bridgeE´internal-data-bridgeÙEAn internal Gatsby plugin which creates data nodes from internal data¥1.0.0¨index.jsÙ%Kyle Mathews <[e-mail]>£MIT�����À�FÙ 5934a54cc0be828e9dabc729bb1c8b07ªSitePlugin´internal-data-bridge�ÀB«CREATE_NODECÙ�/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby/dist/internal-plugins/prod-404-500Ù$e4afae10-f83a-55ec-b2ce-9c40c627e8f8¬prod-404-500¥1.0.0D����Ù�/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby/dist/internal-plugins/prod-404-500E¬prod-404-500Ù�Internal plugin to detect various flavors of 404 pages and ensure there's a 404.html path created as well to ensure compatibility with static hosts¥1.0.0¨index.jsÙ%Kyle Mathews <[e-mail]>£MIT�����À�FÙ 2e142cc1584358a2874a2cf4c9ec45a2ªSitePlugin´internal-data-bridge�ÀB«CREATE_NODECÙ�/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby/dist/internal-plugins/webpack-theme-component-shadowingÙ$6f4b2fa9-66b5-5205-a03f-140005a2cfe5Ù!webpack-theme-component-shadowing¥1.0.0D��µonCreateWebpackConfig��Ù�/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby/dist/internal-plugins/webpack-theme-component-shadowingEÙ!webpack-theme-component-shadowingÙ�An internal Gatsby plugin which handles configuring webpack to ensure theme components fall back from the user's site to theme modules.¥1.0.0¨index.jsÙ.Chris Biscardi <[e-mail]>£MIT�����À�FÙ 43c3f1efa6a7d0a4764fd74c5b0d1779ªSitePlugin´internal-data-bridge�ÀB«CREATE_NODECÙ�/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby/dist/internal-plugins/bundle-optimisationsÙ$fe98981b-1127-5585-bf4d-9c17c9408155´bundle-optimisations¥1.0.0D��³onCreateBabelConfigµonCreateWebpackConfig��Ù�/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby/dist/internal-plugins/bundle-optimisationsÔrG�¤name«description§version¤main§license¬dependencies¯devDependencies°peerDependencies³optionalDependecies²bundledDependecies´bundle-optimisationsÙ4Internal plugin to add a bundle optimisation aliases¥1.0.0¨index.js£MIT�����À�FÙ c07e75f8d740486fe6cc016d1dce4851ªSitePlugin´internal-data-bridge�ÀB«CREATE_NODECÙ�/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby/dist/internal-plugins/functionsÙ$6fb2170e-d707-5f0e-8532-ad998d3d02a6©functions¥1.0.0D��®onPreBootstrap±onCreateDevServer��Ù�/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby/dist/internal-plugins/functionsE©functionsÙCAn internal Gatsby plugin which lets one write serverless functions¥1.0.0¨index.jsÙ+Sid Chatterjee <[e-mail]>£MIT�����À�FÙ f0113a4e435d8ed86d568ba529dd672dªSitePlugin´internal-data-bridge�ÀB«CREATE_NODEÔrHÜ��§resolve¢id¤name§versionªmodulePath¦module®subPluginPaths­pluginOptions¨nodeAPIs«browserAPIs§ssrAPIs®pluginFilepath«packageJson¦parent¨children¨internalÙt/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby-plugin-page-creatorÙ$643103cd-05bc-5148-b9c7-2cc48498c50fºgatsby-plugin-page-creator¦4.14.1Ô��Ô��Ô��ÔrI�§plugins¤path©pathCheck�Ù�/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby/dist/internal-plugins/dev-404-page/src/pagesÂ�µcreatePagesStatefullyºsetFieldsOnGraphQLNodeType¬onPluginInit��Ùt/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby-plugin-page-creatorÔrJ�¤name«description§version¤main¨keywords¦author§license¬dependencies¯devDependencies°peerDependencies³optionalDependecies²bundledDependeciesºgatsby-plugin-page-creatorÙ]Gatsby plugin that automatically creates pages from React components in specified directories¦4.14.1¨index.js�¦gatsby­gatsby-pluginÙ%Kyle Mathews <[e-mail]>£MIT�ÔrK�¤name§version®@babel/runtime§^7.15.4K¯@babel/traverse§^7.15.4Kµ@sindresorhus/slugify¦^1.1.2K¨chokidar¦^3.5.2K°fs-exists-cached¦^1.0.0K±gatsby-core-utils§^3.14.0K±gatsby-page-utils§^2.14.1K³gatsby-plugin-utils¦^3.8.0K°gatsby-telemetry§^3.14.0K¦globby§^11.1.0K¦lodash¨^4.17.21�Kª@babel/cli§^7.15.4K«@babel/core§^7.15.5K»babel-preset-gatsby-package§^2.14.0K©cross-env¦^7.0.3�K¦gatsby«^4.0.0-next��À�FÙ 67c2ff784e953514c4e78d00a6729f4aªSitePlugin´internal-data-bridge�ÀB«CREATE_NODEHÙt/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby-plugin-page-creatorÙ$61c74aef-24e7-5aae-85b6-20650a7de20eºgatsby-plugin-page-creator¦4.14.1Ô��Ô��Ô��I�Ù�/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby/dist/internal-plugins/load-babel-config/src/pagesÂ�µcreatePagesStatefullyºsetFieldsOnGraphQLNodeType¬onPluginInit��Ùt/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby-plugin-page-creatorJºgatsby-plugin-page-creatorÙ]Gatsby plugin that automatically creates pages from React components in specified directories¦4.14.1¨index.js�¦gatsby­gatsby-pluginÙ%Kyle Mathews <[e-mail]>£MIT�K®@babel/runtime§^7.15.4K¯@babel/traverse§^7.15.4Kµ@sindresorhus/slugify¦^1.1.2K¨chokidar¦^3.5.2K°fs-exists-cached¦^1.0.0K±gatsby-core-utils§^3.14.0K±gatsby-page-utils§^2.14.1K³gatsby-plugin-utils¦^3.8.0K°gatsby-telemetry§^3.14.0K¦globby§^11.1.0K¦lodash¨^4.17.21�Kª@babel/cli§^7.15.4K«@babel/core§^7.15.5K»babel-preset-gatsby-package§^2.14.0K©cross-env¦^7.0.3�K¦gatsby«^4.0.0-next��À�FÙ 5b1f8b789c9ddb5b739baa5137663da6ªSitePlugin´internal-data-bridge	ÀB«CREATE_NODEHÙt/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby-plugin-page-creatorÙ$2c4b98cd-b613-57f1-b4d3-89201eb527e4ºgatsby-plugin-page-creator¦4.14.1Ô��Ô��Ô��I�Ù�/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby/dist/internal-plugins/internal-data-bridge/src/pagesÂ�µcreatePagesStatefullyºsetFieldsOnGraphQLNodeType¬onPluginInit��Ùt/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby-plugin-page-creatorJºgatsby-plugin-page-creatorÙ]Gatsby plugin that automatically creates pages from React components in specified directories¦4.14.1¨index.js�¦gatsby­gatsby-pluginÙ%Kyle Mathews <[e-mail]>£MIT�K®@babel/runtime§^7.15.4K¯@babel/traverse§^7.15.4Kµ@sindresorhus/slugify¦^1.1.2K¨chokidar¦^3.5.2K°fs-exists-cached¦^1.0.0K±gatsby-core-utils§^3.14.0K±gatsby-page-utils§^2.14.1K³gatsby-plugin-utils¦^3.8.0K°gatsby-telemetry§^3.14.0K¦globby§^11.1.0K¦lodash¨^4.17.21�Kª@babel/cli§^7.15.4K«@babel/core§^7.15.5K»babel-preset-gatsby-package§^2.14.0K©cross-env¦^7.0.3�K¦gatsby«^4.0.0-next��À�FÙ c6a7174f379b1d34835698e6c2f3fb72ªSitePlugin´internal-data-bridge
ÀB«CREATE_NODEHÙt/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby-plugin-page-creatorÙ$e101a480-3db9-53e5-b92d-e8745de304ddºgatsby-plugin-page-creator¦4.14.1Ô��Ô��Ô��I�Ù�/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby/dist/internal-plugins/prod-404-500/src/pagesÂ�µcreatePagesStatefullyºsetFieldsOnGraphQLNodeType¬onPluginInit��Ùt/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby-plugin-page-creatorJºgatsby-plugin-page-creatorÙ]Gatsby plugin that automatically creates pages from React components in specified directories¦4.14.1¨index.js�¦gatsby­gatsby-pluginÙ%Kyle Mathews <[e-mail]>£MIT�K®@babel/runtime§^7.15.4K¯@babel/traverse§^7.15.4Kµ@sindresorhus/slugify¦^1.1.2K¨chokidar¦^3.5.2K°fs-exists-cached¦^1.0.0K±gatsby-core-utils§^3.14.0K±gatsby-page-utils§^2.14.1K³gatsby-plugin-utils¦^3.8.0K°gatsby-telemetry§^3.14.0K¦globby§^11.1.0K¦lodash¨^4.17.21�Kª@babel/cli§^7.15.4K«@babel/core§^7.15.5K»babel-preset-gatsby-package§^2.14.0K©cross-env¦^7.0.3�K¦gatsby«^4.0.0-next��À�FÙ 7d1d6f5eaa168061772652bb4afab45eªSitePlugin´internal-data-bridge�ÀB«CREATE_NODEHÙt/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby-plugin-page-creatorÙ$d8980946-d542-51e9-84b6-8cbbc53f887dºgatsby-plugin-page-creator¦4.14.1Ô��Ô��Ô��I�Ù¢/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby/dist/internal-plugins/webpack-theme-component-shadowing/src/pagesÂ�µcreatePagesStatefullyºsetFieldsOnGraphQLNodeType¬onPluginInit��Ùt/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby-plugin-page-creatorJºgatsby-plugin-page-creatorÙ]Gatsby plugin that automatically creates pages from React components in specified directories¦4.14.1¨index.js�¦gatsby­gatsby-pluginÙ%Kyle Mathews <[e-mail]>£MIT�K®@babel/runtime§^7.15.4K¯@babel/traverse§^7.15.4Kµ@sindresorhus/slugify¦^1.1.2K¨chokidar¦^3.5.2K°fs-exists-cached¦^1.0.0K±gatsby-core-utils§^3.14.0K±gatsby-page-utils§^2.14.1K³gatsby-plugin-utils¦^3.8.0K°gatsby-telemetry§^3.14.0K¦globby§^11.1.0K¦lodash¨^4.17.21�Kª@babel/cli§^7.15.4K«@babel/core§^7.15.5K»babel-preset-gatsby-package§^2.14.0K©cross-env¦^7.0.3�K¦gatsby«^4.0.0-next��À�FÙ da0442630b5aa62a860739f3d11f3af0ªSitePlugin´internal-data-bridgeÀB«CREATE_NODEHÙt/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby-plugin-page-creatorÙ$5f4c7022-d99a-55bf-96a0-268b23ac516aºgatsby-plugin-page-creator¦4.14.1Ô��Ô��Ô��I�Ù�/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby/dist/internal-plugins/bundle-optimisations/src/pagesÂ�µcreatePagesStatefullyºsetFieldsOnGraphQLNodeType¬onPluginInit��Ùt/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby-plugin-page-creatorJºgatsby-plugin-page-creatorÙ]Gatsby plugin that automatically creates pages from React components in specified directories¦4.14.1¨index.js�¦gatsby­gatsby-pluginÙ%Kyle Mathews <[e-mail]>£MIT�K®@babel/runtime§^7.15.4K¯@babel/traverse§^7.15.4Kµ@sindresorhus/slugify¦^1.1.2K¨chokidar¦^3.5.2K°fs-exists-cached¦^1.0.0K±gatsby-core-utils§^3.14.0K±gatsby-page-utils§^2.14.1K³gatsby-plugin-utils¦^3.8.0K°gatsby-telemetry§^3.14.0K¦globby§^11.1.0K¦lodash¨^4.17.21�Kª@babel/cli§^7.15.4K«@babel/core§^7.15.5K»babel-preset-gatsby-package§^2.14.0K©cross-env¦^7.0.3�K¦gatsby«^4.0.0-next��À�FÙ ba379605c1c4c38a8310faa1bc9b1628ªSitePlugin´internal-data-bridge
ÀB«CREATE_NODEHÙt/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby-plugin-page-creatorÙ$96044d0b-42c7-58a3-adc7-047fe3a77045ºgatsby-plugin-page-creator¦4.14.1Ô��Ô��Ô��I�Ù�/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby/dist/internal-plugins/functions/src/pagesÂ�µcreatePagesStatefullyºsetFieldsOnGraphQLNodeType¬onPluginInit��Ùt/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby-plugin-page-creatorJºgatsby-plugin-page-creatorÙ]Gatsby plugin that automatically creates pages from React components in specified directories¦4.14.1¨index.js�¦gatsby­gatsby-pluginÙ%Kyle Mathews <[e-mail]>£MIT�K®@babel/runtime§^7.15.4K¯@babel/traverse§^7.15.4Kµ@sindresorhus/slugify¦^1.1.2K¨chokidar¦^3.5.2K°fs-exists-cached¦^1.0.0K±gatsby-core-utils§^3.14.0K±gatsby-page-utils§^2.14.1K³gatsby-plugin-utils¦^3.8.0K°gatsby-telemetry§^3.14.0K¦globby§^11.1.0K¦lodash¨^4.17.21�Kª@babel/cli§^7.15.4K«@babel/core§^7.15.5K»babel-preset-gatsby-package§^2.14.0K©cross-env¦^7.0.3�K¦gatsby«^4.0.0-next��À�FÙ 2cbae93998bfdebf9fd60fe0e21a791bªSitePlugin´internal-data-bridge�ÀB«CREATE_NODEHÙr/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby-plugin-typescriptÙ$23e0c653-6325-5156-824d-f561772dbbd1¸gatsby-plugin-typescript¦4.14.0Ô��Ô��Ô��ÔrL�§plugins­allExtensions¥isTSX©jsxPragma�ÂÂ¥React�³pluginOptionsSchema´resolvableExtensions³onCreateBabelConfigµonCreateWebpackConfig��Ùr/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby-plugin-typescriptJ¸gatsby-plugin-typescriptÙ!Adds TypeScript support to Gatsby¦4.14.0¨index.js�¦gatsby­gatsby-pluginªtypescriptÙ%Kyle Mathews <[e-mail]>£MIT�K«@babel/core§^7.15.5KÙ2@babel/plugin-proposal-nullish-coalescing-operator§^7.14.5KÙ(@babel/plugin-proposal-numeric-separator§^7.14.5KÙ(@babel/plugin-proposal-optional-chaining§^7.14.5K¸@babel/preset-typescript§^7.15.0K®@babel/runtime§^7.15.4KÙ#babel-plugin-remove-graphql-queries§^4.14.0�Kª@babel/cli§^7.15.4K«@babel/core§^7.15.5K»babel-preset-gatsby-package§^2.14.0K©cross-env¦^7.0.3�K¦gatsby«^4.0.0-next��À�FÙ 471553834ff2a6a0c23dd154689e4161ªSitePlugin´internal-data-bridge�ÀB«CREATE_NODEÔrM�§resolve¢id¤name§version­pluginOptionsºresolvedCompiledGatsbyNode¨nodeAPIs«browserAPIs§ssrAPIs®pluginFilepath«packageJson¦parent¨children¨internalÙL/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31Ù$7374ebf2-d961-52ee-92a2-c25e7cb387a9³default-site-pluginÙ 42304d354db3e19a8dd5c1dfe5bec821D�Ô���«sourceNodes«createPages��ÙL/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31ÔrN�¤name«description§license¬dependencies¯devDependencies°peerDependencies³optionalDependecies²bundledDependeciesºgatsby-starter-hello-worldºGatsby hello world starter£MIT�K¦gatsby¦^4.3.0Kºgatsby-plugin-gatsby-cloud¦^4.3.0K­inspector-api¦^1.4.1Kªlmdb-store§^1.6.13K°node-html-parser¦^5.2.0K«process-top¦^1.2.0K¥react§^17.0.2K©react-dom§^17.0.2�K´@babel/eslint-parser§^7.16.5K¦eslint¦^8.7.0K¶eslint-config-prettier¦^8.3.0K³eslint-config-react¦^1.1.7K¶eslint-plugin-prettier¦^4.0.0K³eslint-plugin-react§^7.28.0K¨md5-file¦^5.0.0K¨prettier¦^2.5.1���À�FÙ 3b54945f80c588bcd1af6a3b8878a172ªSitePlugin´internal-data-bridge�ÀB«CREATE_NODEHÙt/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby-plugin-page-creatorÙ$a260b412-4f9a-5096-b557-ebf151376a99ºgatsby-plugin-page-creator¦4.14.1Ô��Ô��Ô��I�ÙN/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/src/pagesÂ�µcreatePagesStatefullyºsetFieldsOnGraphQLNodeType¬onPluginInit��Ùt/private/tmp/gatsby-cloud/www/sourcerer-80616c43-05ce-4325-af47-4685474b5f31/node_modules/gatsby-plugin-page-creatorJºgatsby-plugin-page-creatorÙ]Gatsby plugin that automatically creates pages from React components in specified directories¦4.14.1¨index.js�¦gatsby­gatsby-pluginÙ%Kyle Mathews <[e-mail]>£MIT�K®@babel/runtime§^7.15.4K¯@babel/traverse§^7.15.4Kµ@sindresorhus/slugify¦^1.1.2K¨chokidar¦^3.5.2K°fs-exists-cached¦^1.0.0K±gatsby-core-utils§^3.14.0K±gatsby-page-utils§^2.14.1K³gatsby-plugin-utils¦^3.8.0K°gatsby-telemetry§^3.14.0K¦globby§^11.1.0K¦lodash¨^4.17.21�Kª@babel/cli§^7.15.4K«@babel/core§^7.15.5K»babel-preset-gatsby-package§^2.14.0K©cross-env¦^7.0.3�K¦gatsby«^4.0.0-next��À�FÙ 984cbbad2fa4d6d5524d961fea0019b4ªSitePlugin´internal-data-bridge�ÀB«CREATE_NODEÔrO�¬siteMetadata¤port¤host¨polyfillªpathPrefixªjsxRuntime­trailingSlash¢id¦parent¨children¨internalÔrP�¥title«description¦author½Gatsby Benchmark Create Pages¹The createPages benchmark©@gatsbyjsÔ��Ô��à§classic¦legacy¤SiteÀ�FÙ b0b26ea3daf54fe5f244fdc4e8f38424¤Site´internal-data-bridge�ÀB«CREATE_NODEÔrQ�©buildTime¢id¦parent¨children¨internal¸2022-05-20T23:43:38.000Z±SiteBuildMetadataÀ�FÙ aa70b9828c6a8b8660fbf4027a4140f5±SiteBuildMetadata´internal-data-bridge�ÀB«CREATE_NODEÔrR�¢idªbuildCount¦parent¨children¨internal«build-count�À�ÔrS�¤type­contentDigest¥owner§counterªBuildCount­build-count-1³default-site-plugin�ÀB«CREATE_NODEÔrT�¢id¦randomªlongString¦parent¨children¨internal¡0Ë?Ê	û�(ßP À�S©Benchmark²0.2034296510553255³default-site-plugin�ÀB«CREATE_NODET¡1Ë?©ðfK£Å� À�S©Benchmark³0.05066222833000644³default-site-plugin�ÀB«CREATE_NODET¡2Ë?¸�(@Ýà@ À�S©Benchmark³0.09578181823405973³default-site-plugin�ÀB«CREATE_NODET¡3Ë?ÍU�.×3� À�S©Benchmark³0.22915854251041168³default-site-plugin�ÀB«CREATE_NODET¡4Ë?ãñ�¯hÁ� À�S©Benchmark²0.6232317377053282³default-site-plugin�ÀB«CREATE_NODET¡5Ë?Ðí?óII, À�S©Benchmark²0.2644805789797313³default-site-plugin�ÀB«CREATE_NODET¡6Ë?ÊATèº%h À�S©Benchmark³0.20511876453046374³default-site-plugin�ÀB«CREATE_NODET¡7Ë?ï¶�¢¡ & À�S©Benchmark²0.9909809280770829³default-site-plugin�ÀB«CREATE_NODET¡8Ë?âvh� X4 À�S©Benchmark²0.5769539300843065³default-site-plugin�ÀB«CREATE_NODET¡9Ë?äÅ�r
êD À�S©Benchmark±0.649050925006414³default-site-plugin�ÀB«CREATE_NODET¢10Ë?À=��à�è À�S©Benchmark³0.12687815254412205³default-site-plugin�ÀB«CREATE_NODET¢11Ë?â+ö�Ã4¬ À�S©Benchmark²0.5678665745305316³default-site-plugin ÀB«CREATE_NODET¢12Ë?ÞÁSe�Û� À�S©Benchmark²0.4805496680232233³default-site-plugin!ÀB«CREATE_NODET¢13Ë?à°L[ö;� À�S©Benchmark²0.5215207859163109³default-site-plugin"ÀB«CREATE_NODET¢14Ë?Ö�3ÓgÏ° À�S©Benchmark³0.34486098904676066³default-site-plugin#ÀB«CREATE_NODET¢15Ë?ź�ö¿ð� À�S©Benchmark±0.169757123454364³default-site-plugin$ÀB«CREATE_NODET¢16Ë?ç�è2Æ¡( À�S©Benchmark²0.7193490020481805³default-site-plugin%ÀB«CREATE_NODET¢17Ë?í�S�ü( À�S©Benchmark²0.9087310414481851³default-site-plugin&ÀB«CREATE_NODET¢18Ë?äëð}ö�¼ À�S©Benchmark²0.6538011989804526³default-site-plugin'ÀB«CREATE_NODET¢19Ë?§�Ý�ÙI� À�S©Benchmark²0.0450963100708055³default-site-plugin(ÀB«CREATE_NODET¢20Ë?Ùú�e�gL À�S©Benchmark³0.40592131542290555³default-site-plugin)ÀB«CREATE_NODET¢21Ë?Ï5YR��� À�S©Benchmark³0.24381557947276322³default-site-plugin*ÀB«CREATE_NODET¢22Ë?ç�¢xA~� À�S©Benchmark²0.7207805966135667³default-site-plugin+ÀB«CREATE_NODET¢23Ë?áQ5���� À�S©Benchmark²0.5411629734558838³default-site-plugin,ÀB«CREATE_NODET¢24Ë?�Mi¢ÿÄ� À�S©Benchmark³0.02470936829917436³default-site-plugin-ÀB«CREATE_NODET¢25Ë?ío½Yé×´ À�S©Benchmark²0.9198900943872474³default-site-plugin.ÀB«CREATE_NODET¢26Ë?ÇóOÂÈY� À�S©Benchmark²0.1871127797209733³default-site-plugin/ÀB«CREATE_NODET¢27Ë?à Ð�geà À�S©Benchmark²0.5040054645461929³default-site-plugin0ÀB«CREATE_NODET¢28Ë?Õ4ð}�ÄÔ À�S©Benchmark³0.33135616509305277³default-site-plugin1ÀB«CREATE_NODET¢29Ë?��m`db� À�S©Benchmark´0.014944891456751153³default-site-plugin2ÀB«CREATE_NODET¢30Ë?ç"ê{�õ� À�S©Benchmark²0.7230122006000386³default-site-plugin3ÀB«CREATE_NODET¢31Ë?àø2�A}РÀ�S©Benchmark²0.5302975461926795³default-site-plugin4ÀB«CREATE_NODET¢32Ë?Ì»��¡êà À�S©Benchmark³0.22447502968483146³default-site-plugin5ÀB«CREATE_NODET¢33Ë?ê1![Ïüþ À�S©Benchmark±0.818497351952743³default-site-plugin6ÀB«CREATE_NODET¢34Ë?é:�HJ·¶ À�S©Benchmark²0.7883998310041906³default-site-plugin7ÀB«CREATE_NODET¢35Ë?ï±�yugÒ À�S©Benchmark±0.990357148383959³default-site-plugin8ÀB«CREATE_NODET¢36Ë?âÕ�¨½�¶ À�S©Benchmark²0.5885146423032015³default-site-plugin9ÀB«CREATE_NODET¢37Ë?¹3pä'W° À�S©Benchmark³0.09844117708357092³default-site-plugin:ÀB«CREATE_NODET¢38Ë?ß¼ÇIÊÂì À�S©Benchmark²0.4958971233964309³default-site-plugin;ÀB«CREATE_NODET¢39Ë?¶þ4-�DÀ À�S©Benchmark³0.08981634235921643³default-site-plugin<ÀB«CREATE_NODET¢40Ë?ÔF¿��ñ¼ À�S©Benchmark³0.31681814468154434³default-site-plugin=ÀB«CREATE_NODET¢41Ë?êwÃ%�¼$ À�S©Benchmark²0.8271194195166385³default-site-plugin>ÀB«CREATE_NODET¢42Ë?ßén~��à À�S©Benchmark³0.49862253526430145³default-site-plugin?ÀB«CREATE_NODET¢43Ë?í�ëÐ!*� À�S©Benchmark²0.9243067803411709³default-site-pluginÌ@ÀB«CREATE_NODET¢44Ë?ÞJAu	Õ� À�S©Benchmark²0.4732822077706589³default-site-pluginÌAÀB«CREATE_NODET¢45Ë?å%zÏ'qú À�S©Benchmark²0.6608251615503484³default-site-pluginÌBÀB«CREATE_NODET¢46Ë?è½��{�ê À�S©Benchmark²0.7731444937789409³default-site-pluginÌCÀB«CREATE_NODET¢47Ë?á®°¸lªp À�S©Benchmark²0.5525745012321988³default-site-pluginÌDÀB«CREATE_NODET¢48Ë?¤3Å
�<� À�S©Benchmark³0.03945747134377342³default-site-pluginÌEÀB«CREATE_NODET¢49Ë?í�p[ïé� À�S©Benchmark²0.9094774051330188³default-site-pluginÌFÀB«CREATE_NODET¢50Ë?Ü�2H�l� À�S©Benchmark²0.4461179454789894³default-site-pluginÌGÀB«CREATE_NODET¢51Ë?ÝM�wè�¤ À�S©Benchmark³0.45785557470119387³default-site-pluginÌHÀB«CREATE_NODET¢52Ë?°:L.q�� À�S©Benchmark³0.06338955053163375³default-site-pluginÌIÀB«CREATE_NODET¢53Ë?ØÂË��ª¼ À�S©Benchmark²0.3868893413006551³default-site-pluginÌJÀB«CREATE_NODET¢54Ë?¹��ÔV¶à À�S©Benchmark²0.0980530279545877³default-site-pluginÌKÀB«CREATE_NODET¢55Ë?Ò�Z��=Ø À�S©Benchmark³0.28963329652557723³default-site-pluginÌLÀB«CREATE_NODET¢56Ë?¦<�`��  À�S©Benchmark³0.04342667391986432³default-site-pluginÌMÀB«CREATE_NODET¢57Ë?çj�>�úî À�S©Benchmark²0.7317601416433186³default-site-pluginÌNÀB«CREATE_NODET¢58Ë?îèâ�Ã{ð À�S©Benchmark²0.9659283492036348³default-site-pluginÌOÀB«CREATE_NODET¢59Ë?ÑSy�¯:¬ À�S©Benchmark²0.2707198901888812³default-site-pluginÌPÀB«CREATE_NODET¢60Ë?Óyûÿö¢8 À�S©Benchmark²0.3043203353541091³default-site-pluginÌQÀB«CREATE_NODET¢61Ë?ê�íw5Y� À�S©Benchmark²0.8300692871369368³default-site-pluginÌRÀB«CREATE_NODET¢62Ë?åÃ}
àÈ4 À�S©Benchmark±0.680113341432167³default-site-pluginÌSÀB«CREATE_NODET¢63Ë?´ÓvÝo�  À�S©Benchmark³0.08135168939730919³default-site-pluginÌTÀB«CREATE_NODET¢64Ë?Ì�¸�߲РÀ�S©Benchmark³0.21910768881152487³default-site-pluginÌUÀB«CREATE_NODET¢65Ë?¿�èg�ó  À�S©Benchmark³0.12115337875725496³default-site-pluginÌVÀB«CREATE_NODET¢66Ë?ä_¨ázsx À�S©Benchmark²0.6366772083161854³default-site-pluginÌWÀB«CREATE_NODET¢67Ë?Þ�0âË�� À�S©Benchmark³0.47034856937206504³default-site-pluginÌXÀB«CREATE_NODET¢68Ë?Ø�îÝ{)P À�S©Benchmark³0.37652179364217364³default-site-pluginÌYÀB«CREATE_NODET¢69Ë?ïAKx�KÈ À�S©Benchmark²0.9767205575502595³default-site-pluginÌZÀB«CREATE_NODET¢70Ë?Óû
jÂÌ� À�S©Benchmark²0.3121980230891386³default-site-pluginÌ[ÀB«CREATE_NODET¢71Ë?éÉÜÞê4� À�S©Benchmark²0.8058914521985974³default-site-pluginÌ\ÀB«CREATE_NODET¢72Ë?ê±µU�ÎD À�S©Benchmark²0.8341929112488704³default-site-pluginÌ]ÀB«CREATE_NODET¢73Ë?çC�¨wß2 À�S©Benchmark±0.726993397750425³default-site-pluginÌ^ÀB«CREATE_NODET¢74Ë?ì�ü"Ñ�F À�S©Benchmark²0.8939190559435779³default-site-pluginÌ_ÀB«CREATE_NODET¢75Ë?½80�ç�� À�S©Benchmark³0.11413863485315923³default-site-pluginÌ`ÀB«CREATE_NODET¢76Ë?ÊÇ8ÌaÜø À�S©Benchmark²0.2092047689400045³default-site-pluginÌaÀB«CREATE_NODET¢77Ë?�Ʋ°è1� À�S©Benchmark´0.014539142631314617³default-site-pluginÌbÀB«CREATE_NODET¢78Ë?ÙÇ�(�X8 À�S©Benchmark²0.4027717491457028³default-site-pluginÌcÀB«CREATE_NODET¢79Ë?çGÖ·¦�@ À�S©Benchmark²0.7275193774155255³default-site-pluginÌdÀB«CREATE_NODET¢80Ë?ï�°�wÔø À�S©Benchmark²0.9882433572189759³default-site-pluginÌeÀB«CREATE_NODET¢81Ë?ÂÌ�cZ�` À�S©Benchmark³0.14686612940947708³default-site-pluginÌfÀB«CREATE_NODET¢82Ë?áp8Ô¹J¦ À�S©Benchmark²0.5449489741098048³default-site-pluginÌgÀB«CREATE_NODET¢83Ë?äÒøB#ÕÖ À�S©Benchmark²0.6507531444355503³default-site-pluginÌhÀB«CREATE_NODET¢84Ë?ÅÛI� �� À�S©Benchmark²0.1707545549734284³default-site-pluginÌiÀB«CREATE_NODET¢85Ë?áG&AÆ	" À�S©Benchmark²0.5399352345123456³default-site-pluginÌjÀB«CREATE_NODET¢86Ë?ÞÆbíK=¨ À�S©Benchmark³0.48085854695542674³default-site-pluginÌkÀB«CREATE_NODET¢87Ë?áNÔ^Õ�ð À�S©Benchmark²0.5408727504953088³default-site-pluginÌlÀB«CREATE_NODET¢88Ë?å}ý� .X À�S©Benchmark²0.6716294710071891³default-site-pluginÌmÀB«CREATE_NODET¢89Ë?·��m>�� À�S©Benchmark³0.08993652026275911³default-site-pluginÌnÀB«CREATE_NODET¢90Ë?×¼¿q
¯´ À�S©Benchmark³0.37089525261284995³default-site-pluginÌoÀB«CREATE_NODET¢91Ë?ßÝÔ�AÏ� À�S©Benchmark²0.4979144588644371³default-site-pluginÌpÀB«CREATE_NODET¢92Ë?¹W�q¤n� À�S©Benchmark³0.09899189733953762³default-site-pluginÌqÀB«CREATE_NODET¢93Ë?Á��D<Û� À�S©Benchmark²0.1332427581016553³default-site-pluginÌrÀB«CREATE_NODET¢94Ë?ÚÄV(-�ø À�S©Benchmark²0.4182334320412333³default-site-pluginÌsÀB«CREATE_NODET¢95Ë?ËíV~×âH À�S©Benchmark³0.21818047708053911³default-site-pluginÌtÀB«CREATE_NODET¢96Ë?îçÓêýQX À�S©Benchmark±0.965799292529975³default-site-pluginÌuÀB«CREATE_NODET¢97Ë?ìÛ»É�cj À�S©Benchmark²0.9018229424137705³default-site-pluginÌvÀB«CREATE_NODET¢98Ë? z�gy�� À�S©Benchmark³0.03218163264369789³default-site-pluginÌwÀB«CREATE_NODET¢99Ë?½��æÖQРÀ�S©Benchmark³0.11541079888986094³default-site-pluginÌxÀB«CREATE_NODET£100Ë?ènzõ±W( À�S©Benchmark±0.763486366146684³default-site-pluginÌyÀB«CREATE_NODET£101Ë?îk÷�iãÖ À�S©Benchmark²0.9506793234751238³default-site-pluginÌzÀB«CREATE_NODET£102Ë?Ô�äJé�° À�S©Benchmark²0.3140192729533977³default-site-pluginÌ{ÀB«CREATE_NODET£103Ë?¸ �Ê�C` À�S©Benchmark³0.09619193024378836³default-site-pluginÌ|ÀB«CREATE_NODET£104Ë?æéoÿâ0� À�S©Benchmark²0.7159957883573218³default-site-pluginÌ}ÀB«CREATE_NODET£105Ë?ÞRuëQ{  À�S©Benchmark±0.473782996943493³default-site-pluginÌ~ÀB«CREATE_NODET£106Ë?Á}Iìø\� À�S©Benchmark³0.13663600989159308³default-site-pluginÌ�ÀB«CREATE_NODET£107Ë?í"�,¨ô0 À�S©Benchmark²0.9104114410653263³default-site-pluginÌ�ÀB«CREATE_NODET£108Ë?æ�)�ÆÁâ À�S©Benchmark±0.706806718886636³default-site-pluginÌ�ÀB«CREATE_NODET£109Ë?�«�çJÿ� À�S©Benchmark´0.013999029274941455³default-site-pluginÌ�ÀB«CREATE_NODET£110Ë?ß�Vd°�0 À�S©Benchmark³0.49226913292882823³default-site-pluginÌ�ÀB«CREATE_NODET£111Ë?ì�²çc`0 À�S©Benchmark²0.8754515189457432³default-site-pluginÌ�ÀB«CREATE_NODET£112Ë?è��ק+È À�S©Benchmark²0.7520169460270774³default-site-pluginÌ�ÀB«CREATE_NODET£113Ë?ã»O¯Põ  À�S©Benchmark²0.6166151451249426³default-site-pluginÌ�ÀB«CREATE_NODET£114Ë?�zÕï��� À�S©Benchmark´0.014394446714788511³default-site-pluginÌ�ÀB«CREATE_NODET£115Ë?ÕWså�S´ À�S©Benchmark²0.3334626905428266³default-site-pluginÌ�ÀB«CREATE_NODET£116Ë?ËÏëamh� À�S©Benchmark³0.21728269822307666³default-site-pluginÌ�ÀB«CREATE_NODET£117Ë?í[�b��Ê À�S©Benchmark²0.9173595346987551³default-site-pluginÌ�ÀB«CREATE_NODET£118Ë?íÖ�üðÊ À�S©Benchmark²0.9324403051064334³default-site-pluginÌ�ÀB«CREATE_NODET£119Ë?ÝZ¤¸�uÌ À�S©Benchmark²0.4586574365510756³default-site-pluginÌ�ÀB«CREATE_NODET£120Ë?å.�«�Ä À�S©Benchmark²0.6618738410436937³default-site-pluginÌ�ÀB«CREATE_NODET£121Ë?é

[Kyle Mathews]

Oh I should mention too that my test had a million objects in in.

[Kris Zyp]

It looks like your MessagePack document is prepended with a JSON string. And the JSON string is 73 bytes long (74 with the line feed). And yes, decoding JSON with a MessagePack decoder results in numbers (all the bytes in the latin character range are numbers in MessagePack). However, the data after the JSON header does appear to be valid MessagePack with records (albeit a bit garbled by the UTF-8 decoding of copy and pasting).

If you didn't intend to prepend the MessagePack with JSON, (for separate parsing with a JSON parser), than I think the issue would be on the encoding side. I don't think there is anything in msgpackr that would output JSON (there is no reference to JSON in the code), but perhaps something is putting that when setting up the stream to the file?

[Kyle Mathews]

🤦 that's it. I'd been testing using ljson (line delimited json) and had forgotten when switching it over to msgpackr that I still had one bit of code that was directly inserting the first line there into the stream.

You'll be happy to know that while encoding was more-or-less similar speeds (it's hard to tell with all the other overhead that was going on), parsing is something like 30x faster with msgpackr vs. https://www.npmjs.com/package/stream-json 😆

Downloading and parsing the million objects took all of 1.5s!

[Kris Zyp]

Glad to hear it is fast!