thanglq1 / express-xml-parser-large-file

Use express framework to parser large xml file

XML parser

• When working with an XML parser, we should always use 2 libs
  • fast-xml-parser (https://www.npmjs.com/package/fast-xml-parser)
  • xml2js (https://www.npmjs.com/package/xml2js)

Large XML file parser

• If we working with a large XML file, we can not use fast-xml-parser or xml2js because it parser to 1 large Javascript object. We need process an XML stream chunk by chunk. Fortunately, nodejs has XmlStream (https://www.npmjs.com/package/xml-stream)

Pg copy

• After parser a large XML file, we have 2 options for inserting data to postgres:
  • Bulk insert to insert data to postgres
  • Use pg-copy-streams(https://www.npmjs.com/package/pg-copy-streams) to faster than bulk insert. pg-copy-streams will copy from CSV or TSV file into postgres so we use csv-write-stream (https://www.npmjs.com/package/csv-write-stream) to create CSV file

Run

docker compose up