ENHANCEMENT: Allow -gt option to dramatically improve performance
GoogleCodeExporter opened this issue · comments
Google Code Exporter commented
This is not an issue, but an enhancement request.
Using "import QGIS/GDAL layer" from qspatialite you can't specify the ogr/gdal
option of -gt which should improve performance.
The backend, ogr2ogr uses -gt 200 which is very low for importing data to
spatialite. Increasing that value increases performance dramatically.
The only way to use that option is to put it into the "Data Source" creation
options in the layer "Save As..." window. But this way you can only do one
layer at a time, whereas in qspatialite I can do them in batch, but because I
can't specify a custom -gt (like -gt 1024) it is slow to import shapefiles from
qspatialite.
This is a much needed improvement to this utility.
Original issue reported on code.google.com by Sault....@gmail.com
on 8 Jan 2013 at 10:45
Google Code Exporter commented
See "Performance Hints" section at the bottom of
http://www.gdal.org/ogr/drv_sqlite.html
From the site:
"SQLite is a Transactional DBMS; while many INSERT statements are executed in
close sequence, BEGIN TRANSACTION and COMMIT TRANSACTION statements have to be
invoked appropriately in order to get optimal performance. The default OGR
behavior is to COMMIT a transaction every 200 inserted rows. This value is
surely too low for SQLite; and closing too much frequently the current
transaction causes severe performance degradation. The -gt argument allows to
explicitly set the number of rows for each transaction. Explicitly defining -gt
1024 usually ensures a noticeable performance boost; defining an even bigger
-gt 65536 ensures optimal performance while populating some table containing
many hundredth thousand or million rows."
Original comment by Sault....@gmail.com
on 8 Jan 2013 at 10:47
Google Code Exporter commented
Thanks for considering this, it will be a highly appreciated improvement =)
Original comment by Sault....@gmail.com
on 8 Feb 2013 at 8:21
Google Code Exporter commented
QSpatialite doesn't use OGR to load data.
QSpatiaLite load data in QGIS memory, and use QGIS API to read and insert data
into the DB. THis allows QSpatiaLite to deal with many problems such as special
characters, very long column names, and optimising MAPINFO compatibility.
But still, a quick import option, with OGR, should be a good option. I'll be
working on this.
Original comment by romain.r...@gmail.com
on 8 Feb 2013 at 7:49
- Changed state: Accepted