import function too slow
cmedinga opened this issue · comments
Details
I'm importing a csv file of 66.927 records x 34 columns. If I do it with select import function, it takes 1513 seconds. If I do it via Tools > Import menu, it takes less than a second.
Steps to reproduce
Create a text file of that size (due to sensitive information, I cannot share it) separated by semicolons. Try to import to a new table with this query:
SELECT import('c:\path\to\file.csv', 'CSV', 'my_table', 'utf-8', 'CsvImport.FirstRowAsColumns=true CsvImport.Separator=1
CsvImport.NullValues=true');
With the import window, use this params:
Operating system
Windows 10
SQLiteStudio version
3.4.4
I'm not 100% sure, but if I remember correctly, the options should be specified in a 1-per-line manner, like this:
SELECT import('c:\path\to\file.csv', 'CSV', 'my_table', 'utf-8', 'CsvImport.FirstRowAsColumns=true
CsvImport.Separator=1
CsvImport.NullValues=true');
Let me know if that makes any difference. If yes, I will add this information to wiki https://github.com/pawelsalawa/sqlitestudio/wiki/CsvImport
Actually, this 1-per-line option applies to all import plugins, not just CSV and it is already mentioned in documentation for the import()
function: https://github.com/pawelsalawa/sqlitestudio/wiki/User_Manual#built-in-sql-functions
Yes, I tried it, but it makes no difference. It takes more than 20 minutes to load the file.