multiple errors on running mantis2trac.py with Mantis 1.0.1 and Trac 0.10

Question

multiple errors on running mantis2trac.py with Mantis 1.0.1 and Trac 0.10

Aeon opened this issue 7 years ago · comments

On 26-Jan-2007, at 6:47pm, pottingerhj wrote (Trac issue 1150):

a minor annoyance: an extra slash is required at the end of the trac env path, which isn't standard practice; the example given of /path/to/trac/env is wrong.
after fixing 1 above, running the script gets me all the way to step 2:

2. import components...
sql: SELECT category, user_id as owner FROM mantis_project_category_table WHERE project_id '3' or project_id '5' or project_id '7' or project_id '9' or project_id = '10'
Adding user hardy to sessions table
Traceback (most recent call last):
  File "./mantis2trac.py", line 926, in ?
    main()
  File "./mantis2trac.py", line 923, in main
    convert(MANTIS_DB, MANTIS_HOST, MANTIS_USER, MANTIS_PASSWORD, TRAC_ENV, TRAC_CLEAN)
  File "./mantis2trac.py", line 578, in convert
    component['owner'] = trac.getLoginName(mysql_cur, component['owner'])
  File "./mantis2trac.py", line 467, in getLoginName
    print 'could not insert %s into sessions table: sql error %s ' % loginName, self.db().error()

After some debugging, I discover that:

AttributeError: 'pysqlite2.dbapi2.Connection' object has no attribute 'error'

So I drop the fancy error reporting and just see what error Python will throw:

pysqlite2.dbapi2.OperationalError: near "IGNORE": syntax error

According the SQLite page, that is indeed incorrect syntax. Lines 462 and 470 should read "INSERT OR IGNORE" not "INSERT IGNORE".

So, after correcting these lines, I receive this error:

2. import components...
sql: SELECT category, user_id as owner FROM mantis_project_category_table WHERE project_id '3' or project_id '5' or project_id '7' or project_id '9' or project_id = '10'
Adding user hardy to sessions table
Traceback (most recent call last):
  File "./mantis2trac.py", line 926, in ?
    main()
  File "./mantis2trac.py", line 923, in main
    convert(MANTIS_DB, MANTIS_HOST, MANTIS_USER, MANTIS_PASSWORD, TRAC_ENV, TRAC_CLEAN)
  File "./mantis2trac.py", line 578, in convert
    component['owner'] = trac.getLoginName(mysql_cur, component['owner'])
  File "./mantis2trac.py", line 474, in getLoginName
    (result[0]['username'].encode('utf-8'), '1', 'name', result[0]['realname'].encode('utf-8'),
  File "/usr/lib/python2.3/site-packages/trac/db/util.py", line 47, in execute
    return self.cursor.execute(sql_escape_percent(sql), args)
  File "/usr/lib/python2.3/site-packages/trac/db/sqlite_backend.py", line 56, in execute
    args or [])
  File "/usr/lib/python2.3/site-packages/trac/db/sqlite_backend.py", line 48, in _rollback_on_error
    return function(self, *args, **kwargs)
pysqlite2.dbapi2.OperationalError: near ",": syntax error

And, unfortunately, my debug abilities have hit a wall. I'm not sure what might be the problem, so I'm throwing this out there, to see if anyone might be able to help.

Anton Stroganov · Answer 1 · Wed Mar 29 2017 02:58:17 GMT+0800 (China Standard Time)

On 6-Feb-2007, at 9:34pm, anonymous wrote (Trac issue 1150#comment:1):

Had the exact same problems as you did...they seem to have been introduced in the latest version [1724] . I got the previous version [1721] from the following link and it worked great: http://trac-hacks.org/browser/mantisimportscript/mantis2trac.py?rev=1721&format=raw

Anton Stroganov · Answer 2 · Wed Mar 29 2017 02:58:18 GMT+0800 (China Standard Time)

On 6-Feb-2007, at 10:42pm, @Aeon wrote (Trac issue 1150#comment:2):

It's complaining because sqlite doesn't support mutliple insert statements like sql does. Try the new revision (r1927) (I split up the session attribute insert into separate queries and removed the 'IGNORE' part (not sure how to cleanly make alternate sql syntax depending on whether the trac is running a MySQL or SQLite backend.

Anton Stroganov · Answer 3 · Wed Mar 29 2017 02:58:18 GMT+0800 (China Standard Time)

On 7-Feb-2007, at 9:17pm, anonymous wrote (Trac issue 1150#comment:3):

Tried the new revision (r1927). No luck, here's the error:

2. import components...
sql: SELECT category, user_id as owner FROM mantis_project_category_table WHERE project_id '3' or project_id '5' or project_id '7' or project_id '9' or project_id = '10'
Adding user hardy to sessions table
Traceback (most recent call last):
  File "./mantis2trac.py", line 936, in ?
    main()
  File "./mantis2trac.py", line 933, in main
    convert(MANTIS_DB, MANTIS_HOST, MANTIS_USER, MANTIS_PASSWORD, TRAC_ENV, TRAC_CLEAN)
  File "./mantis2trac.py", line 588, in convert
    component['owner'] = trac.getLoginName(mysql_cur, component['owner'])
  File "./mantis2trac.py", line 467, in getLoginName
    print 'could not insert %s into sessions table: sql error %s ' % loginName, self.db().error()
TypeError: not enough arguments for format string

Anton Stroganov · Answer 4 · Wed Mar 29 2017 02:58:19 GMT+0800 (China Standard Time)

On 7-Feb-2007, at 10:00pm, @Aeon wrote (Trac issue 1150#comment:4):

(In [1929]) MantisImportScript: second try to fix #1150

Anton Stroganov · Answer 5 · Wed Mar 29 2017 02:58:20 GMT+0800 (China Standard Time)

On 7-Feb-2007, at 10:02pm, @Aeon wrote (Trac issue 1150#comment:5):

Try now. I didn't actually test my commit last night (yeah, I know, bad bad...) Tested the fix now, should be working correctly. Thanks!

Anton Stroganov · Answer 6 · Wed Mar 29 2017 02:58:20 GMT+0800 (China Standard Time)

On 8-Feb-2007, at 5:41pm, anonymous wrote (Trac issue 1150#comment:6):

still no luck with the new revision (r1929):

2. import components...
sql: SELECT category, user_id as owner FROM mantis_project_category_table WHERE project_id '3' or project_id '5' or project_id '7' or project_id '9' or project_id = '10'
Adding user foo to sessions table
Adding user bar to sessions table
failed executing sql:
INSERT INTO session
                            (sid, authenticated, last_visit)
                        VALUES  ('bar', '1', '1161118853')
Traceback (most recent call last):
  File "./mantis2trac_new.py", line 943, in ?
    main()
  File "./mantis2trac_new.py", line 940, in main
    convert(MANTIS_DB, MANTIS_HOST, MANTIS_USER, MANTIS_PASSWORD, TRAC_ENV, TRAC_CLEAN)
  File "./mantis2trac_new.py", line 595, in convert
    component['owner'] = trac.getLoginName(mysql_cur, component['owner'])
  File "./mantis2trac_new.py", line 476, in getLoginName
    print 'could not insert %s into sessions table: sql error %s ' % (loginName, self.db().error())
  File "/usr/lib/python2.3/site-packages/trac/db/util.py", line 69, in __getattr__
    return getattr(self.cnx, name)
  File "/usr/lib/python2.3/site-packages/trac/db/util.py", line 69, in __getattr__
    return getattr(self.cnx, name)
AttributeError: 'pysqlite2.dbapi2.Connection' object has no attribute 'error'

Anton Stroganov · Answer 7 · Wed Mar 29 2017 02:58:21 GMT+0800 (China Standard Time)

On 8-Feb-2007, at 7:37pm, @Aeon wrote (Trac issue 1150#comment:7):

What the... looks like the sqlite wrapper doesn't provide us with an error() function... grrr.

How large is your mantis db dump? Can you attach it to this ticket so I can test it?

Alternatively, can you go into sqlite3 prompt and run the sql query that it's erroring on, to see if you can get a more intelligible error statement?

INSERT INTO session
 (sid, authenticated, last_visit)
VALUES  ('bar', '1', '1161118853');

Anton Stroganov · Answer 8 · Wed Mar 29 2017 02:58:21 GMT+0800 (China Standard Time)

On 8-Feb-2007, at 7:38pm, @Aeon wrote (Trac issue 1150#comment:8):

Er, that should be

#!sql
INSERT INTO session
    (sid, authenticated, last_visit)
VALUES  ('bar', '1', '1161118853')

Anton Stroganov · Answer 9 · Wed Mar 29 2017 02:58:21 GMT+0800 (China Standard Time)

On 8-Feb-2007, at 10:21pm, anonymous wrote (Trac issue 1150#comment:9):

I'm not comfortable posting my db dump.

However, I did follow the advice of comment 1, and reverted back to r1721. Needed to also turn off the TIME_ADJUSTMENT_HACK and set the defaultencoding to latin-1, but the script ran fine. Looking forward to seeing what I've got in my trac db now.

I'd be willing to share a dump file of my mantis MySQL db via some other method, if you'd like to tinker. I just don't want it available to the world.

Anton Stroganov · Answer 10 · Wed Mar 29 2017 02:58:22 GMT+0800 (China Standard Time)

On 8-Feb-2007, at 10:29pm, @Aeon wrote (Trac issue 1150#comment:10):

No problem. It's really a minor part of the script that's causing all this trouble; more for convenience than anything else. If you like, email me your mantis dump at aeontech at gmail dot com, I'll poke it with a stick, see what I can turn up. The script is running fine against trac on a mysql backend, not sure why it's having so much trouble with sqlite.

Anton Stroganov · Answer 11 · Wed Mar 29 2017 02:58:23 GMT+0800 (China Standard Time)

On 28-Mar-2007, at 3:32pm, david wrote (Trac issue 1150#comment:11):

I'm seeing pretty much the same thing here (though I should admit up front that I'm working on Windows, which may or may not make any difference.)

Mantis database is from 0.19.x, imported into MySQL 5.0.37 no problem; using trac 0.10 with pysqlite-2.3.3.win32-py2.4.exe, set up the trac environment ok, ie database connection is the default sqlite:db/trac.db.

Using latest mantis2trac r1929 (with TRAC_CLEAN option disabled cos windows doesn't understand 'rm' though I was regularly nuking the trac installation to make sure things were clean), I get pretty much the same as anonymous on 9 Feb:

7. import bugs and bug activity...
Adding user jason to sessions table
failed executing sql:
INSERT INTO session
                            (sid, authenticated, last_visit)
                        VALUES  ('jason', '1', '')
Traceback (most recent call last):
  File "C:\Documents and Settings\dec\Desktop\Trac on windows\mantis2trac.1929.py", line 943, in ?
    main()
  File "C:\Documents and Settings\dec\Desktop\Trac on windows\mantis2trac.1929.py", line 940, in main
    convert(MANTIS_DB, MANTIS_HOST, MANTIS_USER, MANTIS_PASSWORD, TRAC_ENV, TRAC_CLEAN)
  File "C:\Documents and Settings\dec\Desktop\Trac on windows\mantis2trac.1929.py", line 648, in convert
    ticket['reporter'] = trac.getLoginName(mysql_cur, bug['reporter_id'])
  File "C:\Documents and Settings\dec\Desktop\Trac on windows\mantis2trac.1929.py", line 476, in getLoginName
    print 'could not insert %s into sessions table: sql error %s ' % (loginName,
 self.db().error())
  File "C:\Python24\lib\site-packages\trac\db\util.py", line 69, in __getattr__
    return getattr(self.cnx, name)
  File "C:\Python24\lib\site-packages\trac\db\util.py", line 69, in __getattr__
    return getattr(self.cnx, name)
AttributeError: 'pysqlite2.dbapi2.Connection' object has no attribute 'error'

However, I had no luck with the suggested comment 1 approach of reverting to r1721 etc, although it did get further:

inserting ticket 9 -- "autogenerate jnlp file from build.paths"
 * adding comment "Take a look at the ant tasks here http:/..."
 * adding comment "its been moved to here http://sourceforg..."
Traceback (most recent call last):
  File "C:\Documents and Settings\dec\Desktop\Trac on windows\mantis2trac.1721.p
y", line 905, in ?
    main()
  File "C:\Documents and Settings\dec\Desktop\Trac on windows\mantis2trac.1721.p
y", line 902, in main
    convert(MANTIS_DB, MANTIS_HOST, MANTIS_USER, MANTIS_PASSWORD, TRAC_ENV, TRAC
_CLEAN)
  File "C:\Documents and Settings\dec\Desktop\Trac on windows\mantis2trac.1721.p
y", line 651, in convert
    trac.addTicketComment(bugid, note['date_submitted'], trac.getLoginName(mysql
_cur, note['reporter_id']), note['note'])
  File "C:\Documents and Settings\dec\Desktop\Trac on windows\mantis2trac.1721.p
y", line 427, in addTicketComment
    (ticket, time.strftime('%s'), author, 'comment', '', comment,))
  File "C:\Python24\lib\site-packages\trac\db\util.py", line 47, in execute
    return self.cursor.execute(sql_escape_percent(sql), args)
  File "C:\Python24\lib\site-packages\trac\db\sqlite_backend.py", line 56, in ex
ecute
    args or [])
  File "C:\Python24\lib\site-packages\trac\db\sqlite_backend.py", line 48, in _r
ollback_on_error
    return function(self, *args, **kwargs)
pysqlite2.dbapi2.IntegrityError: columns ticket, time, field are not unique

... which made me think briefly that the TIME_ADJUSTMENT_HACK would save me, but it didn't.

So, seeing the remarks about pysqlite cf mysql, I followed http://trac.edgewall.org/wiki/MySqlDb and I created the trac database within MySQL. Reran the trac environment initialisation with the database connection string set to: mysql://root:password@localhost/trac

Tried again with r1929 and got essentially the same as with pysqlite:

7. import bugs and bug activity...
Adding user jason to sessions table
failed executing sql:
INSERT INTO session
                            (sid, authenticated, last_visit)
                        VALUES  ('jason', '1', '')
could not insert jason into sessions table: sql error
Traceback (most recent call last):
  File "C:\Documents and Settings\dec\Desktop\Trac on windows\mantis2trac.1929.py", line 943, in ?
    main()
  File "C:\Documents and Settings\dec\Desktop\Trac on windows\mantis2trac.1929.py", line 940, in main
    convert(MANTIS_DB, MANTIS_HOST, MANTIS_USER, MANTIS_PASSWORD, TRAC_ENV, TRAC_CLEAN)
  File "C:\Documents and Settings\dec\Desktop\Trac on windows\mantis2trac.1929.py", line 648, in convert
    ticket['reporter'] = trac.getLoginName(mysql_cur, bug['reporter_id'])
  File "C:\Documents and Settings\dec\Desktop\Trac on windows\mantis2trac.1929.py", line 480, in getLoginName
    c.execute(
  File "C:\Python24\lib\site-packages\trac\db\util.py", line 47, in execute
    return self.cursor.execute(sql_escape_percent(sql), args)
  File "C:\Python24\lib\site-packages\trac\db\util.py", line 47, in execute
    return self.cursor.execute(sql_escape_percent(sql), args)
  File "C:\Python24\lib\site-packages\mysql_python-1.2.2-py2.4-win32.egg\MySQLdb\cursors.py", 
line 151, in execute
TypeError: not all arguments converted during string formatting

For completeness, tried the r1721 approach too:

7. import bugs and bug activity...
inserting ticket 1 -- "Sort Ranges by multiple rows/columns"
Traceback (most recent call last):
  File "C:\Documents and Settings\dec\Desktop\Trac on windows\mantis2trac.1721.p
y", line 905, in ?
    main()
  File "C:\Documents and Settings\dec\Desktop\Trac on windows\mantis2trac.1721.p
y", line 902, in main
    convert(MANTIS_DB, MANTIS_HOST, MANTIS_USER, MANTIS_PASSWORD, TRAC_ENV, TRAC
_CLEAN)
  File "C:\Documents and Settings\dec\Desktop\Trac on windows\mantis2trac.1721.p
y", line 642, in convert
    trac.addTicket(**ticket)
  File "C:\Documents and Settings\dec\Desktop\Trac on windows\mantis2trac.1721.p
y", line 400, in addTicket
    (id, time.strftime('%s'), changetime.strftime('%s'), component.encode('utf-8
'),
  File "C:\Python24\lib\site-packages\trac\db\util.py", line 47, in execute
    return self.cursor.execute(sql_escape_percent(sql), args)
  File "C:\Python24\lib\site-packages\trac\db\util.py", line 47, in execute
    return self.cursor.execute(sql_escape_percent(sql), args)
  File "C:\Python24\lib\site-packages\mysql_python-1.2.2-py2.4-win32.egg\MySQLdb
\cursors.py", line 166, in execute
  File "C:\Python24\lib\site-packages\mysql_python-1.2.2-py2.4-win32.egg\MySQLdb
\connections.py", line 35, in defaulterrorhandler
_mysql_exceptions.OperationalError: (1366, "Incorrect integer value: '' for colu
mn 'time' at row 1")

So I'm now stuck. I've tried all the combinations I can think of and have reached the limit of my understanding of what's going on. Hope this helps?!

Anton Stroganov · Answer 12 · Wed Mar 29 2017 02:58:23 GMT+0800 (China Standard Time)

On 13-Sep-2007, at 7:19am, sngautam wrote (Trac issue 1150#comment:12):

I get this error:

import components...
sql: SELECT category, user_id as owner FROM mantis_project_category_table
warning: unknown mantis userid 0, recording as anonymous
inserting component ' Marketing Request ', owner anonymous
inserting component ' Bugs ', owner anonymous
inserting component ' Customer Support ', owner anonymous
inserting component ' Change Request(Tech) ', owner anonymous
inserting component ' Change Request(Design and content) ', owner anonymous
inserting component ' New Feature ', owner anonymous
inserting component ' Hiring ', owner anonymous
inserting component ' Monitoring ', owner anonymous
inserting component ' Hiring ', owner anonymous

Traceback (most recent call last):
  File "./mantis2trac.py", line 943, in <module>
    main()
  File "./mantis2trac.py", line 940, in main
    convert(MANTIS_DB, MANTIS_HOST, MANTIS_USER, MANTIS_PASSWORD, TRAC_ENV, TRAC_CLEAN)
  File "./mantis2trac.py", line 596, in convert
    trac.setComponentList(components, 'category')
  File "./mantis2trac.py", line 353, in setComponentList
    (comp[key].encode('utf-8'), comp['owner'].encode('utf-8'),))
  File "/usr/lib/python2.5/site-packages/trac/db/util.py", line 50, in execute
    return self.cursor.execute(sql_escape_percent(sql), args)
  File "/usr/lib/python2.5/site-packages/trac/db/sqlite_backend.py", line 56, in execute
    args or [])
  File "/usr/lib/python2.5/site-packages/trac/db/sqlite_backend.py", line 48, in _rollback_on_error
    return function(self, *args, **kwargs)
pysqlite2.dbapi2.IntegrityError: column name is not unique

Anton Stroganov · Answer 13 · Wed Mar 29 2017 02:58:24 GMT+0800 (China Standard Time)

On 23-Nov-2007, at 2:43pm, The Fixer wrote (Trac issue 1150#comment:13):

Hi,
I also had have this problem.

all places with
INSERT INTO ticket_change
should be replaced with
INSERT OR IGNORE INTO ticket_change
and then the script will work.

(BTW: Furtermore, I had to replace all 'utf-8' with 'latin1' to make it work with all my used German umlauts. Maybe this should be made configurable as script command line argument...)

Anton Stroganov · Answer 14 · Wed Mar 29 2017 02:58:24 GMT+0800 (China Standard Time)

On 26-Nov-2007, at 8:28am, The Fixer wrote (Trac issue 1150#comment:14):

Hmm...now with that 'INSERT OR IGNORE ticket_change' it converts well but I get a runtime-error:

Traceback (most recent call last):
  File "C:\Program Files\Python24\Lib\site-packages\trac\web\main.py", line 406, in dispatch_request
    dispatcher.dispatch(req)
  File "C:\Program Files\Python24\Lib\site-packages\trac\web\main.py", line 237, in dispatch
    resp = chosen_handler.process_request(req)
  File "C:\Program Files\Python24\Lib\site-packages\trac\ticket\report.py", line 88, in process_request
    resp = self._render_view(req, db, id)
  File "C:\Program Files\Python24\Lib\site-packages\trac\ticket\report.py", line 346, in _render_view
    value['date'] = format_date(cell)
  File "C:\Program Files\Python24\Lib\site-packages\trac\util\datefmt.py", line 68, in format_date
    return format_datetime(t, format, gmt)
  File "C:\Program Files\Python24\Lib\site-packages\trac\util\datefmt.py", line 58, in format_datetime
    t = time.localtime(float(t))
ValueError: empty string for float()

What can I do now? sigh

(Hmm...and, please, forget the comment about using of 'latin1' because 'utf-8' was correct. Before the mantis2trac script call I had to convert my database to utf-8. Because trac can only handle utf-8.)

Anton Stroganov · Answer 15 · Wed Mar 29 2017 02:58:25 GMT+0800 (China Standard Time)

On 26-Nov-2007, at 8:30am, The Fixer wrote (Trac issue 1150#comment:15):

The whole thing seems to be related to ticket #5278

Anton Stroganov · Answer 16 · Wed Mar 29 2017 02:58:25 GMT+0800 (China Standard Time)

On 26-Nov-2007, at 8:31am, The Fixer wrote (Trac issue 1150#comment:16):

Hmm... #5278 is here

Anton Stroganov · Answer 17 · Wed Mar 29 2017 02:58:26 GMT+0800 (China Standard Time)

On 27-Nov-2007, at 7:21pm, The Fixer wrote (Trac issue 1150#comment:17):

I tried this and converting and web-gui to the tickets run fine now. Thanks to licho@ufo.cz! :-) I think this ticket can be closed now.

Anton Stroganov · Answer 18 · Wed Mar 29 2017 02:58:26 GMT+0800 (China Standard Time)

On 7-Dec-2007, at 8:44am, sngautam wrote (Trac issue 1150#comment:18):

I'm still facing the same problem. I used the patched script given in #5855

++++
pysqlite2.dbapi2.IntegrityError?: column name is not unique
++++

This is my error message.
Same as posted above.

Anton Stroganov · Answer 19 · Wed Mar 29 2017 02:58:27 GMT+0800 (China Standard Time)

On 17-Jul-2012, at 11:21am, @Aeon wrote (Trac issue 1150#comment:19):

The Fixer is right, it's due to the Trac ticket #5278. I don't have a test database to test with right now, but the latest code is now on github, please feel free to submit patches.