Out Of Memory killer triggered
Glucy-2 opened this issue · comments
Describe the bug
Out Of Memory killer triggered when I am trying to transfer a 1.59GB .sqlite file
screenshot of VNC viewer:
Expected behaviour
It works
Actual result
screenshot of VNC viewer:
> ./.local/bin/sqlite3mysql -f ./ledger.sqlite -d ledger -u * -h *.*.*.* --mysql-password ****************
2022-03-22 20:42:10 INFO Transferring table players
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 67/67 [00:00<00:00, 74.73it/s]
2022-03-22 20:42:11 INFO Adding unique index to column "player_id" in table players
2022-03-22 20:42:11 INFO Adding unique index to column "player_id" in table players
2022-03-22 20:42:11 INFO Transferring table ActionIdentifiers
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:00<00:00, 47.51it/s]
2022-03-22 20:42:11 INFO Adding unique index to column "action_identifier" in table ActionIdentifiers
2022-03-22 20:42:11 INFO Adding unique index to column "action_identifier" in table ActionIdentifiers
2022-03-22 20:42:11 INFO Transferring table worlds
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 33.94it/s]
2022-03-22 20:42:11 INFO Adding unique index to column "identifier" in table worlds
2022-03-22 20:42:11 INFO Adding unique index to column "identifier" in table worlds
2022-03-22 20:42:12 INFO Transferring table ObjectIdentifiers
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1314/1314 [00:16<00:00, 80.31it/s]
2022-03-22 20:42:28 INFO Adding unique index to column "identifier" in table ObjectIdentifiers
2022-03-22 20:42:28 INFO Adding unique index to column "identifier" in table ObjectIdentifiers
2022-03-22 20:42:28 INFO Transferring table sources
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 45/45 [00:00<00:00, 73.59it/s]
2022-03-22 20:42:29 INFO Adding unique index to column "name" in table sources
2022-03-22 20:42:29 INFO Adding unique index to column "name" in table sources
2022-03-22 20:42:30 INFO Transferring table actions
19%|████████████████████████▍ | 483474/2494012 [1:41:22<7:33:52, 73.83it/s]
[1] 1017098 killed ./.local/bin/sqlite3mysql -f ./ledger.sqlite -d ledger -u * -h
System Information
debian 11 which has 8 GB of memory
$ sqlite3mysql --version| software | version |
|------------------------|-------------------------------------------------------------|
| sqlite3-to-mysql | 1.4.12 |
| | |
| Operating System | Linux 5.10.0-13-amd64 |
| Python | CPython 3.9.2 |
| MySQL | mysql Ver 8.0.26 for Linux on x86_64 (Source distribution) |
| SQLite | 3.34.1 |
| | |
| click | 8.0.4 |
| mysql-connector-python | 8.0.28 |
| pytimeparse | 1.1.8 |
| simplejson | 3.17.6 |
| six | 1.16.0 |
| tabulate | 0.8.9 |
| tqdm | 4.63.0 |
Hi and thanks for the report, however, I'm not sure how to address this.
Did you check which process used how much memory?
What other processes were you running?
How much did MySQL (the usual suspect) consume?
Have you tried running it with Python 3.10?
Have you tried running it on a system with more RAM and profile it?
What's in the actions table?
Also, next time you run it, run it with --debug which will dump all the exceptions to the console.
I tried again with more free ram and it said it has done, but the size of the db hasn't changed.
log:
2022-03-23 00:06:55 INFO Transferring table players
2022-03-23 00:06:56 INFO Adding unique index to column "player_id" in table players
2022-03-23 00:06:56 INFO Adding unique index to column "player_id" in table players
2022-03-23 00:06:56 INFO Adding unique index to column "player_id" in table players
2022-03-23 00:06:56 INFO Adding unique index to column "player_id" in table players
2022-03-23 00:06:56 INFO Adding unique index to column "player_id" in table players
2022-03-23 00:06:56 INFO Adding unique index to column "player_id" in table players
2022-03-23 00:06:57 INFO Adding unique index to column "player_id" in table players
2022-03-23 00:06:57 INFO Adding unique index to column "player_id" in table players
2022-03-23 00:06:57 INFO Transferring table ActionIdentifiers
2022-03-23 00:06:57 INFO Adding unique index to column "action_identifier" in table ActionIdentifiers
2022-03-23 00:06:57 INFO Adding unique index to column "action_identifier" in table ActionIdentifiers
2022-03-23 00:06:57 INFO Adding unique index to column "action_identifier" in table ActionIdentifiers
2022-03-23 00:06:57 INFO Adding unique index to column "action_identifier" in table ActionIdentifiers
2022-03-23 00:06:57 INFO Adding unique index to column "action_identifier" in table ActionIdentifiers
2022-03-23 00:06:57 INFO Adding unique index to column "action_identifier" in table ActionIdentifiers
2022-03-23 00:06:57 INFO Adding unique index to column "action_identifier" in table ActionIdentifiers
2022-03-23 00:06:57 INFO Adding unique index to column "action_identifier" in table ActionIdentifiers
2022-03-23 00:06:57 INFO Transferring table worlds
2022-03-23 00:06:58 INFO Adding unique index to column "identifier" in table worlds
2022-03-23 00:06:58 INFO Adding unique index to column "identifier" in table worlds
2022-03-23 00:06:58 INFO Adding unique index to column "identifier" in table worlds
2022-03-23 00:06:58 INFO Adding unique index to column "identifier" in table worlds
2022-03-23 00:06:58 INFO Adding unique index to column "identifier" in table worlds
2022-03-23 00:06:58 INFO Adding unique index to column "identifier" in table worlds
2022-03-23 00:06:58 INFO Adding unique index to column "identifier" in table worlds
2022-03-23 00:06:58 INFO Adding unique index to column "identifier" in table worlds
2022-03-23 00:06:58 INFO Transferring table ObjectIdentifiers
2022-03-23 00:07:13 INFO Adding unique index to column "identifier" in table ObjectIdentifiers
2022-03-23 00:07:13 INFO Adding unique index to column "identifier" in table ObjectIdentifiers
2022-03-23 00:07:13 INFO Adding unique index to column "identifier" in table ObjectIdentifiers
2022-03-23 00:07:13 INFO Adding unique index to column "identifier" in table ObjectIdentifiers
2022-03-23 00:07:13 INFO Adding unique index to column "identifier" in table ObjectIdentifiers
2022-03-23 00:07:13 INFO Adding unique index to column "identifier" in table ObjectIdentifiers
2022-03-23 00:07:13 INFO Adding unique index to column "identifier" in table ObjectIdentifiers
2022-03-23 00:07:13 INFO Adding unique index to column "identifier" in table ObjectIdentifiers
2022-03-23 00:07:14 INFO Transferring table sources
2022-03-23 00:07:14 INFO Adding unique index to column "name" in table sources
2022-03-23 00:07:14 INFO Adding unique index to column "name" in table sources
2022-03-23 00:07:14 INFO Adding unique index to column "name" in table sources
2022-03-23 00:07:14 INFO Adding unique index to column "name" in table sources
2022-03-23 00:07:14 INFO Adding unique index to column "name" in table sources
2022-03-23 00:07:14 INFO Adding unique index to column "name" in table sources
2022-03-23 00:07:14 INFO Adding unique index to column "name" in table sources
2022-03-23 00:07:14 INFO Adding unique index to column "name" in table sources
2022-03-23 00:07:15 INFO Transferring table actions
2022-03-23 08:23:29 INFO Adding index to column "x, y, z, world_id" in table actions
2022-03-23 08:23:30 INFO Adding index to column "x, y, z, world_id" in table actions
2022-03-23 08:24:54 INFO Adding foreign key to actions.player_id referencing players.id
2022-03-23 08:26:10 INFO Adding foreign key to actions.source referencing sources.id
2022-03-23 08:27:20 INFO Adding foreign key to actions.old_object_id referencing ObjectIdentifiers.id
2022-03-23 08:28:34 INFO Adding foreign key to actions.object_id referencing ObjectIdentifiers.id
2022-03-23 08:29:51 INFO Adding foreign key to actions.world_id referencing worlds.id
2022-03-23 08:31:00 INFO Adding foreign key to actions.action_id referencing ActionIdentifiers.id
2022-03-23 08:32:10 INFO Done!
Whoa, transferring the actions table for 8h 15min 😱 What do you have in there LOL?
I'm afraid I'll need more info here to give you some more feedback and/or potentially fix this.