`skeema init` appears hung if large .sql data dump file in working directory

[Mike Gray]

Describe the bug
I must be missing something because I know I've gotten this tool to work before, however I also know I had the same issue where --debug doesn't print out any logging.

Environment

• Skeema version: 1.8.1-community, commit 320c1f4, released 2022-07-28T18:57:46Z
• Database vendor, version, OS/platform: MariaDB 10.5.15 on Debian 10
• Any other environmental notes or special configuration:

To Reproduce
Not sure how this could be reproduced, but here are the following commands I've tried:

• ./skeema init -h localhost -S /var/run/mysqld/mysqld.sock --schema tc -d database -u root -p --debug production
• ./skeema init -h localhost -S /var/run/mysqld/mysqld.sock -u root -p --schema tc --debug
• Both of the above with sudo
• As well as both of the above from the root account
• As well as many different iterations with and without --debug

Each time it asks for the db password (which I know is correct because I can login via mysql), and then does nothing. The cpu usage goes to 100% and there is no output and nothing happens. Am I missing something?

[Evan Elias]

That's odd, it's definitely not a failure mode I've ever seen before. skeema init is fairly simple so it doesn't have any extra debug logging. The --debug option is pretty limited in scope anyway (see docs) -- it's currently geared for developer debugging rather than end-user debugging.

A few things to check please:

• Is Skeema going to 100% cpu usage, or is it MariaDB server doing that?

• Can you switch to using tcp/ip instead of a unix socket? e.g. ./skeema init -h 127.0.0.1 -u root -p --schema tc

  • You may need to confirm your db is listening on tcp/ip first, check its config or just do a simple test using the mysql command-line client

• If you intentionally enter the wrong password, does Skeema exit with an error right away? (just trying to figure out whether the problem is pre- or post- auth handshake)

• If you manually run a SHOW PROCESSLIST query in the database while Skeema is hanging, what does it show?

• If you try an older version of Skeema do you get the same result?

[Mike Gray]

• Skeema goes to 100% (fairly certain that's just one core)
• Able to use 127.0.0.1, so switched to that
• Wrong password doesn't do anything differently
• Didn't find "access denied" message in the logs when using wrong password
• It shows this one, which isn't the logged in process, not sure what it is
  | 2203628 | root | localhost | tc | Sleep | 4 | | NULL | 0.000 |
• Went back to 1.7.1 as I believe that is the one I used before
• Tried everything with both ip and localhost/sock

:(

[Evan Elias]

Thanks, that's informative -- if you're seeing the exact same behavior even with the wrong password, it means the problem might be located in the Golang mysql driver.

To be clear, are you seeing the same problem with 127.0.0.1 (tcp/ip), or is that working properly?
Are you seeing the problem with skeema 1.7.1 as well, or is that working properly?
Thanks again!

[Mike Gray]

It has the same behavior with IP as it does with the socket. 1.7.1 had the same issues interestingly. I wish I could remember how I got it to work previously, but it was very frustrating even then

I just tried both 1.8.1 and 1.7.1 on my mac and they have the same behavior. I must be doing something wrong haha

[Evan Elias]

Very strange. So initially I was thinking this was networking-related, but on further thought it might be filesystem-related. skeema init does some verification of your current working directory (to ensure there's not already *.sql or .skeema files there) and parent directory hierarchy (to look for parent .skeema files) as an early step. Maybe it's getting caught in an infinite loop due to circular symlinks or something similar.

A few ideas on that front:

• Try making a new empty directory in a not-terribly-deep directory tree, and cd to that directory before invoking skeema init. Does it still hang?

• If you omit the host option -h entirely, do you get an error right away or does it still hang?

• Try omitting any non-essential options like --schema and -d just to confirm these aren't impacting things

• If you're familiar with strace on linux, you could try that to see what system calls skeema is making

And a few more ideas on the networking side, in case that's actually the problem:

• Try adding --ssl-mode=disabled to the command-line just to rule out any issues with SSL/TLS

• From your Mac, are you still connecting to the mariadb server on your Linux box? Or do you mean you get the same Skeema stall behavior on your Mac when connecting to a db locally on your Mac? If the Linux machine is the common denominator, maybe try restarting the server, check firewall settings, SELinux / AppArmor etc if relevant.

• If you're comfortable with tcpdump you could see what network packets are being sent to the db on your machine (when connecting via tcp/ip instead of a unix socket)

[Evan Elias]

Related to the first part of my previous reply -- if you're running skeema init (or most other Skeema commands really) from a directory that also happens to contain a large multi-GB .sql file from mysqldump, you might see behavior like what you're describing.

Skeema attempts to process *.sql files in your working directory since it assumes they contain CREATE statements managed by Skeema. But the processing logic for this is currently very inefficient for large files, especially with long INSERT statements generated by a logical data dump tool like mysqldump. This will be greatly improved in v1.9 as part of the seed data support described in #22 (comment).

[Mike Gray]

Huzzah! It was the .sql files in the current directory. On my mac I was trying to connect to a database in docker, but on both systems there was at least one sql file in the directory.

And I'm now remembering that was the same conclusion I came to previously when I was playing around with it.

I would think init wouldn't want to scan sql files for any reason?

(Also, yes the files are pretty large, so probably why it hung for a while)

[Evan Elias]

Awesome, glad that was it, thanks for filing the issue and for all your patience with debugging!

I'll retitle this issue to reflect the root cause, and will keep open, with an expected fix going into Skeema v1.9 along with the other filesystem management improvements. [Update: this fix is getting bumped to v1.10 since it's a substantial refactor]

You're absolutely correct that init doesn't need the sql file contents. It's only scanning them because some logic still needs to be decoupled. Currently the function that instantiates a directory struct value always interprets the .skeema option file as well as scanning the *.sql files. Most commands need both things to happen, but init is a special case which only needs the former. Should be fixable with a little refactoring soon. Thanks again.

[Evan Elias]

This is now fixed in the main branch by commit 2481bed. The next release isn't scheduled yet, but will likely be in 1 to 2 months. Thanks again for the issue report, as well as all the troubleshooting to pinpoint the cause!

[Evan Elias]

Skeema v1.10 is now available and includes the fix for this issue. Thanks again for the report and troubleshooting!