Giters

dbcli / mycli

A Terminal Client for MySQL with AutoCompletion and Syntax Highlighting.

Home Page:http://mycli.net

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Read more options from my.cnf

romuald opened this issue · comments

commented

Just a list of options I'm currently using and that would be nice to have automatcially in mycli

[mysql]
pager = mypager  # https://github.com/romuald/mypager
safe-updates
max-join-size=1000000000

They all seem to work nicely within the CLI, but aren't loaded automatically

commented

The pager option in the my.cnf is already supported by mycli. But we only read the [client] section of my.cnf not the [mysql] section. So try moving the pager line under the [client] section.

We will consider expanding the my.cnf support to add the other options as well.

commented

mycli should also read a mycli section in both .my.cnf and .mylogin.cnf in addition to client as MySQL programs do.

It could also read mycli sections with a given suffix as MySQL program do when given the --defaults-group-suffix option.
For example with mysql here is how I can store all my connection settings in .my.cnf/.mylogin.cnf:

$ mysqleditor set --login-path=mysql_testing --user root --host testing-host --password
$ cat >> ~/.my.cnf <<EOF
[mysql_testing]
database = testing
prompt = "\u@\h/\d>"
EOF
# Will load user, host, password from ~/.mylogin.cnf and database, prompt from ~/.my.cnf
$ mysql --defaults-group-suffix=_testing
commented

Actually, using [client] causes some problems. e.g. set this:

[client]
prompt=\n\D\n\u@\h:\p/\d\n>\_

then:

$ mysqldump
mysqldump: unknown variable 'prompt=
\D
\u@\h:\p/\d
>\_'

I know prompt is probably better in .myclirc, but you get the general idea. I'd second the use of a dedicated [mycli] section.

commented

That's an interesting idea. @tsroten What do you think?

I'd actually vote for dropping support for ~/.my.cnf altogether and only using ~/.myclirc instead.

But I'm worried it'll break compatibility for older versions.

commented

Hmm, this is tricky. For one thing, I wish MySQL's clients didn't quit with an error message if they see an option they don't recognize (e.g. like mysqldump above). See the MySQL docs:

Be sure not to put an option in the [client] group unless it is recognized by all client programs that you use. Programs that do not understand the option quit after displaying an error message if you try to run them.

Like @amjith said, I like the idea of keeping this simple. If we add a dedicated [mycli] section, we're reading options from yet another place. The thing is, and I think this is related to @amjith's point, we are in no way obligated to read ~/.my.cnf at all. The MySQL docs say:

The [client] option group is read by all client programs provided in MySQL distributions.

Of course, mycli isn't provided in the distributions. As far as I can tell, reading from MySQL's option files is not an official recommendation for third-party clients.

Reading the [client] section is a nice feature because it allows mycli to be, on a basic level, a drop-in replacement for the official client. The same thing goes for reading the login path file. But, reading another section from ~/.my.cnf introduces unnecessary complexity just for the users that want to handle all their mysql-related config in a single file when that file is not extensible enough for all third-party features. Ultimately, we cannot replace .myclirc with .my.cnf, so I'd argue we should maintain the minimum-level of .my.cnf support we feel is required.

So, I think my opinion is to continue reading the [client] section and the login path file, but leave it at that.

commented

@dbolser-ebi I think mysqldump (and other mysql tools) dislike unquoted config options, that's why you get "unknown variable". What if you use prompt="\n\D\n\u@\h:\p/\d\n>\_" in [client]?

commented

Exact same error. I think [client] is the wrong place for options like this.

commented

@dbolser-ebi I think you're right. That's what .myclirc is for -- actual mycli options.

commented

One related thing is that mycli reads the pager config from the [client] section -- something we've advised users to do in the past.

Basically, unless I'm missing something, you can't specify a default pager option for mycli using config files if you are a user of any of the following official clients:

  • mysqldump
  • mysqlcheck
  • mysqlpump
  • mysqlimport
  • mysqladmin
  • mysqlshow

I'm beginning to seriously dislike .my.cnf ;)

As far as I can tell, MySQL does not have a master list of options recognized by all official clients. There are client-specific lists however.

The thing is, MySQL puts the burden on the user to make sure they keep their [client] section compatible across the apps they use (emphasis mine):

Be sure not to put an option in the [client] group unless it is recognized by all client programs that you use.

The problem is, since mycli only reads these settings from .my.cnf's [client] section, we potentially break users' official clients by requiring the use of a .my.cnf [client] section for these options. mysqldump can't even handle the database option, let alone some of the other ones. The official clients listed above all support specifying options in a client-specific section of the config file.

Ultimately, in the context of the MySQL documentation, I don't see evidence that the intent was ever that third-party clients would rely on MySQL config files (outside of those that use the C API). So, I think our support of .my.cnf is primarily a convenience to the user, but it should not be the only (or even primary) route to specify configuration options.

I'm seeing a few options long-term (and, please suggest others!):

  1. Drop support of .my.cnf altogether. This does lead to a negative onboarding experience for users -- we'd lose any drop-in replacement benefits. Doing this also begs the question of whether or not we should drop login path support. Overall, this is simpler for mycli's code.
  2. Add support for these options to .myclirc so that people who use particular official clients can continue to use them without breaking them. Do this while continuing support for .my.cnf. This introduces code complexity around reading configuration options.
  3. Support a [mycli] section and a [client] section in the .my.cnf. This introduces the most code complexity, but it allows the user to keep many of their settings in one spot. Then, .myclirc is used for mycli UI-type settings (and favorite queries, of course!). If we do this without adding support for these options to myclirc, then mycli continues to rely on a config file that's not its own.

Like I said in my earlier post, I'm in favor of number 2:

  • Essentially, expand myclirc to be the designated config location for mycli -- it's missing support for quite a few options we read from .my.cnf that break other clients (e.g. pager, database, host, socket, ssl info, etc.)
  • Maintain support of .my.cnf for users who want to use the [client] section.
  • Refactor the code when mycli runs to load/prioritize the various configs all at once so it's not in multiple places in main.py.

I'd also support number 1 if that seems like a better option (like @amjith suggested above).

I'd be very hesitant to support number 3. I think that's out of scope for mycli.

commented

Sorry to add another option:

  1. Support [mysql] section of .my.cnf (and possibly [mycli]) and allow all options there to be overridden by their use in .myclirc

Use of [client] is just wrong IMHO. Use of [mysql] is probably what you need to mitigate the "negative onboarding experience for new users", and probably what people expect from "drop-in replacement benefits".

Other than that you need to make documentation clearer. I only just learned .myclirc exists coming back here to complain about .my.cnf for the second time.

If I had to vote for 1-3, I'd pick 1 + improved docs.

commented

@tsroten Thank you for taking the time to list the possibilities.

I'm with @dbolser-ebi on this one. Let's do option 1. I know this will piss people off, but I think it was a mistake to add my.cnf support in the first place. I made a mistake by hastily adding my.cnf support and we're now paying the price for it in complexity.

I say we drop my.cnf support, add proper documentation about ~/.myclirc file and release it as mycli 2.0.0 since it is a breaking change.

There might be a bit of backlash for a little bit but if we're consistent and patient in our response users will understand.

commented

Great, I'm on board with dropping .my.cnf support. So, it sounds like we'd need to:

  • Add missing options to .myclirc (basically all the options we support from .my.cnf).
  • Drop the .my.cnf-related code
  • Update the documentation

And, what about MySQL's login path support? If we're dropping .my.cnf support, it kind of makes sense to get rid of that feature as well. However, it's the only option (that I know of) for storing encrypted authentication information with mycli. If we want to get rid of it, I suggest we do it in a standalone pull request so as to not muddy up the other changes. And, if we get rid of it, it would be nice to provide a new way to store encrypted login information.

commented

I'm on board with dropping support for ~/.my.cnf support. But can you elaborate why we want to drop support for login path?

It seems like we're not asking users to add mycli specific config options in that file.

commented

Dropping login path support is definitely out of scope for this issue. But, here are a few thoughts on it.

I might have been getting ahead of myself there. I guess I was thinking that if we are dropping support for my.cnf, does that mean we're eventually going to drop support for all mysql option files?

Ultimately, keeping login path support is a huge convenience without the same problems as the [client] section of the my.cnf file, so I can see a good argument for keeping it.

But, long-term it would be good to provide a better way use encrypted login credentials. The login path files are essentially a weakly-encrypted version of the my.cnf file.

In @grooverdan's words, "Fine for compatibility but please don't portray this as security feature."

Anyways, maybe this is a discussion for another day.

On a different note, does dropping my.cnf support also mean that we drop support for the defaults-file and defaults-group-suffix options? Or, do they still have utility after my.cnf support goes away?

commented

You bring up a ton of useful points. I'm starting to backtrack on my suggestion.

Let's reevaluate why we added support for my.cnf and why we're getting rid of it. Here's the original ticket that requested it: #17

The interesting bit is that the original implementation never supported the pager option. It only supported these:

[client]
database
user
password
host
port
socket
default-character-set

I think those are valid options that we don't have in our .myclirc file. Hence there is no conflict.

We (by that I mean, I) went and added support for pager in my.cnf which was a big mistake. I think that is a mycli specific option that does NOT belong in my.cnf file.

What if we take the pager option from my.cnf and move it to myclirc and keep the support for my.cnf file?

If both my.cnf and myclirc has a pager option then myclirc will override the pager value from my.cnf. Which essentially means we might break compatibility for people who have come to rely on pager option being available in my.cnf. But we can answer those folks and assuage their fears.

In the future, we take a more cautious approach for adding support for things to read from my.cnf.

Thoughts?

commented

If we're not dropping support for .my.cnf files, we need to try and set rules for what options in .my.cnf we support. Would it be only those supported by PyMySQL?

commented

I can't see what exactly they support from the [client] section.

I vote for just removing support for the pager config and moving it to myclirc.

commented

Differences are highlighted.

Here is the list from PyMySQL's code:

  • user
  • password
  • host
  • database
  • socket
  • port
  • bind-address
  • default-character-set
  • ssl-ca
  • ssl-capath
  • ssl-cert
  • ssl-key
  • ssl-cipher

Here is the list that mycli supports now:

  • user
  • password
  • host
  • database
  • socket
  • port
  • default-character-set
  • ssl-ca
  • ssl-cert
  • ssl-key
  • ssl-cipher
  • ssl-verify-server-cert
  • local-infile/loose-local-infile
  • prompt
  • pager
  • skip-pager
commented

Oh yeah, I forgot about prompt, I think we should remove prompt support from my.cnf as well, since we have a prompt config in our myclirc.

I don't know what these two things do,

ssl-verify-server-cert
local-infile/loose-local-infile

I'll leave it up to you folks to decide whether to keep those two or not.

commented

Fixed the link to relevant part in PyMySQL above.

commented

Well, on these remaining options:

  • ssl-verify-server-cert is deprecated. But, we still need some way to toggle check_hostname in PyMySQL's ssl config. The new way is ssl-mode=VERIFY_IDENTITY.
  • we need some way to toggle the local infile option for pymysql. It's probably easier to keep it as is unless we decide it's better suited for myclirc.
commented

Sorry for an off topic question, but how does a third party client like mycli work with login path? At work we have 10+ servers and a bunch of scripts that consume the credentials in one way or another. If they could use login path, that would be ideal.

commented

@dbolser-ebi mycli's config module can read and decrypt the login path file.

The code we use for decrypting is in config.py.

Here is how you could use mycli to open the login path file and read the user/password/host from a section called [qa1]

from mycli.config import get_mylogin_cnf_path, open_mylogin_cnf, read_config_file

login_path_file = open_mylogin_cnf(get_mylogin_cnf_path())
config = read_config_file(login_path_file)

print(config['qa1']['user'])
print(config['qa1']['host'])
print(config['qa1']['password'])
commented

@dbolser-ebi requested we can read local-infile via the .myclirc as well.
#434 (comment)

commented

I'm closing this issue in favor of an overhaul to the config files: #581