vttablet: vreplication_retry_delay should be configurable

[Matthew Kent]

We observed the following on RDS:

If vreplication has an issue like a duplicate key error it will create numerous 'Binlog Dump GTID' threads when reconnecting. These can eventually consume all resources on a server and hang new connections. This is a known mysql behaviour per this issue. We can lessen the likelihood of this by bumping -vreplication_retry_delay up from it's 5s default, though we'll need to add monitoring to detect the underlying issue.

Suggest we allow vreplication_retry_delay to be configurable.

[Peter Farr]

This sounds easy enough. I'd be happy to review a PR if you submit it.

[Anthony Yeh]

In general, you can set any vttablet flag with extraFlags. We tend to add something specific to the CRD API only when there's an opportunity to abstract the config in some way, such as when multiple flags need to be set in concert to achieve some high-level goal. This seems more like a one-off adjustment, which is what extraFlags was intended for. Does that make sense for your use case?

[Matthew Kent]

In general, you can set any vttablet flag with extraFlags. We tend to add something specific to the CRD API only when there's an opportunity to abstract the config in some way, such as when multiple flags need to be set in concert to achieve some high-level goal. This seems more like a one-off adjustment, which is what extraFlags was intended for. Does that make sense for your use case?

It does thanks! I didn't realize I could pass arbitrary options. I think I'm all set here.

Fixed via

          - cell: useast1
            type: externalmaster
            replicas: 1
            vttablet:
              extraFlags:
                db_charset: utf8mb4
                queryserver-config-pool-size: "5"
                queryserver-config-stream-pool-size: "5"
                queryserver-config-transaction-cap: "5"
                vreplication_retry_delay: "30s"