Giters

departurerb / departure

Percona's pt-online-schema-change runner for ActiveRecord migrations.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

pt-osc - output maxing at 8 chars

sauloperez opened this issue · comments

commented

Running migrations locally, I do not encounter this problem. However, when my demo and production servers run migrations during deployment (on VM running Linux, with migration triggered by Capistrano), the logs are always messed up. Here is a gist of one such migration -

https://gist.github.com/garettarrowood/42449b48709d85017a13bd5e522183ab

I think the issue is the read_nonblock(8) method here - https://github.com/redbooth/departure/blob/master/lib/departure/command.rb#L47 .

I'll open a PR, tie my repo to it, and see if just raising the number to 40 solves the problem. But before I do, can you provide some reference why a max length of 8 is used here? What problem might I create by changing it?

Upon further thought - it seems more likely logger.write_no_newline(data) is not working correctly.

commented

Hi @garettarrowood, I'll take a look later today

commented

Thanks @sauloperez !

I ran out of time for doing the leg work necessary to fork, merge a change, publish my own version of the gem, then open another work-repo PR to test that new version. I've not abandoned debugging this myself, but any ideas or fixes are appreciated.

commented

read_nonblock will read at most N bytes, but may read less. There is no reason to only read 8 characters at a time. It will read whatever is available, up to N, and in this case there is no downside to reading larger chunks. We are experiencing this problem in our logs, and it is very irritating. I will do some legwork and testing both with larger chunks for the read, and with investingations into the behavior of write_no_newline and will deliver a patch.

commented

There don't seem to be any bugs in logger.write_no_newline(data).

My suspicion is that what we are seeing is an interaction between the select loop in Departure that is reading tiny little 8 character chunks, and the IO handling somewhere in the pipeline from Net:SSH -> SSHKit -> Airbrussh. I have been looking at a lot of code, and there is no specific thing that jumps out at me, but I am confident that all that we really need to do to address this is to permit Departure to read more IO. There is no reason for reading those tiny blocks of data in combination with read_nonblock. I will give you a PR for this change, and it should be sufficient to solve the problem.