Giters

shipitjs / shipit-deploy

Set of deployment tasks for Shipit based on git and rsync commands.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Deployments slow on large repos

Truemedia opened this issue · comments

commented

Hi,
I am having issues with deployments being slow with one of my projects which has a large number of tags and commits.

I can understand why it would be slow if it is downloading the entire repo every time it deploys, but is it possible for the deploy command to use a cached version of the repo and simply pull the tags and deploy the latest one?

commented

Hi,
Yes I have had this setting enabled on my project for a while. Is there anything else that could be slowing it down?

commented

Which part of the process is slow? Specifically - is the local checkout, or the remote copy?

I've had instances where I had gitignored things (db backups) that I was inadvertently transferring on deploy, as they were not in my ignores setting

commented

I think it is the local checkout. On this command it seems to get stuck for a while
Running "git fetch shipit --prune --depth=1 && git fetch shipit --prune "refs/tags/*:refs/tags/*"" on local.
Then I get a list of new branches and tags, but what it says is new is basically every single branch and tag that exists in the repo.

commented

Try and run those commands separately (without &&) and see which one hangs?

commented

Try running git gc (garbage collection) as well.
I would run this on your main repo and also rm your tmp workspace folder so when you deploy it does it from scratch.

commented

@timkelty Thanks, I'll give those a try and reply once I see the results. This may be useful info for people with similar issue.

commented

@Truemedia did that solve the problem for you? I'm also seeing this issue

commented

I am having issues with speed of "deploy:fetch" too and i think that "deploy:fetch" does too much. @timkelty can you please explain why (when shallowClone is true) instead of

git clone -b $branchName --depth=1 $repoUrl

Shipit-deploy does

1. Repository initialize. 
2. Remote update 
3. Repository fetch. 
4. Check out 
5. Reset working tree. 
5. Branch merge
commented

@kroleg We're splitting git clone into git init, git fetch git checkout so we have more granular control over what we're doing. That's really what git clone is doing, anyway. Can you tell which step of deploy:fetch it is getting hung up on?

commented

@timkelty hi, I found it got stuck on git fetch shipit --prune --depth=1, below is the screenshot.

Is it mean the repository too big?

commented

@timkelty i am on medium speed internet connections (public wifi) 95% of the time and single git clone ... takes 5s when deploy:fetch takes 18s and more.

I wrote a simple (and rough) replacement for deploy:fetch which can be pasted inside deploy function shipitfile.js to replace original one

const mkdirp = require('mkdirp'),
    chalk = require('chalk'),
    Bluebird = require('bluebird');

shipit.blTask('deploy:fetch', () => {

        return createWorkspace()
            .then(cloneRepository)
            .then(function () {
                shipit.emit('fetched');
            });

        function createWorkspace() {
            return shipit.local('rm -rf ' + shipit.config.workspace)
                .then(() => {
                    shipit.log('Create workspace "%s"', shipit.config.workspace);
                    return Bluebird.promisify(mkdirp)(shipit.config.workspace)
                        .then(function () {
                            shipit.log(chalk.green('Workspace created.'));
                        });
                });
        }

        function cloneRepository() {
            return shipit.local(`git clone -b ${shipit.config.branch} --depth=1 ${shipit.config.repositoryUrl} .`, {cwd: shipit.config.workspace})
                .then(function () {
                    shipit.log(chalk.green('Repository fetched.'));
                });
        }
    });

For me this function is ~4 times faster. Should i publish it as separate package or it can be included as optional replacement for deploy:fetch in shipit-deploy?

commented

@bryantAXS Sorry I kept forgetting to reply to this. Had no look speeding up my deployments. I am tempted to try something simillar to @kroleg.

commented
commented

@Truemedia i just published a shipit-deploy-simple-fetch package. Hope it will be useful for other people too :)

commented

Thanks for doing that @kroleg.
Sounds like you have something there...I feel like there are reasons that we're doing it this way currently, I just don't have the time right now to dig in at the moment.

commented

@kroleg Cheers :) will have a try of it when I have some spare time.

commented

If still a problem, please reopen an issue on shipit repository.