Giters

labs42io / circleci-monorepo

An example of monorepo with CircleCI using conditional workflows and pipeline parameters.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Question: change detection

pkyeck opened this issue · comments

commented

Hi, first of: thanks for the medium article + the repo!
Looks promising.

I was playing around with your setup and was wondering how you handle feature-branches. I hab a couple of commits on the branch which changed files in two packages. The script picked up the changes ok and ran the two jobs -> errors. Ok, lets fix them in the first one. Great fixed, but it didn't check the second package (build still failed) and on the PR page it said "all ok, no errors, you can merge into master ...".

My question: shouldn't it - when you are working on a branch - always check against parent branch to run tests for all packages that have changes compared to parent branch?

commented

When it calculates changes, it starts only those workflows in which you have code changes.
In your case you fixed the first package and it ran the corresponding workflow, but the second package is not touched as there are no code changes. Question is why would you like to rebuild the second package if you have changes only in the first one? Does that mean second package depends on first package?

Supporting custom dependencies between packages would be a nice feature. You could then configure that second package should be rebuilt each time code is changed in the first package. Other solution would be to rebuild each time all the workflows for which there are code changes, and those that failed in the feature branch. But currently I'm not sure if CircleCI API supports to get the failing workflows in a given git branch.

My question: shouldn't it - when you are working on a branch - always check against parent branch to run tests for all packages that have changes compared to parent branch?

Checking always against parent branch might not be an optimal solution. What if in this case you have commits in develop branch?

commented

thanks for the quick reply!

In your case you fixed the first package and it ran the corresponding workflow, but the second package is not touched as there are no code changes.

  1. I created a new branch
  2. made changes to 2 packages
  3. commit
  4. CI found two errors
  5. made changes to 1 package to fix error
  6. commit
  7. CI found no errors b/c last commit only changed 1 package and left the other package from step 2 untouched

If you'd compare against parent branch, step 7 would trigger job for both packages and still find that 1 has errors.

Supporting custom dependencies between packages

It's more: if you change multiple packages in one PR, there are some dependencies - CI doesn't have to find them, the developer tells you about it b/c they changed files in more than one package (I think that this is one of the biggest benefits of a monorepo).

Checking always against parent branch might not be an optimal solution. What if in this case you have commits in develop branch?

Should it be last commit on parent-branch or last commit on parent branch before creating branch?

plus: PR should ask for a merge of master before allowing to merge the branch into master anyway, right?!

commented

Should it be last commit on parent-branch or last commit on parent branch before creating branch?

Suppose I create a feature branch and apply a refactoring over all packages (like renaming some variables, or doing some lint fixes.) . I commit the changes, CI will be triggered for all packages. Even if all pass, after that if I continue working in this branch and apply another small commit in only one package, then according to your suggestion it will still build all packages.
That doesn't sound very optimal.

Ideally it should run CI only for packages that have changes or have a fail status in current branch.

commented

@pkyeck I've done few adjustments in branch feature/include_failed.

What changed: I use the CircleCI API to get last 100 builds in current branch, group them by workflow names and take those for which the last build failed. I then include also these packages to the ones with changes.
I then tried to reproduce your exact use case to see if it works:

  • I created a feature branch, feaure/include_failed
  • Committed changes in api and app (#21)
  • Both api and app failed (#22 and #23)
  • I committed a change in app package (#24)
  • CI triggered for both, api and app (#25...#28 )

It also tells you if the package was included because the CI is in failed state:
image

See the CircleCI pipeline:
https://circleci.com/gh/labs42io/circleci-monorepo/tree/feature/include_failed
https://circleci.com/gh/labs42io/circleci-monorepo/24

Current limitation is that CircleCI API returns only up to 100 builds (jobs). There is possibility for pagination, but this is not handled in current script.

Could you please check if the adjusted script works for you?

commented

Ideally it should run CI only for packages that have changes or have a fail status in current branch.

That would be ideal.

The update looks good - will take a look whenever I have the time to play around with it some more. Thanks!