Package.xml only has existing modified files, missing newly created files

[Matt Farrell]

Issue verification check:

• is the current repository fully deployable at the commit SHA provided with the 'from' parameter of the command?

When I create a PR from fromBranch to toBranch I can see over 1k commits and 2k files changed. Not sure if there is any limits to sfdx-git-delta?

What is the problem?

We are testing changing our CICD process to include sfdx-git-delta and facing issues with a large PR not including newly created files, only modified files.

What is the parameter and the value you used with it?

In this testing the fromBranch and toBranch are being provided by the user.

sf sgd:source:delta --from origin/${{ github.event.inputs.fromBranch}} --to origin/${{ github.event.inputs.toBranch }} --output .

What is the expected result?

Expect it to create a package.xml file with all created and modified files listed.
As an example I will refer to flows (although it happens across all objects/created files), we can see there is 2 modified flows, and n newly created files.

What is the actual result?

It is only adding modified files to package.xml, not newly created files.

<types>
    <members>Account_Create_DSR</members>
    <members>AM_Handoff</members>
    <name>Flow</name>
</types>

Steps to reproduce

I'm not entirely sure how to share a reproduction as I am unable to share the code due to workplace security. Full Github Workflow below.

Execution context

OS: ubuntu-latest
Node v18.18.2
Git: 2.42.0
SFDX: @salesforce/cli/2.14.6 linux-arm64 node-v18.18.2
sfdx-git-delta 5.27.0 (latest-rc)

More information (optional)

This is a very large difference, over 1k commits a 2k modified files.

Running git diff --name-status --no-renames origin/fromBranch origin/toBranch correctly lists all files.

Git Workflow below.

name: Manual Deployment

on:
    workflow_dispatch:
        inputs:
            fromBranch:
                description: 'FROM Branch'
                required: true
            toBranch:
                description: 'TO Branch'
                required: true
            toOrg:
                description: 'TO Org'
                required: true

jobs:
    validate:
        runs-on: ubuntu-latest
        steps:
            - name: 'Checkout source code'
              uses: actions/checkout@v4
              with:
                fetch-depth: 0

            - name: 'Determine org'
              run: |
                case "${{ github.event.inputs.toOrg }}" in
                    "QA") echo ${{ secrets.QA_SFDX_AUTH_URL }} > ./auth.txt ;;
                    "UAT") echo ${{ secrets.UAT_SFDX_AUTH_URL }} > ./auth.txt ;;
                    "PROD") echo ${{ secrets.PROD_SFDX_AUTH_URL }} > ./auth.txt ;;
                    *) exit 1 ;;
                esac
            
            - name: 'Install npm dependencies'
              run: |
                npm install @salesforce/cli --global --silent
                npm install --save-dev @salesforce/eslint-config-lwc @lwc/eslint-plugin-lwc @salesforce/eslint-plugin-lightning eslint-plugin-import eslint-plugin-jest --silent
                sf plugins install @salesforce/sfdx-scanner 2>/dev/null
                echo y | sf plugins install sfdx-git-delta@latest-rc 2>/dev/null
            
            - name: 'Authenticate with SF'
              run: sf org login sfdx-url -f auth.txt -a sfOrg -d

            - name: 'Create package.xml'
              run: sf sgd:source:delta --from origin/${{ github.event.inputs.fromBranch}} --to origin/${{ github.event.inputs.toBranch }} --output .
            
            - name: 'Log Package.xml'
              run: |
                ls package
                cat package/package.xml

            # - name: 'Deploy'
            #   run: sf project deploy start --dry-run -g -o sfOrg -x package/package.xml 

If there is any further information I can share please let me know.

[Sebastien]

Hi @MattFaz !

Thanks for raising this issue and thanks for contributing in making this project better!

Investigation

Interesting issue you have here. I do not see any issue with the pipeline configuration. It uses fetch-depth: 0 so it gets everything.

Do you reproduce the issue on your laptop ?
Is there any way I could have access to the repository to reproduce the issue ?
If not, could you send metrics about the diff ?

• how many commits are between from and to sha ? git log origin/fromBranch..origin/toBranch --oneline | wc -l
• how many files are involved ?
  • Added: git diff --name-status --no-renames origin/fromBranch origin/toBranch | grep A | wc -l
  • Deleted: git diff --name-status --no-renames origin/fromBranch origin/toBranch | grep D | wc -l
  • Modified: git diff --name-status --no-renames origin/fromBranch origin/toBranch | grep M | wc -l

I suspect there may be a max buffer size issue hiding somewhere or a memory/heap size limit reached related to CI context.

To unblock you during the investigation and potential fix

It seems to be the first deployment as you wrote it "lists all files". If it is the case, it could be a good idea to do an initial full deployment (as the delta will result in full) and then do incremental deployment starting from there.

[Matt Farrell]

Thanks for the fast reply @scolladon

Do you reproduce the issue on your laptop ?

When developing/testing github workflows we use Act to simulate the workflow running on Github. I can confirm I get the same behavior on my local machine using Act.

Is there any way I could have access to the repository to reproduce the issue ?

Unfortunately not as it's within our organization, I'd be happy to show you over a google meet?
[e-mail] or if there is some method I can use to contact you?

If not, could you send metrics about the diff ?

Absolutely, please see the output of those commands below.

• git log origin/fromBranch..origin/toBranch --oneline | wc -l: 1120 commits
• git diff --name-status --no-renames origin/fromBranch origin/toBranch | grep A | wc -l: 1644 added files
• git diff --name-status --no-renames origin/fromBranch origin/toBranch | grep D | wc -l: 2110 deleted files
• git diff --name-status --no-renames origin/fromBranch origin/toBranch | grep M | wc -l: 1176 modified files

I will say that those deleted numbers don't look completely correct, but perhaps that's due to our process of cherry-picking.

I suspect there may be a max buffer size issue hiding somewhere or a memory/heap size limit reached related to CI context.

Yes that is my suspicion as well, but I wouldn't know where to begin on solving it 😅

It seems to be the first deployment as you wrote it "lists all files". If it is the case, it could be a good idea to do an initial full deployment (as the delta will result in full) and then do incremental deployment starting from there.

It's not our first deployment, we've actually done thousands but this is part of a migration project and has made this significantly larger than normal. Generally we would have < 25 commits and <100 files changed. This deployment is unusually large, however we want to ensure our CICD process handles it no matter the size.

[Sebastien]

Thank you very much for those information @MattFaz.

Could you try on your laptop without Act ? Just using the plugin directly from the terminal please. Let me know the result.

Can I schedule something on Friday ? I'm on CET timezone, what time would be ok for you ?

[Matt Farrell]

Could you try on your laptop without Act ? Just using the plugin directly from the terminal please. Let me know the result.

Exactly the same result running the plugin straight from MacOS terminal.

Can I schedule something on Friday ? I'm on CET timezone, what time would be ok for you ?

I'm AEDT timezone, so we don't have very good overlap 🙁
Additionally my Monday/Tuesday are public holidays, if you want to send me an invite that works for you for tomorrow or sometime next week, I can just propose a new time if it doesn't work 🙂

[Sebastien]

Thank you very much for your help and reactivity @MattFaz , really helpful.

I sent you a meeting invitation for tomorrow, it is early for me and late for you... If it does not work let's schedule something next week.

I'll try to reproduce the issue locally this week-end, starting from the metrics you shared.
I'll let you know the result.

[Matt Farrell]

Thank you so much for the call & your help @scolladon
Turns out it was a misunderstanding on the --from and --to branches!
I was thinking of it as 'move changes from x branch to y branch' (i.e. from Release to Prod), when it’s actually 'from this current state, to new state' (i.e. from Prod, to Release)

Really happy, thank you :)