Improvements for managing untracked projects

[Joakim Andersson]

Whenever the manifest file is modified by removing entries in the manifest or by moving the manifest entry path the project folder may end up containing folders that are no longer needed or duplicated entries.

An example of this is the TF-M repository split, where the folder name was changed from tfm to tf-m.
Users doing west update after this change will end up with duplicated entries in their folder structure when the new project is imported by the old one still remains.

Usually what the user wants to do at this point is to remove the not-used project as it now has duplicated entries and this causes problems when searching in the projects for symbols.

Since the folders usually are git repositories cleaning this up should not be handled automatically by the tool as the user my still have branches in the old folder that they wish to clean up first.

Providing the user with the information about these old folders with by useful so that they can do the appropriate action to get rid of them.

Listing the untracked folders in a similar way as git does when doing west status would be useful.
West should skip folders that are inside existing projects, as this would otherwise be listed by git.

Example:

west status
=== status of manifest (zephyr):
On branch main
Your branch is up to date with 'upstream/main'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	build/

nothing added to commit but untracked files present (use "git add" to track)
=== 

Untracked folders:
	modules/tee/tfm

Add files to west.yml to track

This is very useful when doing bisect to find a bug, or checking out previous releases to reproduce reported bugs.
This could add older projects to the checkout that should be cleaned up afterwards.

[Marc Herbert]

Listing the untracked folders in a similar way as git does when doing west status would be useful.

This like a great idea to me, simple and useful. However we may need some kind of new .west[/]ignore. We typically put build folders in the west topdir; it's a very convenient place for project specific stuff that is not version controlled (or not yet).

Untracked folders:
modules/tee/tfm

I would keep it simple and report only the top level directories (e.g.: modules/ here). Maybe top level files too for consistency with git?

[Joakim Andersson]

I would keep it simple and report only the top level directories (e.g.: modules/ here). Maybe top level files too for consistency with git?

But modules/ and modules/tee contain projects that are tracked, i.e. modules/tee/tf-m/trusted-firmware-m for example in this case.

[Marc Herbert]

Sorry I wrote too fast. I meant: if there is a modules/foo/bar/ directory, then it should only report modules/foo/ but not modules/foo/bar to keep the output short (assuming there is no west project anywhere below modules/foo/).

Different git clients report the similar situation with different verbosity.

On the other hand, if there is a west project modules/foo/project/, then it should report modules/foo/bar/ (and obviously not modules/foo/)

So the pseudo-code should probably look like this:

def west_find_untracked(parent):
   for child in parent:
        if child is a west module:
             continue # nothing to report
        if west module(s) below child:
             west_find_untracked(child)
        else: # e.g.: zephyr_top/modules
             report child as untracked # unless westignored

west_find_untracked(west_top)

Nested projects should not matter: anything that git status --ignored can report already should not be reported a second time by west.