git-bisect, but for N repos

Question

Our software is modular and I have about 20 git repos in one project.

If a test fails, it is sometimes hard to find the matching commit since several developers work on these 20 repos.

I know the test worked yesterday and fails reproachable today.

Sometimes I use git-bisec, but this works only for one git repo.

Often changes in two git repos make a test fail.

I could write a dirty script which loops over my N git repos myself, but before doing so, I would like to know how experts would solve this.

I use Python, Django and pytest, but AFAIK this does not matter for this question.

Answer 1

I personally prefer to use repo tool to manage complex projects. Put those 20 repos in manifest.xml and each time when build starts create patch manifest if build fails do repo diff manifests to see what was changed and where.

Answer 2

There is category of QA tool for "reverse dependency" CI builds. So your higher level projects get rebuilt every time a lower level change is made. At scale it can be resource intensive.

The entire class of problem is removed if you stop dealing with repo-to-repo relationships and start following version release methodology for the subcomponents. Then you can track the versions of lower-level dependencies and know when you go to upgrade that it broke. Your CI could build against several versions of dependencies if you wanted to systematize it.

Git submodules accomplish that tracking for individual commits, so you again get to decide when to incorporate changes from lower levels. (Notably, that can also be used like released versions if you only ever update to tagged release commits.)