git status takes too long

Question

I'm working on a project where the version control system is SVN and I want to use git. I did a git svn clone but git status works terribly slow (around 8 minutes). The repository has around 63000 files and most of them are libraries ignored by git. Is this normal? I did a git prune && git gc to perform a cleanup of unreachable objects and a garbage collect. I also did a git repack -Adf but this made things even worse. It takes even longer(more than 20 minutes).

What am I doing wrong? This is a visual studio project and I assume that the .gitignore file does not contain the right things. Is it possible to find out exactly which files are generated from a visual studio build and which have to be versioned?

If the .gitignore file is not the problem, how can I make my git status faster, is it normal for a project with 65000 files (around 10GB) to work that slow with git?

Answer 1

For a repository of that size, git status and associated commands can be very slow. Git works much better when projects are teased apart and separated, while Subversion tends to encourage using single behemoth repositories containing multiple projects, so this sort of problem isn't uncommon when using Git-SVN.

Nonetheless, there're a few different solutions you can use to speed things up:

• If you haven't already, upgrade to using a solid state disk rather than a magnetic disk. This single change made a massive difference to Git's speed when I was working on a similar repository

• Look at the Configuration section of git help svn. That describes setting up Git-SVN to use track subfolders in the Subversion repository (eg trunk/project-a, branches/*/project-a, tags/*/project-a, …) rather than the entire repository. If this makes sense for your repository, it'll mean you can have much smaller checkouts and so much faster runs of git status.

• Look at the Sparse Checkout section of git help read-tree. That'll talk you through setting up Git to use a sparse working copy, similar to a Subversion sparse checkout. Again, this means that there'll be fewer files Git's tracking in your working copy, and hence checking them all will again be quicker.

• Consider setting the "assume unchanged" flag on large sections of your working copy. This will tell Git to not bother checking if the files have changed. There's two ways of doing this:

  1. To set the flag for specific folders, run something like the following:

     find <folder-name>... -type f -exec git update-index --assume-unchanged {} +
     

  2. To set the flag for the entire repository (note this will lose uncommitted changes):

     git config core.ignorestat true
     git reset --hard HEAD
     

  Take a look at the --assume-unchanged option in git help update-index and the config.ignoreStat section in git help config for some more information about how these work.

  Using these will mean you need to specify paths to commands like git diff and git add explicitly, ie commands like a bare git diff, git commit -a &c won't work.

• Change your operating system and/or file system. According to the Git man pages (the same ones as in the previous bullet), Windows' lstat is slow, as is the CIFS file system. I suspect the ideal is something like ext3 or ext4 on Linux or some other *nix.

Answer 2

Since 2013, you now (2020) have a dedicated git sparse-checkout which can help clone only the relevant subset of a Git repository.

Not only git status will be faster, but, with Git 2.28 (Q3 2020), "git status" learned to report the status of sparse checkout.

See commit afda36d, commit 30b00f0 (21 Jun 2020), and commit 051df3c (18 Jun 2020) by Elijah Newren (newren).
^{(Merged by Junio C Hamano -- gitster -- in commit 0cc4dca, 06 Jul 2020)}

wt-status: show sparse checkout status as well

^{Signed-off-by: Elijah Newren}

Some of the early feedback of folks trying out sparse-checkouts at $dayjob is that sparse checkouts can sometimes be disorienting; users can forget that they had a sparse-checkout and then wonder where files went.

Add some output to 'git status' in the form of a simple line that states:

You are in a sparse checkout with 35% of files present.  

where, obviously, the exact figure changes depending on what percentage of files from the index do not have the SKIP_WORKTREE bit set.

---

Note: The bash prompt script (in contrib/) did not work under "set -u", which has been fixed with Git 2.32 (Q2 2021).

See commit 5c0cbdb (13 May 2021) by Elijah Newren (newren).
^{(Merged by Junio C Hamano -- gitster -- in commit 02112fc, 20 May 2021)}

git-prompt: work under set -u

^{Signed-off-by: Elijah Newren}

Commit afda36d ("git-prompt: include sparsity state as well", 2020-06-21, Git v2.28.0-rc0 -- merge listed in batch #7) added the use of some variables to control how to show sparsity state in the git prompt, but implicitly assumed that undefined variables would be treated as the empty string.

This breaks users who run under 'set -u'; fix the code to be more explicit.