How to clone all repos at once from GitHub?

Question

I have a company GitHub account and I want to back up all of the repositories within, accounting for anything new that might get created for purposes of automation. I was hoping something like this:

git clone [email protected]:company/*.git 

or similar would work, but it doesn't seem to like the wildcard there.

Is there a way in Git to clone and then pull everything assuming one has the appropriate permissions?

Answer 1

On Windows and all UNIX/LINUX systems, using Git Bash or any other Terminal, replace YOURUSERNAME by your username and use:

CNTX={users|orgs}; NAME={username|orgname}; PAGE=1
curl "https://api.github.com/$CNTX/$NAME/repos?page=$PAGE&per_page=100" |
  grep -e 'clone_url*' |
  cut -d \" -f 4 |
  xargs -L1 git clone

• Set CNTX=users and NAME=yourusername, to download all your repositories.
• Set CNTX=orgs and NAME=yourorgname, to download all repositories of your organization.

The maximum page-size is 100, so you have to call this several times with the right page number to get all your repositories (set PAGE to the desired page number you want to download).

Here is a shell script that does the above: https://gist.github.com/erdincay/4f1d2e092c50e78ae1ffa39d13fa404e

Answer 2

Simple script using GitHub CLI (no API keys)

Here's a simple solution using the official GitHub CLI tool, gh - no need for API keys and can handle any number of private repos.

First time only: login with gh for private repos, and follow prompts:

gh auth login

This will clone any number of repos under a new ./myorgname folder. Replace myorgname with your org name:

gh repo list myorgname --limit 1000 | while read -r repo _; do
  gh repo clone "$repo" "$repo"
done

Setup

To get the GitHub CLI tool:

• Mac - brew install gh
• Linux or Windows - see GitHub install guide

The GitHub CLI tool will be supported long-term as and when the GitHub API changes.

Optional: update existing checkouts

To update repo folders already on disk, as well as cloning new repos, the script needs to check for failure of the gh repo clone, like this:

gh repo list myorgname --limit 1000 | while read -r repo _; do
  gh repo clone "$repo" "$repo" -- -q 2>/dev/null || (
    cd "$repo"
    # Handle case where local checkout is on a non-main/master branch
    # - ignore checkout errors because some repos may have zero commits, 
    # so no main or master
    git checkout -q main 2>/dev/null || true
    git checkout -q master 2>/dev/null || true
    git pull -q
  )
done

Tips

• Don't want to create repos in the ./myorgname folder? Drop the second "$repo" argument to gh repo clone to create them in current directory
• More than 1000 repos? Increase the --limit parameter

Useful options:

• --no-archived - omit archived repositories
• --source - show only non-forks

Background

• GitHub CLI login doc
• Script commands above were derived from an issue comment and gist by davegallant

Answer 3

I don't think it's possible to do it that way. Your best bet is to find and loop through a list of an Organization's repositories using the API.

Try this:

• Create an API token by going to Account Settings -> Applications
• Make a call to: http://${GITHUB_BASE_URL}/api/v3/orgs/${ORG_NAME}/repos?access_token=${ACCESS_TOKEN}
• The response will be a JSON array of objects. Each object will include information about one of the repositories under that Organization. I think in your case, you'll be looking specifically for the ssh_url property.
• Then git clone each of those ssh_urls.

It's a little bit of extra work, but it's necessary for GitHub to have proper authentication.