Automatically copy pushed files from one GitHub repository to another

Question

I have two GitHub repositories.

I'd like to automatically (probably using hooks and/or github API) commit and push files to the second repository when they are pushed to the first one.

The second repository is not a clone of the first one, their folder layout is not necessarily the same there is just a bunch of files in common.

What is the simplest way for doing this?

Bonus points if I don't have to install an http server or learn perl :)

Answer 1

If you are looking for something robust and easy to maintain, I'd encourage you to develop a solution around GitHub Webhooks. Yes it will require you to deploy a HTTP server, say a Node.js server, and it will require a small amount of development (your requirements are fairly specific), but I think it will pay-off if you need something reliable and low-maintenance. That's if you decide this file-mirroring approach is still the right thing to do, having considered the approaches and the set-up effort.

Let the source repositories (on GitHub) be S1, S2 ... with (non-overlapping) file-sets to mirror F1, F2 ..., to be sent to a target repo T (also on GitHub), where the corresponding files are considered read-only. Your requirements are unusual in that Sn and T sound like they aren't cloned from each-other, they might not even have any common commit, in which case this is not a push/fetch scenario. You also haven't guaranteed that the source file-updates occur one-per-commit, or even grouped but isolated from non-replicating changes, so this is not about cherry-picking commits.

The trigger for replication is on push of certain files to S1, S2 ..., not a commit on any developer-clone of those repos, so client-side hooks won't help (and they might be awkward to maintain). GitHub doesn't allow generic hooks of course, so Webhooks are your best solution. You could consider another, polling clone which is regularly pulling from S1 ..., performing logic and then committing to T, but this sounds awkward compared to Webhooks, which will give you reliable delivery, replay capability, a decent audit-trail etc.

The upside is that there is a lot of already-built infrastructure to support this type of set-up, so the actual code you would have to write could be quite small. Say you go with a Node.js type setup:

• Deploy github-webhook-handler. This cool little library is a pre-built handler for GitHub Webhooks, and handles the HMAC X-Hub-Signature verification and provides simple event-listener hooks for all the Webhooks events. You could have one end-point per S, or it's probably easier to fan them in.
• Have some local file (keep it in a Git repo) which maps Sn to Fn.
• Register a handler for X-GitHub-Event: push, and inspect repository/name and commits[]/modified[] for paths matching your local map.
• Deploy node-github, an implementation of the GitHub APIv3 for Node.js.
• For each matching file:
  • Call getBlob to read a utf-8 or base64 copy of the file from Sn.
  • Call createBlob to re-create that file in T.
  • Make a series of calls to T to getReference (current commit), getTree, createTree (create a new one from the base and the new blob), createCommit and finally updateReference . That's one workflow - a lower-collision one would be to branch/merge.

This approach allows you to do everything without needing a local clone of T. You might find it better to use a local clone, I'd see how easy things go with the API method first.

Answer 2

We had a similar problem - we wanted to automatically copy documentation files between project's and common documentation's repositories. We've built tool that listens to GitHub's webhooks, parses commits and creates Pull Request to a chosen destination. We've open sourced it - https://github.com/livechat/copycat - it can be used on any node platform server.