http upload of huge files over slow network (interruptible)

Question

I want to implement a client+server which allows uploading of big files over very slow and faulty networks.

This means the upload needs to be interruptible.

Example: If 80% of the upload data was already transferred, then (after the tcp connection was lost and created again) the second request should only transfer the missing 20%.

In my case the client-server communication needs to use https.

An upload can last up 12 hours.

Client and server will be implemented with Python.

Of course I could invent my own protocol on top of http. I guess this would be simple.

But, I would like to implement a standard/spec (if there is any).

What does the http spec already provide which can help to implement this?

Answer 1

There is already a spec for this: https://tus.io/

Quoting above url:

People are sharing more and more photos and videos every day. Mobile networks remain fragile however. Platform APIs are also often a mess and every project builds its own file uploader. There are a thousand one-week projects that barely work, when all we need is one real project. One project done right.

We are the ones who are going to do this right. Our aim is to solve the problem of unreliable file uploads once and for all. tus is a new open protocol for resumable uploads built on HTTP. It offers simple, cheap and reusable stacks for clients and servers. It supports any language, any platform and any network.

It may seem to be an impossible dream. Perhaps that is because no-one has managed to solve it yet. Still, we are confident and we are going to give it our best shot. Join us on GitHub and help us make the world a better place. Say "No!" to lost cat videos! Say "Yes!" to tus!