Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Process Uploaded file on web server without storing locally first?

I am trying to process the user uploaded file real time on the websever, but it seems, APACHE invokes PHP, only once complete file is uploaded.

When i uploaded the file using CURL, and set

Transfer-Encoding : "Chunked"

I had some success, but can't do same thing via browser.

  • I used Dropzone.js but when i tried to set same header, it said Transfer -Encoding is an unsafe header, hence not setting it.

This answer explains what is the issue there. Can't set Transfer-Encoding :"Chunked from Browser"

In a Nutshell problem is , when a user uploads the file to webserver, i want webserver to start processing it as soon as first byte is available. by process i mean, PIPING it to a Named Pipe.

Dont want 500mb first getting uploaded to a server, then start processing it.

But with current Webserver (APACHE - PHP), I cant seem to be able to accomplish it.

could someone please explain, what technology stack or workarounds to use, so that i can upload the large file via browser and start processing it, as soon as first byte is available.

like image 963
Himanshu97 Avatar asked Mar 21 '17 10:03

Himanshu97


People also ask

How do I upload files to a web server?

Right-click the folder and select “Upload other file here. . .“. Browse the server for the file you want to upload. Select the file and click Open. Now, you will see the file in the folder location on the server.

What danger is there in allowing uploads over the web?

Upload forms on web pages can be dangerous because they allow attackers to upload malicious code to the web server. Attackers can then use tricks to execute this code and access sensitive information or even take control of the server.

What is file upload Bypass?

File upload bypass. File upload mechanisms are very common on websites, but sometimes have poor validation. This allows attackers to upload malicious files to the web server, which can then be executed by other users or the server itself.

How do I upload my files to the server?

You need to navigate to the specified directory in your right pane. This directory is effectively the root of your website — where your index.htmlfile and other assets will go. Once you've found the correct remote directory to put your files in, to upload your files to the server you need to drag-and-drop them from the left pane to the right pane.

Can I download web files to cloud storage without downloading them?

However, most of the users download the files they find on the web to their PC for mobile devices, and then they upload them to the cloud server. There is no way to save web files directly to cloud storage without downloading them.

How to upload files to cloud storage?

Authenticate with your cloud storage account (Google Drive, DropBox or Box.com). Then enter the file URL to save and then click the upload button. It is one of the widely used cloud saving platforms.

How do I push files to a server?

Learn how to push files to a server using the various file transfer tools available. Summary If you have built a simple web page (see HTML basicsfor an example), you will probably want to put it online, on a web server. In this article we'll discuss how to do that, using various available options such as SFTP clients, RSync and GitHub. SFTP


3 Answers

It is possible to use NodeJS/Multiparty to do that. Here they have an example of a direct upload to Amazon S3. This is the form, which sets content type to multipart/form-data. And here is the function for form parts processing. part parameter is of type ReadableStream, which will allow per-chunk processing of the input using data event.

More on readable streams in node js is here.

like image 61
Karatheodory Avatar answered Oct 20 '22 17:10

Karatheodory


If you really want that (sorry don`t think thats a good idea) you should try looking for a FUSE Filesystem which does your job.

Maybe there is already one https://github.com/libfuse/libfuse/wiki/Filesystems

Or you should write your own.

But remember as soon as the upload is completed and the post script finishes his job the temp file will be deleted

like image 25
Jean Fiedler Avatar answered Oct 20 '22 17:10

Jean Fiedler


you can upload file with html5 resumable upload tools (like Resumable.js) and process uploaded parts as soon as they received.

or as a workaround , you may find the path of uploaded file (usually in /tmp) and then write a background job to stream it to 3rd app. it may be harder.

there may be other solutions...

like image 28
Ehsan Chavoshi Avatar answered Oct 20 '22 17:10

Ehsan Chavoshi