I am trying to put a file into a Google Cloud Storage (GCS) bucket from the command line. At a later stage this shall be used in a deployed script at the user end without any type of user-visible authentication.
So far I generate a signed url like this:
gsutil signurl -p notasecret -m PUT -d 1d myserviceaccount.p12 gs://mybucket/testfile
which will generate something like
https://storage.googleapis.com/mybucket/testfile?GoogleAccessId=myserviceaccount@developer.gserviceaccount.com&Expires=1430963040&Signature=gMf2h95bNmolizUGYrsQ%2F%2F%2FiHxW14I%2F0EOU3ZSFWtfCwNqSyok3iweQiuPxYXH4b26FeDSrmFOXB58%2B%2B%2BiAOJ%2B1gdLC9Y%2BkeUdbrjH0eGTW0NVsM1AWY2LsQ3dYf5Ho%2Bos1Fk26EsLJlD096Ku9aWqLW%2FpL%2FBSsUIfHijrFJPdI%3D
The next step (at the user end) would be curl
uploading the file with a PUT request. Like so:
curl -X PUT --data-binary @testfile 'https://storage.googleapis.com/mybucket/testfile?GoogleAccessId=myserviceaccount@developer.gserviceaccount.com&Expires=1430963040&Signature=gMf2h95bNmolizUGYrsQ%2F%2F%2FiHxW14I%2F0EOU3ZSFWtfCwNqSyok3iweQiuPxYXH4b26FeDSrmFOXB58%2B%2B%2BiAOJ%2B1gdLC9Y%2BkeUdbrjH0eGTW0NVsM1AWY2LsQ3dYf5Ho%2Bos1Fk26EsLJlD096Ku9aWqLW%2FpL%2FBSsUIfHijrFJPdI%3D'
I can get this to work with an existing file in the bucket and a GET request (for downloading), but it does not seem to work for uploading. curl
throws the server's response with error messages like this at me:
<?xml version='1.0' encoding='UTF-8'?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided.
Check your Google secret key and signing method.</Message>
<StringToSign>PUT
application/x-www-form-urlencoded
1430963040
/mybucket/testfile</StringToSign>
</Error>
And this makes sense to me, as obviously I am not just making a bare PUT request, but one for a particular file of a specific size, whereas the signature computed by 'gsutil signurl' would not know about these details at the time it is computed.
Somehow I was under the impression (e.g., based on the last usage case described in gsutil signurl
documentation and also in the post How to allow anonymous uploads to cloud storage) that it should be possible to generate a generic signed url for uploading purposes and then use it later. Am I just mistaken about this point or is there a way to fix the curl
request?
Any thoughts about this are appreciated. However, I'd like this to work with "minimal tools", i.e., ideally shell and curl only, but no other programming languages.
EDIT: Organising one's thoughts by formulating the exact problem is the first step towards the solution. I realise now that
curl -X PUT -T - [request-url] < testfile
does actually solve the immediate problem. However, this means multiple users would write to the same file if they use the same signed url. The documentation suggests you can omit the object name in the creation of the signed url, i.e., use
gsutil signurl -p notasecret -m PUT -d 1d myserviceaccount.p12 gs://mybucket/
This, supposedly, would allow anyone with the resulting signed url to put any object of any type into my bucket. Only I do not get this work, as I don't see how you can then tell GCS which object you are actually writing to.
Cloud Storage provides the Signed URL feature to let individual end users perform specific actions. Signed URL makes it possible to generate temporary credentials valid only for a specific end user to securely upload a file. The Google Cloud official client library makes it easy to generate a Signed URL.
Overview. A signed URL is a URL that provides limited permission and time to make a request. Signed URLs contain authentication information in their query string, allowing users without credentials to perform specific actions on a resource.
To create a valid pre-signed URL for your object, you must provide your security credentials, specify a bucket name, an object key, specify the HTTP method (for instance the method is "GET" to download the object) and expiration date and time. Anyone who receives the pre-signed URL can then access the object.
This was driving me mad too. Turns out it was the binary-file
part of the curl command. Try this instead:
curl -X PUT --upload-file me.jpeg $SIGNED_URL
If the resource does not specify a single object, you can do so on an individual basis by adding a URL param to the request with the name of the object. For example:
curl -X PUT -T - [request-url]?name=[object-name] < testfile
This surely works with storage/v1
, although I have not tried myself with a signed URL yet.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With