Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Error "No URLs matched" When copying Google cloud bucket data to my local computer?

I am trying to download a folder which is inside my Google Cloud Bucket, I read from google docs gsutil/commands/cp and executed below the line.

gsutil cp -r appengine.googleapis.com gs://my-bucket

But i am getting the error

CommandException: No URLs matched: appengine.googleapis.com

Edit

By running below command

gsutil cp -r gs://logsnotimelimit .

I am getting Error

IOError: [Errno 22] invalid mode ('ab') or filename: u'.\logsnotimelimit\appengine.googleapis.com\nginx.request\2018\03\14\14:00:00_14:59:59_S0.json_.gstmp'

like image 584
Sudhanshu Gaur Avatar asked Mar 26 '18 12:03

Sudhanshu Gaur


People also ask

Which command line tool works with Google cloud storage buckets?

gsutil is a Python application that lets you access Cloud Storage from the command line. You can use gsutil to do a wide range of bucket and object management tasks, including: Creating and deleting buckets. Uploading, downloading, and deleting objects.

How do I access public Google Storage bucket?

In the Google Cloud console, go to the Cloud Storage Buckets page. Click on the name of the bucket that contains the object you want to make public, and navigate to the object if it's in a subdirectory.


3 Answers

What is the appengine.googleapis.com parameter in your command? Is that a local directory on your filesystem you are trying to copy to the cloud bucket?

The gsutil cp -r appengine.googleapis.com gs://my-bucket command you provided will copy a local directory named appengine.googleapis.com recursively to your cloud bucket named my-bucket. If that's not what you are doing - you need to construct your command differently.

I.e. to download a directory named folder from your cloud bucket named my-bucket into the current location try running gsutil cp -r gs://my-bucket/folder .

-- Update: Since it appears that you're using a Windows machine (the "\" directory separators instead of "/" in the error message) and since the filenames contain the ":" character - the cp command will end up failing when creating those files with the error message you're seeing.

like image 147
Mihail Russu Avatar answered Sep 21 '22 21:09

Mihail Russu


Just wanted to help people out if they run into this problem on Windows. As administrator:

  • Open C:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\platform\gsutil\gslib\utils
  • Delete copy_helper.pyc
  • Change the permissions for copy_helper.py to allow writing
  • Open copy_helper.py
  • Go to the function _GetDownloadFile
  • On line 2312 (at time of writing), change the following line
download_file_name = _GetDownloadTempFileName(dst_url)

to (for example, objective is to remove the colons):

download_file_name = _GetDownloadTempFileName(dst_url).replace(':', '-')
  • Go to the function _ValidateAndCompleteDownload
  • On line 3184 (at time of writing), change the following line
final_file_name = dst_url.object_name

to (for example, objective is to remove the colons):

final_file_name = dst_url.object_name.replace(':', '-')
  • Save the file, and rerun the gsutil command
  • FYI, I was using the command gsutil -m cp -r gs://my-bucket/* . to download all my logs, which by default contain : which does not bode well for Windows files!

Hope this helps someone, I know it's a somewhat hacky solution, but seeing as you never need (should have) colons in Windows filenames, it's fine to do and forget. Just remember that if you update the Google SDK you'll have to redo this.

like image 44
Eugene Avatar answered Sep 24 '22 21:09

Eugene


I got same issue and resolved it as below.

  1. Open a cloud shell, and copy objects by using gsutil command.

gsutil -m cp -r gs://[some bucket]/[object] .

  1. On the shell, zip those objects by using zip command.

zip [some file name].zip -r [some name of your specific folder]

  1. On the shell, copy the zip file into GCS by using gsutil command.

gsutil cp [some file name].zip gs://[some bucket] .

  1. On a Windows Command Prompt, copy the zip file in GCS by using gsutil command.

gsutil cp gs://[some bucket]/[some file name].zip .

I wish this information helps someone.

like image 25
Delta Avatar answered Sep 22 '22 21:09

Delta