AWS CodeBuild buildspec.yml get all files and subfolders recursively

Question

I'm trying to use AWS CodeBuild to get all files and subfolders inside a nested public folder and deploy to an S3 bucket using CodePipeline. I was able to hook them all together but struggling to configure the buildspec.yml file to get the output I want.

My folder structure:

<path>/public/

├── 404.html
├── css
│   ├── ...
├── fonts
│   ├── bootstrap
│   │   ├── ...
│   ├── icomoon
│   │   ├── icomoon
│   │   │   ├── ...
│   └── simple-line-icons
│       ├── ...
├── images
│   ├── ...
├── index.html
├── index.xml
├── js
│   ├── ...
└── tags
    └── index.xml

I need to put everything (including the subfolders) inside the public folder into the root of an S3 bucket.

So far I've tried following the docs here, here and here. I've tried using:

• **/* to get everything recursively inside the folder public but the S3 bucket will have the path to the folder so the index.html is not in the root.
• discard-paths: yes to remove the path to the public folder but inside the S3 bucket, all files are there in the root, no sub-folder structure is kept.
• base-directory: as described here.
• artifacts: secondary-artifacts: artifact1: files: - directory/file artifact2: files: - directory/file2 to keep the folder structure but my build failed.
• Different combinations of all the syntaxes above but my build just failed.

Answer 1

TL;DR You probably don't need to use "discard-paths: yes". So, drop "discard-paths: yes" and instead use "base-directory" with "**/*" for glob pattern. Read on for more reasoning why.

So yeah, as it usually happens, the culprit was pretty trivial. In my case the directory structure looked like this:

dist/
|-- favicon.ico
|-- index.html
|-- js
|   `-- app.js
`-- revision.txt

So in order for the artifact to contain everything from the "dist" directory whilst preserving the nested directories structure, the buildspec had to look like this:

artifacts:
  files:
    - '**/*'
  base-directory: 'dist'

Essentially "base-directory" parameter behaves as unix's "cd" command, so at first CodeBuild jumps to that directory and only after that resolves the glob pattern, in this case - **/*. If you also happen to use "discard-paths" then what happens is that, presumably, CodeBuild will change current directory to "dist", resolve the files structure correctly but then remove all path prefixes from the resolved paths, thus the result will be a plain directory structure with no nesting.

Hope that helps!

Answer 2

I've figured out a way to work around it. I don't think it's the best way but it works. I hope you can find this solution useful buildspec.yml:

version: 0.2

phases:
  build:
    commands:
      - mkdir build-output
      - cp -R <path>/public/ build-output
  post_build:
    commands:
      - mv build-output/**/* ./
      - mv build-output/* ./
      - rm -R build-output *.yml LICENSE README* .git*
artifacts:
  files:
    - '**/*'

In words:

• I copy everything inside the nested <path>/public folder out to a folder called ./build-output.

• Then in post_build I move everything out of folder build-output.

• Delete files pulled from my GitHub's repo which aren't needed to host the static website in the S3 bucket

• Then I get the result I want: all files inside public in the root of the S3 bucket with the right folder tree.

Update:

• You can also use my buildspec.yml example file here. Since the example is out of the scope of this question, I don't paste the code here.
• I explain in details here.