How to embed file for later parsing execution use

Question

I am essentially trying to walk through a folder of html files. I want to embed them into the binary file and be able to parse them upon request for template execution purposes. (Please excuse me if im not wording this properly).

Any ideas, tips, tricks or better way of accomplishing this is much appreciated.

// Template Files
type TempFiles struct {
    Files map[string]string
}

// Loop through view files and load them
func LoadTempFiles() {
    t := new(TempFiles)

    // Load template files
    filepath.Walk("application/views", func(path string, info os.FileInfo, err error) error {
    if !info.IsDir() {
        content, _ := ioutil.ReadFile(path)
        t.Files[path] = string(content)
    }
    return nil
    })
}

func ViewTemp(w http.ResponseWriter, path string) {
    t := new(TempFiles)

    temp, err := template.New().Parse(t.Files[path])
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
    } else {
        temp.Execute(w, nil)
    }
}

Answer 1

I do this with most of my Go web apps. I use go-bindata to auto-generate Go source code from all the files I want to embed and then compile them into the binary. All this is done automatically during build.

One downside is that the current go build tools do not offer a way to hook into the build process, so I use a Makefile for this purpose. When the makefile is invoked, it runs go-bindata to generate the sources for all necessary files, then usually performs some additional code generation bits and bobs (notably, creating a Go source file which lists all the embedded files in a map.. A Table of Contents if you will). It then proceeds to compile the actual program.

This can become a little messy, but you only have to set it all up once. Another downside, is that the use of a Makefile means the software is not compatible with the go get command. But since most of my web apps are not meant to be shared anyway, this has not been a problem so far.

When it comes to debugging/developing such an application, there is another issue that arises from embedding the static web content: I can't just edit an HTML or CSS file and refresh the browser to see its effects. I would have to stop the server, rebuild it and restart it with every edit. This is obviously not ideal, so I split the Makefile up into a debug and release mode. The release mode does what I described above. The debug mode, however, wil not actually embed the static files. It does generate source files for each of them, but instead of having them contain the actual file data, it contains a stub which simply loads the data from the filesystem.

As far as the server code is concerned, there is no difference in the generated code. All it does is call a function to fetch the contents of a given static file. It does not care whether that content is actually embedded in the binary, or if it's loaded from an external source. So the two build modes are freely interchangeable.

For example, the same generated function to fetch static file content in release and debug mode would look as follows:

Release mode:

func index_html() []byte {
    return []byte {
        ....
    }
}

Debug mode:

func index_html() []byte {
   data, err := ioutil.ReadFile("index.html")
   ...
   return data
}

The interface in both cases is identical. This allows for easy and care-free development and debugging.

Answer 2

Another tool to consider: Another recent good tool comes from esc: Embedding Static Assets in Go (GitHub repo)

a program that:

• can take some directories and recursively embed all files in them in a way that was compatible with http.FileSystem
• can optionally be disabled for use with the local file system for local development
• will not change the output file on subsequent runs
• has reasonable-sized diffs when files changed
• is vendoring-friendly

Vendoring-friendly means that when I run godep or party, the static embed file will not change.
This means it must not have any third-party imports (since their import path will be rewritten during goimports, and thus different than what the tool itself produces), or a specifiable location for the needed third-party imports.

It generates nice, gzipped strings, one per file.
There is a simple flag to enable local development mode, which is smart enough to not strip directory prefixes off of filenames (an option in esc that is sometimes needed).
The output includes all needed code, and does not depend on any third-party libraries for compatibility with http.FileSystem.

Answer 3

I made a package that makes switching between debug and production easier. It also provides an http.FileSystem implementation, making it easy to server the files. And it has several ways of adding the files to the binary (generate go code, or append as zip). https://github.com/GeertJohan/go.rice

Answer 4

Go now has builtin support for this:

package main

import (
   "embed"
   "os"
)

//go:embed *.html
var content embed.FS

func main() {
   b, e := content.ReadFile("index.html")
   if e != nil {
      panic(e)
   }
   os.Stdout.Write(b)
}

https://golang.org/pkg/embed