OSInet

Bundling templates with embed

howto embed 2021-02-16 - 6 min. to read

Table of contents

Resource embedding tools like embed allow developers to include static assets and other unpublished files like templates in their executable programs.

Let’s see how.

Why would I use a bundler like embed ?

One of the reasons why Go is so popular is the ease of deployment of its applications: just copy the compiled binary and you’re done1.

However, as soon as one needs to copy a bunch of non-code files like CSS, JS, or Go templates, that advantages disappears unless one finds a way to embed them in the executable.

To that purpose, over the years, developers from the Go community have created various packages, amongst which rakyll/statik, markbates/pkger, and many others.

Facing the multiplicity of such solutions, and their varying levels of efficiency and ease of use, the Go project eventually chose to include such a feature within the language as its standard library, as the //go:embed directive and the embed package.

In an earlier article, we saw how to bundle templates with pkger. The news embed mechanism is the conclusion of ticket Github logo35950 , already mentioned in that article, which is still relevant regarding the motivations for embedding mechanisms.

Let us therefore see how to use embed instead of markbates/pkger for the exact same example used in the previous article, to illustrate how to convert from one approach to the other one.

How can I use embed in my projects ?

Unlike pkger and various other embedding packages:

  • the new mechanism requires no external dependency, just depending on Go 1.16 or newer
  • there is no difference between a development and a production build, at least when it comes to assets embedding
  • no API is needed to load a single resource like a template file
  • the API to load a resource tree is made of only 3 methods

Once compiled, the resulting project has no remaining runtime dependency on pkger or the original resources.

For embed to include resources in the bundle (executable, library, plugin), the module sources must be adapted. Let us see how.

The code examples in this post are excerpted from the github.com/fgm/pkger_demo V21, which is ready to compile. You can check it out to follow the explanations in your own IDE:

$ GO111MODULE=off go get github.com/fgm/pkger_demo
$ cd $GOPATH/src/github.com/fgm/pkger_demo/v2

Loading an isolated resource

All it takes to load an isolated resource (a single file) with embed is declaring a string or []byte variable in package scope, labeling it with the //go:embed directive.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

package main

import _ "embed"

//go:embed hello.txt
var hello string

func main() {
	println(hello)
}

The embed API: embed.FS

To load resources from an embedded resource tree, and load files from it selectively, the embedding directive remains unchanged, e.g. //go:embed templates, but the variable type must be embed.FS.

This new type was introduced in Go 1.16 and implements 3 new interfaces belonging to the new io/fs package of the standard library, as a result of the file I/O rationalization in that version.

Methodembed.FSfs.FSfs.ReadDirFSfs.ReadFileFS
Open(name string) (fs.File, error)XXvia fs.FSvia fs.FS
ReadDir(name string) ([]fs.DirEntry, error)XX
ReadFile(name string) ([]byte, error)XX

This API is simpler than the one provided by earlier contributed packages, as it allows using standard library functions to access resources listed in a //go:embed directive.

These embedding directives may include a * wildcard to embed multiple files, with just one pattern, and there can be multiple names or patterns, either space-separated on a single directive line, or on adjacent //go:embed lines, with the latter being more readable and easier to maintain over time in version control.

All files thus listed in a single //go:embed block will be automatically loaded in the embed.FS variable, under their respective relative path.

Loading templates from a directory subtree

The pkger_demo V2 demo program is a basic Web application, which returns a page formatted by two templates, as is common practice.

templates/page.gohtml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

{{ define "page" }}
{{- /*gotype: github.com/fgm/pkger_demo/v2.PageData*/ -}}
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>{{ .Path }} callback</title>
</head>
<body>
  <p>Called on {{ .Path }}</p>
  {{ template "layout/footer" . }}
</body>
</html>
{{ end }}

templates/layout/footer.gohtml

1
2
3
4
5
6

{{define "layout/footer"}}
{{- /*gotype: github.com/fgm/pkger_demo/v2.PageData*/ -}}
  <footer>
    &copy; {{ .Year }} Frederic G. MARAND for OSInet
  </footer>
{{end}}

Let us look into the main.go file salient bits. Lines 1-13 contain the program preamble (package, import), importing the embed package referenced further down. Nothing interesting there.

Lines 14-16 show how to embed an isolated resource as a string variable:

14
15
16

// To embed a random file, just go:embed it as a string or []byte
//go:embed hello.txt
var hello string

While lines 18-20 show how to embed multiple resources as a subtree, automatically loaded in an embed.FS variable.

18
19
20

// To use an embed hierarchy, use go:embed with an embed.FS.
//go:embed templates
var templates embed.FS

The program loads templates it finds in the /templates directory (the path is as provided in the //go:embed directives), but also in all its subdirectories. To discover them, it will have to crawl the substree starting at /templates, and parse all discovered templates:

28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

func compileTemplates(dir string) (*template.Template, error) {
	const fun = "compileTemplates"
	tpl := template.New("")
	// Since filepath.Walk only handles filesystem directories, we use the new
	// and optimized fs.WalkDir introduced in Go 1.16, which takes an fs.FS.
	err := fs.WalkDir(templates, dir, func(path string, info fs.DirEntry, err error) error {
		// Skip non-templates.
		if info.IsDir() || !strings.HasSuffix(path, ".gohtml") {
			return nil
		}
		// Load file from embed virtual file, or use the shortcut
		// templates.ReadFile(path).
		f, _ := templates.Open(path)
		// Now read it.
		sl, _ := io.ReadAll(f)
		// It can now be parsed as a string.
		tpl.Parse(string(sl))
		return nil
	})
	return tpl, err
}
  • line 33: this works much like filepath.Walk: the WalkDir function walks the subtree, invoking the walkFn callback on each resource met. In this example, the callback is the anonymous function at lines 33-46.
  • lines 34-37: the function ignores non-files, and files not matching the expected template name format.
  • lige 40, it opens the file using templates.Open, getting a standard fs.File, which happens to be an io.Reader implementation.
  • line 42, since f is an io.Reader the function reads its content using the standard io.ReadAll function. These two calls on lines 40-42 could also be replaced by a single templates.ReadFile() call.
  • ligne 44, with the result being a plain string, the function can now parse it as a Go template.

The final result of the function is a valid set of compiled templates, or an error value.

Once the program obtains the compiled templates at line 57, the HTTP request handlers can use the template without any notion of their loading mechanism:

59
60
61
62
63
64
65
66

	// Now serve pages
	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		ctx := PageData{Path: r.URL.Path, Year: time.Now().Local().Year()}
		tpl.ExecuteTemplate(w, "page", ctx)
	})
	log.Printf("Listening on %s", addr)
	log.Fatal(http.ListenAndServe(addr, nil))
}

Additional resources

  1. yes, there are some limitations regarding shared networks libraries, but they are not relevant in this context. ↩︎ ↩︎

Fullscreen text UIs How to plot Go test coverage over time