Bundling templates with pkger

Resource embedding tools like pkger allow developers to include static assets and other unpublished files like templates in their executable programs.

Let’s see how.

Why would I use a bundler like markbates/pkger ?

One of the most visible differences between traditional Web development using the LAMP stack or alternatives like Node.JS, Python, or Ruby on Rails, lies in the fact that Go applications are - as much as possible - compiled to a single executable file, unlike the myriad of script files and resources comprising deployed applications in these other technologies.

When it comes to business logic, this has little impact, as these files will be out of the document root anyway, but Web applications also use files beyond executable code and database content. These include static assets, which get served straight to the user browser, be it under control of the application, on a route bound to the native http.FileServer Handler, or by means of a specialized Web server set in front of the Go applications, or even by a completely independent cloud file service, like Amazon S3, Google Cloud Storage, ou Azure Files.

Templates are a slightly different case, though : not only is the on-disk layout usually in a hierarchy, unrelated with the application URL structure - unlike assets - but they are not sent to the user browser anyway, so they have no reason to exist outside the program code¹. Their embedding in the binary can even in some cases considered to be a security requirement.

For these cases, or for any other data a program needs to embed, the tried-and-true method is one pre-compilation step transforming the data file(s) to source code, which will then be compiled and linked into the binary just like any other piece of code. Just think of go generate with the stringer command.

Many contributed tools have come and gone to answer that need, so much so that since 12/2019, the Go project has been discussing how to provide it as a standard feature, in ticket 35950 , considering multiple approaches.

Until the autumn of 2019, the packr tool included in the Buffalo web framework was the go-to solution for new projects, if only because Buffalo itself is so popular. It comes with several limitations, though, and its designer Mark Bates created a newer solution, the markbates/pkger module (cf. Additional resources below).

Most examples available on line to date, however, use the Buffalo-specific plush template engine. This post shows how to build an example without any Buffalo specifics, using traditional Go templates located in two distinct directories.

How can I use pkger in my projects ?

During development:

• Go must be at version 1.13 or higher.
• The project must use VGO modules, upon which pkger relies to resolve paths, so be sure to initialize the module system. The module name is optional if the working directory is within the GOPATH.

$ go mod init [module]

• The pkger command must be installed on the development machine, in order to generate the resource-equivalent source code from the resources, and to provide debugging information about resources.

$ go get github.com/markbates/pkger/cmd/pkger

Once compiled, the resulting project has no remaining runtime dependency on pkger or the original resources.

Step 2: building during development

While developing, pkger accesses resources through the underlying file system API, and does not embed them in the compiled program.

To have the code is run in development mode, make sure that the pkged.go is actually not present in the project root directory. Have no qualms deleting it, as it will be regenerated the next time you compile a deployment build.

Having pkger access files through the file system during deployment means, especially in the case of templates, that you won’t have to recompile while developing the templates themselves: just re-run the compiled program, and it will load the new versions of the templates, until you compile for deployment.

The pkger list command outputs a list of files which pkger will be bundling in the program once it is build for deployment². As a side effect, it also deletes the pkged.go if it exists. This allows you to check what pkger will be bundling in the final program, so you can modify your program to ensure all needed files are included.

Step 3: building for deployment

Once the list of bundled files matches your expectations, you can have pkger generate the pkged.go file, by invoking the pkger command without any arguments. If all goes well, it outputs nothing, and silently creates pkged.go at the module root.

$ ls -F ; pkger ; ls -F
LICENSE		README.md	go.mod		go.sum		main.go		templates/
LICENSE		README.md	go.mod		go.sum		main.go		pkged.go	templates/

This works nicely with the go generate command if your main source file has been coded for this, allowing a more Go-idiomatic build process:

$ ls -F ; go generate ; ls -F
LICENSE		README.md	go.mod		go.sum		main.go		templates/
LICENSE		README.md	go.mod		go.sum		main.go		pkged.go	templates/

As is customary with any command used by go generate, in order to make it possible for machines with a Go toolchain but without the pkger command (e.g. a CI server) to build the program, you should commit the pkged.go to your VCS repository along with the rest of the source code.

Once this file is generated, you can compile as you prefer, ideally with just a plain go build : the only thing pkger changes to your build process is the extra pkged.go added to the project when built for deployment.

How do I embed resources with pkger ?

For pkger to bundle resources with your code, the module code must use its APIs. Let’s see how this is done.

_The code examples in this post are excerpted from the https://github.com/fgm/pkger_demo demo project, which is ready to compile: you can check it out to follow the explanations in your own IDE:

$ go get github.com/fgm/pkger_demo

Step 1: making the project compatible with go generate

In order to use go generate to trigger bundle generation, your code must include a generation instruction in one file at the root of the module. For this example, it is present in the root main.go file:

1
2
3
4

//go:generate pkger

package main
// ...snip...

Adding this line will allow the standard go generate command to run pkger when invoked, which will create the pkged.go, ready to compile.

Step 2: understanding the pkger API design

The way pkger identifies the files it needs to bundle is by parsing the Go source code in the module. During this parsing, it locates calls to its API functions, and gathers the arguments with which they are invoked, when they are specified as being disk paths. When the values of these arguments are constant strings, they are resolved relative to the location of the parsed file within the module. If the paths are absolute (e.g. /foo.gohtml), the file system root used for these absolute paths is held to be the root directory of the module. Thus :

• for the demo, when checked out to

/Users/fgm/go/src/github.com/fgm/pkger_demo

• …an argument passed as

"/templates/layout.gohtml"

• …is interpreted by pkger as a real absolute path:

/Users/fgm/go/src/github.com/fgm/pkger_demo/templates/layout.gohtml

During development, this constant absolute path will be passed to the standard runtime library calls³.

In deployment builds, the disk paths are no longer used : instead, the pkger functions resolve the arguments from the data bundled into the executable program, allowing the built program to be completely standalone.

Step 3: using the pkger API functions

The big idea for the pkger API is to be tiny, to take and return data with types very close to the native Go runtime library API for input/output, for a subset of the os and filepath packages, and use the runtime library for everything else. The only general use functions are the following ones:

A few specific functions exist, to tweak the bundling process:

• Include(name string) string is an identity function doing nothing but return its single argument.
• Current(), Info(string), and Parse(string) are used to manager the pkger paths and implementations for advanced development scenarios.

The pkger.Include function is by far the most useful of them : although all it does is return the path it receives as an argument without doing anything with it, the very fact that it is used allows the pkger command to mark a string as a path to bundle while it parses the module source code.

As the previous table shows, the main difference between the functions in the pkger API and those in the Go runtime is their returning a pkging.File interface value instead of an *os.File pointer. This is actually a not a significant cause for concern: most functions in the Go runtime do not depend on the concrete os.File for their arguments, but on the various io interfaces it implements ; interfaces which the pkging.File type actually embeds:

Step 4: loading templates from a directory subtree

The pkger_demo demo application is a very basic Go web application, which just returns a page laid out by two templates, following a common practice of variant body + shared footer.

templates/page.gohtml:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

{{ define "page" }}
{{- /*gotype: github.com/fgm/pkger_demo.PageData*/ -}}
<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>{{ .Path }} callback</title>
</head>
<body>
  <p>Called on {{ .Path }}</p>
  {{ template "layout/footer" . }}
</body>
</html>
{{ end }}

templates/layout/footer.gohtml

1
2
3
4
5
6

{{define "layout/footer"}}
  {{- /*gotype: github.com/fgm/pkger_demo.PageData*/ -}}
  <footer>
    &copy; {{ .Year }} Frederic G. MARAND for OSInet
  </footer>
{{end}}

Let us look into the main.go file salient bits. Lines 1-22 contain the program preamble (package, import) and define the PageData type, used to pass data to the executed templates. Nothing interesting there.

The program loads templates it finds in the /templates directory (the / is relative to the module root, not the disk root), but also in all its subdirectories. To discover them, it will have to crawl the substree starting at /templates, and parse all discovered templates:

23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

func compileTemplates(dir string) (*template.Template, error) {
	const fun = "compileTemplates"
	tpl := template.New("")
	// Since Walk receives a dynamic value, pkger won't be able to find the
	// actual directory to package from the next line, which is why we used
	// pkger.Include() in main().
	err := pkger.Walk(dir, func(path string, info os.FileInfo, _ error) error {
		// Skip non-templates.
		if info.IsDir() || !strings.HasSuffix(path, ".gohtml") {
			return nil
		}
		// Load file from pkpger virtual file, or real file if pkged.go has not
		// yet been generated, during development.
		f, _ := pkger.Open(path)
		// Now read it.
		sl, _ := ioutil.ReadAll(f)
		// It can now be parsed as a string.
		tpl.Parse(string(sl))
		return nil
	})
	return tpl, err
}

• line 29: this works just like filepath.Walk: the Walk function walks the subtree, invoking the walkFn callback on each directory entry met. In this example, the callback is the anonymous function at lines 29-42.
• lines 30-33: the function ignores non-files, and files not matching the expected template name format.
• line 36, it opens the file using pkger.Open, getting a pkging.File which happens to be an io.Reader implementation.
• line 37, since f is an io.Reader, the function reads its content using the standard ioutil.ReadAll function.
• line 40, with the result being a plain string, the function can now parse it as a Go template.

The final result of the function is a valid set of compiled templates, or an error value.

When generating the bundle, pkger will notice the pkger.Open call, and try to deduce the path of a resource to add to the bundle. But at this point, we have a problem…

Because pkger is a static analyzer, and the argument passed to pkger.Open is a variable (as opposed to a constant), it can not determine its value, which will only be known at runtime. So it has to ignore that call, and will not bundle the templates. Should the program limit itself to this mechanism, the templates would go missing in the pkged.go file, and the program would fail.

This is where the pkger.Include identity function comes in at line 50, informing pkger that a path with the constant value /templates needs to be bundled:

46
47
48
49
50
51
52

func main() {
	const addr = ":8080"

	// Tell pkger that it has to package that directory.
	dir := pkger.Include("/templates")
    // Only compile templates on startup.
	tpl, _ := compileTemplates(dir)

When pkger discovers this function during static analysis, it notices that its path argument is a constant string, the value of which it can evaluate as it is at generate time. It will therefore open that path, notice it is a directory, and include it with all its subdirectory. Let us check the result:

$ pkger list
github.com/fgm/pkger_demo
 > github.com/fgm/pkger_demo:/templates
 > github.com/fgm/pkger_demo:/templates/layout
 > github.com/fgm/pkger_demo:/templates/layout/footer.gohtml
 > github.com/fgm/pkger_demo:/templates/page.gohtml

Since the templates are now bundled in the program once compiled for deployment, the pkger.Open calls will succeed, and function will successfully read, compile, and return the template set.

⚠️ Notice that - unlike the Walk callback function - pkger cannot filter anything out of the sub-tree : when passing directories to pkger.Include, they must only contain files which are actually meant to be used at runtime, as everything is these directories will be bundled in the program, increasing its size and memory footprint.

Once the program obtains the compiled templates at line 52, the HTTP request handlers can use the template without any notion of their loading mechanism:

54
55
56
57
58
59
60
61

// Now serve pages
	http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
		ctx := PageData{Path: r.URL.Path, Year: time.Now().Local().Year()}
		tpl.ExecuteTemplate(w, "page", ctx)
	})
	log.Printf("Listening on %s", addr)
	log.Fatal(http.ListenAndServe(addr, nil))
}

Additional resources by Mark Bates

• Introductory blog post (2019-11)
  https://blog.gobuffalo.io/introducing-pkger-static-file-embedding-in-go-1ce76dc79c65
• Introduction video for pkger (2019-11)
  That video details the goals Mark Bates followed when designing this new tool to improve upon his previous gobuffalo/packr/v2.

• To summarize the video : the pkger API is idiomatic, copying the io and filepath packages, whereas the packr2 API was idiosyncratic ; it is also lossless, preserving file information, where the packr2 API was lossy, losing the information about the original file.