Resource embedding tools like
pkger allow developers to include static assets
and other unpublished files like templates in their executable programs.
Let’s see how.
Why would I use a bundler like
One of the most visible differences between traditional Web development using the LAMP stack or alternatives like Node.JS, Python, or Ruby on Rails, lies in the fact that Go applications are - as much as possible - compiled to a single executable file, unlike the myriad of script files and resources comprising deployed applications in these other technologies.
When it comes to business logic, this has little impact, as these files will be
out of the document root anyway, but Web applications also use files beyond
executable code and database content.
These include static assets, which get served straight to the user browser,
be it under control of the application, on a route bound to the native
or by means of a specialized Web server set in front of the Go applications, or
even by a completely independent cloud file service, like
Google Cloud Storage, ou
Templates are a slightly different case, though : not only is the on-disk layout usually in a hierarchy, unrelated with the application URL structure - unlike assets - but they are not sent to the user browser anyway, so they have no reason to exist outside the program code1. Their embedding in the binary can even in some cases considered to be a security requirement.
For these cases, or for any other data a program needs to embed, the tried-and-true
method is one pre-compilation step transforming the data file(s) to source code,
which will then be compiled and linked into the binary just like any other piece
of code. Just think of
go generate with the
Many contributed tools have come and gone to answer that need, so much so that since 12/2019, the Go project has been discussing how to provide it as a standard feature, in ticket 35950 , considering multiple approaches.
Until the autumn of 2019, the
packr tool included in the Buffalo web framework
was the go-to solution for new projects, if only because Buffalo itself is so
popular. It comes with several limitations, though, and its designer Mark Bates
created a newer solution, the
(cf. Additional resources below).
Most examples available on line to date, however, use the Buffalo-specific
plush template engine.
This post shows how to build an example without any Buffalo specifics, using
traditional Go templates located in two distinct directories.
How can I use
pkger in my projects ?
Step 1: preparing requirements
- Go must be at version 1.13 or higher.
- The project must use VGO modules, upon which
pkgerrelies to resolve paths, so be sure to initialize the module system. The module name is optional if the working directory is within the
$ go mod init [module]
pkgercommand must be installed on the development machine, in order to generate the resource-equivalent source code from the resources, and to provide debugging information about resources.
$ go get github.com/markbates/pkger/cmd/pkger
Once compiled, the resulting project has no remaining runtime dependency on pkger or the original resources.
Step 2: building during development
pkger accesses resources through the underlying file system
API, and does not embed them in the compiled program.
To have the code is run in development mode, make sure that the
actually not present in the project root directory. Have no qualms deleting it,
as it will be regenerated the next time you compile a deployment build.
pkger access files through the file system during deployment means,
especially in the case of templates, that you won’t have to recompile while
developing the templates themselves: just re-run the compiled program, and it
will load the new versions of the templates, until you compile for deployment.
pkger list command outputs a list of files which
pkger will be bundling
in the program once it is build for deployment2. As a side effect, it also deletes
pkged.go if it exists.
This allows you to check what
pkger will be bundling in the final program,
so you can modify your program to ensure all needed files are included.
Step 3: building for deployment
Once the list of bundled files matches your expectations, you can have
pkged.go file, by invoking the
pkger command without any arguments.
If all goes well, it outputs nothing, and silently creates
pkged.go at the module
$ ls -F ; pkger ; ls -F LICENSE README.md go.mod go.sum main.go templates/ LICENSE README.md go.mod go.sum main.go pkged.go templates/
This works nicely with the
go generate command if your main source file has been
coded for this, allowing a more Go-idiomatic build process:
$ ls -F ; go generate ; ls -F LICENSE README.md go.mod go.sum main.go templates/ LICENSE README.md go.mod go.sum main.go pkged.go templates/
As is customary with any command used by
go generate, in order to make it
possible for machines with a Go toolchain but without the
pkger command (e.g.
a CI server) to build the program, you should commit the
pkged.go to your VCS
repository along with the rest of the source code.
Once this file is generated, you can compile as you prefer, ideally with just
go build : the only thing
pkger changes to your build process is the
pkged.go added to the project when built for deployment.
How do I embed resources with
pkger to bundle resources with your code, the module code must use its
APIs. Let’s see how this is done.
_The code examples in this post are excerpted from the https://github.com/fgm/pkger_demo demo project, which is ready to compile: you can check it out to follow the explanations in your own IDE:
$ go get github.com/fgm/pkger_demo
Step 1: making the project compatible with
In order to use
go generate to trigger bundle generation, your code must include
a generation instruction in one file at the root of the module. For this example,
it is present in the root
Adding this line will allow the standard
go generate command to run
invoked, which will create the
pkged.go, ready to compile.
Step 2: understanding the
pkger API design
pkger identifies the files it needs to bundle is by parsing the Go
source code in the module. During this parsing, it locates calls to its API
functions, and gathers the arguments with which they are invoked, when they are
specified as being disk paths.
When the values of these arguments are constant strings, they are resolved relative
to the location of the parsed file within the module. If the paths are absolute
/foo.gohtml), the file system root used for these absolute paths is held
to be the root directory of the module. Thus :
- for the demo, when checked out to
- …an argument passed as
- …is interpreted by
pkgeras a real absolute path:
During development, this constant absolute path will be passed to the standard runtime library calls3.
In deployment builds, the disk paths are no longer used : instead, the
functions resolve the arguments from the data bundled into the executable program,
allowing the built program to be completely standalone.
Step 3: using the
pkger API functions
The big idea for the
pkger API is to be tiny, to take and return data with types
very close to the native Go runtime library API for input/output, for a subset
filepath packages, and use the runtime library for everything else.
The only general use functions are the following ones:
A few specific functions exist, to tweak the bundling process:
Include(name string) stringis an identity function doing nothing but return its single argument.
Parse(string)are used to manager the
pkgerpaths and implementations for advanced development scenarios.
pkger.Include function is by far the most useful of them : although all it
does is return the path it receives as an argument without doing anything with it,
the very fact that it is used allows the
pkger command to mark a string as
a path to bundle while it parses the module source code.
As the previous table shows, the main difference between the functions in the
pkger API and those in the Go runtime is their returning a
value instead of an
This is actually a not a significant cause for concern: most functions in the Go
runtime do not depend on the concrete
os.File for their arguments, but on the
io interfaces it implements ; interfaces which the
Step 4: loading templates from a directory subtree
The pkger_demo demo application is a very basic Go web application, which just returns a page laid out by two templates, following a common practice of variant body + shared footer.
Let us look into the
main.go file salient bits. Lines 1-22 contain the program
import) and define the
PageData type, used to pass data
to the executed templates. Nothing interesting there.
The program loads templates it finds in the
/templates directory (remember the
/ is relative to the module root, not the disk root), but also in all its
subdirectories. To discover them, it will have to crawl the substree starting at
/templates, and parse all discovered templates:
- line 29: this works just like
Walkfunction walks the subtree, invoking the
walkFncallback on each directory entry met. In this example, the callback is the anonymous function at lines 29-42.
- lines 30-33: the function ignores non-files, and files not matching the expected template name format.
- line 36, it opens the file using
pkger.Open, getting a
pkging.Filewhich happens to be an
- line 37, since
io.Reader, the function reads its content using the standard
- line 40, the result being a plain string, the function can now parse it as a Go template.
The final result of the function is a valid set of compiled templates, or an error value.
When generating the bundle,
pkger will notice the
pkger.Open call, and try
to deduce the path of a resource to add to the bundle. But at this point, we have
pkger is a static analyzer, and the argument passed to
a variable (as opposed to a constant), it can not determine its value, which will
only be known at runtime. So it has to ignore that call, and will not bundle the
Should the program limit itself to this mechanism, the templates would go missing
pkged.go file, and the program would fail.
This is where the
pkger.Include identity function comes in at line 50, informing
pkger that a path with the constant value
/templates needs to be bundled:
pkger discovers this function during static analysis, it notices that its
path argument is a constant string, the value of which it can evaluate as it is
at generate time. It will therefore open that path, notice it is a directory, and
include it with all its subdirectory. Let us check the result:
$ pkger list github.com/fgm/pkger_demo > github.com/fgm/pkger_demo:/templates > github.com/fgm/pkger_demo:/templates/layout > github.com/fgm/pkger_demo:/templates/layout/footer.gohtml > github.com/fgm/pkger_demo:/templates/page.gohtml
Since the templates are now bundled in the program once compiled for deployment,
pkger.Open calls will succeed, and function will successfully read, compile,
and return the template set.
⚠️ Notice that - unlike the
Walk callback function -
pkger cannot filter anything
out of the sub-tree : when passing directories to
pkger.Include, they must only
contain files which are actually meant to be used at runtime, as everything is
these directories will be bundled in the program, increasing its size and memory
Once the program obtains the compiled templates at line 52, the HTTP request handlers can use the template without any notion of their loading mechanism:
Additional resources by Mark Bates
- Introductory blog post (2019-11)
- Introduction video for
That video details the goals Mark Bates followed when designing this new tool to improve upon his previous
- To summarize the video : the
pkgerAPI is idiomatic, copying the
filepathpackages, whereas the
packr2API was idiosyncratic ; it is also lossless, preserving file information, where the
packr2API was lossy, losing the information about the original file.
When designing applications to accept templates which can change after deployment, they will have to be present, making them more similar to static assets, although they remain outside the end user reach. ↩︎
The command can also output a detailed view of the bundle in JSON format when used with the JSON flag:
pkger list -json↩︎
This has a sometimes surprising consequence : a binary built in development mode can be run from any current working directory and still find its resources at their original location on the development machine, because their path is resolved to an actual absolute path, instead of being relative to the current working directory. ↩︎