psxsdk

The compiler in the browser

The playground builds a program and runs it without anything installed, and so does yaroze. The compiler behind both is TinyGo itself, compiled to WebAssembly, with LLVM and lld linked into it the way a release build of TinyGo links them. In the browser it runs against a file system held in memory, and on the desktop yaroze runs it in-process under wazero. web/compiler holds the recipe, and build.sh is the whole of it.

TinyGo compiles TinyGo#

TinyGo is a Go program bound to LLVM through cgo, and Go's own WebAssembly ports have no cgo. TinyGo's does: its wasip1 target compiles the C parts with clang and links them with wasm-ld. So TinyGo compiles TinyGo, against LLVM built for wasm32-wasi:

  • LLVM (llvm/configure.sh): LLVM, clang and lld as static libraries for wasm32-wasi, with only the Mips backend, no threads and no tools, using nixpkgs' wasi cross toolchain. llvm-wasi.patch and wasi-compat/ are what LLVM's Support library needs to compile without signals, processes, sockets or users: a few declarations and stubs, and the code paths that catch crashes or run subprocesses cut out. libcxx.sh builds libc++ with its threading API on, because LLVM's headers use std::mutex whatever LLVM_ENABLE_THREADS says. Nothing ever starts a thread.
  • Glue (glue/): the C and C++ that is not LLVM's. That is go-llvm's bindings, an lld driver with only the ELF linker, TinyGo's clang driver for the full build, and wasi-stubs.c, the POSIX functions LLVM links against and WASI does not have, each failing the way a system without the feature would, and an sbrk that gives wasi-libc's allocator pieces of TinyGo's heap. C's memory is that allocator's (the custommalloc tag), since LLVM makes millions of small allocations per build and TinyGo's allocator is slow at that; the pieces are manual allocations the collector neither scans nor frees, so the Go side is collected as before, and the runtime's heap starts past the initial memory the allocator takes for itself first (tinygo_morecore and heapStartAddr in the runtime). dlmalloc assumes sbrk is contiguous until a call fails, so the first is refused.
  • The target (target/gen.sh): wasip1, with every library above on the link line and a 32MB stack, since LLVM's recursion needs far more than the usual 64K. With less, it overflows and corrupts memory, and fails differently on every build. The compiler is built with no scheduler (-scheduler=none, and -panic-unwind=explicit so that recover works without asyncify): the patch has the builder run its jobs one after another rather than each on a goroutine, and go/ssa build packages in turn, and gives os/exec and the parking in internal/poll shapes that start nothing. With the asyncify scheduler every goroutine got a 32MB stack and switching unwound the whole stack, which was most of a build's time, and asyncify's instrumentation was a third of the module. DWARF is left out of the IR too, since nothing in the playground reads it; -debug keeps it.
  • TinyGo (tinygo-wasm.patch, applied by prepare.sh to upstream TinyGo at the flake's revision with the PlayStation targets from internal/toolchain/_tinygo laid over it): those files are only additions, as a pull request to TinyGo would be, and the patch holds what running the compiler as a WebAssembly program needs. That is cmd/tinygo-wasm, the build command alone, and under the wasm build tag a loader that lists packages with go/build instead of running go list, takes GOROOT and GOVERSION from the environment instead of go env, uses a GOROOT merged in advance, and runs lld in-process instead of as a subprocess. cmd/tinygo-sysroot, also in the patch, gathers that merged GOROOT, the psx packages and the target files into the tree the compiler reads. prepare.sh says how to regenerate the patch.

Leaving clang out#

A release TinyGo compiles a target's C and assembly with an in-process clang, and clang is more than half of the module. So there are two builds. The shipped tinygo.wasm leaves clang out and is 75MB, 22MB gzipped. tinygo-full.wasm keeps it, at 169MB, and never leaves the build machine: mkcache.sh runs it under wasmtime over every example, against the same sysroot paths the browser uses, and the cache it fills ships in sysroot.tar.gz, keyed by the shipped compiler's hash. A first build then finds the target's C and the standard library already compiled. The playground's sysroot holds the cache at TinyGo's default -opt=z and is 5MB; yaroze's holds -opt=2 as well and is 8.8MB.

A build in the playground#

compile-worker.js fetches the compiler, the tools and the sysroot once, and pipeline.js runs each build in a worker: the program's //go:generate lines, then the compiler, then mips1fix, eco2exe and mkdisc. The tools are the repository's own, compiled to wasip1 by the Go toolchain and run through wasi.js, a WASI host over an in-memory file system. The result is a disc the player boots like an example. The packages a program imports are linked into one bitcode module the first time that set is seen and kept in the cache (prelink-*.bc, keyed by the packages' own cache entries), since a build changes the main package alone nearly always and parsing one module is quicker than parsing and linking thirty; the cache the sysroot ships holds the examples' sets. The compiler is loaded once and asked to build as often as needed (Compiler in pipeline.js): the module is a reactor, with tinygo_args and tinygo_build exports in place of a command line, and between builds its loader keeps the standard library and the imported packages parsed and type-checked, in one file set and one SSA program (loader.Keep, loader/keep.go), since a build changes the main package alone nearly always. A build of the square example takes about 0.3s on a desktop after the first in a page, which takes about a second while the browser compiles the module's code as it runs and the compiler reads the standard library. So once the compiler has loaded, which the editor's first use starts, the page builds the project as it stands in the background, and again a little after each change to it while the page is in view; when the program is the same by the time Run is pressed, that build is used and Run is immediate (prebuild and schedulePrebuild in playground.js).

What is built is a project (project.js): the program's Go files, a tab each over the editor (files.js), and its data files, in the Files sidebar (sidebar.js, with inspect.js to view them), uploaded or copied from an example. The data goes under assets/, for mkdisc to put on the disc where the program registers it with package disc. Sources under assets/ are converted by the program's own //go:generate lines (convert.js), which the worker runs with the converters built to wasip1. A scene is placed in the Scene tab (scene-tab.js, drawing with preview.js) and cooked by mkscene the same way, for package scene to read on the console. Projects are kept in the browser's IndexedDB, and go out and come in as zips laid out like the examples, so a download runs with go run.

While a project runs, each cooked scene is also written into the emulator's RAM over the file the program loaded, and package scene's live variable tells the program to read it again, so the console follows the editor without a rebuild. That goes through the worker's symbol, peek and poke messages to psx.symbol, psx.read and psx.write in web/main.go. The worker's break, unbreak, step and registers messages set breakpoints on the program's functions by the same symbols, pause it at one with a break event, and step it an instruction at a time.

Hot reload#

A program built with -hot (yaroze run --hot, or the playground's Hot box) can be given new code while it runs. The compiler, in a pass added to TinyGo (transform/hot.go in internal/toolchain/_tinygo, with the flags and hooks in internal/toolchain/tinygo-hot.patch), renames each function of the main package F to F$body, keeps it out of line, and puts a stub F in its place that loads F$slot, a word holding the body's address, and tail-calls it. Every call, function value and method table goes through the slot, so writing the slot changes what runs next, including the frame loop's body, which Screen.Frames calls through its function value every frame. The load is volatile, so the optimizer keeps it. The link script reserves 256KB above the program for patches, and the Go heap starts above that; a program so large that this would leave the heap under 256KB gets a smaller region, down to none, which yaroze run --hot and the playground say.

A patch is the same program compiled again with -hot-base, the ELF of the running program, and -hot-at, an address in its region. After the optimizer, a second pass binds the module to the base: every function the base has becomes its address there, every writable variable too, since that is the running program's state, and every type descriptor, since interfaces compare types by address. Read-only data is compiled afresh, so a changed table or string is the new one. The bodies are kept and linked at the address, with __hot_patch, a table of the base's slots to point at them, and anything the base lacks, a new function or variable, is compiled in with it. A constant is folded into the code that uses it, so a changed constant is in the new functions; a variable set from one keeps its value, since it is the running program's state, except that __hot_inits carries the old and new initial bytes of every bound variable whose initial value changed, and the loader gives the new value to a variable that still holds the old one, which the program has not written since it started. The hot package (internal/hot) reads the patch's sections and table, and loads it the next time the program calls the library's VSync, where every frame of the main package's code has returned: yaroze run through psxplay, and the playground through the worker's patch message to psx.patch. Each patch is linked after the one before, since the program may still be running that one's code, until the region is full.

What a patch cannot do, the compiler reports as an error starting hot:, and the program has to be built and started again: a variable that changed size, a function and a variable swapping names. What it cannot see is the user's: a type whose layout changed while objects of the old layout are live, a frame loop whose captured variables changed, or a running function, main itself, whose old activation goes on in the old code until it returns. Those are the ways a patched game misbehaves, so the feature is opt-in. The cost of the slots is a load and an indirect jump at each call into the main package, which at the size optimization is the calls the compiler would otherwise have inlined.

Building it#

task compiler builds the two modules, which takes most of an hour, and task playground adds them, the tools, gofmt and the sysroot to the site. The fork is checked out at the revision flake.nix pins, so the compiler in the browser is the TinyGo the examples were built with, and task yaroze copies the same compiler and a sysroot into cmd/yaroze/wasm for yaroze to embed.

In CI, the Compiler workflow builds the modules when web/compiler, the flake or the PlayStation files in internal/toolchain/_tinygo change, or on request. It caches them under a key made of those, less the Markdown (key.sh), publishes them as a release named by the key (release.sh), which outlives the cache's week, and then runs the Pages workflow. Pages restores the cache, or downloads the release on a miss, through the same composite action, and never builds the compiler itself: with neither, a push to main skips the deploy until the Compiler workflow has published one, and a pull request builds the site without the playground. A release published from a local build (release.sh .build/compiler) before the change is pushed saves the workflow the build.

Limits#

  • There is no clang, so a program cannot use cgo or anything whose C is not already in the cache. Only the ELF flavor of lld is linked in, for size.
  • The cache and the sysroot are made together, by task playground, and a compiler with another hash ignores the cache.
  • The compiler builds only at the optimization levels its cache holds.
  • Memory is TinyGo's garbage collector's, LLVM's allocations included, and is scanned conservatively.
  • A project is kept in the browser that made it, until it is downloaded.