Spectra gets Rust scripting!

spectra is a crate I’ve been maintaining for a few months / years now. It’s a crate that I mainly use for demoscene productions (I released two with it, Céleri Rémoulade and Outline Invitation) but I also use it to play around and experiment new rendering, animation and video game techniques.

The crate features a few things that I enjoy daily. Among them:

The last thing I’ve been working on is being productive. That might seem counter-intuitive, but when you start building “frameworks” and “engines”, you actually end up writing a lot of code for “the beauty and power of it” and lose focus on what’s important: releasing things. I know that and looked for what I could enhance to augment my productivity.

Among interesting topics I came up with, I stated:

Clearly, there’s something about live coding.

Introducing scripting in Rust

I went through several thought processes. I first had a look at Lua, because lots of people think it’s cool. However, I don’t like the syntax nor the overall safety the language gives doesn’t give you.

I remembered that almost a decade ago, when I was 15 or 16 year old, I implemented some kind of a plugins system in C++ for a side project of mine. The idea was simple:

This is not magic. When you link a crate / library, by default, a lot of compilers will perform static linking. That’s the case of ghc or rustc, in most situations.

If you don’t know yet, a library / crate / dependency statically-linked in a program means that all of its symbols (the ones used in the program at least) are placed in a specific section of the generated executable, so that those symbols have a proper address and “come with the binary”.

Dynamic linking, on the other side, is a way to express a dependency between your binary and some code, which is most of the time living in a relocatable archive (.so on Linux or macOSX, .dll on Windows – macOSX also uses .framework but it’s just for the overall idea). When your binary needs to call a function defined in a dynamically linked library, it’ll open the shared library with the dlopen function, try to locate the function by giving its (unmangled) name to dlsym or dladdr for instance, if the function exists, you’ll be able to run its code.

There exists attacks and interesting things you can do with dynamic libraries. Because the code doesn’t live in the binary, you can for instance replace a legit and safe .dll on Windows with a malicious one. Or you can patch a buggy dynamic library by shipping a new .dll / .so without having its dependent to re-compile their applications (which would be needed in case of static linking). Another funny thing you can do: fraps, a real-time recorder for your video games, performs some kind of .dll injection by pushing an OpenGL .dll into your binary (Windows allows this) and replacing some known functions. For instance, the function responsible in swapping your renderbuffer chains. It can then intercepts pixels regions, adds overlay, etc. Fascinating! ;)

Anyway, the idea is that whenever you link a dynamic library, your compiler / linker will just insert the required code in your binary (think of dlopen) so that your binary can load code at runtime. It’s then pretty easy to implement a plugin system:

So you could imagine, as a very simple start example, a loop that would simply invokes such a function. Whenever you change the library, the code getting ran will automatically changes as well!

Scripting in spectra

So I decided to reproduce that in Rust using the few crates of mine:

For unix plateforms, you can see that libloading is just a smart wrapper over the functions I mentionned. See for yourself.

Ok, so, let’s try the experiment!

Let’s write plugins in Rust!

Foreword

The first thing we must accomplish is very simple: we want to be able to load some (Rust) code at runtime, inside our application. For achieving that goal, we need to load a dynamic library (.so on Linux) with libloading::Libray::open. Once we have the library, we can just look a symbol up with libloading::Library::get. In case of a successful lookup, that function returns a value of type Symbol, which implements Deref for the symbol you’re asking.

Dynamic library typically gathers functions, so we’ll be looking for fn symbols.

However, we don’t have any dynamic library yet. We only have… a Rust file – .rs. How can we get a dynamic library out of it?

Generating a dynamic library

This is actually pretty simple. We’re gonna start easily by making a .so that will just contain a function called hello_world that will just print "Hello, world!" on stdout. You all know how to implement such a function:

/// hello_world.rs
pub fn hello_world() {
  println!("Hello, world!");
}

We make the function pub so that it gets actually exported when we generate the dynamic library.

Copy that code and put it in, for instance, /tmp.

Then, let’s generate a dynamic library!

cd /tmp
rustc --crate-type dylib hello_world.rs

Once rustc returned, you should see a new file in /tmp: /tmp/libhello_world.so! Here we are! We have a dynamic library! Let’s try to find our function in it with the handy nm program.

I reckon nm is already installed on your machine. If not, it should come with packages like base-devel, build-essentials or that kind of meta packages.

nm /tmp/libhello_world.so | grep hello_world
0000000000000000 N rust_metadata_hello_world_8787f43e282added376259c1adb08b80
0000000000040ba0 T _ZN11hello_world11hello_world17h49fe1e199729658eE

Urgh. We have a problem. Rust has mangling for its symbol names. It means that it will alter the symbols so that it can recognize them in a dynamic library. For instance, the mangle version of a function name foo defined for a type A won’t be the same as the one of foo defined for a type B. However, in our case, we don’t want mangling, because, well, we won’t be able to lookup the name up – no one will even try to guess that _ZN11hello_world11hello_world17h49fe1e199729658eE function name.

rustc has a very simple solution to that: just tell it you don’t want a symbol’s name to be mangled. It’ll be stored in the dynamic library the way you write it. This is done with the #[no_mangle] attribute.

/// hello_world.rs
#[no_mangle]
pub fn hello_world() {
  println!("Hello, world!");
}

Now recompile with the same rustc line, and run the nm + grep oneliner again.

nm /tmp/libhello_world.so | grep hello_world
0000000000040b70 T hello_world
0000000000000000 N rust_metadata_hello_world_8787f43e282added376259c1adb08b80

Now you can see there exists a symbol called hello_world: this is our symbol!

Load the library in Rust and read a symbol

For our little example here, we’re just gonna load and run the code in a new project.

cd /tmp
cargo new --bin dyn-hello-world
     Created binary (application) `dyn-hello-world` project
cd dyn-hello-world

Edit the Cargo.toml file to include libloading = "0.5" in the [dependencies] section. Ok, we’re good to go. Run a second terminal in which you run this command to automatically check whether your code is okay:

cd /tmp/dyn-hello-world
cargo watch

Let’s edit our main.rs file.

extern crate libloading;

use libloading::Library;

fn main() {
  let lib = Library::new("/tmp/libhello_world.so").unwrap();
  let symbol = unsafe { lib.get::<extern fn ()>(b"hello_world").unwrap() };

  symbol();
}

It should check. A bit of explanation:

Now compile and run the code:

cargo build
   Compiling cc v1.0.4
   Compiling libloading v0.5.0
   Compiling dyn-hello-world v0.1.0 (file:///tmp/dyn-hello-world)
    Finished dev [unoptimized + debuginfo] target(s) in 2.63 secs
cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.0 secs
     Running `target/debug/dyn-hello-world`
Hello, world!

Hurray! We’ve just built our first dynamic library loader!

Let’s compile Rust… from Rust!

Ok, now, let’s iterate: we need to compile the Rust code from our Rust code (haha). For doing that, we’ll need to generate the dynamic library at some place. Because I don’t like to put artifacts in random places, I like to use the tempdir crate, which gives you a TempDir that creates a new temporary directory with a random name when you ask for it, and removes it from your filesystem when the TempDir goes out of scope. We’ll also be using the std::process module to run rustc.

Add the following to your [dependencies] section in your Cargo.toml:

tempdir = "0.3"

Ok, let’s compile a Rust source file into a dynamic library from our Rust source code!

extern crate libloading;
extern crate tempdir;

use libloading::Library;
use std::process::Command;
use std::str::from_utf8_unchecked;
use tempdir::TempDir;

fn main() {
  let dir = TempDir::new("").unwrap(); // we’ll drop the .so here
  let target_path = dir.path().join("libhello_world.so");
  
  let compilation =
    Command::new("rustc")
    .arg("--crate-type")
    .arg("dylib")
    .arg("/tmp/hello_world.rs")
    .arg("-o")
    .arg(&target_path)
    .output()
    .unwrap();

  if compilation.status.success() {
    let lib = Library::new(&target_path).unwrap();
    let symbol = unsafe { lib.get::<extern fn ()>(b"hello_world").unwrap() };

    symbol();
  } else {
    let stderr = unsafe { from_utf8_unchecked(compilation.stderr.as_slice()) };
    eprintln!("cannot compile {}", stderr);
  }
}

Compile, run, and…

cargo run
   Compiling dyn-hello-world v0.1.0 (file:///tmp/dyn-hello-world)
    Finished dev [unoptimized + debuginfo] target(s) in 0.58 secs
     Running `target/debug/dyn-hello-world`
Hello, world!

This piece of code is the first premise of our plugin system. You can see interesting properties:

However, there’s a problem. Try adding an extern crate to hello_world.rs, like, for instance:

extern crate spectra;

#[no_mangle]
pub fn hello_world() {
  println!("Hello, world!");
}

Run dyn-hello-world.

cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.0 secs
     Running `target/debug/dyn-hello-world`
cannot compile error[E0463]: can't find crate for `spectra`
 --> /tmp/hello_world.rs:1:1
  |
1 | extern crate spectra;
  | ^^^^^^^^^^^^^^^^^^^^^ can't find crate

error: aborting due to previous error

Damn, how are we gonna solve this?

Finding dependencies!

The key is to understand how cargo deals with the dependencies you declare in your Cargo.toml’s [dependencies] section. To do this, clean your project, and recompile in verbose mode – we’re like… reverse engineering cargo build!

cargo clean -p dyn-hello-world
cargo build --verbose
       Fresh libc v0.2.36
       Fresh winapi-build v0.1.1
       Fresh winapi v0.2.8
       Fresh cc v1.0.4
       Fresh rand v0.4.2
       Fresh kernel32-sys v0.2.2
       Fresh libloading v0.5.0
       Fresh remove_dir_all v0.3.0
       Fresh tempdir v0.3.6
   Compiling dyn-hello-world v0.1.0 (file:///tmp/dyn-hello-world)
     Running `rustc --crate-name dyn_hello_world src/main.rs --crate-type bin --emit=dep-info,link -C debuginfo=2 -C metadata=e8b1e5e96f709abe -C extra-filename=-e8b1e5e96f709abe --out-dir /tmp/dyn-hello-world/target/debug/deps -C incremental=/tmp/dyn-hello-world/target/debug/incremental -L dependency=/tmp/dyn-hello-world/target/debug/deps --extern tempdir=/tmp/dyn-hello-world/target/debug/deps/libtempdir-9929bcad6dc8cc47.rlib --extern libloading=/tmp/dyn-hello-world/target/debug/deps/liblibloading-a2854ce154eb4d6f.rlib -L native=/tmp/dyn-hello-world/target/debug/build/libloading-4553b9f132aa813a/out`
    Finished dev [unoptimized + debuginfo] target(s) in 0.50 secs

We can see a few things going on here. First, there are some Fresh lines we’re not interested about. Then cargo tries to compile our application. You can see an invokation to rustc with a long list of arguments. Among them, two interest us:

There’s also a -L native=… one. This is explained in the manual of rustc and corresponds to native library we must link against, like a C library, for instance.

As you can see, the path used in -L dependencie=… is pretty constant. It seems it has the form:

/path/to/project/target/<build type>/deps

However, remember that the initial intent was to write a plugin system for spectra, which is a library. We don’t know the path to the project that will be using spectra, so we must find it in a way.

Two options:

I chose the second option because it was pretty easy and straight-forward to implement. However, I’m not a huge fan of it – it’s pretty… non-elegant to me. Here’s the code that gives me the root path of the current project a function is defined in:

/// Try to find the project root path so that we can pick dependencies.
fn find_project_root() -> Result<PathBuf, PluginError> {
  let result =
    Command::new("cargo")
    .arg("locate-project")
    .output()
    .unwrap();

  if !result.status.success() {
    Err(PluginError::CannotFindMetaData("cannot locate root project".to_owned()))
  } else {
    let json = unsafe { from_utf8_unchecked(result.stdout.as_slice()) };
    let root =
      json.split(':').nth(1).and_then(|x| {
        if x.len() >= 3 {
          Path::new(&x[1..x.len()-3]).parent().map(|x| x.to_owned())
        } else {
          None
        }
      });

    root.ok_or_else(|| PluginError::CannotFindMetaData("cannot extract root project path from metadata".to_owned()))
  }
}

As you can see, I use the cargo locate-project feature, that gives you the root path of the project the cargo invokation is in. It supports calling it from a subdirectory, which is neat (a bit like git, actually). Most of the code is removing the JSON sugar over it. It’s pretty unsafe because if the output format of cargo locate-project changes, this code will basically break.

Pro reminder for myself: write unit tests for that piece of code. :D

I won’t show the PluginError type, it’s not important and it’s spectra current code for the feature.

Ok, we lack two things:

Finding the build type is pretty easy: you can use the debug_assertions cfg! argument. It’s set to true on debug target and false on release. You can actually witness it works with the following oneliner:

println!("build target is debug: {}", cfg!(debug_assertions));

Ok, now, how do we find our spectra crate’s path? You saw it has a metadata glued to its name in the previous verbose output.

I didn’t find any satisfying solution. I just came up with a solution. I warn you: it’s not elegant, it’s highly unsafe, but for now, it works. I’ll try to find a better way later.

Here’s the code.

/// Find the spectra crate in the given directory.
fn find_spectra_crate(path: &Path) -> Result<PathBuf, PluginError> {
  // we open the directory we pass in as argument to the function
  if let Ok(dir) = path.read_dir() {
    // we iterate over all of its contained files to find if it has our spectra crate
    for entry in dir {
      let path = entry.unwrap().path();

      match path.file_name() {
        // we try to pattern match its name; this is a bit unsafe because it won’t support two
        // versions of libspectra… meh.
        Some(filename) if filename.to_str().unwrap().starts_with("libspectra") => {
          return Ok(path.to_owned());
        }

        _ => ()
      }
    }

    Err(PluginError::CannotFindMetaData("cannot find the spectra crate; try to build your project’s dependencies".to_owned()))
  } else {
    Err(PluginError::CannotFindMetaData("cannot find read dependencies".to_owned()))
  }
}

If you put those three functions altogether, you can now implement the rustc call without hardcoding any paths, since they all will be found at runtime and injected in the call!

A bit of hindsight. I didn’t explain that, but it’s a bit obvious: you won’t be able to use all the crates you want in your plugins, for a very simple reason: you must have them installed somewhere. In order for the find_spectra_crate to find anything, you must have spectra = "…" in the [dependencies] section of your Cargo.toml. If you want to use any crate, you need to add them in the [dependencies] and write a smarter function that parses the Cargo.toml’s [dependencies] section and add them to the rustc invokation… which is basically like re-writting a feature of cargo itself!

How do you make auto-reload again?

I didn’t speak about that, but in spectra, it’s very easy to have a resource to auto-reload if it changes on the disk. This is done via the warmy crate. You just implement the Load trait and ask for your type at a Store by providing a typed key. I won’t talk too much about warmy – if you’re interested, go read the online documentation here!

The idea is that I just created a type that wraps both libloading::Library and tempdir::TempDir. Something like this:

pub struct Plugin {
  lib: Library,
  dir: TempDir
}

I still don’t know whether I need to implement Drop or not – if the TempDir dies first, I don’t know whether the Library object is in an unstable state. If not, that means that the whole library was loaded in RAM at the end of the Library::new call, and that I don’t even need to keep the temporary directory around!

It’s just the beginning

The interface of a Plugin is not yet defined. I’m writing this blog entry on Sunday night / Monday morning, while I had that plugin experiment over the weekend. A few thoughts, though:

I have a lot of things to say, especially lately. I’ll be posting more thoughts and experiments of mine soon! Thanks for having read through, and as always, keep the vibes!

↑ Spectra gets Rust scripting!
spectra, Rust, plugins, dynamic library
Mon Feb 19 02:57:00 2018 UTC