Pluginval is a plugin dev’s best friend

Pluginval is a cross-platform open source app from Tracktion.

Think of it as quality control for your plugin. It loads your audio plugin as a DAW would, runs myriad tests against it, and reports any problems.

Whether you are having problems with your plugin failing DAW validation or just want to catch issues before your users do, it’s worth learning how to use!

Here are a few tips on working with pluginval:

Pluginval Tip 1: Compile locally

You could download and run the latest release from GitHub — but in order to efficiently debug and resolve issues that pluginval unearths, you’ll want to run it in a debugger.

So build and run the project locally.

Pluginval Tip 2: Run it in CI

You’ll probably want to run pluginval continually while developing a plugin to proactively catch potential problems you are introducing. Otherwise, problems will bunch up and you won’t be quite sure when they were introduced, making resolution more troublesome.

There are some straightforward docs about running in CI, which boils down to sucking down the binary and running it against your built plugin.

You can see a real life config example my Pamplejuce repo which runs pluginval on GitHub Actions.

Pluginval Tip 3: “Validate in process” to catch asserts and set breakpoints

It can feel a bit nerve wracking the first few times you run into your first FAIL. What exactly failed? How should you resolve it?

The process to resolve a failure is:

  1. Run pluginval from your IDE with a debugger
  2. Make sure “Validate in process” is checked so that the debugger can “see through” to your code from pluginval. (If you are on the command line, specify --validate-in-process).
  3. Place breakpoints in pluginval itself. For example, search the source for testRunner.setAssertOnFailure (false) and set it to true.
  4. Inspect pluginval’s code and the state of your plugin to resolve the issue.
Be sure “Validate in process” is checked

Pluginval Tip 4: Specify a seed to reproduce issues

Pluginval uses randomness throughout the code base (and optionally the test order). This means that you might see intermittent failures and slightly different results on each run.

If you want to reproduce a problematic run, grab the random seed that’s output at the start of the log:

Started validating: AudioUnit-MyPlugin-1682755d-61203125
Random seed: 0x374115a

You can then feed this seed to pluginval as an option, ensuring the exact same set of tests are run. This is especially useful when reproducing an intermittent problem on CI.

Common Problem: No types found

Starting test: pluginval / Scan for known types: /home/runner/work/myplugin/Builds/MyPlugin_artefacts/Release/VST3/MyPlugin.vst3...
Num types found: 0
!!! Test 1 failed: No types found

This translates to “We couldn’t find your plugin.”

Double check the following:

  1. The plugin is actually built and your plugin exists.
  2. The path is correct.
  3. Does your plugin .vst/.component have spaces in it? If you are using JUCE, be aware that the name of the plugin is set by PRODUCT_NAME which allows spaces.

I’ve been bitten by all 3 of these problems!

Common FAIL: Not destroying listeners

Pluginval catches all sorts of issues. Here’s a popular one in JUCE: parameter listeners being referenced after they are destroyed.

This usually boils down to “you forgot to call removeListener in a Component’s destructor”.

It will show up as a memory-related complaint where listeners are called, for example in AudioProcessorValueTreeState.cpp‘s implementation of parameterValueChanged.

Is Pluginval enough?

Occasionally, a plugin will fail validation in a DAW like logic, but pass a strictness level 10 with pluginval. Ideally, those test cases are then identified and contributed back to pluginval itself.

On MacOS, you can use apple’s auval tool on the command line to first get more detail on the failure.

auval basics

To get more detail than what logic shows you, run auval -v in Terminal.

auval -v TYPE SUBT MANU

TYPE is aufx for an audio effect or aumu for an audio instrument.

SUBT is what JUCE refers to as your 4 character PLUGIN_CODE

MANU is what JUCE refers to as your 4 character `PLUGIN_MANUFACTURER_CODE`

Here are a couple examples:

aumu Vita Tyte  -  Matt Tytel: Vital
aufx TP3X GDHZ  -  Goodhertz: Ghz Tupe 3

Debugging auval

I’ve never had to deal with this. If you get stuck, here’s a JUCE forum post for you.

Leave a comment

Your email address will not be published.