Discussion:
[linux-audio-dev] Plugin APIs (again)
Tim Hockin
2002-12-03 01:09:00 UTC
Permalink
Somehow I got unsubbed and didn't realize it. I've missed a few month of
discussion, and it seems like I've missed a lot, and yet not much has
changed :)

I regret that I can not completely read 4 months of archive, so if I need to
read something more than the skimming I've done, just point me at it, please
:)

It seems that people are still bandying about the idea of extending or
building from LADSPA, in several respects. A while back, I said I was, too.
I thought it only fair to solicit input from people here.

My goal has always been to implement a music app that tastes somewhat like
FruityLoops. Something that is easy to get started with, but has much depth
and complexity if you know how to ask for it. I envisioned it as an
all-in-one music studio application with synths, FX, etc.

LADSPA has a strong base, but I wanted a plugin API that was explicit about
note-control, multiple inputs/outputs and multi-channel audio. I wanted to
keep it pure C and truly as simple as possible, but no more. I wanted to
take away some complexity from the host without burdening the plugin too
much.

That said, here is the header file I've been dinking with. I haven't spent
a whole lot of time on it, just a few weekends thinking and
proof-of-concepting. I'd really like to get feedback on it. I've tried to
make it resemble LADSPA, as much as possible. There are a couple FIXME
notes where I've been undecided or haven't experimented completely yet.

I know there are probably other people who are writing similar apps. If we
have similar goals, I'd be happy to collaborate instead.

Thanks for the time. I'm tired of waiting for new versions of Windows music
apps, only to find the features I want STILL not implemented. Linux needs a
TRULY GOOD music studio project.

Tim



/*
* Tim Hockin <***@hockin.org>
* Copyright (c) 2001-2002 Tim Hockin
*
* This header file defines the API for the Open Audio Plugin Interface
* (OAPI).
*
* Goals:
* The main goal of this API is to provide a system that is full-featured
* enough to be the primary plugin system for an audio-development
* application, while remaining as simple, lightweight, and self-contained
* as possible.
*
* Credits:
* This API is based largely on LADSPA by Richard W.E. Furse et al.
*
* Overview:
* Plugins are loaded from shared object files. A shared object file holds
* one or more plugin descriptors, accessed by index. Each descriptor holds
* all the information about a single plugin - it's identification,
* capabilities, controls, and access methods.
*/
#ifndef OAPI_H__
#define OAPI_H__

#ifdef __cplusplus
extern "C" {
#endif

#include <stdint.h>

/*
* A single audio sample: this datatype is used for passing audio data
* between plugins and hosts. Data is normalized between -1.0 and 1.0
* as the 0dB value, with 0.0 being silence.
*/
typedef float OAPI_sample;

/* forward declarations of the fundamental OAPI types */
typedef struct OAPI_descriptor OAPI_descriptor;
typedef struct OAPI_control OAPI_control;
typedef struct OAPI_ctrl_int OAPI_ctrl_int;
typedef struct OAPI_ctrl_float OAPI_ctrl_float;
typedef struct OAPI_ctrl_enum OAPI_ctrl_enum;
typedef struct OAPI_ctrl_string OAPI_ctrl_string;
typedef struct OAPI_state OAPI_state;

/* Plugin types: all OAPI plugins are one or more of these. */
#define OAPI_TYPE_SOURCE 0x01
#define OAPI_TYPE_EFFECT 0x02
#define OAPI_TYPE_SINK 0x04

/*
* A plugin descriptor: this structure describes what an plugin can do.
* Every plugin has exactly one descriptor, which the host must treat as
* READ-ONLY.
*/
struct OAPI_descriptor {
/*
* The label is a file-unique, non-whitespace, string identifier for
* the plugin. Hosts can use the filename and label to identify a
* plugin uniquely.
*/
const char *oapi_label;

/*
* The type is a bitmask of OAPI_TYPE_* values. A plugin may
* specify more than one type, for example a synth that can be a
* vocoder.
*/
//FIXME: needed? if note_on: SRC, if ins+outs: FX, if !outs: SINK
uint_32_t oapi_type;

/*
* The serial is simply a number by which the host can compare two
* versions of a plugin and pick the later version. The actual
* value has no meaning to the host. The only requirement for this
* field is that the value never gets smaller in new releases.
*/
unsigned oapi_serial;

/*
* The name is an arbitrary string, which hosts can display to
* users.
*/
const char *oapi_name;

/*
* These are display-friendly fields, which hosts can display to
* users.
*/
const char *oapi_version;
const char *oapi_author;
const char *oapi_copyright;
const char *oapi_license;
const char *oapi_url;
const char *oapi_notes;

/*
* Plugins can have multiple inputs and outputs (in/outs). These
* fields identify the capabilities of the plugin. Negative values
* mean no limit, all other values are literal.
*/
int oapi_inputs_min;
int oapi_inputs_max;
int oapi_outputs_min;
int oapi_outputs_max;

/*
* Each input or output consists of 1 or more audio channels. These
* values are the limits per in/out.
*/
int oapi_channels_min;
int oapi_channels_max;

/*
* Plugins can have controllable variables, things that would be
* knobs, buttons, and switches in hardware. Controls are presented
* as an array of the base control type, with enough information to
* be cast to the proper specific type.
*/
int oapi_ncontrols;
OAPI_control **oapi_controls;

//FIXME: instead of void *, make it return a descriptor, which is a
//clone of this with a 'priv' field, and status fields ? eliminates
//'query' method, which is broken anyway

//FIXME: explicit negative return values == better error reporting

/*
* create: instantiate the plugin
*
* This method is used to create an instance of the plugin described
* by the descriptor. Memory will be allocated and state will be
* initialized in this method. After this method is called, but
* before the plugin's activate() method is called, the host must
* call the plugin's set_rate() method.
*
* Arguments:
* descriptor: a pointer to the plugin descriptor.
*
* Returns:
* This method returns a pointer to a plugin-private handle. If
* initialization fails for any reason, this method must return
* NULL.
*/
void *(*oapi_create)(OAPI_descriptor *descriptor);

/*
* destroy: destroy an instance
*
* This method is used to destroy and clean up after an instance of
* the plugin. All allocated resources must be released. After
* this method is invoked, the plugin handle is no longer valid.
* This function can not fail.
*
* Arguments:
* plugin: a pointer to the plugin instance.
*
* Returns:
* This method does not return a value.
*/
void (*oapi_destroy)(void *plugin);

/*
* set_rate: set the sample rate
*
* This method must be called between the create() and activate()
* methods, or the plugin will not know the host's sample rate.
* It may be called again at any time, though changing the sample
* rate on an activated plugin may be delayed by the plugin until it
* is no longer active. There is no required set of supported
* sample rates, but plugins should support the common sample
* rates (44100, 48000, 96000) to be generally useful. Hosts should
* always check that all plugins support the desired sample rate.
*
* Arguments:
* plugin: a pointer to the plugin instance.
* rate: the desired sample rate.
*
* Returns:
* This method returns 0 on success and -1 if the sample rate is
* not supported or some error occurred.
*/
int (*oapi_set_rate)(void *plugin, int rate);

/*
* activate: prepare a plugin for running
*
* This method is used to prepare a plugin before being run. This
* is an optional method, and if provided, will be called once
* before the first call to the run() method. This method will not
* be called again by the host until the deactivate() method is
* called.
*
* Arguments:
* plugin: a pointer to the plugin instance.
*
* Returns:
* This method returns 0 on success and -1 on error.
*/
int (*oapi_activate)(void *plugin);

/*
* deactivate: stop a plugin after running
*
* This method is used to shut down a plugin when the host is done
* running it. This method, if provided, will be called at least
* once some time after a call to the activate() method and before
* the destroy() method. Hosts may use the deactivate()/activate()
* pair as a way to reset the plugin's state.
*
* Arguments:
* plugin: a pointer to the plugin instance.
*
* Returns:
* This method returns 0 on success and -1 on error.
*/
int (*oapi_deactivate)(void *plugin);

/*
* connect_input,connect_output: connect buffers for data
*
* Each in/out consists of one or more audio channels. Hosts must
* connect in/outs to buffers in order for a plugin to operate. The
* plugin should use these methods to flag in/outs as active or
* inactive and to recognize the desired number of channels (up to
* the plugin maximums specified in the descriptor). Plugins are
* expected to handle the case of the input buffer being the same as
* the output buffer (in-place operation) transparently to the host.
*
* Arguments:
* plugin: a pointer to the plugin instance.
* index: the in/out number (zero-based value)
* channel: the audio channel within the in/out (zero-based value)
* buffer: the buffer in which to read/write audio data
*
* Returns:
* These methods return 0 on success and -1 on error.
*/
int (*oapi_connect_input)(void *plugin, int index, int channel,
OAPI_sample *buffer);
int (*oapi_connect_output)(void *plugin, int index, int channel,
OAPI_sample *buffer);

/*
* run: use the plugin
*
* This method invokes the plugin for a number of audio samples,
* which are provided on connected in/outs.
*
* Arguments:
* plugin: a pointer to the plugin instance.
* nsamples: the number of samples available on in/out buffers.
*
* Returns:
* This method returns 0 on success and -1 on error.
*/
int (*oapi_run)(void *plugin, int nsamples);

#if 1 //two options for note-control
/*
* voice_on: trigger a note
*
* This method is provided by source plugins, such as synthesizers
* and samplers.
*
* Arguments:
* plugin: a pointer to the plugin instance
* note: a MIDI compatible note number. MIDI specifies that note
* number 60 is middle C, and all other notes are relative, with
* half-steps assigned at whole number increments. Hosts should
* support at least 128 note values (0-127) as designated by
* MIDI, but are free to support more. Plugins may respond to
* less than the full 128 note range.
*
* Returns:
* This method returns a non-negative voice-id on success and -1 on
* error. The voice-id is per-instance and has no meaning to the
* host. Voice-ids are unique, and if a plugin returns a duplicate
* voice-id, the prior instance of that voice has been stopped.
* Plugins and hosts can both use this to control polyphony.
*/
int (*oapi_voice_on)(void *plugin, int note);

/*
* manipulate a playing voice
*/
int (*oapi_note_off)(void *plugin, int voice);
//FIXME: how to indicate that a voice has finished?
#else
typedef enum OAPI_voice_op OAPI_voice_op;
enum OAPI_voice_op {
OAPI_VOICE_NONE = 0
OAPI_VOICE_ON = 1,
OAPI_VOICE_OFF = 2,
OAPI_VOICE_ISON = 3,
};
int (oapi_voice_ctrl)(void *plugin, OAPI_voice_op, ...);
//voice_ctrl(plug, OAPI_VOICE_ON, note); /* voice-id or -1 */
//voice_ctrl(plug, OAPI_VOICE_OFF, voice); /* 0, 1 if already off, -1 */
//voice_ctrl(plug, OAPI_VOICE_ISON, voice); /* 0 or 1 */
#endif

/* query the current state */
//FIXME: need to identify channels per in/out - this is b-r-o-k-e-n
int (*oapi_query)(void *plugin, OAPI_state *state);
};

/* the various types that a control may be */
typedef enum {
OAPI_CTRL_INT = 1,
OAPI_CTRL_FLOAT,
OAPI_CTRL_ENUM,
OAPI_CTRL_STRING,
} OAPI_ctrl_type;

/* for declaring ctrl sub-types */
#define OAPI_CONTROL_COMMON_FIELDS \
/* unique (within this plugin), non-whitespace string identifier */ \
const char *ctrl_label; \
/* display-friendly name */ \
const char *ctrl_name; \
/* the type of control */ \
OAPI_ctrl_type ctrl_type; \
/*
* all ctrl types have the following methods
* int (*ctrl_set)(void *instance, <type> value);
* <type> (*ctrl_get)(void *instance);
*/

/* a knob, button, etc. of a plugin */
struct OAPI_control {
OAPI_CONTROL_COMMON_FIELDS
};
/* to help in declaring arrays of pointers to structs statically */
#define OAPI_DECL_CONTROL(type) (OAPI_control *)&(OAPI_ctrl_ ## type)

/* the expanded structs for each control type */
struct OAPI_ctrl_int {
OAPI_CONTROL_COMMON_FIELDS
int ctrl_min;
int ctrl_max;
int ctrl_default;
uint_32_t ctrl_flags;
int (*ctrl_set)(void *plugin, int value);
int (*ctrl_get)(void *plugin);
};

struct OAPI_ctrl_float {
OAPI_CONTROL_COMMON_FIELDS
float ctrl_min;
float ctrl_max;
float ctrl_default;
uint_32_t ctrl_flags;
int (*ctrl_set)(void *plugin, float value);
float (*ctrl_get)(void *plugin);
};

struct OAPI_ctrl_enum {
OAPI_CONTROL_COMMON_FIELDS
int ctrl_min;
int ctrl_max;
int ctrl_default;
const char **ctrl_options;
int (*ctrl_set)(void *plugin, int value);
int (*ctrl_get)(void *plugin);
};

struct OAPI_ctrl_string {
OAPI_CONTROL_COMMON_FIELDS
const char *ctrl_default;
uint_32_t ctrl_flags;
int (*ctrl_set)(void *plugin, const char *value);
char *(*ctrl_get)(void *plugin);
};

/* flags for various control types */
#define OAPI_CTRL_FL_SCALAR 0x01 /* is scalar to the sample rate */
#define OAPI_CTRL_FL_LOG 0x02 /* is logarithmic */
#define OAPI_CTRL_FL_FILE 0x04 /* is a file name */
#define OAPI_CTRL_FL_DIR 0x08 /* is a directory */
#define OAPI_CTRL_FL_BOOL 0x10 /* is a boolean */
#define OAPI_CTRL_FL_FRAMES 0x20 /* is a measurement of frames */
#define OAPI_CTRL_FL_RDONLY 0x40 /* is read-only */

/* how a host queries the state of a plugin */
struct OAPI_state {
int st_inputs;
int st_outputs;
int st_channels;
};

/* how the host learns about the plugin */
OAPI_descriptor *oapi_descriptor(int index);

#ifdef __cplusplus
} /* extern "C" */
#endif

#endif /* OAPI_H__ */
Paul Davis
2002-12-03 03:46:01 UTC
Permalink
Post by Tim Hockin
I know there are probably other people who are writing similar apps. If we
have similar goals, I'd be happy to collaborate instead.
Thanks for the time. I'm tired of waiting for new versions of Windows music
apps, only to find the features I want STILL not implemented. Linux needs a
TRULY GOOD music studio project.
come on over and get coding on ardour! it will be as good as you care
to help make it.

--p
Conrad Parker
2002-12-03 07:00:01 UTC
Permalink
Hi Tim,

I like the style of your proposed API. A few comments:

*) Scope: your stated goal is to produce an API that is "full-featured
enough to be the primary plugin system for an audio-development
application". I think it would be more correct to say that this is
an API designed for virtual instrument plugins, with a programmatic
rather than GUI interface (as LADSPA is for effects plugins).

This is not a bad thing, in fact I think we are better served with
small, focussed plugin APIs for specific tasks -- and virtual
instruments and effects are two very worthwhile specific tasks.

Also, there are plugins that cannot be accomodated by this API; eg.
causal plugins (eg. reversal of large buffers), plugins for which
nsamples in != nsamples out (eg. time stretching and rate conversion),
and plugins which need to do a number of passes (eg. GWC's noise
reduction needs to profile some quiet data in a first pass); not to
mention that this plugin implies time domain manipulation only.

I don't think the scope of your API should be expanded to include
these, as it would lose much of its elegance -- it would be awful to
clutter the common cases (instruments and effects) just for the needs
of offline and highly analytic plugins. However I would suggest that
you refine the stated scope (eg. instruments and inline effects) and
concentrate on doing a good job of that.

*) Note control: I prefer your first method (separate voice_on() and off()
functions, rather than a single voice_ctrl() function) -- for the
same reason that "ioctls are bad, mmkay" and syscalls are good, ie.
there's no need for a cryptic overloaded control function, especially
here to implement only a well defined set of operations.

*) Identifying controls: It would be quite useful to share the same get()
and set() functions between a number of controls (eg. similar sliders
of an equaliser), for which you would want eg. an int parameter
identifying the control index. (In those situations, making a bunch of
tiny interface functions within the plugin is pretty ugly).

*) Descriptions: It'd be nice to supply both short and long descriptions
for controls as well as for the whole plugin.

that's it from me for now; nice work overall, I like that it treats
multiple channels coherently and handles enum and string inputs, that
makes generated GUIs a lot nicer :)

Conrad.
Tim Hockin
2002-12-03 10:16:01 UTC
Permalink
Post by Paul Davis
Post by Tim Hockin
I know there are probably other people who are writing similar apps. If we
have similar goals, I'd be happy to collaborate instead.
Thanks for the time. I'm tired of waiting for new versions of Windows music
apps, only to find the features I want STILL not implemented. Linux needs a
TRULY GOOD music studio project.
come on over and get coding on ardour! it will be as good as you care
to help make it.
So I took a look, and Ardour seems great, but somewhat orthogonal to my
idea, which is much more about sequencing virtual instuments. YOu're trying
to replace ProTools (and doing reasonably well, it seems) and I'm trying to
replace FruityLoops or Reason.

Do you see a melding of Ardour with that ideal? My general design for my
app has been VERY plugin-centric. I imagine the mixer, audio-recording (if
any), audio-output, and everything in between as plugins. Also, I don't
know C++ too well, though I guess I should learn it.

Before I dig too far into your code - do you want me to try to steer it
towards my own ideas, or would you rather we just use JACK to each do what
we do well and still play together?

Tim

p.s. Whether I work on Ardour or not, I'm stealing some of your code/ideas :)
Erik de Castro Lopo
2002-12-03 10:34:01 UTC
Permalink
On Tue, 3 Dec 2002 01:11:11 -0800 (PST)
Post by Tim Hockin
Before I dig too far into your code - do you want me to try to steer it
towards my own ideas, or would you rather we just use JACK to each do what
we do well and still play together?
Thats the way. I don't think its smart to try and make one app which does
everything. However multiple apps work togther and share components like
libsndfile and Secret Rabbit Code is the way to go.

Erik
--
+-----------------------------------------------------------+
Erik de Castro Lopo ***@mega-nerd.com (Yes it's valid)
+-----------------------------------------------------------+
"Perl : this is a blue collar language" - Angus Lees
Steve Harris
2002-12-03 13:00:01 UTC
Permalink
Fisrt off, I agree with Conrad, its better to make this fucused on, and
really good at, one task (ie. instruments), that be OK at lots of things.
Post by Tim Hockin
/* forward declarations of the fundamental OAPI types */
typedef struct OAPI_descriptor OAPI_descriptor;
typedef struct OAPI_control OAPI_control;
typedef struct OAPI_ctrl_int OAPI_ctrl_int;
typedef struct OAPI_ctrl_float OAPI_ctrl_float;
typedef struct OAPI_ctrl_enum OAPI_ctrl_enum;
Enum's good, I'm not so sure about int or string, a well defined enum type
probably makes string unneccesary and int is probably not needed.
Post by Tim Hockin
/* Plugin types: all OAPI plugins are one or more of these. */
#define OAPI_TYPE_SOURCE 0x01
#define OAPI_TYPE_EFFECT 0x02
#define OAPI_TYPE_SINK 0x04
I dont think this helps, it should be obvious by inspection whether an
instrument is appropraite for the situation the host wants to fill.
Post by Tim Hockin
/*
* These are display-friendly fields, which hosts can display to
* users.
*/
const char *oapi_version;
const char *oapi_author;
const char *oapi_copyright;
const char *oapi_license;
const char *oapi_url;
const char *oapi_notes;
If you're going to support metadata, support the Dublin Core, but generally
extrnal metadata is better than internal. Just provide a unique ID and
metadata can refer to that.
Post by Tim Hockin
int oapi_inputs_min;
int oapi_inputs_max;
int oapi_outputs_min;
int oapi_outputs_max;
Variable numbers of i/o is interesting, but I'm not sure if it
overcomplicates things from the instruments point of view. Can you explain
why you think this is a good idea?
Post by Tim Hockin
int oapi_channels_min;
int oapi_channels_max;
I'm pretty confident this is not the right way to go. If it matters to you
what output belongs to what channel then well known labels would be
better, I think.
Post by Tim Hockin
/*
* create: instantiate the plugin
Maybe better to stiuck to LADSPA nomenclature? It might not be perfect,
but it would avoid confusion.
Post by Tim Hockin
/*
* set_rate: set the sample rate
LADSPA definatly got this right, make it an argument to instantiate.
Changing the sample rate of system is not an RT operation, so theres no
need to make it ultra efficient, and instruments will have a very hard
time rejigging everything. Simpler to just remove and instatiate a new one
with the same settings.
Post by Tim Hockin
#if 1 //two options for note-control
/*
* voice_on: trigger a note
*/
int (*oapi_voice_on)(void *plugin, int note);
/*
* manipulate a playing voice
*/
int (*oapi_note_off)(void *plugin, int voice);
I prefer this one.

After reading this I think the best solution is something very like LADSPA
but with note_on and note_off calls. Other useful things like enum types
/can/ be added via external metadata, eg RDF ;)

- Steve
David Olofson
2002-12-03 15:03:01 UTC
Permalink
(Steve has made some good points that I agree with mostly, so I chose
to comment on his reply.)
Post by Steve Harris
Fisrt off, I agree with Conrad, its better to make this fucused on,
and really good at, one task (ie. instruments), that be OK at lots
of things.
Yes. Though, I think using a virtual instruments capable API for
effects can be great, especially when it comes to automation (both
need to deal with accurate timing and control interpolation), but of
course, there's always the ease-of-use/complexity trade-off; a
virtual instruments API will always be more complex than LADSPA.
Post by Steve Harris
Post by Tim Hockin
/* forward declarations of the fundamental OAPI types */
typedef struct OAPI_descriptor OAPI_descriptor;
typedef struct OAPI_control OAPI_control;
typedef struct OAPI_ctrl_int OAPI_ctrl_int;
typedef struct OAPI_ctrl_float OAPI_ctrl_float;
typedef struct OAPI_ctrl_enum OAPI_ctrl_enum;
Enum's good, I'm not so sure about int or string, a well defined
enum type probably makes string unneccesary and int is probably not
needed.
Well, I'm not sure it's worthwhile going beyond floats + "hints"...
Strings could be interesting, as but how should the host deal with
them? Are they file names, just names, scripts, or what? How to edit
them? How to store them in "preset files" and the like?

I have a feeling that going down this route either results in complex
hosts, or a pointless "official" API for stuff that only matters to
plugins and their custom GUIs anyway - or both.

How about just supporting a "raw data block" control type, so that
hosts can store the data in preset files or automation, but without
understanding the data? Custom (G)UIs would be needed for editing the
data - just as with VST, although the (G)UIs are not part of the
plugin binaries here. (I think we have concluded long ago that GUI
and DSP code should be entirely separated.)
Post by Steve Harris
Post by Tim Hockin
/* Plugin types: all OAPI plugins are one or more of these. */
#define OAPI_TYPE_SOURCE 0x01
#define OAPI_TYPE_EFFECT 0x02
#define OAPI_TYPE_SINK 0x04
I dont think this helps, it should be obvious by inspection whether
an instrument is appropraite for the situation the host wants to
fill.
Agreed. The VST API has something like this, and it seems to have
been more harmful and confusing than helpful. Not sure if this was
actually specified in the API originally, but people have come to
assume that VST plugins and VSTi plugins have other implicit
differences than the latter accepting timestamped events (like VSTis
not being able to have audio inputs) - which results in trouble when
people start exploring "unusual" configurations.

If it has outputs, it's a source. If it has inputs, it's a sink. If
it has both, it's (probably) an effect. Either way, the relation
between input and output can be of *any* kind (within the
restrictions of the API), so there's not much point in trying to
describe it to the host.

Likewise with timestamped "MIDI style" control events - either you
accept them, or you don't. Whether you have audio inputs, outputs or
both is another story entirely.
Post by Steve Harris
Post by Tim Hockin
/*
* These are display-friendly fields, which hosts can display to
* users.
*/
const char *oapi_version;
const char *oapi_author;
const char *oapi_copyright;
const char *oapi_license;
const char *oapi_url;
const char *oapi_notes;
If you're going to support metadata, support the Dublin Core, but
generally extrnal metadata is better than internal. Just provide a
unique ID and metadata can refer to that.
I'd definitely go for Unique IDs + external data here. Oh, and it's
probably a good idea to structure the ID as "vendor" and "product"
fields, so you don't have to aquire a global unique ID for every
plugin you release... Just get a vendor ID, and then manage your
plugin IDs yourself.
Post by Steve Harris
Post by Tim Hockin
int oapi_inputs_min;
int oapi_inputs_max;
int oapi_outputs_min;
int oapi_outputs_max;
Variable numbers of i/o is interesting, but I'm not sure if it
overcomplicates things from the instruments point of view. Can you
explain why you think this is a good idea?
Well, I don't know Tim's reasons, I could give it a try: "Horizontal"
mixer modules, multipart synths and samplers and that kind of stuff.
Sure, you could probably just say you have 128 ins and 128 outs (or
whatever) - but then, would hosts assume that all are actually
initialized and in use? I see a risk of complicating port management
here, if ports can be "disabled" as well as "connected" or
"disconnected". It might be better if they just don't exist if you
don't need them.

Besides, I think fixed size arrays and the like are generally a bad
idea, and should be avoided whenever practically possible. You alway
hit the limit sooner or later...
Post by Steve Harris
Post by Tim Hockin
int oapi_channels_min;
int oapi_channels_max;
I'm pretty confident this is not the right way to go. If it matters
to you what output belongs to what channel then well known labels
would be better, I think.
Yeah, just as with input/output relations, there's no useful and
reasonably simple way of describing channel/input/output relations.
This just adds a dimension of indexing, for no gain.

Besides, you would at least have to have one pair of limits for each
input and each output for this to be of any use. (Unless all inputs
are required to be identical, and likewise for outputs.)

Consider a mixer, where you have insert sends and returns, as well as
bus sends and returns. Obviously, you'd probably want to be able to
use some 128 channels or so - but most probably not 128 busses!
Post by Steve Harris
Post by Tim Hockin
/*
* create: instantiate the plugin
Maybe better to stiuck to LADSPA nomenclature? It might not be
perfect, but it would avoid confusion.
Here's another idea (from Audiality):

typedef enum
{
FX_STATE_CLOSED = 0,
FX_STATE_OPEN,
FX_STATE_READY,
FX_STATE_PAUSED,
FX_STATE_RUNNING,
FX_STATE_SILENT,
FX_STATE_RESTING
} a_fxstates_t;

typedef struct a_plugin_t
{
...
/* Plugin state management */
int (*state)(struct a_plugin_t *p, a_fxstates_t new_state);
...
} a_plugin_t;

The host is required to "climb" the state ladder, calling the
plugin's state() callback, never skipping a state - but there's a
handy wrapper in the API that deals with that. (So you can switch
directly from CLOSED to RUNNING with a single call if you like.)

Dunno, but I've found this to be clean, simple and safe so far. It's
hard to accidentally have the host confuse plugins by doing things in
the wrong order - and it's also strictly specified what plugins may
and may not do in each state, and when switching between any two
adjacent states.
Post by Steve Harris
Post by Tim Hockin
/*
* set_rate: set the sample rate
LADSPA definatly got this right, make it an argument to
instantiate. Changing the sample rate of system is not an RT
operation, so theres no need to make it ultra efficient, and
instruments will have a very hard time rejigging everything.
Simpler to just remove and instatiate a new one with the same
settings.
I totally agree.
Post by Steve Harris
Post by Tim Hockin
#if 1 //two options for note-control
/*
* voice_on: trigger a note
*/
int (*oapi_voice_on)(void *plugin, int note);
/*
* manipulate a playing voice
*/
int (*oapi_note_off)(void *plugin, int voice);
I prefer this one.
After reading this I think the best solution is something very like
LADSPA but with note_on and note_off calls. Other useful things
like enum types /can/ be added via external metadata, eg RDF ;)
Here, I prefer timestamped events, for various reasons. Most
importantly, timestamped events allow sample accurate control without
the host splitting buffers, and it allows plugins to implement event
handling in any way they like - including just executing all events
before processing each audio buffer, for quick hacks. (You could even
throw in a macro for that, for quick LADSPA-><new plugin API>
porting.)

I've been using this for a while in Audiality (very similar to the
event system designed for MAIA), and there's no turning back;
timestamped events are simply superior, IMNSHO.

Might look a bit more complicated than direct function calls at
first, but in my experience, it actually *simplifies* all but the
very basic stuff. Filtering, proccesing and routing of events can be
done by inserting plugins in between other plugins, and lots of it
becomes trivial, as you can disregard the timestamps altogether in
many cases.

I thought an event system would result in some overhead (linked
lists, gloabal event pool and stuff), but in fact, the old "once per
block" function call based implementation (*) was slower - and
obviously, it was not sample accurate. (No buffer splitting - that
would have slowed things down further.) It seems that staying within
the inner loops (event processing and DSP) for one unit at a time
more than makes up for the overhead that the event system brings.

(*) Note that that was mostly inlines messing with the internals of
the respective unit - which obviously won't work with dynamically
loaded plugins. There, every "action" has to be a real function
call via function pointer. With an event system like the one in
Audiality, there's no difference for dynamically loaded plugins,
as you're still just passing little structs around.

Oh, BTW, events can easilly be tunnelled through whatever protocol
you like (between threads, processe or systems), as the timing info
is there already. Just make sure they arrive in time, and you're
fine; order and accuracy is preserved, provided the "time base" is
synchronized. (In a studio, you'd sync the audio interfaces to ensure
that.)


//David Olofson - Programmer, Composer, Open Source Advocate

.- Coming soon from VaporWare Inc...------------------------.
| The Return of Audiality! Real, working software. Really! |
| Real time and off-line synthesis, scripting, MIDI, LGPL...|
`-----------------------------------> (Public Release RSN) -'
.- M A I A -------------------------------------------------.
| The Multimedia Application Integration Architecture |
`----------------------------> http://www.linuxdj.com/maia -'
--- http://olofson.net --- http://www.reologica.se ---
David Olofson
2002-12-03 17:13:00 UTC
Permalink
(Commenting on my own reply. Just thinking too much as usual. :-)
Post by Tim Hockin
Post by Steve Harris
Post by Tim Hockin
/*
* create: instantiate the plugin
Maybe better to stiuck to LADSPA nomenclature? It might not be
perfect, but it would avoid confusion.
typedef enum
{
FX_STATE_CLOSED = 0,
FX_STATE_OPEN,
FX_STATE_READY,
FX_STATE_PAUSED,
FX_STATE_RUNNING,
FX_STATE_SILENT,
FX_STATE_RESTING
} a_fxstates_t;
typedef struct a_plugin_t
{
...
/* Plugin state management */
int (*state)(struct a_plugin_t *p, a_fxstates_t new_state);
...
} a_plugin_t;
Maybe I should point out that a_plugin_t is the *instance* type for
an Audiality plugin. Currently, I use a single entry to instantiate a
plugin. Example:

void delay_init(struct a_plugin_t *p)
{
p->state = delay_state;
p->control = delay_control;
p->process = delay_process;
p->process_r = delay_process_r;
}

I suppose one could add a state FX_STATE_VOID (or something) below
FX_STATE_CLOSED, and move instantiation and destruction (which is a
NOP as of now, as the host manages the a_plugin_t structs) into the
state() callback. Then just export the state() call, instead of an
extra init func. (Or use a "fake" state() call if desired, which
installs the real one in the provided a_plugin_t struct - coder's
choice.)

However, one should probably have some more arguments for the init
calls, to support multiple plugins, variants and stuff in a single
.so - so I guess it's better kept the way it is... :-)


//David Olofson - Programmer, Composer, Open Source Advocate

.- Coming soon from VaporWare Inc...------------------------.
| The Return of Audiality! Real, working software. Really! |
| Real time and off-line synthesis, scripting, MIDI, LGPL...|
`-----------------------------------> (Public Release RSN) -'
.- M A I A -------------------------------------------------.
| The Multimedia Application Integration Architecture |
`----------------------------> http://www.linuxdj.com/maia -'
--- http://olofson.net --- http://www.reologica.se ---
Tim Hockin
2002-12-03 23:04:00 UTC
Permalink
I'm replying to all the replies in one email, hope no one minds :)
Post by Steve Harris
Fisrt off, I agree with Conrad, its better to make this fucused on, and
really good at, one task (ie. instruments), that be OK at lots of things.
Agreed withone exception - It is meant to be good at instruments, effects
and outputs. In my worldview almost everything is a plugin.
Post by Steve Harris
Post by Tim Hockin
typedef struct OAPI_ctrl_int OAPI_ctrl_int;
typedef struct OAPI_ctrl_float OAPI_ctrl_float;
typedef struct OAPI_ctrl_enum OAPI_ctrl_enum;
Enum's good, I'm not so sure about int or string, a well defined enum type
probably makes string unneccesary and int is probably not needed.
Strings are for things like files, primarily. There is no way to
autogenerate any sort of UI for anything dealing with files without some
understanding of strings natively. Int is merely for completeness. There
are things that are better represented as ints, and since there is this
notion of type-specific controls, it makes some things just a hair easier.
I had bool in there too, but realized bool is a special-case int. Maybe int
is a special-case float, but I don't really think so. Simple, but not too
simple.
Post by Steve Harris
Post by Tim Hockin
/* Plugin types: all OAPI plugins are one or more of these. */
#define OAPI_TYPE_SOURCE 0x01
#define OAPI_TYPE_EFFECT 0x02
#define OAPI_TYPE_SINK 0x04
I dont think this helps, it should be obvious by inspection whether an
instrument is appropraite for the situation the host wants to fill.
I think I agree with this - I left a fixme to that effect in the header..
Post by Steve Harris
Post by Tim Hockin
/*
* These are display-friendly fields, which hosts can display to
* users.
*/
const char *oapi_version;
const char *oapi_author;
const char *oapi_copyright;
const char *oapi_license;
const char *oapi_url;
const char *oapi_notes;
If you're going to support metadata, support the Dublin Core, but generally
extrnal metadata is better than internal. Just provide a unique ID and
metadata can refer to that.
Firstly, I hate the idea that a plugin writer needs to procure a unique ID.
Secondly, I think some meta-data needs to be IN the plugin. Now, maybe
'notes', and 'url' can be outside the plugin, but
author/copyright/license/version can't. Think of commercial plugin (I know,
pipe dreams, but just pretend :). As a commercial vendor you DO NOT want it
to be trivial for a person to 'lose' the info about the license. I can
think of a several reasons to keep it all together, and no good reason NOT
to. Specifically, this data is inseperable from the plugin. As an author,
I want this data joined at the hip with the plugin. Further, the idea that
a plugin consists of a single .so file appeals to me. Presets? I guess those
can live entirely in external files. I can't see ANY good reason to make
such simple meta-data reside externally. Maybe I can be convinced - a few
people chided in on this point - can anyone tell me WHY?
Post by Steve Harris
Post by Tim Hockin
int oapi_inputs_min;
int oapi_inputs_max;
int oapi_outputs_min;
int oapi_outputs_max;
Variable numbers of i/o is interesting, but I'm not sure if it
overcomplicates things from the instruments point of view. Can you explain
why you think this is a good idea?
It stems from experiences with virtual drum machines and synths. It is very
nice to provide a plugin that does things like slicing (thing .rex) or
drum-pads and route each pad/slice to a different set of FX. Further, I
envision a mixer effect which has multiple inputs, and potentially multiple
outputs, and wants to process each input discretely before mixing. I didn't
think it made sense to make a hard limit, much less a limit of 1. The
reality is that MOST instruments/effects will have 1 in and/or 1 out. Those
with more than 1 should be presented by the host as needing special
attention, and can make assumptions as the author desires. For example, a
drum-machine with 16 outs which has only been connected on the first input
can assume that the other 15 pads should be internally routed to the first
output.

Perhaps it is useful to design some way for the host to specify this:
oapi_route_output(plug, 1..15, 0);. Or perhaps it is just policy that if a
plug has multiple outs but only the first out is connected, all outs
re-route to out0. Needs more thought.
Post by Steve Harris
Post by Tim Hockin
int oapi_channels_min;
int oapi_channels_max;
I'm pretty confident this is not the right way to go. If it matters to you
what output belongs to what channel then well known labels would be
better, I think.
I wanted to make sure that this API handles mono plugs, stereo plugs, 5.1
plugs, and whatever turns up next without re-write. I kind of assumed that
1 channel = MONO, 2 = LEFT,RIGHT, 6 = FL,FC,FR,RL,RR,LFE or some such
mapping. Again the idea being that a plugin may handle more than 1 channel
in different ways, and that each in/out may not have the same # of channels.
A multi-pad drum machine may have a stereo snare, but a mono kick. Maybe
this is not a good idea, I haven' decided. That's why I sent it here for
discussion :) Can you expound on the well-known labels idea? I'm not sure
I fully understand what you're thinking.
Post by Steve Harris
Post by Tim Hockin
* create: instantiate the plugin
Maybe better to stiuck to LADSPA nomenclature? It might not be perfect,
but it would avoid confusion.
Mmm, perhaps.
Post by Steve Harris
Post by Tim Hockin
/*
* set_rate: set the sample rate
LADSPA definatly got this right, make it an argument to instantiate.
Changing the sample rate of system is not an RT operation, so theres no
need to make it ultra efficient, and instruments will have a very hard
time rejigging everything. Simpler to just remove and instatiate a new one
with the same settings.
ok, need to rework this then. I had done it that way but decided this was
simpler, but I suppose you are correct :)
Post by Steve Harris
Post by Tim Hockin
int (*oapi_voice_on)(void *plugin, int note);
int (*oapi_note_off)(void *plugin, int voice);
I prefer this one.
I do too, but I don't want to have 10 APIs that are purely for instruments
and not effects. Add to those two the need for a voice_ison() api for proper
voice accounting of things like triggered samples, and eventually a
voice_param() to change dynamic parameters per-voice (something I have not
introduced in this API yet). I'd like to be able to say 'increase the
volume of this existing voice'. Fruityloops does it, and I've found it
to be quite conventient for things like strings and pads. FL does
Velocity, Pan, Filter Cut, Filter Res, and Pitch Bend. Not sure which of
those I want to support, but I like the idea.
Post by Steve Harris
After reading this I think the best solution is something very like LADSPA
but with note_on and note_off calls. Other useful things like enum types
/can/ be added via external metadata, eg RDF ;)
Lots CAN be added via meta-data, but my goal was to be as simple as makes
sense, and no simpler. Keeping LADSPA (imho) overly simple and using
external metadata is kind of ugly (again, imho). We control LADSPA (well,
Richard does). He can evolve it or change it and make a LADSPA v2 which
implements more advanced features. BUT there are lots of reasons not too.
The idea of this API was to take advntage of all the great LADSPA plugins in
a more complex host. One of the first plugins I will write to this API is a
LADSPA n-way adapter. You can take a mono LADSPA effect and apply it to as
many channels as the host cares to pass it. The host is simple. The LADSPA
is simple. Complexity hidden in a plugin.
Post by Steve Harris
Yes. Though, I think using a virtual instruments capable API for
effects can be great, especially when it comes to automation (both
need to deal with accurate timing and control interpolation), but of
course, there's always the ease-of-use/complexity trade-off; a
virtual instruments API will always be more complex than LADSPA.
I don't think it needs to be TOO much more complex. The areas I've added
complexity:
* multi-in/out
* multi-channel
* type-specific controls
* voice_on/off
Post by Steve Harris
Well, I'm not sure it's worthwhile going beyond floats + "hints"...
Strings could be interesting, as but how should the host deal with
them? Are they file names, just names, scripts, or what? How to edit
them? How to store them in "preset files" and the like?
String is a primitive - the control flags can specify a filename or a
directory name. I'm sure that filename will be the most common case. It
seems like String controls probably do not want to be automated. But I see
no reason they CAN'T be. All this complexity is to make things that plugins
want to say (enumerated values, integral values) less complex and less
hacked on the back of something else. As simple as makes sense and no
simpler.
Post by Steve Harris
I have a feeling that going down this route either results in complex
hosts, or a pointless "official" API for stuff that only matters to
plugins and their custom GUIs anyway - or both.
Custom GUIs don't even need to identify filenames as controls - they can do
it themselves. The point of making it a control is to allow them to NOT
have a custom GUI
Post by Steve Harris
How about just supporting a "raw data block" control type, so that
hosts can store the data in preset files or automation, but without
I don't get what you're devolving it to. The point of Strings was to say 'I
want a filename' or 'Any string will do for this field named SPEECH'. Maybe
a raw data block is useful, but I haven't seen an example. The types I
specified are based on experience using FruityLoops and similar apps and
dealing with VST instruments and other plugins and trying to proived the
simplest subset of what they need without oversimplifying.
Post by Steve Harris
plugin binaries here. (I think we have concluded long ago that GUI
and DSP code should be entirely separated.)
absolutely.
Post by Steve Harris
If it has outputs, it's a source. If it has inputs, it's a sink. If
it has both, it's (probably) an effect. Either way, the relation
between input and output can be of *any* kind (within the
restrictions of the API), so there's not much point in trying to
describe it to the host.
As above, I think I agree here.
Post by Steve Harris
Likewise with timestamped "MIDI style" control events - either you
accept them, or you don't. Whether you have audio inputs, outputs or
both is another story entirely.
In my worldview MIDI is ENTIRELY the host's problem. The
voice_on/voice_off/voice_change API is intended to provide the majority of
MIDI controlability. The host can translate from actual MIDI.
Post by Steve Harris
I'd definitely go for Unique IDs + external data here. Oh, and it's
but why?
Post by Steve Harris
probably a good idea to structure the ID as "vendor" and "product"
I though about doing a Vendor Code. But why burden some person with
distributing vendor IDs and maintaining that database, when storing a
string is just as simple. It's not like a few bytes of memory are going to
matter. Now, I'm NOT trying to be flippant about memory usage. I just want
to be realistic.
Post by Steve Harris
Well, I don't know Tim's reasons, I could give it a try: "Horizontal"
mixer modules, multipart synths and samplers and that kind of stuff.
Sure, you could probably just say you have 128 ins and 128 outs (or
whatever) - but then, would hosts assume that all are actually
initialized and in use? I see a risk of complicating port management
here, if ports can be "disabled" as well as "connected" or
"disconnected". It might be better if they just don't exist if you
don't need them.
Morte or less - yes. A plugin sets it's maximum/minimum. The host can use
or not use them. A modular system (a la Reason) would let the user drag
cables. A simple system would set up a lot of it automagically.
Post by Steve Harris
Besides, I think fixed size arrays and the like are generally a bad
idea, and should be avoided whenever practically possible. You alway
hit the limit sooner or later...
Agreed 100000% I refuse to fixed-size limit anything, if possible. Hence
->oapi_ncontrols and ->oapi_controls
Post by Steve Harris
Post by Tim Hockin
I'm pretty confident this is not the right way to go. If it matters
to you what output belongs to what channel then well known labels
would be better, I think.
Yeah, just as with input/output relations, there's no useful and
reasonably simple way of describing channel/input/output relations.
This just adds a dimension of indexing, for no gain.
Perhaps. Perhaps different channels-counts per in.out is a bad idea. But I
see a definate need for different channels-counts on input vs. output. What
of a mono-izer. Or a mono->stereo. Or Stereo->5.1.
Post by Steve Harris
Besides, you would at least have to have one pair of limits for each
input and each output for this to be of any use. (Unless all inputs
are required to be identical, and likewise for outputs.)
Limits are limits. Current value is different. The host already knows the
current value for everything but instruments. It did the connecting. I
admit to not having a good answer yet for asking an instrument how many
channels on a given out. It could be simple: oapi_nchannels(plug, outnum);
Post by Steve Harris
Consider a mixer, where you have insert sends and returns, as well as
bus sends and returns. Obviously, you'd probably want to be able to
use some 128 channels or so - but most probably not 128 busses!
I want to avoid pure hardware paradigms. But you are correct in that it can
get complex. Maybe it needs to be curtailed, but it definitely needs to
support multi-channel audio. Ideas welcome.
Post by Steve Harris
typedef enum
{
FX_STATE_CLOSED = 0,
FX_STATE_OPEN,
FX_STATE_READY,
FX_STATE_PAUSED,
FX_STATE_RUNNING,
FX_STATE_SILENT,
FX_STATE_RESTING
} a_fxstates_t;
Similar to gstreamer. Need to comtemplate this. Can you elucidate on the
different states? It seems like the same as multiple callbacks, just in one
callback. How about params?
Post by Steve Harris
the wrong order - and it's also strictly specified what plugins may
and may not do in each state, and when switching between any two
adjacent states.
can you post docs on this?
Post by Steve Harris
Here, I prefer timestamped events, for various reasons. Most
importantly, timestamped events allow sample accurate control without
the host splitting buffers, and it allows plugins to implement event
handling in any way they like - including just executing all events
before processing each audio buffer, for quick hacks. (You could even
throw in a macro for that, for quick LADSPA-><new plugin API>
porting.)
I've been using this for a while in Audiality (very similar to the
event system designed for MAIA), and there's no turning back;
timestamped events are simply superior, IMNSHO.
Might look a bit more complicated than direct function calls at
first, but in my experience, it actually *simplifies* all but the
very basic stuff. Filtering, proccesing and routing of events can be
done by inserting plugins in between other plugins, and lots of it
becomes trivial, as you can disregard the timestamps altogether in
many cases.
Can you talk more about it - I don't know the paradigm. I'm open to ideas
:)


I REALLY appreciate all the feedback. I will incorporate the bits I
definitely agree with and hopefully I can get more discussion about the bits
I am unsure about. I'm willing to be convinced if you are :)


Tim
David Olofson
2002-12-04 03:10:01 UTC
Permalink
On Tuesday 03 December 2002 22.58, Tim Hockin wrote:
[...unique IDs...]
Post by David Olofson
Maybe I
can be convinced - a few people chided in on this point - can
anyone tell me WHY?
Well, a guaranteed unique ID is really rather handy when you want to
load up a project on another system and be *sure* that you're using
the right plugins... That's about the only strong motivation I can
think of right now, but it's strong enough for me.

Now, you could claim that author, plugin name and version should do,
but there are no guarantees... Unless you throw a unique author ID in
there *as well*. How about this:

int author_id; /* Plugin author's unique ID */
const char *name; /* Plugin name */
int version; /* 8:8:16 major:minor:patchlevel */

(And then whatever optional data you may want, which is not
considered by the host.)

Plugins are assumed to be compatible/equivalent if all of the
following are true:

* They have identical author_id fields.
* They have identical names.
* They have identical major version numbers.

When loading a net, the host must also ensure that the minor version
numbers of loaded plugins are equal to, or greater than the minor
versions in the net description. (Older minor verisons may lack
features, but newer versions should only *add* capabilities, that
shouldn't matter unless you use them. New major number effectively
means that you have a new, different plugin.)


[...variable # of ins and outs...]
Post by David Olofson
It stems from experiences with virtual drum machines and synths.
It is very nice to provide a plugin that does things like slicing
(thing .rex) or drum-pads and route each pad/slice to a different
set of FX. Further, I envision a mixer effect which has multiple
inputs, and potentially multiple outputs, and wants to process each
input discretely before mixing. I didn't think it made sense to
make a hard limit, much less a limit of 1. The reality is that
MOST instruments/effects will have 1 in and/or 1 out. Those with
more than 1 should be presented by the host as needing special
attention, and can make assumptions as the author desires.
For
example, a drum-machine with 16 outs which has only been connected
on the first input can assume that the other 15 pads should be
internally routed to the first output.
IMHO, plugins should not worry about whether or not their outputs are
connected. (In fact, there are reasons why you'd want to always
guarantee that all ports are connected before you let a plugin run.
Open inputs would be connected to a dummy silent output, and open
outputs would be connected to a /dev/null equivalent.)

Indeed, one could use the "connected"/"not connected" info of ports
as some kind of interface to the plugin, but to me, it seems way too
limited to be of any real use. You still need a serious interface
that lets you configure the internals of the plugin the way you want,
just like you program a studio sampler to output stuff to the outputs
you want. This interface may be standardized or not - or there may be
both variants - but either way, it has to be more sophisticated than
one bit per output.
Post by David Olofson
Perhaps it is useful to design some way for the host to specify
this: oapi_route_output(plug, 1..15, 0);. Or perhaps it is just
policy that if a plug has multiple outs but only the first out is
connected, all outs re-route to out0. Needs more thought.
I think there should be as little policy as possible in an API. As
in; if a plugin can assume that all ins and outs will be connected,
there are no special cases to worry about, and thus, no need for a
policy.

Note that you can still tell the host that you can have anything from
1 through 16 outputs. The plugin may do whatever it likes when the
host says "i want only one out" - including being "smart" about the
internal routing, disregarding any user presets. (Some hardware boxes
seem to do this in some cases, but it seems to be meant more as a
helper when you're in that "where the h*ll is my signal!?" situation
in the studio. A good software studio would let you turn on various
indicators and meters all over the place instead, and preferably also
provide a "Free Listen" feature that lets you monitor anything you
click on.)
Post by David Olofson
Post by Steve Harris
Post by Tim Hockin
int oapi_channels_min;
int oapi_channels_max;
I'm pretty confident this is not the right way to go. If it
matters to you what output belongs to what channel then well
known labels would be better, I think.
I wanted to make sure that this API handles mono plugs, stereo
plugs, 5.1 plugs, and whatever turns up next without re-write. I
kind of assumed that 1 channel = MONO, 2 = LEFT,RIGHT, 6 =
FL,FC,FR,RL,RR,LFE or some such mapping. Again the idea being that
a plugin may handle more than 1 channel in different ways, and that
each in/out may not have the same # of channels. A multi-pad drum
machine may have a stereo snare, but a mono kick. Maybe this is
not a good idea, I haven' decided. That's why I sent it here for
discussion :)
Strongest resason *not* to use multichannel ports: They don't mix
well with how you work in a studio. If something gives you multiple
channels as separate mono signals, you can connect each one wherever
you want, independently, which gives you total control without the
need for the rest of your plugin chain to understand any multichannel
formats.

Strongest reason *for*: When implementing it as interleaved data, it
may server as a performance hack in some situations. It is still
unclear to me, though, whether or not this applies to "your average
plugin" in a real virtual studio, and even if it does, whether the
complexity/performance ration is anywhere near justifiable. (I
suspect not.)
Post by David Olofson
Can you expound on the well-known labels idea? I'm
not sure I fully understand what you're thinking.
Like on a studio mixing desk; little notes saying things like
"bdrum", "snare upper", "snare lower", "overhead left", "overhead
right" etc.

In a plugin API, you could standardize the way of "naming" these in
some way, such as always marking things MONO, LEFT, RIGHT, CENTER,
BLEFT, BRIGHT, SUB etc etc. Use a string, or a struct with some
enums, or whatever; just put it in the API, so hosts can make sense
of it.

Point being that if the host understands the labels, it can figure
out what belongs together and thus may bundle mono ports together
into "multichannel cables" on the user interface level.


[...]
Post by David Olofson
Post by Steve Harris
Post by Tim Hockin
/*
* set_rate: set the sample rate
LADSPA definatly got this right, make it an argument to
instantiate. Changing the sample rate of system is not an RT
operation, so theres no need to make it ultra efficient, and
instruments will have a very hard time rejigging everything.
Simpler to just remove and instatiate a new one with the same
settings.
ok, need to rework this then. I had done it that way but decided
this was simpler, but I suppose you are correct :)
Well, that's the hard part with APIs. Sometimes you forget to
consider the implications that minor API changes may have upon
implementations. :-)
Post by David Olofson
Post by Steve Harris
Post by Tim Hockin
int (*oapi_voice_on)(void *plugin, int note);
int (*oapi_note_off)(void *plugin, int voice);
I prefer this one.
I do too, but I don't want to have 10 APIs that are purely for
instruments and not effects. Add to those two the need for a
voice_ison() api for proper voice accounting of things like
triggered samples, and eventually a voice_param() to change dynamic
parameters per-voice (something I have not introduced in this API
yet). I'd like to be able to say 'increase the volume of this
existing voice'.
Well, I don't quite understand the voice_ison() call. I think voice
allocation best handled internally by each synth, as it's highly
implementation dependent.

If you want more direct ("tracker style") control over voices, just
use monophonic patches, or use some kind of "note ID" system that
lets you address individual notes.

In Audiality, I'm currently using actual voice numbers + "owner
fingerprints" (channel numbers, IIRC) as note IDs, but that means you
have to wait until the voice is actually allocated until you know
your voice ID.

Not a major problem within the engine core (*), but it means you
can't keep track of individual notes from, say, another machine over
LAN. Starting a voice and getting it's ID (so you can control the
voice) requires a client/engine roundtrip, and that's a bad idea in a
distributed real time system...

(*) Actually, there is one minor problem: Voice allocation/stealing
is restricted to buffer granularity, since when someone in the net
allocates a voice, there is no guarantee that that voice doesn't
belong to someone that runs *later* in the net. This hasn't been
a real problem so far, but I dot't like it. I want *everything*
sample accurate! ;-)


What I'll do in some future version is probably let each channel
allocate a range of Virtual Voice IDs (plain integers; 0..N-1), and
then have the engine keep track of these VVIDs on a per-channel
basis. That way, a channel can always rely on it's allocated VVIDs to
be valid - whether or not they have real voices connected to them. In
fact, this potentially enables the engine to "unsteal" voices - not
that I know if that would be worth the effort. (Normally, you should
just get more power when you run out of voices... Voice stealing -
however elegant - is just a brutal overload handling system.)
Post by David Olofson
Fruityloops does it, and I've found it to be
quite conventient for things like strings and pads. FL does
Velocity, Pan, Filter Cut, Filter Res, and Pitch Bend. Not sure
which of those I want to support, but I like the idea.
"None of those, but instead, anything" would be my suggestion. I
think it's a bad idea to "hardcode" a small number of controls into
the API. Some kind of lose "standard" such as the MIDI CC allocation,
could be handy, but the point is that control ports should just be
control ports; their function is supposed to be decided by the plugin
author.


[...]
Post by David Olofson
Post by Steve Harris
Yes. Though, I think using a virtual instruments capable API for
effects can be great, especially when it comes to automation
(both need to deal with accurate timing and control
interpolation), but of course, there's always the
ease-of-use/complexity trade-off; a virtual instruments API will
always be more complex than LADSPA.
I don't think it needs to be TOO much more complex. The areas I've
* multi-in/out
Yeah, that's required.
Post by David Olofson
* multi-channel
Should be handled on the UI level, IMHO. (See above.) Doing it down
here only complicates the connection managment for no real gain.
Post by David Olofson
* type-specific controls
Yeah... But this is one subject where I think you'll have to search
for a long time to find even two audio hackers that agree on the same
set of data types. ;-)
Post by David Olofson
* voice_on/off
Just a note here: Most real instruments don't have an absolute start
or end of each note. For example, a violin has it's pitch defined as
soon as you put your finger on the string - but when is the note-on,
and *what* is it? I would say "bow speed" would be much more
appropriate than on/off events.
Post by David Olofson
Post by Steve Harris
Well, I'm not sure it's worthwhile going beyond floats +
"hints"... Strings could be interesting, as but how should the
host deal with them? Are they file names, just names, scripts, or
what? How to edit them? How to store them in "preset files" and
the like?
String is a primitive - the control flags can specify a filename or
a directory name. I'm sure that filename will be the most common
case. It seems like String controls probably do not want to be
automated. But I see no reason they CAN'T be. All this complexity
is to make things that plugins want to say (enumerated values,
integral values) less complex and less hacked on the back of
something else. As simple as makes sense and no simpler.
Well, yes. There *has* to be a set of basic types that cover
"anything we can think of". (Very small set; probably just float and
raw data blocks.) I'm thinking that one might be able to have some
"conveniency types" implemented on top of the others, rather than a
larger number of actual types.

Dunno if this makes a lot of sense - I just have a feeling that
keeping the number of different objects in a system to a functional
minimum is generally a good idea. What the "functional minimum" is
here remains to see...
Post by David Olofson
Post by Steve Harris
I have a feeling that going down this route either results in
complex hosts, or a pointless "official" API for stuff that only
matters to plugins and their custom GUIs anyway - or both.
Custom GUIs don't even need to identify filenames as controls -
they can do it themselves. The point of making it a control is to
allow them to NOT have a custom GUI
Yeah, I know. It's just that I get nervous when something tries to do
"everything", but leaves out the "custom format" fallback for cases
that cannot be forseen. :-)
Post by David Olofson
Post by Steve Harris
How about just supporting a "raw data block" control type, so
that hosts can store the data in preset files or automation, but
without
I don't get what you're devolving it to. The point of Strings was
to say 'I want a filename' or 'Any string will do for this field
named SPEECH'.
Right. I think we can conclude that "raw data block" is not a
replacement for strings, but either a special kind of strings, or a
different type entirely.
Post by David Olofson
Maybe a raw data block is useful, but I haven't
seen an example. The types I specified are based on experience
using FruityLoops and similar apps and dealing with VST instruments
and other plugins and trying to proived the simplest subset of what
they need without oversimplifying.
Well, you can put stuff in external files, but that seems a bit risky
to me, in some situations. Hosts should provide per-project space for
files that should always go with the project, and some rock solid way
of ensuring that

1) all required files can be found and stored in an
archive, for moving a project to another system,
or for backup, and that

2) users don't accidentally edit global data used by
other projects while working with a project. (This
is where the per-project file space comes in.)


[...]
Post by David Olofson
Post by Steve Harris
Likewise with timestamped "MIDI style" control events - either
you accept them, or you don't. Whether you have audio inputs,
outputs or both is another story entirely.
In my worldview MIDI is ENTIRELY the host's problem. The
voice_on/voice_off/voice_change API is intended to provide the
majority of MIDI controlability. The host can translate from
actual MIDI.
I totally agree. I'm not suggesting that the actual MIDI protocol
should be used in any way in the API. (Although people on the VST
list seem to believe MIDI is great as a control protocol for plugins,
it's never going into another plugin API, if I can help it... What a
mess - and how many plugins actually implement more than the most
basic stuff? *heh*)
Post by David Olofson
Post by Steve Harris
I'd definitely go for Unique IDs + external data here. Oh, and it's
but why?
See above. (And note that I'm not so sure about the "external data"
part, actually. I just think the ID is required for user data
portability reasons.)
Post by David Olofson
Post by Steve Harris
probably a good idea to structure the ID as "vendor" and
"product"
I though about doing a Vendor Code. But why burden some person
with distributing vendor IDs and maintaining that database, when
storing a string is just as simple.
The string has to be unique. Just Google for some common "unique"
brand names, and you'll see what I'm worried about. :-)
Post by David Olofson
It's not like a few bytes of
memory are going to matter. Now, I'm NOT trying to be flippant
about memory usage. I just want to be realistic.
*hehe* Well, no, the size of strings won't matter - and unless I'm
missing something, nor does the overhead of "parsing" them.


[...]
Post by David Olofson
Post by Steve Harris
Post by Tim Hockin
I'm pretty confident this is not the right way to go. If it
matters to you what output belongs to what channel then well
known labels would be better, I think.
Yeah, just as with input/output relations, there's no useful and
reasonably simple way of describing channel/input/output
relations. This just adds a dimension of indexing, for no gain.
Perhaps. Perhaps different channels-counts per in.out is a bad
idea. But I see a definate need for different channels-counts on
input vs. output. What of a mono-izer. Or a mono->stereo. Or
Stereo->5.1.
Yes, definitely. A plugin must be able to have anything from 0 and up
of each, independently. But I prefer to see it as M mono ins and N
mono outs, labeled in a standardized and sensible way, so you and the
host know what they are.
Post by David Olofson
Post by Steve Harris
Besides, you would at least have to have one pair of limits for
each input and each output for this to be of any use. (Unless all
inputs are required to be identical, and likewise for outputs.)
Limits are limits. Current value is different.
Right - but I'm *am* talking about limits. :-)

If you have a heavily optimized mixer plugin, it's rather likely that
it will only support say, 8 busses (no loop for bus sends!), but
"any" number of strips. How do you describe that using the same limit
for both strip insert loops and bus (or master) insert loops?


[...]
Post by David Olofson
Post by Steve Harris
Consider a mixer, where you have insert sends and returns, as
well as bus sends and returns. Obviously, you'd probably want to
be able to use some 128 channels or so - but most probably not
128 busses!
I want to avoid pure hardware paradigms.
So do I. I'm just trying to base my examples on well known equipment
and terminology.
Post by David Olofson
But you are correct in
that it can get complex. Maybe it needs to be curtailed, but it
definitely needs to support multi-channel audio. Ideas welcome.
Multichannel == multiple mono ins and/or outs... That's how you do it
in analog, and it's how most plugin APIs (of the kind we're
discussing here) do it. It's simple and very flexible.
Post by David Olofson
Post by Steve Harris
typedef enum
{
FX_STATE_CLOSED = 0,
FX_STATE_OPEN,
FX_STATE_READY,
FX_STATE_PAUSED,
FX_STATE_RUNNING,
FX_STATE_SILENT,
FX_STATE_RESTING
} a_fxstates_t;
Similar to gstreamer. Need to comtemplate this. Can you elucidate
on the different states?
Steve Harris
2002-12-04 10:41:00 UTC
Permalink
Post by David Olofson
IMHO, plugins should not worry about whether or not their outputs are
connected. (In fact, there are reasons why you'd want to always
guarantee that all ports are connected before you let a plugin run.
Open inputs would be connected to a dummy silent output, and open
outputs would be connected to a /dev/null equivalent.)
In modular synth systems there are good reasons why you want to know
whether things are connected (and good reasons why they wouldn't be), but
this spec so far is not useful to a modular synth, so in this case I
agree.
Post by David Olofson
I think there should be as little policy as possible in an API. As
in; if a plugin can assume that all ins and outs will be connected,
there are no special cases to worry about, and thus, no need for a
policy.
Yes.
Post by David Olofson
Strongest resason *not* to use multichannel ports: They don't mix
well with how you work in a studio. If something gives you multiple
channels as separate mono signals, you can connect each one wherever
you want, independently, which gives you total control without the
need for the rest of your plugin chain to understand any multichannel
formats.
Strongest reason *for*: When implementing it as interleaved data, it
may server as a performance hack in some situations. It is still
unclear to me, though, whether or not this applies to "your average
plugin" in a real virtual studio, and even if it does, whether the
complexity/performance ration is anywhere near justifiable. (I
suspect not.)
Interleaved data is more trouble that its worth IMNSHO. I think
multichannel outputs are extra complexity with no benefit.

I'm not sure what peoples opinions on numbers of outputs are, obviously the
number needs to be variable per instrument at development time, but I dont
think it should be variable at instantiation time, that doesn't sound
useful, and it would be hell to optimise.
Post by David Olofson
Post by Tim Hockin
Can you expound on the well-known labels idea? I'm
not sure I fully understand what you're thinking.
...
Post by David Olofson
In a plugin API, you could standardize the way of "naming" these in
some way, such as always marking things MONO, LEFT, RIGHT, CENTER,
BLEFT, BRIGHT, SUB etc etc. Use a string, or a struct with some
enums, or whatever; just put it in the API, so hosts can make sense
of it.
Yes, thanks, thats exactly what I meant. I also proposed a system like
this for standardising panners (eg. stereo in/5.1 out) in LADSPA. Noones
tried it yet, but it should work :)

On controls:

It seems like you need two sets, per instrument and per voice, they should
share the same interface, and the per voice ones definatly should not be
limited to modulation,chorus,whatever.

My feeling is that just float+"arbitrary opaque binary data" is enough.
The float can be augmented with hints, enumerations, whatever.

- Steve
David Olofson
2002-12-04 18:36:37 UTC
Permalink
Post by Steve Harris
Post by David Olofson
IMHO, plugins should not worry about whether or not their outputs
are connected. (In fact, there are reasons why you'd want to
always guarantee that all ports are connected before you let a
plugin run. Open inputs would be connected to a dummy silent
output, and open outputs would be connected to a /dev/null
equivalent.)
In modular synth systems there are good reasons why you want to
know whether things are connected (and good reasons why they
wouldn't be), but this spec so far is not useful to a modular
synth, so in this case I agree.
Well, I actually agree with disconnected outputs making sense - but I
think plugins should have a chance of deciding how to deal with it,
preferably before/during instantiation. (See my previous post.)


[...]
Post by Steve Harris
I'm not sure what peoples opinions on numbers of outputs are,
obviously the number needs to be variable per instrument at
development time, but I dont think it should be variable at
instantiation time, that doesn't sound useful, and it would be hell
to optimise.
I don't see why a mixer plugin would have serious trouble supporting
1+ strips. Basic mixers strips could indeed be implemented as
separate plugins, but if you want PFL, busses and stuff, I think
you're much better off implementing and optimizing that within a
single plugin.

Besides, I bet someone can come up with something with much more
intricate inter-channel relations than a mixer...


[...]
Post by Steve Harris
It seems like you need two sets, per instrument and per voice, they
should share the same interface, and the per voice ones definatly
should not be limited to modulation,chorus,whatever.
My feeling is that just float+"arbitrary opaque binary data" is
enough. The float can be augmented with hints, enumerations,
whatever.
I agree.


//David Olofson - Programmer, Composer, Open Source Advocate

.- Coming soon from VaporWare Inc...------------------------.
| The Return of Audiality! Real, working software. Really! |
| Real time and off-line synthesis, scripting, MIDI, LGPL...|
`-----------------------------------> (Public Release RSN) -'
.- M A I A -------------------------------------------------.
| The Multimedia Application Integration Architecture |
`----------------------------> http://www.linuxdj.com/maia -'
--- http://olofson.net --- http://www.reologica.se ---
Conrad Parker
2002-12-04 03:31:02 UTC
Permalink
Post by Tim Hockin
Post by Steve Harris
If you're going to support metadata, support the Dublin Core, but generally
extrnal metadata is better than internal. Just provide a unique ID and
metadata can refer to that.
Firstly, I hate the idea that a plugin writer needs to procure a unique ID.
Secondly, I think some meta-data needs to be IN the plugin. Now, maybe
'notes', and 'url' can be outside the plugin, but
author/copyright/license/version can't. Think of commercial plugin (I know,
pipe dreams, but just pretend :). As a commercial vendor you DO NOT want it
to be trivial for a person to 'lose' the info about the license. I can
think of a several reasons to keep it all together, and no good reason NOT
to. Specifically, this data is inseperable from the plugin. As an author,
I want this data joined at the hip with the plugin. Further, the idea that
a plugin consists of a single .so file appeals to me.
I have to say I agree with Tim here -- internal, inseparable strings are
much simpler for both the host and plugin authors to deal with, and don't
force apps to link against random external libraries (xml or rdf or
whatever) just to get at a few necessary strings. The core stuff (name,
description etc.) is not actually "meta" at all, they're fields which just
happen to be of string type.

If a host needs to parse xml, or even hit the network, just to access
core information, that significantly raises the barrier to entry for
developing compatible software. The barrier to entry is already
high enough (the ability to write or load dynamic libraries and design
a callback based plugin) -- high enough that I've seen a few people shy
away from it already. Adding to this will only slow the uptake, or put
us in a situation where all our half-developed apps can only do half of
what's required of the standard, which will be awful.

For stuff that is actually metadata (keywords, location of publication etc.)
... if you want to reference external metadata by unique id, I propose that
the data type of the unique id be a URL :)

Conrad.
Steve Harris
2002-12-04 10:20:01 UTC
Permalink
Post by Conrad Parker
I have to say I agree with Tim here -- internal, inseparable strings are
much simpler for both the host and plugin authors to deal with, and don't
force apps to link against random external libraries (xml or rdf or
whatever) just to get at a few necessary strings. The core stuff (name,
description etc.) is not actually "meta" at all, they're fields which just
happen to be of string type.
For the stuff that is acutally vital, UID, version, instrument name,
whatever, then yes. But anything more sophisticated than that /really/
should be external. Internalised metadata systems invariably end up being
inadequet, clumsy or just useless (ala LADSPA).

Better to accept form the start that you are going to have to use external
metadata if you want to express anything complex. Once the instrument
standard is complex enough that it requires a library anyway (eg. for
timestamped event handling), you may as well provide decent, extensible
metadata support.

Do a web search if you want to find a million and one bad experiences with
internal metadata. Its pretty much a non-decision in the digital libraries
and knowledge management world now.
Post by Conrad Parker
For stuff that is actually metadata (keywords, location of publication etc.)
... if you want to reference external metadata by unique id, I propose that
the data type of the unique id be a URL :)
URI maybe, but a URL requires a network connection, so its not ideal.

- Steve
Tim Hockin
2002-12-04 10:15:01 UTC
Permalink
Post by David Olofson
Well, a guaranteed unique ID is really rather handy when you want to
load up a project on another system and be *sure* that you're using
the right plugins... That's about the only strong motivation I can
think of right now, but it's strong enough for me.
Ok, I see your motivation for this. I hate the idea of 'centrally assigned'
anythings for something as open as this. I'll think more on it..
Post by David Olofson
IMHO, plugins should not worry about whether or not their outputs are
connected. (In fact, there are reasons why you'd want to always
guarantee that all ports are connected before you let a plugin run.
Open inputs would be connected to a dummy silent output, and open
outputs would be connected to a /dev/null equivalent.)
I disagree with that - this is a waste of DSP cycles processing to be sent
nowhere.
Post by David Olofson
as some kind of interface to the plugin, but to me, it seems way too
limited to be of any real use. You still need a serious interface
if it has no other use than 'ignore this signal and spare the CPU time' it
is good enough for me.
Post by David Olofson
just like you program a studio sampler to output stuff to the outputs
you want. This interface may be standardized or not - or there may be
both variants - but either way, it has to be more sophisticated than
one bit per output.
Ehh, again, I think it is simpler. Lets assume a simple sampler. It has a
single output with 0 or more channels (in my terminology). If you load a
stereo sample, it has 2 channels. A 5.1 sample has 6 channels. Let's
consider an 8-pad drum machine. It has 8 outputs each with 0-2 channels.
Load a stereo sample, that output has 2 channels. Now, as I said, maybe
this is a bad idea. Maybe it should be assumed that all outputs have 2
channels and mono gets duplicated to both or (simpler) LEFT is MONO.

What gets confusing is what we're really debating. If I want to do a simple
stereo-only host, can I just connect the first pair of outs and the plugin
will route automatically? Or do I need to connect all 8 to the same buffer
in order to get all the output. In the process of writing this I have
convinced myself you are right :) If the host does not connect pad #2, pad
#2 is silent.
Post by David Olofson
I think there should be as little policy as possible in an API. As
in; if a plugin can assume that all ins and outs will be connected,
there are no special cases to worry about, and thus, no need for a
policy.
Slight change - a plugin only needs to handle connected inouts. If an inout
is not connected, the plugin can skip it or do whatever it likes.
Post by David Olofson
Strongest resason *not* to use multichannel ports: They don't mix
well with how you work in a studio. If something gives you multiple
I considered that. At some point I made a conscious decision to trade off
that ability for the simplicity of knowing that all my stereo channels are
bonded together. I guess I am rethinking that.
Post by David Olofson
Strongest reason *for*: When implementing it as interleaved data, it
bleh - I always assumed an inout was n mono channels. The only reason for
grouping them into inouts was to 'bond' them.
Post by David Olofson
Like on a studio mixing desk; little notes saying things like
"bdrum", "snare upper", "snare lower", "overhead left", "overhead
right" etc.
Should I use a different word than 'port'? is it too overloaded with
LADSPA?

Hrrm, so how does something like this sound?

(metacode)

struct port_desc {
char *names;
};
simple sampler descriptor {
...
int n_out_ports = 6;
struct port_desc *out_ports[] = {
{ "mono:left" }
{ "right" }
{ "rear:center" }
{ "rear-left" }
{ "rear-right" }
{ "sub:lfe" }
};
...
};

So the host would know that if it connects 1 output, the name is "mono", and
if it connects 2 ouptuts, the names are "left", "right", etc. Then it can
connect "left" to "left" on the next plugin automatically. And if you want
to hook it up to a mono output, the user could be asked, or assumptions can
be made. This has the advantage(?) of not specifying a range of acceptable
configs, but a list. It can have 1, 2, or 6 channels.

another example:

drum machine descriptor {
...
int n_out_ports = 16;
struct port_desc *out_ports[] = {
{ "left:left(0):mono(0)" }, { "right:right(0)" },
{ "left(1):mono(1)" }, { "right(1)" },
{ "left(2):mono(2)" }, { "right(2)" },
{ "left(3):mono(3)" }, { "right(3)" },
{ "left(4):mono(4)" }, { "right(4)" },
{ "left(5):mono(5)" }, { "right(5)" },
{ "left(6):mono(6)" }, { "right(6)" },
{ "left(7):mono(7)" }, { "right(7)" },
};
...
};

and finally:

mixer descriptor {
...
int n_in_ports = -1;
struct port_desc *in_ports[] = {
{ "in(%d)" }
}
int n_out_ports = 2;
struct port_desc *out_ports[] = {
{ "left:mono" }
{ "right" }
}
}

Or something similar. It seems that this basic code would be duplicated in
almost every plugin. Can we make assumptions and let the plugin leave it
blank if the assumptions are correct?

In thinking about this I realized a potential problem with not having bonded
channels. A mixer strip is now a mono strip. It seems really nice to be
able to say "Input 0 is 2-channels" and load a stereo mixer slot, "Input 1 is
1-channel" and load a mono mixer slot, "Input 2 is 6-channel" and load a 5.1
mixer slot.

I'm back to being in a quandary. Someone convince me!
Post by David Olofson
Point being that if the host understands the labels, it can figure
out what belongs together and thus may bundle mono ports together
into "multichannel cables" on the user interface level.
This is what the "inout is a bundle of mono channels" idea does.
Post by David Olofson
Well, I don't quite understand the voice_ison() call. I think voice
allocation best handled internally by each synth, as it's highly
implementation dependent.
My ideas wrt polyphony:

* note_on returns an int voice-id
* that voice-id is used by the host for note_off() or note_ctrl()
* you can limit polyphony in the host
- when I trigger the 3rd voice on an instrument set for 2-voices, I
can note_off() one of them
* you can limit polyphony in the instrument
- host has triggered a 3rd voice, but I only support 2, so I
internally note_off() one of them and return that voice_id again.
The host can recognise that and account for polyphony accurately
(even if it is nothing more than a counter).
* note_off identifies to the host if a voice has already ended (e.g. a sample)
* note_ison can be called by the host periodically for each voice to see
if it is still alive (think of step-sequenced samples). If a sample ends,
the host would want to decrement it's voice counter. The other option is a
callback to the host. Not sure which is less ugly.

I am NOT trying to account for cross-app or cross-lan voices, though a JACK
instrument which reads from a JACK port would be neat.

Side note: is there a mechanism in jack for me to pass (for example) a
'START NOW' message or a 'GO BACK 1000 samples' message to a Jack port?
Post by David Olofson
a real problem so far, but I dot't like it. I want *everything*
sample accurate! ;-)
Actually, our focus is slightly different. I'm FAR less concerned with
sample-accurate control. Small enough buffers make tick-accurate control
viable in my mind. But I could be convinced. It sure is SIMPLER. :)
Post by David Olofson
Post by Tim Hockin
quite conventient for things like strings and pads. FL does
Velocity, Pan, Filter Cut, Filter Res, and Pitch Bend. Not sure
which of those I want to support, but I like the idea.
"None of those, but instead, anything" would be my suggestion. I
think it's a bad idea to "hardcode" a small number of controls into
the API. Some kind of lose "standard" such as the MIDI CC allocation,
could be handy, but the point is that control ports should just be
control ports; their function is supposed to be decided by the plugin
author.
I've contemplated an array of params that are configurable per-note. Not
everything is. What if we had something like

struct int_voice_param {
int id;
char *name;
int low;
int high;
};

and specify an array of them. The host can use this array to build a list
of per-note params to display to the user. This starts to get messy with
type-specific controls. Perhaps this info belongs as part of the control
structure. Yes, I think so.
Post by David Olofson
Should be handled on the UI level, IMHO. (See above.) Doing it down
here only complicates the connection managment for no real gain.
I want to ignore as much of it as possible in the UI. I want to keep it
simple at the highest level so a musician spends his time making music, not
dragging virtual wires. Ideally if there is a stereo instrument and I want
to add a stereo reverb, I'd just drop it in place, all connections made
automatically. If I have a mono instrument and I want a stereo reverb, I'd
drop the reverb in place and it would automatically insert a mono-stereo
panner plugin between them.
Post by David Olofson
Yeah... But this is one subject where I think you'll have to search
for a long time to find even two audio hackers that agree on the same
set of data types. ;-)
I think INT, FLOAT, and STRING suffice pretty well. And I MAY be convinced
that INT is not needed. Really, I prefer int (maps well to MIDI). What
kinds of knobs need to be floats?
Post by David Olofson
Just a note here: Most real instruments don't have an absolute start
or end of each note. For example, a violin has it's pitch defined as
soon as you put your finger on the string - but when is the note-on,
and *what* is it? I would say "bow speed" would be much more
appropriate than on/off events.
I'd assume a violin modeller would have a BOWSPEED control. The note_on()
would tell it what the eventual pitch would be. The plugin would use
BOWSPEED to model the attack.
Post by David Olofson
Well, yes. There *has* to be a set of basic types that cover
"anything we can think of". (Very small set; probably just float and
raw data blocks.) I'm thinking that one might be able to have some
"conveniency types" implemented on top of the others, rather than a
larger number of actual types.
I agree - Bool is a flag on INT. File is a flag on String.
Post by David Olofson
Dunno if this makes a lot of sense - I just have a feeling that
keeping the number of different objects in a system to a functional
minimum is generally a good idea. What the "functional minimum" is
here remains to see...
With this I agree. One of the reasons I HATE so many APIs is that they are
grossly over normalized. I don't need a pad_factory object and a pad object
and a plugin_factory object and a parameter object and an
automatable_parameter object and a scope object... I want there to be as
FEW structs/objects as possible.

That said, one I am considering adding is a struct oapi_host. This would
have callbacks for things like malloc, free, and mem_failure (the HOST
should decide how to handle memory allocation failures, not the plugin) as
well as higher level stuff like get_buffer, free_buffer, and who knows what
else. Minimal, but it puts control for error handling back in the hands of
the host.
Post by David Olofson
Yeah, I know. It's just that I get nervous when something tries to do
"everything", but leaves out the "custom format" fallback for cases
that cannot be forseen. :-)
We're speaking of controls here. In my mind controls have three
characteristics. 1) They have to specify enough information that the host can
draw a nice UI automatically. 2) They are automatable (whether it is sane
or not is different!). 3) They alone compose a preset. What would a
raw_data_block be?
Post by David Olofson
Well, you can put stuff in external files, but that seems a bit risky
to me, in some situations. Hosts should provide per-project space for
files that should always go with the project, and some rock solid way
of ensuring that
I don't really want the plugins writing files. I'd rather see the host
write a preset file by reading all the control information, or by the host
calling a new char *oapi_serialize() method to store and a new
oapi_deserialize(char *data) method to load.
Post by David Olofson
should be used in any way in the API. (Although people on the VST
list seem to believe MIDI is great as a control protocol for plugins,
it's never going into another plugin API, if I can help it... What a
:) good good
Post by David Olofson
So do I. I'm just trying to base my examples on well known equipment
and terminology.
Honestly, I don't know all the terminology. I have never worked with much
studio gear. Most of what I have done is in the software space. So I may
be making mistakes by that, but I may also be tossing obsolete-but-accepted
notions for the same reason :)

[ state-machine header ... ]

Very interesting. I actually like it very much. I am going to have athink
on that. It may be a better paradigm.
Post by David Olofson
Well, it's basically about sending structured data around, with
timestamps telling the receiver when to process the data. As an
example, instead of calling
voice_start(v, ev->arg1);
directly, at exactly the right time (which would mean you have to
aev_send1(&v->port, 0, VE_START, wave);
where aev_send1() is an inline function that grabs an event struct
from the pool, fills it in and sends it to the voice's event port.
The sender does nothing more about it for now; it just keeps
processing it's entire buffer and then returns. Just as if it had
been processing only audio buffers. In fact, the timestamped events
are very similar to audio data in that they contain both actual data
and timing information - it's just that the timing info is explicit
in events.
Interesting. How important is this REALLY, though. Let me break it into
two parts: note control and parameter control. Note control can be tick
accurate as far as I am concerned :) As for param control, it seems to me
that a host that will automate params will PROBABLY have small ticks. If
the ticks are small (10-50 samples), is there a REAL drawback to
tick-accurate control? I know that philosophically there is, but REALLY.

In the event model, if I want a smooth ramp for a control between 0 and 100
across 10 ticks of 10 samples, do I need to send 10 'control += 1' events
before each tick?
Post by David Olofson
Seriously, it's probably time to move on to the VSTi/DXi level now.
LADSPA and JACK rule, but the integration is still "only" on the
audio processing/routing level. We can't build a complete, seriously
useful virtual studio, until the execution and control of synths is
as rock solid as the audio.
Well, I really want to do it, so let's go. You keep talking about
Audiality, but if we're designing the same thing, why aren't we working on
the same project?

Lots of ideas to noodle on and lose sleep on. Looking forward to more
discussion

Thanks
Tim
Steve Harris
2002-12-04 11:08:00 UTC
Permalink
Post by Tim Hockin
Post by David Olofson
Well, a guaranteed unique ID is really rather handy when you want to
load up a project on another system and be *sure* that you're using
the right plugins... That's about the only strong motivation I can
think of right now, but it's strong enough for me.
Ok, I see your motivation for this. I hate the idea of 'centrally assigned'
anythings for something as open as this. I'll think more on it..
The central part can be very open, eg in URIs you're allowed to assign
URIs from any domain you have control over, so I could label somthing
"http://plugin.org.uk/foo#23". This means that anyone with a website can
assign URIs.

[NB I know it starts with http:, but it /not/ a URL, me and some others
argued it should have a non-resolvable protocol name, but...]

I dont really like using strings for UIDs though. Its too easy to read
semantics into them.
Post by Tim Hockin
Post by David Olofson
IMHO, plugins should not worry about whether or not their outputs are
connected. (In fact, there are reasons why you'd want to always
guarantee that all ports are connected before you let a plugin run.
Open inputs would be connected to a dummy silent output, and open
outputs would be connected to a /dev/null equivalent.)
I disagree with that - this is a waste of DSP cycles processing to be sent
nowhere.
No, its a waste of DSP cycles to check whether something's there or not.
Post by Tim Hockin
Slight change - a plugin only needs to handle connected inouts. If an inout
is not connected, the plugin can skip it or do whatever it likes.
This is still complexity (if we have a timestamp system), if ther are no
events on the input hte instrument will ingore it anyway.
Post by Tim Hockin
Should I use a different word than 'port'? is it too overloaded with
LADSPA?
No, thats exactly a good reason to use the word port (unless youre
describing something thats actually different).
Post by Tim Hockin
So the host would know that if it connects 1 output, the name is "mono", and
if it connects 2 ouptuts, the names are "left", "right", etc. Then it can
connect "left" to "left" on the next plugin automatically. And if you want
to hook it up to a mono output, the user could be asked, or assumptions can
be made. This has the advantage(?) of not specifying a range of acceptable
configs, but a list. It can have 1, 2, or 6 channels.
I'm not sure I like the idea of varying the behaviour of the system
depending on the number of connections, I have hardware that does this and
it drives me nuts. Especially when patchbays come into it. ITs kind of
forgivable in hardware, as noone wants to buy a sperate mono and stereo
reverb, but in software I dont think theres any such excuse.
Post by Tim Hockin
I'm back to being in a quandary. Someone convince me!
Dont bundle up channels! People dont do that in studios for a good reason.
a single channel on a single connections is simple, unambiguous, obvious.
But say you have 2 channels is that L+R, M+S, C+M? how about 8, is it 7.1,
8.1 pre filter, 5.1 in ADAT? What do you do when you have a 5.1 bundle
and you want to send it as 5.1 in ADAT? Does that require a 5.1 rebundler?

I think its much better just to have discrete, mono channels that can be
assigned as needed.
Post by Tim Hockin
* you can limit polyphony in the host
- when I trigger the 3rd voice on an instrument set for 2-voices, I
can note_off() one of them
I'm not sure about the other arguments, but polyphony control is complex
and probably only the instrument can do it usefully. If you want this you
might have to do it by telling the instrument what its max poly should be.
Post by Tim Hockin
Side note: is there a mechanism in jack for me to pass (for example) a
'START NOW' message or a 'GO BACK 1000 samples' message to a Jack port?
Jack has strinport comtrols that can be use to move the conceptual time of
the jack system, loop, sync etc.
Post by Tim Hockin
Actually, our focus is slightly different. I'm FAR less concerned with
sample-accurate control. Small enough buffers make tick-accurate control
viable in my mind. But I could be convinced. It sure is SIMPLER. :)
I thought this too unltil recent, but Benno demonstrated to me that sample
accurate is not significantly harder, and is more efficient in general.
Post by Tim Hockin
I've contemplated an array of params that are configurable per-note. Not
everything is. What if we had something like
If the per-voice params are different to he poer-instrument params then
something has gone seriously wrong.
Post by Tim Hockin
I think INT, FLOAT, and STRING suffice pretty well. And I MAY be convinced
that INT is not needed. Really, I prefer int (maps well to MIDI). What
kinds of knobs need to be floats?
Seen as plugins internally work in float, all knobs need to be float ;)
Post by Tim Hockin
I'd assume a violin modeller would have a BOWSPEED control. The note_on()
would tell it what the eventual pitch would be. The plugin would use
BOWSPEED to model the attack.
voice_on is proably more instrument neutral. Not everything is a note (eg.
my LADSPA gong).
Post by Tim Hockin
I agree - Bool is a flag on INT. File is a flag on String.
I dont think you can do anything useful with a generic file string, what
could be host use it for?
Post by Tim Hockin
With this I agree. One of the reasons I HATE so many APIs is that they are
grossly over normalized. I don't need a pad_factory object and a pad object
and a plugin_factory object and a parameter object and an
automatable_parameter object and a scope object... I want there to be as
FEW structs/objects as possible.
Good, so ditch int and string and replace them with a generic binary type ;)
Post by Tim Hockin
That said, one I am considering adding is a struct oapi_host. This would
have callbacks for things like malloc, free, and mem_failure (the HOST
should decide how to handle memory allocation failures, not the plugin) as
Memeory allocation failures dont really happen in RT systems, you get
massive overruns first.
Post by Tim Hockin
Post by David Olofson
Yeah, I know. It's just that I get nervous when something tries to do
"everything", but leaves out the "custom format" fallback for cases
that cannot be forseen. :-)
We're speaking of controls here. In my mind controls have three
characteristics. 1) They have to specify enough information that the host can
draw a nice UI automatically. 2) They are automatable (whether it is sane
or not is different!). 3) They alone compose a preset. What would a
raw_data_block be?
I dont bvelieve the host can draw a nice UI automatically, the best it can
do is a usable UI.
Post by Tim Hockin
I don't really want the plugins writing files. I'd rather see the host
write a preset file by reading all the control information, or by the host
calling a new char *oapi_serialize() method to store and a new
oapi_deserialize(char *data) method to load.
Agreed.

- Steve
Steve Harris
2002-12-04 11:40:58 UTC
Permalink
Post by Steve Harris
URIs from any domain you have control over, so I could label somthing
Here I meant URL/URI, not domain.

And as mailto:***@ecs.soton.ac.uk#myInstrument is a valid URI, everyone
here can do that. Though, I still prefer integers ;)

- Steve
David Olofson
2002-12-04 19:04:00 UTC
Permalink
Post by Steve Harris
Post by Tim Hockin
Slight change - a plugin only needs to handle connected inouts.
If an inout is not connected, the plugin can skip it or do
whatever it likes.
This is still complexity (if we have a timestamp system), if ther
are no events on the input hte instrument will ingore it anyway.
True. At least in the case of Audiality, a plugin doesn't really have
to care whether or not it's event port is connected or not - it just
checks for events.

BTW, yes, that's *port*. One DSP loop - one event port. This is
because plugins checking multiple event ports and deciding which one
goes first doesn't sound like a whole lot of fun. I definitely prefer
having the host merge/sort, or even having senders sort/insert when
sending events. I'm doing the latter in some cases currently, but I'm
not sure it'll stay that way. When you know there's only one sender
to a port, you can just tack events onto the end of the list, but
that only works for very simple plugin nets. The main problem with
sender sort/insert is that it more or less requires a function call,
while "add as last event" is a very simple operation that is
preferably inlined.


[...]
Post by Steve Harris
Post by Tim Hockin
I'd assume a violin modeller would have a BOWSPEED control. The
note_on() would tell it what the eventual pitch would be. The
plugin would use BOWSPEED to model the attack.
voice_on is proably more instrument neutral. Not everything is a
note (eg. my LADSPA gong).
Good point. In fact, I would say that pitch and velocity are just two
arbitrary "instantiation parameters" for notes. For some instruments,
they're hardly applicable at all, while for others, they're
completely insufficient. That's why I'm leaning more and more towards
entirely doing away with on/off events as such, and just using
controls. A synth can interpret a "velocity" control like an on/off
switch with note on velocity - or as bow speed, or air flow/pressur,
or whatever...
Post by Steve Harris
Post by Tim Hockin
I agree - Bool is a flag on INT. File is a flag on String.
I dont think you can do anything useful with a generic file string,
what could be host use it for?
How about the host knowing that this is the name of a file that
should go with the project? I would suggest the following file name
"flags":

FILE_PUBLIC:
The file should be considered public on the system,
meaning that the user should be warned that modifying
it may affect other projects. Preferably, the user
should be given the option of editing a FILE_PROJECT
copy of the file.

FILE_LIBRARY: (is FILE_PUBLIC)
This file is from a standard library, and can
be assumed to be available on other systems.
It should not be included in bundles for backup
or transfer, unless the user explicitly asks that
*all* files are included.

FILE_USER: (is FILE_PUBLIC)
This is user created data, or other data that may
not be available to the public, or on other systems.
It should be included in bundles for backup and
transfer.

FILE_PROJECT:
The file belongs specifically to this project, and
can be assumed to be part of the project. The user
will not be warned about changing the file. The
file should be included in bundles for backup and
transfer.


[...]
Post by Steve Harris
Post by Tim Hockin
That said, one I am considering adding is a struct oapi_host.
This would have callbacks for things like malloc, free, and
mem_failure (the HOST should decide how to handle memory
allocation failures, not the plugin) as
Memeory allocation failures dont really happen in RT systems, you
get massive overruns first.
Right, either you get memory - or you're screwed.

That said, it's possibly to implement real time memory managers, and
I think the host designer should have the option.
Post by Steve Harris
Post by Tim Hockin
Post by David Olofson
Yeah, I know. It's just that I get nervous when something tries
to do "everything", but leaves out the "custom format" fallback
for cases that cannot be forseen. :-)
We're speaking of controls here. In my mind controls have three
characteristics. 1) They have to specify enough information that
the host can draw a nice UI automatically. 2) They are
automatable (whether it is sane or not is different!). 3) They
alone compose a preset. What would a raw_data_block be?
I dont bvelieve the host can draw a nice UI automatically, the best
it can do is a usable UI.
Good point. I still think it's nice if hosts can construct basic GUIs
for simple plugins, but I see no point in taking any major design
measures with host constructed GUIs in mind.


//David Olofson - Programmer, Composer, Open Source Advocate

.- Coming soon from VaporWare Inc...------------------------.
| The Return of Audiality! Real, working software. Really! |
| Real time and off-line synthesis, scripting, MIDI, LGPL...|
`-----------------------------------> (Public Release RSN) -'
.- M A I A -------------------------------------------------.
| The Multimedia Application Integration Architecture |
`----------------------------> http://www.linuxdj.com/maia -'
--- http://olofson.net --- http://www.reologica.se ---
Steve Harris
2002-12-04 19:37:00 UTC
Permalink
Post by David Olofson
I definitely prefer
having the host merge/sort, or even having senders sort/insert when
sending events. I'm doing the latter in some cases currently, but I'm
not sure it'll stay that way. When you know there's only one sender
to a port, you can just tack events onto the end of the list, but
that only works for very simple plugin nets. The main problem with
sender sort/insert is that it more or less requires a function call,
while "add as last event" is a very simple operation that is
preferably inlined.
I agree, you dont want event sorting code in every instrument, you will
end up with some very odd behaviour due to bugs.
Post by David Olofson
Good point. In fact, I would say that pitch and velocity are just two
arbitrary "instantiation parameters" for notes. For some instruments,
they're hardly applicable at all, while for others, they're
completely insufficient. That's why I'm leaning more and more towards
entirely doing away with on/off events as such, and just using
controls. A synth can interpret a "velocity" control like an on/off
switch with note on velocity - or as bow speed, or air flow/pressur,
or whatever...
Yes, but voice_on/off is a commonly useful abstraction. Hoever making sure
htat the API is scale neutral (unlike generic MIDI) is important. I'm not
sure what the best way to ensure this is.

Obviously pitch and velicty are optional, not all instruments have either.
Thats one reason if favour of using well know portt names to indicate what
voice parameters normally map to what MIDI controls.
Post by David Olofson
How about the host knowing that this is the name of a file that
should go with the project? I would suggest the following file name
OK, that is a good point, it would make the host able to store the session
more easily, and package it up my copying the file into local space, and
changin the path.
Post by David Olofson
Right, either you get memory - or you're screwed.
That said, it's possibly to implement real time memory managers, and
I think the host designer should have the option.
But if the api is like LADSPA you're not allowed to do allocation in the
RT part anyway, so you may as well stick to malloc and friends. If hte
host is desperate to use something else it can override them with
LD_PRELOAD.
Post by David Olofson
Good point. I still think it's nice if hosts can construct basic GUIs
for simple plugins, but I see no point in taking any major design
measures with host constructed GUIs in mind.
Sure, but simple instruments dont require file i/o or any of that stuff
anyway.

- Steve
David Olofson
2002-12-04 21:50:01 UTC
Permalink
On Wednesday 04 December 2002 19.31, Steve Harris wrote:
[...]
Post by Steve Harris
Post by David Olofson
Good point. In fact, I would say that pitch and velocity are just
two arbitrary "instantiation parameters" for notes. For some
instruments, they're hardly applicable at all, while for others,
they're completely insufficient. That's why I'm leaning more and
more towards entirely doing away with on/off events as such, and
just using controls. A synth can interpret a "velocity" control
like an on/off switch with note on velocity - or as bow speed, or
air flow/pressur, or whatever...
Yes, but voice_on/off is a commonly useful abstraction.
Well, yes, but I'm becoming more and more convinced that it only
makes sense in the internals of some
Post by Steve Harris
Hoever
making sure htat the API is scale neutral (unlike generic MIDI) is
important. I'm not sure what the best way to ensure this is.
I use "linear pitch". 1.0 would be your MIDI standard middle C. +1.0
would give you the next higher semitone, and +12.0 would give you the
next higher octave.

Being that note IDs have nothing to do with pitch, you can use
whatever scale you like, as well as starting all notes at the same
pitch, or whatever.
Post by Steve Harris
Obviously pitch and velicty are optional, not all instruments have
either. Thats one reason if favour of using well know portt names
to indicate what voice parameters normally map to what MIDI
controls.
Yes, indeed. Most poeple will just want to throw a synth in and start
playing, so hosts must have a way of figuring out some sensible basic
routing. Musicians in general probably don't want to connect ten
"cables" on the screen before they can even get the basic data from
the MIDI controller into the synth. :-)
Post by Steve Harris
Post by David Olofson
How about the host knowing that this is the name of a file that
should go with the project? I would suggest the following file
OK, that is a good point, it would make the host able to store the
session more easily, and package it up my copying the file into
local space, and changin the path.
Yes, that's the idea. Getting the best of two worlds: Separate
(possibly human readable) files, as well as useful project management
capabilities.
Post by Steve Harris
Post by David Olofson
Right, either you get memory - or you're screwed.
That said, it's possibly to implement real time memory managers,
and I think the host designer should have the option.
But if the api is like LADSPA you're not allowed to do allocation
in the RT part anyway, so you may as well stick to malloc and
friends.
That's not the whole story. When you hit malloc(), we're talking
*unbounded* latency. That means it may take 10 seconds to instantiate
a plugin. This may be ok if you never instantiate plugins "live", but
if you do (for live gigs, or whatever), you should be nervous about
this.

Maybe it's no big deal; most people can probably just instantiate all
plugins before starting to play. I just don't like assuming that this
will never be an issue, when the solution is as simple as two host
calls that would normally wrap malloc() and free().
Post by Steve Harris
If hte host is desperate to use something else it can
override them with LD_PRELOAD.
Yeah - on some platforms. And it's a rather messy hack for something
like that, IMHO.
Post by Steve Harris
Post by David Olofson
Good point. I still think it's nice if hosts can construct basic
GUIs for simple plugins, but I see no point in taking any major
design measures with host constructed GUIs in mind.
Sure, but simple instruments dont require file i/o or any of that
stuff anyway.
Well, then there's no problem. It's just that if we design an API for
simple instruments, I can quarantee that it won't be too long before
we're here again, discussing a third API for "advanced instruments".
(That will probably happen anyway, but we can at least *consider*
covering what we know already.)

Or we just design an event system and strap that onto JACK as well as
a stripped version of LADSPA. :-)


//David Olofson - Programmer, Composer, Open Source Advocate

.- Coming soon from VaporWare Inc...------------------------.
| The Return of Audiality! Real, working software. Really! |
| Real time and off-line synthesis, scripting, MIDI, LGPL...|
`-----------------------------------> (Public Release RSN) -'
.- M A I A -------------------------------------------------.
| The Multimedia Application Integration Architecture |
`----------------------------> http://www.linuxdj.com/maia -'
--- http://olofson.net --- http://www.reologica.se ---
Steve Harris
2002-12-04 22:33:00 UTC
Permalink
Post by David Olofson
Post by Steve Harris
Hoever
making sure htat the API is scale neutral (unlike generic MIDI) is
important. I'm not sure what the best way to ensure this is.
I use "linear pitch". 1.0 would be your MIDI standard middle C. +1.0
would give you the next higher semitone, and +12.0 would give you the
next higher octave.
Thats somewhat semitone biased, if we use +1.0 is middle C, +2.0 is one
octave higher then we are tuning neutral, and, coincidentally compatible
with 1.0volt/octave analogue synths.
Post by David Olofson
Well, then there's no problem. It's just that if we design an API for
simple instruments, I can quarantee that it won't be too long before
we're here again, discussing a third API for "advanced instruments".
(That will probably happen anyway, but we can at least *consider*
covering what we know already.)
Well, no, because for advanced instruments (eg. a full featured sampler)
we have jack.
Post by David Olofson
Or we just design an event system and strap that onto JACK as well as
a stripped version of LADSPA. :-)
I think its reasonable to expect a jack app to be able to natively
understand MIDI/OSC/CV/whatever.

- Steve
David Olofson
2002-12-04 22:45:01 UTC
Permalink
Post by Steve Harris
Post by David Olofson
Post by Steve Harris
Hoever
making sure htat the API is scale neutral (unlike generic MIDI)
is important. I'm not sure what the best way to ensure this is.
I use "linear pitch". 1.0 would be your MIDI standard middle C.
+1.0 would give you the next higher semitone, and +12.0 would
give you the next higher octave.
Thats somewhat semitone biased, if we use +1.0 is middle C, +2.0 is
one octave higher then we are tuning neutral, and, coincidentally
compatible with 1.0volt/octave analogue synths.
Well, I just decided to use 12.0 per octave in Audiality. No
particular reason (yeah, right! ;-), and it's easy enough to change.
With fixed point numbers, this affects accuracy slightly, but with
floats, it makes no difference whatsoever in that regard.
Post by Steve Harris
Post by David Olofson
Well, then there's no problem. It's just that if we design an API
for simple instruments, I can quarantee that it won't be too long
before we're here again, discussing a third API for "advanced
instruments". (That will probably happen anyway, but we can at
least *consider* covering what we know already.)
Well, no, because for advanced instruments (eg. a full featured
sampler) we have jack.
Well, yes - modulo a sample accurate event system that deals with
stuff beyond MIDI.
Post by Steve Harris
Post by David Olofson
Or we just design an event system and strap that onto JACK as
well as a stripped version of LADSPA. :-)
I think its reasonable to expect a jack app to be able to natively
understand MIDI/OSC/CV/whatever.
Sure - but how do I connect a sequencer to it, and how do I make it
all stay in sync? I think people will soon expect timing and reliable
of softsynths to be *better* than h/w synths + MIDI; not worse.


//David Olofson - Programmer, Composer, Open Source Advocate

.- Coming soon from VaporWare Inc...------------------------.
| The Return of Audiality! Real, working software. Really! |
| Real time and off-line synthesis, scripting, MIDI, LGPL...|
`-----------------------------------> (Public Release RSN) -'
.- M A I A -------------------------------------------------.
| The Multimedia Application Integration Architecture |
`----------------------------> http://www.linuxdj.com/maia -'
--- http://olofson.net --- http://www.reologica.se ---
Steve Harris
2002-12-04 23:03:00 UTC
Permalink
Post by David Olofson
Post by Steve Harris
Well, no, because for advanced instruments (eg. a full featured
sampler) we have jack.
Well, yes - modulo a sample accurate event system that deals with
stuff beyond MIDI.
OSC?
Post by David Olofson
Post by Steve Harris
Post by David Olofson
Or we just design an event system and strap that onto JACK as
well as a stripped version of LADSPA. :-)
I think its reasonable to expect a jack app to be able to natively
understand MIDI/OSC/CV/whatever.
Sure - but how do I connect a sequencer to it, and how do I make it
all stay in sync? I think people will soon expect timing and reliable
of softsynths to be *better* than h/w synths + MIDI; not worse.
Well, alsa-seq, DMIDI or OSC, the linux MIDI sequencers I've seen support
alsa-seq, and some support OSC. All of these will give better timing than
hardware MIDI IIRC.

- Steve
David Olofson
2002-12-05 00:46:00 UTC
Permalink
Post by Steve Harris
Post by David Olofson
Post by Steve Harris
Well, no, because for advanced instruments (eg. a full featured
sampler) we have jack.
Well, yes - modulo a sample accurate event system that deals with
stuff beyond MIDI.
OSC?
Maybe. It has been mentioned, IIRC. What happened?
Post by Steve Harris
Post by David Olofson
Post by Steve Harris
Post by David Olofson
Or we just design an event system and strap that onto JACK as
well as a stripped version of LADSPA. :-)
I think its reasonable to expect a jack app to be able to
natively understand MIDI/OSC/CV/whatever.
Sure - but how do I connect a sequencer to it, and how do I make
it all stay in sync? I think people will soon expect timing and
reliable of softsynths to be *better* than h/w synths + MIDI; not
worse.
Well, alsa-seq, DMIDI or OSC, the linux MIDI sequencers I've seen
support alsa-seq, and some support OSC. All of these will give
better timing than hardware MIDI IIRC.
Well, I guess I'll just try one or more of them. I want to have
Audiality running as a JACK client ASAP, and the most important part
of that is the replacement of raw MIDI input with something
timestamped that some useful sequencer can handle. (Linux only of
course; I don't have, and don't want a Windows machine for sequencing
any more.)


//David Olofson - Programmer, Composer, Open Source Advocate

.- Coming soon from VaporWare Inc...------------------------.
| The Return of Audiality! Real, working software. Really! |
| Real time and off-line synthesis, scripting, MIDI, LGPL...|
`-----------------------------------> (Public Release RSN) -'
.- M A I A -------------------------------------------------.
| The Multimedia Application Integration Architecture |
`----------------------------> http://www.linuxdj.com/maia -'
--- http://olofson.net --- http://www.reologica.se ---
Steve Harris
2002-12-05 11:16:01 UTC
Permalink
Post by David Olofson
Post by Steve Harris
Post by David Olofson
Well, yes - modulo a sample accurate event system that deals with
stuff beyond MIDI.
OSC?
Maybe. It has been mentioned, IIRC. What happened?
Its around, there is some support in software for it.
Post by David Olofson
Post by Steve Harris
Well, alsa-seq, DMIDI or OSC, the linux MIDI sequencers I've seen
support alsa-seq, and some support OSC. All of these will give
better timing than hardware MIDI IIRC.
Well, I guess I'll just try one or more of them. I want to have
Audiality running as a JACK client ASAP, and the most important part
of that is the replacement of raw MIDI input with something
timestamped that some useful sequencer can handle. (Linux only of
course; I don't have, and don't want a Windows machine for sequencing
any more.)
Well, alsa-seq seems to be the system of choice. Never used it myself
though.

- Steve
David Olofson
2002-12-04 18:35:00 UTC
Permalink
Post by Tim Hockin
Post by David Olofson
Well, a guaranteed unique ID is really rather handy when you want
to load up a project on another system and be *sure* that you're
using the right plugins... That's about the only strong
motivation I can think of right now, but it's strong enough for
me.
Ok, I see your motivation for this. I hate the idea of 'centrally
assigned' anythings for something as open as this. I'll think more
on it..
I see your point, but I'm not sure if it makes sense to take the
measures needed to guarantee uniqueness in some other way. Developer
IDs could probably be 16 bit, and I would assume that even if we
knock out both VST and DX (!), we're not going to have hundreds of
new developers registering every day constantly - so one could just
have someone administer this manually via email.

Either way, it only takes one unique ID per developer... How about
using the h/w-address of one of your NICs? But then, what if you
don't have a NIC? :-)
Post by Tim Hockin
Post by David Olofson
IMHO, plugins should not worry about whether or not their outputs
are connected. (In fact, there are reasons why you'd want to
always guarantee that all ports are connected before you let a
plugin run. Open inputs would be connected to a dummy silent
output, and open outputs would be connected to a /dev/null
equivalent.)
I disagree with that - this is a waste of DSP cycles processing to
be sent nowhere.
So, why would you ask the plugin to set up outputs that you won't
connect, and then force the plugin to have another conditional to
check whether the output is connected or not?

Besides, it's not always as simple as adding a conditional to
*efficiently* eliminate the work related to an output. And even if it
is, chances are you have to put that conditional in the inner loop,
to avoid too much code duplication.

A stronger motivation, however, is that this would be another feature
of the API; one that every plugin is required to support.

That said, thinking about multiport plugins with complex internal
routing (ie output x may be affected by other inputs than input x), I
can see where something like this would be useful.

I would propose that the pre-instantiation host/plugin "negotiations"
including:

* A way for the host to ask the plugin what different types
of ports it has.
* A way of getting the maximum and minimum number of ports
of each type, as well as the granularity for port counts.
* A way for the host to tell the plugin how many ports of
each type it wants for a particular instance of the plugin.
* A way for the host to *ask* the plugin to disable certain
ports if possible, so they can be left disconnected.


The important point is that however you do it, you end up with a
plugin with two 1D, contiguous arrays (although possibly with some
ports disabled, if the plugin supports it); one for inputs and one
for outputs. That will simplify the low level/DSP code, and I think
*that's* where complexity matters the most. (Have as little code as
possible in the main path, to avoid optimizing stuff that can be done
elsewhere.)

As an example, say you have a plugin that takes mono input and
generates 5.1 output. Port info could look something like this:

Input type: In;MONO
Min count: 1
Max count: -1 (unlimited)
Granularity: 1
DisableSingle: FALSE (doesn't make sense)
DisableGroup: FALSE (group == single)

Output type: Out;5.1
Min count: 6
Max count: -1 (unlimited)
Granularity: 6
DisableSingle: TRUE
DisableGroup: FALSE (doesn't make sense)


So, if you want to process one mono input and have 4 channel output
(no bass and center channels), you'd ask for this when instantiating:

Inputs: In;MONO
Count: 1
Disabled: <None>

Outputs: Out;5.1
Count: 6
Disabled: <1, 5> (center, bass)

Now, if the plugin didn't support DisableSingle on the output ports
of type Out;5.1, you'd have to accept getting all 6 outs, and just
route the bass and centel channels to "/dev/null". It should be easy
enough for the host, and it could simplify and/or speed up the
average case (all outputs used, assumed) of the plugin a bit, since
there's no need for conditionals in the inner loop, mixing one buffer
for each output at a time, or having 63 (!) different versions of the
mixing loop.
Post by Tim Hockin
Post by David Olofson
as some kind of interface to the plugin, but to me, it seems way
too limited to be of any real use. You still need a serious
interface
if it has no other use than 'ignore this signal and spare the CPU
time' it is good enough for me.
Well, it sounds useful, but I'm afraid it's not always that easy to
make use of in real DSP code - and in the cases where it isn't, I
think it's a bad idea to *require* that plugins support it.

IMO, it should be optional, and plugin coders should be strongly
*recommended* to consider this feature when it means a speed-up in
useful configurations without penalizing the average case. Plugins
should have the chance of setting it up during initialization or
similar context, rather than checking for it in the "process()"
callback.
Post by Tim Hockin
Post by David Olofson
just like you program a studio sampler to output stuff to the
outputs you want. This interface may be standardized or not - or
there may be both variants - but either way, it has to be more
sophisticated than one bit per output.
Ehh, again, I think it is simpler. Lets assume a simple sampler.
It has a single output with 0 or more channels (in my terminology).
If you load a stereo sample, it has 2 channels. A 5.1 sample has
6 channels. Let's consider an 8-pad drum machine. It has 8
outputs each with 0-2 channels. Load a stereo sample, that output
has 2 channels. Now, as I said, maybe this is a bad idea. Maybe
it should be assumed that all outputs have 2 channels and mono gets
duplicated to both or (simpler) LEFT is MONO.
Well, I've never seen a synth or sampler (hardware or software)
change it's output "format" based on the format of the loaded
waveform - and I can't see that it would make sense, or even be
particularly useful.
Tim Hockin
2002-12-04 10:43:04 UTC
Permalink
Post by Steve Harris
For the stuff that is acutally vital, UID, version, instrument name,
whatever, then yes. But anything more sophisticated than that /really/
should be external. Internalised metadata systems invariably end up being
inadequet, clumsy or just useless (ala LADSPA).
The only meta-data I included is stuff that might pop up in an 'About' box.
Presets and the likes BEG to be external.

Tim
Dr. Matthias Nagorni
2002-12-04 12:02:01 UTC
Permalink
Hi,

Recently Frank Neumann pointed me at some very impressive CSound
algorithms implemented by Sean Costello. I was especially looking for a
digital implementation of the Bode Frequency Shifter which is among the
coolest modules you can have in a modular synthesizer. There is also a
vintage stringphaser which sounds incredible.

Writing email to Sean Costello to ask whether he would mind if his algorithms
were ported to C and published under GPL, I discovered that his email
address is no longer valid. Googling for it was unsucessful because there
are many Sean Costellos and the CSound author seems to have disappeared.

Now my question is whether anybody on this list happens to know Sean
Costello and/or knows of a valid email address of him ?

Matthias
--
Dr. Matthias Nagorni
SuSE GmbH
Deutschherrnstr. 15-19 phone: +49 911 74053375
D - 90429 Nuernberg fax : +49 911 74053483
Tim Hockin
2002-12-04 18:25:01 UTC
Permalink
Post by Steve Harris
In modular synth systems there are good reasons why you want to know
whether things are connected (and good reasons why they wouldn't be), but
this spec so far is not useful to a modular synth, so in this case I
agree.
Can you expound, if only for curiosity? Both reasons why it's useful for
modular synths and why this API is not?
Post by Steve Harris
Interleaved data is more trouble that its worth IMNSHO. I think
I agree with that.
Post by Steve Harris
multichannel outputs are extra complexity with no benefit.
I'm coming round
Post by Steve Harris
I'm not sure what peoples opinions on numbers of outputs are, obviously the
number needs to be variable per instrument at development time, but I dont
think it should be variable at instantiation time, that doesn't sound
useful, and it would be hell to optimise.
I'd agree with this, but for a few exceptions:
* Mixer plugin. I really do want it to have a variable number of inputs. I
don't want to say 48-channels or 128-channels.
Post by Steve Harris
It seems like you need two sets, per instrument and per voice, they should
share the same interface, and the per voice ones definatly should not be
limited to modulation,chorus,whatever.
what if each control has an int per_voice; field. If the field is 0, that
param is not modulatable per-voice. If it is > 0, it is an instrument-unique
ID which can be used by the host to modify that param:
plug->voice_change(plug, voice_id, param_id, value);
(or whatever it will look like if we go the way of all events).

Can we assume that all voices have a standard control mapped to velocity? Or
should the per-voice controls actually be another set of controls, and each
instrument needs to specify velocity,pitchbend (and others)?

As a similar: should instrument plugins be providing hints as to their
MIDI-equivalent functionality? CC-equivalent numbers per control, perhaps?
Post by Steve Harris
My feeling is that just float+"arbitrary opaque binary data" is enough.
The float can be augmented with hints, enumerations, whatever.
String is needed if we want to deal with filenames without a custom GUI.
Opaque binary data has no meaning to the host, and the host is what
manipulates controls. Where does it get that data?
Steve Harris
2002-12-04 19:30:00 UTC
Permalink
Post by Tim Hockin
Post by Steve Harris
In modular synth systems there are good reasons why you want to know
whether things are connected (and good reasons why they wouldn't be), but
this spec so far is not useful to a modular synth, so in this case I
agree.
Can you expound, if only for curiosity? Both reasons why it's useful for
modular synths and why this API is not?
Its very common to have modules that change thier behviour whne cables are
attached, eg. to override a knob, or to tuern a knob form a value to a
level modulation.

The reason this is not useful in a modular systme is (mostly) the way they
deal with polyphony.

In a modular system, each oscilator etc, is monophonic and you clone whole
blocks of the system to make it polyphonic. Also, pitch is controlled via
CV control and gate, which doesnt map well to midi style note events.

Sure, you could coerce the described system into a modular voewpoint but
there would be a lot of overhead and neadless complexity.
Post by Tim Hockin
Post by Steve Harris
I'm not sure what peoples opinions on numbers of outputs are, obviously the
number needs to be variable per instrument at development time, but I dont
think it should be variable at instantiation time, that doesn't sound
useful, and it would be hell to optimise.
* Mixer plugin. I really do want it to have a variable number of inputs. I
don't want to say 48-channels or 128-channels.
Right, but do mixer plugins belong in an instrument API? Be good at one
thing...

If you can give me an example of an instrument that benefits from variable
numbers of i/o and doesn't deserve to be a standalone jack client then I'l
agree with you.
Post by Tim Hockin
Can we assume that all voices have a standard control mapped to velocity? Or
should the per-voice controls actually be another set of controls, and each
instrument needs to specify velocity,pitchbend (and others)?
Here you could make use of well known labels again, note-on-velocity,
note-off-velocity, pitchbend etc. The host can map these to the normal MIDI
controls of it likes.

It probably makes sense to list the per-voice controls speratly from the
per-instrument. Its just important that they are the same at a fundamental
level (otherwise you end up with very confused developers and code).
Post by Tim Hockin
Post by Steve Harris
My feeling is that just float+"arbitrary opaque binary data" is enough.
The float can be augmented with hints, enumerations, whatever.
String is needed if we want to deal with filenames without a custom GUI.
Opaque binary data has no meaning to the host, and the host is what
manipulates controls. Where does it get that data?
The binary data comes from "GUIs", I still dont see how a generic UI can
usefully deal with filenames, it can pop up a file selector dialgoue, but
the user doesnt know what thier looking for.

- Steve
David Olofson
2002-12-04 21:22:01 UTC
Permalink
On Wednesday 04 December 2002 19.24, Steve Harris wrote:
[...]
Post by Steve Harris
In a modular system, each oscilator etc, is monophonic and you
clone whole blocks of the system to make it polyphonic. Also, pitch
is controlled via CV control and gate, which doesnt map well to
midi style note events.
Another reason why having those events at all is a bad idea...

Note that with an event based system (or a callback based system -
but not a shared variable based system, such as LADSPA), you still
actually have *events* for everything. It's just a matter of
interprettation. If you feel like a MIDI synth, just interpret the
first velocity change to a non-zero value as NoteOn with a velocity
of the argument value. Then wait for a velocity change to 0, and
treat that as NoteOff. (Or rather, NoteOn with vel == 0, which is
what most controllers and sequencers send, unfortunately. :-/ )
Post by Steve Harris
Sure, you could coerce the described system into a modular
voewpoint but there would be a lot of overhead and neadless
complexity.
Well, all I have to say on that is that I think it's a bad idea to
draw a strict line between the two kinds of systems. I think one API
that handles both would be a lot more like real analog stuff, where
you don't really have to worry about protocols and APIs - it's all
just analog signals. I'd like to be able to use virtual studio style
plugins inside my modular synth networks, and vice versa, as far as
practically possible.
Post by Steve Harris
Post by Tim Hockin
Post by Steve Harris
I'm not sure what peoples opinions on numbers of outputs are,
obviously the number needs to be variable per instrument at
development time, but I dont think it should be variable at
instantiation time, that doesn't sound useful, and it would be
hell to optimise.
* Mixer plugin. I really do want it to have a variable number of
inputs. I don't want to say 48-channels or 128-channels.
Right, but do mixer plugins belong in an instrument API? Be good at
one thing...
Yes - but if you remove the features needed to support things like
mixers, you also remove features that instruments may need. Not
everyone wants to build instruments out of lots of little plugins.

Besides, mixers need automation...
Post by Steve Harris
If you can give me an example of an instrument that benefits from
variable numbers of i/o and doesn't deserve to be a standalone jack
client then I'l agree with you.
Good point - if we were talking about a plugin API strictly designed
for synths.
Steve Harris
2002-12-04 22:25:00 UTC
Permalink
Post by David Olofson
Well, all I have to say on that is that I think it's a bad idea to
draw a strict line between the two kinds of systems. I think one API
that handles both would be a lot more like real analog stuff, where
you don't really have to worry about protocols and APIs - it's all
just analog signals. I'd like to be able to use virtual studio style
plugins inside my modular synth networks, and vice versa, as far as
practically possible.
Well LADSPA can handle modular synth style modules perfectly well, modulo
the lack of GUIs nad conneciton detection (c.f. AMS), so theres no need
to support thier features elsewhere.
Post by David Olofson
Post by Steve Harris
If you can give me an example of an instrument that benefits from
variable numbers of i/o and doesn't deserve to be a standalone jack
client then I'l agree with you.
Good point - if we were talking about a plugin API strictly designed
for synths.
David Olofson
2002-12-04 20:48:01 UTC
Permalink
On Wednesday 04 December 2002 18.19, Tim Hockin wrote:
[...]
Post by Tim Hockin
Post by Steve Harris
It seems like you need two sets, per instrument and per voice,
they should share the same interface, and the per voice ones
definatly should not be limited to modulation,chorus,whatever.
what if each control has an int per_voice; field. If the field is
0, that param is not modulatable per-voice. If it is > 0, it is an
instrument-unique ID which can be used by the host to modify that
plug->voice_change(plug, voice_id, param_id, value);
(or whatever it will look like if we go the way of all events).
Sounds like a good idea. The overlap between voice and channel
controls on most synths is big enough that I can't see a proper
motivation for two separate interfaces. It *might* make sense to have
two variants of the events/calls, though;

plug->channel_change(plug, param_id, value);
plug->voice_change(plug, voice_id, param_id, value);

The alternative would be what you suggested; testing voice_id for 0,
or -1 or whatever. (I prefer -1, as everything starts at 0. :-)
Post by Tim Hockin
Can we assume that all voices have a standard control mapped to
velocity?
No. Organs don't have velocity, for example, so why should virtual
organs implement it, or even get the information? (Ok, I know that
most synths implement velocity for organ sounds anyway...)
Post by Tim Hockin
Or should the per-voice controls actually be another set
of controls, and each instrument needs to specify
velocity,pitchbend (and others)?
Well... In the case of Audiality, you can assume *nothing* about a
patch/instrument. A channel is routed right into a patch driver,
which can be a standard mono or poly synth, a MIDI file player, a
script parser/VM or whatever. If you loud the patch "CoffeeBrewer",
it may *really* be an interface to your coffee brewing machine,
rather than something that drives the synth engine.

So, I would say it can be a bit hard for some synths to report a
useful list of controls. On most synths, it all depends on the loaded
patch/instrument. There may be a fixed set of controls, but that
doesn't mean all patches/instruments make use of all of them. I would
really prefer if the API would enable the host to display a list of
controls that are *actually* supported; not just a list of what
*might* be supported.

This is one aspect of MIDI that sucks. There are 128 controls, but
most synths only support a fraction of them - and then most patches
only support a fraction of *those* controls. As if that's not bad
enough, most synths and patches aren't properly documented, so you
simply have to test all controls, or dive into the programming mode
of the synth, to find out what is supported, and how it works.

Wouldn't it be great if every patch/instrument could report the
controls it has, and perhaps even a short description of each one of
them?
Post by Tim Hockin
As a similar: should instrument plugins be providing hints as to
their MIDI-equivalent functionality? CC-equivalent numbers per
control, perhaps?
I'd prefer if we don't even mention "MIDI" in the API - but the
terminology and the controls mentioned there might be a good starting
point, though.

Either way, I think we need more than one enumeration, or at least,
the enumeration needs to be "partitioned" in some way, so one can
tell whether or not two controls with different type hints are
approximately equivalent or not. For example, one might be interested
in knowing whether a control affects level, frequency, modulation
deepth or "effect depth". One might also want to know whether we're
talking about volume, frequency, cutoff or whatever. Further, linear,
quadratic, cubic, logarithmic etc could be of interest.

That said, we're still talking about *hints*. It's not like you're
not going to be able to use a plugin just because the host doesn't
understand what the controls are all about. The important thing is
that the *user* understands what the controls are for - and a
sensible name is usually sufficient for that.
Post by Tim Hockin
Post by Steve Harris
My feeling is that just float+"arbitrary opaque binary data" is
enough. The float can be augmented with hints, enumerations,
whatever.
String is needed if we want to deal with filenames without a custom GUI.
Just to make a general point about worrying about host constructed
GUIs:

You need various special GUI widgets to handle things like envelopes,
graphical oscillator waveform editors, spectrum editors and other
cool stuff. In some cases, you also need special control types to
deal with these in a sane way. Are we supposed to put all that in the
API and all hosts, or are we going to lock that type of plugins out
entirely?

My point is just that a line has to be drawn somewhere. IMO, it
should be drawn so that communication that the host can have actual
use of understanding should go through standardized protocols, while
the rest can go through "raw data blocks" and similar generic
solutions.

In this case; it *is* useful for the host to know about external
files that plugins reference, since this information can server as a
complete list of all data that is referenced by a project.

In general; it is *not* useful for the host to understand data that
it isn't expected to be able to actually *manipulate*. IMHO, there
should be as little as possible of this in plugins, but I can't see
any way of eliminating it.

And mind you; I really tried. That's basically what the MAIA project
was about. The project is effectively dead now, although many of the
ideas live on in Audiality now. (The basic event system is almost
identical, for example.)

That said, I actually came up with a rather simple solution, put in
relation to the magnitude of the problem:

* Put unique IDs on the data types, so hosts can know which
ports are compatible without understanding the data, or
even recognising the data type IDs.

* Plugin coders should be encouraged to first look for
built-in standard data types that fit the bill, and only
invent new types when they're really required.

* After inventing a new data type, you're encouraged to
implement a set of "mini plugins" (a toolkit, if you will)
to perform basic operations on the new data type. This
would include editors, displays, mixers and that sort of
things. These "mini plugins" should be registered under
special names or IDs, so hosts can recognize them as
"compatibility plugins", when they're faced with their
respective custom data type.


We could support a system like this for the "raw data blocks". Host
and plugin authors may support it or not; the only differece is that
when there is support in the host, and the plugin author cared to
implement the "compatibility mini plugins", these mystical ports may
actually be manipulated by the host to some extent. (If not, the
standard procedures for preset saving and loading apply, and that's
about it.)
Post by Tim Hockin
Opaque binary data has no meaning to the host, and the host is
what manipulates controls. Where does it get that data?
I don't totally agree with the idea that it is the host that
manipulates controls. I would agree that having *only* controls, and
no mystical private patch data, whould be cool - but I don't think
restricting the API to supporting only known control types is a good
idea. Some sort of "raw data blocks" will be needed by some plugins.

Whether or not this should be implemented as an actual raw data block
type, or as external files (loaded by the host or the plugin?) is
another matter.

I'm actually leaning more and more towards external files, as it's a
simple and flexible solution, that allows easy sharing of data across
projects, use of standard external editors and other stuff - and the
host may still be nice and actually *understand* where these files
belong, and what to do with them when saving projects for backup or
transfer. (See one of my previous posts.)


//David Olofson - Programmer, Composer, Open Source Advocate

.- Coming soon from VaporWare Inc...------------------------.
| The Return of Audiality! Real, working software. Really! |
| Real time and off-line synthesis, scripting, MIDI, LGPL...|
`-----------------------------------> (Public Release RSN) -'
.- M A I A -------------------------------------------------.
| The Multimedia Application Integration Architecture |
`----------------------------> http://www.linuxdj.com/maia -'
--- http://olofson.net --- http://www.reologica.se ---
Tim Hockin
2002-12-04 20:11:00 UTC
Permalink
Post by Steve Harris
The central part can be very open, eg in URIs you're allowed to assign
URIs from any domain you have control over, so I could label somthing
"http://plugin.org.uk/foo#23". This means that anyone with a website can
assign URIs.
I dont really like using strings for UIDs though. Its too easy to read
semantics into them.
I agree about semantics, but self-maintenance makes this much more
appealing. Of course, domain registration expires.
Post by Steve Harris
Post by Tim Hockin
I disagree with that - this is a waste of DSP cycles processing to be sent
nowhere.
No, its a waste of DSP cycles to check whether something's there or not.
can we meet in the middle and say that for some cases it is easier to assume
silence, and in others it is easier to check for presence?
Post by Steve Harris
Dont bundle up channels! People dont do that in studios for a good reason.
a single channel on a single connections is simple, unambiguous, obvious.
But say you have 2 channels is that L+R, M+S, C+M? how about 8, is it 7.1,
8.1 pre filter, 5.1 in ADAT? What do you do when you have a 5.1 bundle
and you want to send it as 5.1 in ADAT? Does that require a 5.1 rebundler?
ok, ok, I'm coming around again :)
Post by Steve Harris
I'm not sure about the other arguments, but polyphony control is complex
and probably only the instrument can do it usefully. If you want this you
might have to do it by telling the instrument what its max poly should be.
Perhaps this, with some way of the instrument telling the host about numbers
of voices.
Post by Steve Harris
voice_on is proably more instrument neutral. Not everything is a note (eg.
my LADSPA gong).
sorry, note_on is MIDI, voice_on is what I put in the API :)
Post by Steve Harris
I dont think you can do anything useful with a generic file string, what
could be host use it for?
The host could provide a standard file-open dialog for filenames. The host
could provide a text-box for speech-synthesis.
Steve Harris
2002-12-04 20:53:00 UTC
Permalink
Post by Tim Hockin
Post by Steve Harris
I dont really like using strings for UIDs though. Its too easy to read
semantics into them.
I agree about semantics, but self-maintenance makes this much more
appealing. Of course, domain registration expires.
Thats why its inportant to choose your URIs carefully. It could also be
viewed as another rason to issue central ints, they dont expire ;) As long
as there is a reserved block for experimentation, its its easy to get real
IDs it should all be OK.
Post by Tim Hockin
Post by Steve Harris
Post by Tim Hockin
I disagree with that - this is a waste of DSP cycles processing to be sent
nowhere.
No, its a waste of DSP cycles to check whether something's there or not.
can we meet in the middle and say that for some cases it is easier to assume
silence, and in others it is easier to check for presence?
OK, maybe. But (as discussed elsewhere) in an event system theres no need
to check, there wont ever be any events on unconnected inputs.
Post by Tim Hockin
Post by Steve Harris
I'm not sure about the other arguments, but polyphony control is complex
and probably only the instrument can do it usefully. If you want this you
might have to do it by telling the instrument what its max poly should be.
Perhaps this, with some way of the instrument telling the host about numbers
of voices.
If you think its neccesary, it is meaningless in some cases though and I'm
not clear on what it would be useful for. Every instrument feature adds to
the barrier to entry and needs to be justified.
Post by Tim Hockin
Post by Steve Harris
I dont think you can do anything useful with a generic file string, what
could be host use it for?
The host could provide a standard file-open dialog for filenames. The host
could provide a text-box for speech-synthesis.
OK, I'm starting to come round on this one.

- Steve
David Olofson
2002-12-04 22:11:01 UTC
Permalink
[...]
Post by Steve Harris
Post by Tim Hockin
can we meet in the middle and say that for some cases it is
easier to assume silence, and in others it is easier to check for
presence?
OK, maybe. But (as discussed elsewhere) in an event system theres
no need to check, there wont ever be any events on unconnected
inputs.
It's different with audio ports, though. (Unless they were in fact
abstract objects, controlled by the host only through events sent to
the plugin. I were thinking along those lines in MAIA.)

An audio input or output either needs to have a valid buffer (which
may be a fake silence buffer, or a trash "/dev/null" buffer), or you
must ensure that the plugin checks the pointer.

I would prefer that plugins get to handle this themselves in any way
they like.

Dead simple:
Just say you don't support disconnected ports.
The host will have to give you fake buffers if
it wants to leave ports "open".

Nice but simple:
Accept disconnected ports, and just check the
pointers, or whatever. (Note that passing NULL
buffers might have other users. In Audiality,
it's legal for inputs, and means "silence for
one frame". IIRC, you can turn the feature off,
if you like, forcing the host to always pass
valid buffers.)

Elegant (implementation dependent):
Accept disconnected ports. If the plugin is
a complex beast, chances are it has an internal
list, tree or network of things to do. All you
do is deal with disconnected ports when
contructing that list, tree or network.
Post by Steve Harris
Post by Tim Hockin
Post by Steve Harris
I'm not sure about the other arguments, but polyphony control
is complex and probably only the instrument can do it usefully.
If you want this you might have to do it by telling the
instrument what its max poly should be.
Perhaps this, with some way of the instrument telling the host
about numbers of voices.
If you think its neccesary, it is meaningless in some cases though
and I'm not clear on what it would be useful for. Every instrument
feature adds to the barrier to entry and needs to be justified.
Agreed. I think you either use a monophonic patch/instrument, or you
just let the synth make the best use of it's voice reserve.

Note that per-note/voice controls have nothing to do with this -
unless you make the same mistake I did in Audiality; use voice
numbers as note IDs... A voice is one thing, and a note is another. A
note may not have a voice at all times.
Post by Steve Harris
Post by Tim Hockin
Post by Steve Harris
I dont think you can do anything useful with a generic file
string, what could be host use it for?
The host could provide a standard file-open dialog for filenames.
The host could provide a text-box for speech-synthesis.
OK, I'm starting to come round on this one.
Well, the file dialog would need some extra info, but it does make
sense to use the same file selector (provided by the host) for
everything it could handle, within reasonable limits. (If you want
the preview button to work for Audiality AGW scripts, you need to
wait for me to hack a custom file selector. ;-)


//David Olofson - Programmer, Composer, Open Source Advocate

.- Coming soon from VaporWare Inc...------------------------.
| The Return of Audiality! Real, working software. Really! |
| Real time and off-line synthesis, scripting, MIDI, LGPL...|
`-----------------------------------> (Public Release RSN) -'
.- M A I A -------------------------------------------------.
| The Multimedia Application Integration Architecture |
`----------------------------> http://www.linuxdj.com/maia -'
--- http://olofson.net --- http://www.reologica.se ---
Steve Harris
2002-12-04 22:35:01 UTC
Permalink
Post by David Olofson
Accept disconnected ports, and just check the
pointers, or whatever. (Note that passing NULL
buffers might have other users. In Audiality,
it's legal for inputs, and means "silence for
one frame". IIRC, you can turn the feature off,
if you like, forcing the host to always pass
valid buffers.)
This one sounds good to me, Mattias suggested something similar for LADSPA.

- Steve
David Olofson
2002-12-04 22:54:00 UTC
Permalink
Post by Steve Harris
Post by David Olofson
Accept disconnected ports, and just check the
pointers, or whatever. (Note that passing NULL
buffers might have other users. In Audiality,
it's legal for inputs, and means "silence for
one frame". IIRC, you can turn the feature off,
if you like, forcing the host to always pass
valid buffers.)
This one sounds good to me, Mattias suggested something similar for LADSPA.
Especially for the inputs, where it coincides with the really rather
useful "NULL input instead of a buffer means silence" feature - which
actually is the same thing as "this input is not connected right
new", for all practical purposes. (Unless you want to emulate pops
and clicks generated when pulling and plugging analog cables, that
is. ;-)

However, it's not fun when you're in an inner loop like this:

for(i = 0; i < frames; ++i)
{
float s = inputs[0][i];

...interesting code here...

outputs[0][i] = s * controls[0];
outputs[1][i] = s * controls[1];
outputs[2][i] = s * controls[2];
outputs[3][i] = s * controls[3];
outputs[4][i] = s * controls[4];
}

Sure, you could turn the loop inside out, but if the "interesting
code" is complicated, you'll want an intermediate buffer, not to have
5 copies of that code.

Why not keep it simple, and have the ability to say that the host
shouldn't pass NULL pointers for output buffers to this plugin?


//David Olofson - Programmer, Composer, Open Source Advocate

.- Coming soon from VaporWare Inc...------------------------.
| The Return of Audiality! Real, working software. Really! |
| Real time and off-line synthesis, scripting, MIDI, LGPL...|
`-----------------------------------> (Public Release RSN) -'
.- M A I A -------------------------------------------------.
| The Multimedia Application Integration Architecture |
`----------------------------> http://www.linuxdj.com/maia -'
--- http://olofson.net --- http://www.reologica.se ---
Steve Harris
2002-12-04 23:15:01 UTC
Permalink
Post by David Olofson
Why not keep it simple, and have the ability to say that the host
shouldn't pass NULL pointers for output buffers to this plugin?
Sure, sounds fine to me.

- Steve
Tim Hockin
2002-12-04 22:39:00 UTC
Permalink
Post by David Olofson
Post by Tim Hockin
I disagree with that - this is a waste of DSP cycles processing to
be sent nowhere.
So, why would you ask the plugin to set up outputs that you won't
connect, and then force the plugin to have another conditional to
check whether the output is connected or not?
This confuses me. A plugin says it can handle 1-6 channels. The host only
connects 2 channels. The plugin loops for i = 0 to i = me->nchannels.
There isn't any checking. If the plugin says it can handle 2-6 channels and
the host only connects 1, it is an error. Connect at least the minimum, up
to the maximum. In typing this, I've seen that discontiguous connections
do, in fact, require condionals. Maybe it is safe to say you have to
connect ports in order?
Post by David Olofson
I would propose that the pre-instantiation host/plugin "negotiations"
* A way for the host to tell the plugin how many ports of
each type it wants for a particular instance of the plugin.
This is exactly what I'm talking about with the connect methods. Before we
go into PLAY mode, we ask for a certain number of channels.
Post by David Olofson
* A way for the host to *ask* the plugin to disable certain
ports if possible, so they can be left disconnected.
hmm, this is interesting, but now we're adding the conditional
Post by David Olofson
plugin with two 1D, contiguous arrays (although possibly with some
ports disabled, if the plugin supports it); one for inputs and one
for outputs. That will simplify the low level/DSP code, and I think
Yes, I've come around to this. The question in my mind is now about
disabling (or just not connecting) some ports.
Post by David Olofson
Now, if the plugin didn't support DisableSingle on the output ports
of type Out;5.1, you'd have to accept getting all 6 outs, and just
route the bass and centel channels to "/dev/null". It should be easy
enough for the host, and it could simplify and/or speed up the
average case (all outputs used, assumed) of the plugin a bit, since
there's no need for conditionals in the inner loop, mixing one buffer
for each output at a time, or having 63 (!) different versions of the
mixing loop.
ok, I see now. If the plugin supports disabling, the host can use it. If
the plugin is faster to assume all ports connected, it does that instead. I
think I rather like that.
Post by David Olofson
think it's a bad idea to *require* that plugins support it.
This is key, again, you've convinced me.
Post by David Olofson
strongly prefer working with individual mono waveforms, each on a
voice of their own, as this offers much more flexibility. (And it's
also a helluva' lot easier to implement a sampler that way! :-)
just so we're clear, 'voice' in your terminology == 'channel' in mine?
Post by David Olofson
...provided there is a quarantee that there is a buffer for the port.
Or you'll segfault unless you check every port before messing with
it. :-)
Do we need to provide a buffer for ports that are disabled?
Post by David Olofson
Post by Tim Hockin
Hrrm, so how does something like this sound?
(metacode)
Yeah, something like that. Add "count granularity", and you'll make
life for the plugin coder a lot easier, I think. (Again, see above.)
ok, I'll include something to this effect in the next draft.
Post by David Olofson
Post by Tim Hockin
{ "left(4):mono(4)" }, { "right(4)" },
Does this mean the plugin is supposed to understand that you want a
"mono mix" if you only connect the left output?
If the host connects this pad to a mono effect, it knows that the
'left' channel is also named 'mono'. I do not expect the plugin to mono-ize
a stereo sample (though it can if it feels clever).
Post by David Olofson
all that much to it. There's no sensible way of describing the
input/output relations of every possible plugin, so it's debatable
whether we should care to try at all.
I'm agreeing now..
Post by David Olofson
Post by Tim Hockin
* note_on returns an int voice-id
* that voice-id is used by the host for note_off() or note_ctrl()
That's the way I do it in Audiality - but it doesn't mix well with
timestamped events, not even within the context of the RT engine
core.
how so - it seems if you want to send a voice-specific event, you'd need
this
Post by David Olofson
I don't think that's a good idea. The synth has a much better chance
of knowing which voice is "best" to steal - and if smart voice
stealing is not what you want, you shouldn't use a polyphonic synth
or sound.
ok, ok.
Post by David Olofson
Besides, VSTi has it. DXi has it. I bet TDM has it. I'm sure all
major digital audio editing systems (s/w or h/w) have it. Sample
accurate timing. I guess there is a reason. (Or: It's not just me! :-)
yeah, VSTi also has MIDI - need I say more? I'm becoming convinced, though.
Post by David Olofson
What kind of knobs need to be ints? And what range/resolution should
they have...? You don't have to decide if you use floats.
They should have the same range as floats - whatever their control struct
dictates.
Post by David Olofson
Post by Tim Hockin
I'd assume a violin modeller would have a BOWSPEED control. The
note_on() would tell it what the eventual pitch would be. The
plugin would use BOWSPEED to model the attack.
Then how do you control pitch continously? ;-)
with a per-voice pitchbend
Post by David Olofson
Some controls may not be possible to change in real time context -
but I still think it makes sense to use the control API for things
like that.
I don't know if I like the idea of controls being flagged RT vs NONRT, but
maybe it is necessary. Or maybe it's not, and a user who changes a sample
in real time can expect a glitch.
Post by David Olofson
An Algorithmically Generated Waveform script...?
BUt what I don't get is: who loads the data into the control?

a) hast will call deserialize() with a string or other standard format
b) plugin will load it from a file, in which case host passes the filename
to the cotrol
c) host loads a chunk of arbitrary data which it read from the plugin before
saving/restoring - in which case how did it get there in the first place?
(see a or b)
Post by David Olofson
Well, then I guess you'll need the "raw data block" type after all,
since advanced synth plugins will have a lot of input data that
cannot be expressed as one or more "normal" controls in any sane way.
Such as? Where does this data come from in the first place?
Post by David Olofson
Just as with callback models, that depends entirely on the API and
the plugin implementation. AFAIK, DXi has "ramp events". The
Audiality synth has linear ramp events for output/send levels.
So does Apple Audio Units. I am starting to like the idea..
Post by David Olofson
Post by Tim Hockin
Audiality, but if we're designing the same thing, why aren't we
working on the same project?
Well, that's the problem with Free/Open Source in general, I think.
The ones who care want to roll their own, and the ones that don't
care... well, they don't care, unless someone throws something nice
and ready to use at them.
As to Audiality, that basically came to be "by accident". It started
Interesting how it came about, but why are you helping me turn my API into
yours, instead of letting me work on yours? Just curious. I do like to
roll my own, but I don't want to waste time..

Tim
David Olofson
2002-12-05 00:41:00 UTC
Permalink
Post by Tim Hockin
Post by David Olofson
Post by Tim Hockin
I disagree with that - this is a waste of DSP cycles processing
to be sent nowhere.
So, why would you ask the plugin to set up outputs that you won't
connect, and then force the plugin to have another conditional to
check whether the output is connected or not?
This confuses me. A plugin says it can handle 1-6 channels. The
host only connects 2 channels. The plugin loops for i = 0 to i =
me->nchannels. There isn't any checking.
If you have a "group" of channels, and just want to skip one in the
middle, that won't work.
Post by Tim Hockin
If the plugin says it can
handle 2-6 channels and the host only connects 1, it is an error.
Connect at least the minimum, up to the maximum. In typing this,
I've seen that discontiguous connections do, in fact, require
condionals.
Yes, that's exactly what I'm thinking about.
Post by Tim Hockin
Maybe it is safe to say you have to connect ports in
order?
Safe, bit it wouldn't be sufficient. Consider my mono->5.1 example.
Post by Tim Hockin
Post by David Olofson
I would propose that the pre-instantiation host/plugin
* A way for the host to tell the plugin how many ports of
each type it wants for a particular instance of the plugin.
This is exactly what I'm talking about with the connect methods.
Before we go into PLAY mode, we ask for a certain number of
channels.
Post by David Olofson
* A way for the host to *ask* the plugin to disable certain
ports if possible, so they can be left disconnected.
hmm, this is interesting, but now we're adding the conditional
Well, the point is that the conditional doesn't have to end up in the
inner loop of the plugin. The host could throw in a silent or garbage
buffer, if the plugin coder decides it's too hairy to implement the
plugin in a different way.

Then again, the plugin could use a private array of buffer pointers,
and throw in silence/garbage buffers itself. The buffers should still
be supplied by the host, though, to reduce memory use and cache
thrashing. (This is just hopefully just a "rare" special case, but
anyway...)
Post by Tim Hockin
Post by David Olofson
plugin with two 1D, contiguous arrays (although possibly with
some ports disabled, if the plugin supports it); one for inputs
and one for outputs. That will simplify the low level/DSP code,
and I think
Yes, I've come around to this. The question in my mind is now
about disabling (or just not connecting) some ports.
Post by David Olofson
Now, if the plugin didn't support DisableSingle on the output
ports of type Out;5.1, you'd have to accept getting all 6 outs,
and just route the bass and centel channels to "/dev/null". It
should be easy enough for the host, and it could simplify and/or
speed up the average case (all outputs used, assumed) of the
plugin a bit, since there's no need for conditionals in the inner
loop, mixing one buffer for each output at a time, or having 63
(!) different versions of the mixing loop.
ok, I see now. If the plugin supports disabling, the host can use
it. If the plugin is faster to assume all ports connected, it does
that instead.
Yes, that's the idea. And for the host, these are decisions made when
building the net, so it doesn't matter performance wise. It's code
that needs to be there, indeed - but it's simple and generic enough
that it could go in the host SDK. (Like that state changing function
of Audiality, for example; it comes with the plugin API.)
Post by Tim Hockin
I think I rather like that.
Good - then it might not be totally nonsense. :-)
Post by Tim Hockin
Post by David Olofson
think it's a bad idea to *require* that plugins support it.
This is key, again, you've convinced me.
Post by David Olofson
strongly prefer working with individual mono waveforms, each on a
voice of their own, as this offers much more flexibility. (And
it's also a helluva' lot easier to implement a sampler that way!
:-)
just so we're clear, 'voice' in your terminology == 'channel' in mine?
Well... If your definition of channel is like in the (classic)
tracker days, yes. What I call a voice is what plays a single
waveform in a synth or sampler. Depending on the design, it may only
have waveform and pitch controls - or it may include filters,
envelope generators, LFOs, distortion, panning and whatnot.

In fact, a voice could theoretically even combine multiple waveforms,
but that borders to something I'd call a "voice structure" - and that
should probably be published as multiple voices in a voice oriented
API.

I Audiality however, a 'channel' doesn't have a fixed relation to
audio processing. It's basically like a channel in MIDI speak, and
when dealing with notes, you're really dealing with *notes* - not
voices. (Remember the CoffeeBrewer patch? ;-)


Anyway, my original comment was really about synth/sampler
programming, where I prefer to construct stereo sounds from multiple
mono samples (each sample on it's own voice), as opposed to working
with voices that are capable of playing stereo waveforms.

That said, Audiality supports stereo voices. Don't know if I'll keep
that feature, though. It's a performance hack for sound effects in
games, mostly, and at some point, I'll probably have to sacrifice
some low end scalability for the high end.
Post by Tim Hockin
Post by David Olofson
...provided there is a quarantee that there is a buffer for the
port. Or you'll segfault unless you check every port before
messing with it. :-)
Do we need to provide a buffer for ports that are disabled?
Yes, if the plugin says it can't deal with disconnected ports. If it
says it can, it's supposed to check the pointers at some point during
the process() call - preferably once, before the event/DSP loop.

As to ports that are outside the number of ports requested by the
host; well those are outside the loops, and simply don't exist.


[...]
Post by Tim Hockin
Post by David Olofson
Post by Tim Hockin
{ "left(4):mono(4)" }, { "right(4)" },
Does this mean the plugin is supposed to understand that you want
a "mono mix" if you only connect the left output?
If the host connects this pad to a mono effect, it knows that the
'left' channel is also named 'mono'. I do not expect the plugin to
mono-ize a stereo sample
Ok.
Post by Tim Hockin
(though it can if it feels clever).
This makes me nervous. :-)

Ok; it's rather handy that some synths automatically transform the
left line output into a mono output if the right output is not
connected (the JV-1080 does that, IIRC) - but when it comes to
software, it's not like you need to drag in a mixer and cables to
implement that outside the machine. I'd really rather not have
plugins do all sorts of "smart" things without explicitly being asked
to.


[...]
Post by Tim Hockin
Post by David Olofson
Post by Tim Hockin
* note_on returns an int voice-id
* that voice-id is used by the host for note_off() or
note_ctrl()
That's the way I do it in Audiality - but it doesn't mix well
with timestamped events, not even within the context of the RT
engine core.
how so - it seems if you want to send a voice-specific event, you'd
need this
No, there are other ways. All you really need is a unique ID for each
voice, for addressing per-voice.

The simple and obvious way is to just get the voice number from the
voice allocator. The problem with that is that you need to keep track
of whether or not that voice still belongs to you before trying to
talk to it. In Audiality, I do that by allowing patch plugins (the
units that drives one or more voices based on input from a 'channel')
to "mark" voices they allocate with an ID - which is usually just the
channel number.

That's not perfect, though: If you change the patch on that channel
while you have old notes hanging, you either have to kill those notes
first (JV-1080 style - not good), or you have to keep the old patch
around, so it can control it's voices until they die. The latter is
what I'm trying to do in Audiality, but it requires that patches can
recognize their own voices *even* if there are other patches still
working on the same channel.

I could mark voices with *both* channel and patch numbers, but I have
a feeling that would only work until I discover *another* way that
patches could lose track of their voices. A cleaner and more generic
solution is needed. (Especially for a similar system for use in
public plugin API!)

...And then there's still this roundtrip issue, of course.

So, what I'm considering for Audiality is 'Virtual Voice Handles'.
When a patch is initialized, it asks the host for a number of these
(contigous range), which will then function as the "virtual voice
reserve" for the patch. When you want to allocate a voice, you send a
VVH with the "request", and just assume that you got one.
Tim Hockin
2002-12-05 05:48:01 UTC
Permalink
Post by David Olofson
No, there are other ways. All you really need is a unique ID for each
voice, for addressing per-voice.
Unless I'm missing something - all you really need is a voice-id that is
Unique within that instance of that plugin.

[... RT vs NONRT controls ...]
Post by David Olofson
1) realloc() the buffers when the certain parameters
change, or
2) decide on an absolute maximum buffer size, and always
allocate that during instantiation, or
3) realloc() the buffers when a NONRT "max_delay" or
similar parameter is changed.
3 works and is relatively clean and simple - but it requires an
"extra" interface for setting NONRT parameters. I would definitely
prefer this to be basically the same as the normal control interface,
as the only significant difference is in which context the control
changes are executed.
Post by Tim Hockin
Or maybe it's not, and a user
who changes a sample in real time can expect a glitch.
So you're not supposed to be able to implement delays that can take
modulation of the delay parameters, and still cope with arbitrary
delay lengths? (Just an example; this has been discussed before, so I
guess some people have more, and real examples.)
So how do VST effects do it? I can increase the delay feedback all I like and
it doesn't hiccup, stutter, or misbehave at all. You're saying that this is
a non-rt control because it might have to realloc() buffers?
Post by David Olofson
that plugins should have "safe defaults" built-in (to prevent them
absolutely - this is why controls have default values.
David Olofson
2002-12-05 07:45:01 UTC
Permalink
Post by Tim Hockin
Post by David Olofson
No, there are other ways. All you really need is a unique ID for
each voice, for addressing per-voice.
Unless I'm missing something - all you really need is a voice-id
that is Unique within that instance of that plugin.
Yes, that's correct.
Post by Tim Hockin
[... RT vs NONRT controls ...]
So how do VST effects do it? I can increase the delay feedback all
I like and it doesn't hiccup, stutter, or misbehave at all. You're
saying that this is a non-rt control because it might have to
realloc() buffers?
Yes. I'm not sure exactly how VST plugins (those that actually handle
this "properly" ) implement it, but inless it simply relies on
malloc() "being fast" most of the time, I'm quite sure it involves
doing any allocations and reinitialization in another thread. If it's
done from within the audio thread, it will not work reliably.

Also note that you might just have been lucky. There is no guarantee
that malloc() *will* cause missed deadlines when you're testing, just
because it can. However, it *can* on any operating system that has
virtual memory + swap - and probably most others (the few that still
exist, if any) as well.


//David Olofson - Programmer, Composer, Open Source Advocate

.- Coming soon from VaporWare Inc...------------------------.
| The Return of Audiality! Real, working software. Really! |
| Real time and off-line synthesis, scripting, MIDI, LGPL...|
`-----------------------------------> (Public Release RSN) -'
.- M A I A -------------------------------------------------.
| The Multimedia Application Integration Architecture |
`----------------------------> http://www.linuxdj.com/maia -'
--- http://olofson.net --- http://www.reologica.se ---
Steve Harris
2002-12-05 11:20:00 UTC
Permalink
Post by Tim Hockin
So how do VST effects do it? I can increase the delay feedback all I like and
it doesn't hiccup, stutter, or misbehave at all. You're saying that this is
a non-rt control because it might have to realloc() buffers?
Delay feedback is a RT control, delay time might not be (depending on how
its implemented).

I'm not sure how this can be handled easily, there are lots of things like
loading samples that will be impossible to handle in an RT way, but I'm
not sure that something like a sampler belongs in this kind of API. I know
there are VSTi samplers, but windows people dont have jack.

I could be that everything that can be usfully implemented in this kind of
API doesnt need non RT controls.

- Steve
Dave Griffiths
2002-12-05 12:18:01 UTC
Permalink
Post by Steve Harris
Post by Tim Hockin
So how do VST effects do it? I can increase the delay feedback all I like and
it doesn't hiccup, stutter, or misbehave at all. You're saying that this is
a non-rt control because it might have to realloc() buffers?
Delay feedback is a RT control, delay time might not be (depending
on how its implemented).
I'm not sure how this can be handled easily, there are lots of
things like loading samples that will be impossible to handle in an
RT way, but I'm not sure that something like a sampler belongs in
this kind of API. I know there are VSTi samplers, but windows people
dont have jack.
This is actually a current problem with spiralsynthmodular, the sampler is rt
safe, except when loading/saving samples. I'm thinking that it's solvable by
using a async loader on a different thread that the plugin can poll. As long
as it does something sane until the sample is loaded (play silence) it should
be Ok.

At the moment, you get chucked off jack when loading large samples.

dave
Steve Harris
2002-12-05 12:46:01 UTC
Permalink
Post by Dave Griffiths
Post by Steve Harris
I'm not sure how this can be handled easily, there are lots of
things like loading samples that will be impossible to handle in an
RT way, but I'm not sure that something like a sampler belongs in
this kind of API. I know there are VSTi samplers, but windows people
dont have jack.
This is actually a current problem with spiralsynthmodular, the sampler is rt
safe, except when loading/saving samples. I'm thinking that it's solvable by
using a async loader on a different thread that the plugin can poll. As long
as it does something sane until the sample is loaded (play silence) it should
be Ok.
Yes, this will work in a system like SSM, but I'm not sure how well
generic .so plugins and threads will play together... I guess theres
nothing to stop you from cloning a new thread inside a .so, but it may
come as a supprise to the host ;)

- Steve
Dave Griffiths
2002-12-05 12:54:00 UTC
Permalink
Post by Steve Harris
Post by Dave Griffiths
Post by Steve Harris
I'm not sure how this can be handled easily, there are lots of
things like loading samples that will be impossible to handle in an
RT way, but I'm not sure that something like a sampler belongs in
this kind of API. I know there are VSTi samplers, but windows people
dont have jack.
This is actually a current problem with spiralsynthmodular, the sampler is rt
safe, except when loading/saving samples. I'm thinking that it's solvable by
using a async loader on a different thread that the plugin can poll. As long
as it does something sane until the sample is loaded (play silence) it should
be Ok.
Yes, this will work in a system like SSM, but I'm not sure how well
generic .so plugins and threads will play together... I guess theres
nothing to stop you from cloning a new thread inside a .so, but it
may come as a supprise to the host ;)
ssm plugins are .so plugins - and the midi plugin already spawns a thread to
collect the midi data ok (the host knows nothing about it) so hopefully it's
doable.

dave
Steve Harris
2002-12-05 14:00:01 UTC
Permalink
Post by Dave Griffiths
ssm plugins are .so plugins - and the midi plugin already spawns a thread to
collect the midi data ok (the host knows nothing about it) so hopefully it's
doable.
Yes, its fine in SSM, because you know that there is only one host, and
you know how it behaves. I dont know, but I'm guessing that it could mess
up some generic hosts. Maybe its always safe.

- Steve
David Olofson
2002-12-05 16:37:01 UTC
Permalink
[...loading samples in different thread...]
Post by Steve Harris
Yes, this will work in a system like SSM, but I'm not sure how well
generic .so plugins and threads will play together... I guess
theres nothing to stop you from cloning a new thread inside a .so,
but it may come as a supprise to the host ;)
Good point.

The way it's supposed to be done in Audiality (but isn't right now)
is that the *host* is responsible for not executing non RT save
parameter changes in RT context. It basically has to rip the plugin
out of the net and pass it to a butler thread which performs the
operation, sending the plugin back when done.

However, this only works for certain cases, obviously. It's not as
simple as that if you only need to run *part* of the plugin in
another thread, while the RT part remains running.

I'm considering some kind of RPC style thing to deal with this. I'd
much rather have the host handle the details, as anything involving
plugins and direct use of threading APIs is inherently very
nonportable - and potentially rather hairy.


//David Olofson - Programmer, Composer, Open Source Advocate

.- Coming soon from VaporWare Inc...------------------------.
| The Return of Audiality! Real, working software. Really! |
| Real time and off-line synthesis, scripting, MIDI, LGPL...|
`-----------------------------------> (Public Release RSN) -'
.- M A I A -------------------------------------------------.
| The Multimedia Application Integration Architecture |
`----------------------------> http://www.linuxdj.com/maia -'
--- http://olofson.net --- http://www.reologica.se ---
David Olofson
2002-12-05 16:30:01 UTC
Permalink
On Thursday 05 December 2002 12.07, Dave Griffiths wrote:
[...non RT controls, VST & delay ...]
Post by Dave Griffiths
This is actually a current problem with spiralsynthmodular, the
sampler is rt safe, except when loading/saving samples. I'm
thinking that it's solvable by using a async loader on a different
thread that the plugin can poll. As long as it does something sane
until the sample is loaded (play silence) it should be Ok.
At the moment, you get chucked off jack when loading large samples.
Applies to Audiality as well. It needs humongous amounts of CPU power
for "loading" most sounds as well, since they're actually scripts
used to render the waveforms using the off-line synth.

There's also another, similar problem: Plugin instantiation. FX
inserts are selected through MIDI CC events, which means their
instantiation is triggered from within RT context.

Currently, they're also instantiated in there, which is just a quick
hack. (Which, AFAIK, prevents this engine from running at all on Mac
OS classic - can't malloc() from within interrupt context.)

What I'll do is throw in a "butler thread" (Benno's invention :-), to
which I pass requests to instantiate or destroy plugins, load or
unload waveforms and other non RT safe things. There will probably
just be an sfifo in each direction, since those are extremely
portable and easy to use. The method of waking up the butler thread
will probably vary a little between platforms. On systems without
threads, the butler will just be a callback that you're supposed to
call from the main loop every now and then.


//David Olofson - Programmer, Composer, Open Source Advocate

.- Coming soon from VaporWare Inc...------------------------.
| The Return of Audiality! Real, working software. Really! |
| Real time and off-line synthesis, scripting, MIDI, LGPL...|
`-----------------------------------> (Public Release RSN) -'
.- M A I A -------------------------------------------------.
| The Multimedia Application Integration Architecture |
`----------------------------> http://www.linuxdj.com/maia -'
--- http://olofson.net --- http://www.reologica.se ---
David Olofson
2002-12-05 16:16:00 UTC
Permalink
Post by Steve Harris
Post by Tim Hockin
So how do VST effects do it? I can increase the delay feedback
all I like and it doesn't hiccup, stutter, or misbehave at all.
You're saying that this is a non-rt control because it might have
to realloc() buffers?
Delay feedback is a RT control, delay time might not be (depending
on how its implemented).
Right - I didn't even notice that he wrote "feedback" and not "delay
time"... *heh* That's what I was thinking about, to clarify things.
Post by Steve Harris
I'm not sure how this can be handled easily, there are lots of
things like loading samples that will be impossible to handle in an
RT way, but I'm not sure that something like a sampler belongs in
this kind of API. I know there are VSTi samplers, but windows
people dont have jack.
Well, they have ReWire, but AFAIK, that supports only the "load as
DLL and run as callback" method...
Post by Steve Harris
I could be that everything that can be usfully implemented in this
kind of API doesnt need non RT controls.
I strongly doubt it's as simple as that. Maybe those controls will
really be more like "instantiation time parameters" - but they still
need to be *somewhere*. An extra bit in an existing API seems like
less complexity than a parallel API with essentially the same
functionality.


//David Olofson - Programmer, Composer, Open Source Advocate

.- Coming soon from VaporWare Inc...------------------------.
| The Return of Audiality! Real, working software. Really! |
| Real time and off-line synthesis, scripting, MIDI, LGPL...|
`-----------------------------------> (Public Release RSN) -'
.- M A I A -------------------------------------------------.
| The Multimedia Application Integration Architecture |
`----------------------------> http://www.linuxdj.com/maia -'
--- http://olofson.net --- http://www.reologica.se ---
Steve Harris
2002-12-05 16:49:00 UTC
Permalink
Post by David Olofson
Post by Steve Harris
I'm not sure how this can be handled easily, there are lots of
things like loading samples that will be impossible to handle in an
RT way, but I'm not sure that something like a sampler belongs in
this kind of API. I know there are VSTi samplers, but windows
people dont have jack.
Well, they have ReWire, but AFAIK, that supports only the "load as
DLL and run as callback" method...
And it doesn't appear to be as successful, Cubase and Protools dont use it
for thier primary i/o for example (c.f. Muse and Ardour).

- Steve
David Olofson
2002-12-05 18:31:00 UTC
Permalink
Post by Steve Harris
Post by David Olofson
Post by Steve Harris
I'm not sure how this can be handled easily, there are lots of
things like loading samples that will be impossible to handle
in an RT way, but I'm not sure that something like a sampler
belongs in this kind of API. I know there are VSTi samplers,
but windows people dont have jack.
Well, they have ReWire, but AFAIK, that supports only the "load
as DLL and run as callback" method...
And it doesn't appear to be as successful, Cubase and Protools dont
use it for thier primary i/o for example (c.f. Muse and Ardour).
Well, that's to be expected. Win32 DLLs are a mess, and implementing
entire applications as DLLs probably brings in a whole lot of extra
problems. And if you're doing DLL + callbacks anyway, why not just
use VSTi or DXi? Guess that explains most of it.


//David Olofson - Programmer, Composer, Open Source Advocate

.- Coming soon from VaporWare Inc...------------------------.
| The Return of Audiality! Real, working software. Really! |
| Real time and off-line synthesis, scripting, MIDI, LGPL...|
`-----------------------------------> (Public Release RSN) -'
.- M A I A -------------------------------------------------.
| The Multimedia Application Integration Architecture |
`----------------------------> http://www.linuxdj.com/maia -'
--- http://olofson.net --- http://www.reologica.se ---
Tim Hockin
2002-12-07 09:31:01 UTC
Permalink
So, I'm incorporating the things that were discussed this week into my API
proposal. It's starting to take shape, and I really appreciate the feedback
I got (why I can't get that level of feedback even at work, is beyond me).

I've hit a couple of large questions that I want to bounce off you all.

Per-voice controls vs. Master controls:
----
This is what I am thinking: One of the flags on each control is whether it
is a CTRL_MASTER, or a CTRL_VOICE (or both). This allows the plugin to
define a ctrl labelled (for example) VELOCITY and the host can connect it to
any MIDI input, if it wants.

Let's assume a plugin provides a control named FOO which is both VOICE and
MASTER. Let's assume the current state of FOO is the value 50.0. Let's
assume the sequencer triggers a voice and sets FOO for that voice to 80.0.
The user then turns the master FOO down to 0. What happens to the value of
the voice's FOO.
a) goes to 0
b) ends up at 30.0
c) stays at 80.0

Maybe controls that are both MASTER and VOICE should be absolute values for
MASTER and scalars against the MASTER value per-voice?

Needs discussion.

Triggering a voice
----
I see a few methods, each has pros and cons.

1) * Host calls plug->note_on(timestamp, note, velocity), and
gets a voice-id.
* Host sends n VOICE_PARAM events to set up any params it wants
* Host calls plug->run()
* Plugin activates the voice at the right timestamp with the
right params.
* Host calls plug->get_polyphony() to know how many voices are
playing.
* Host calls plug->voice_off(voice, velocity)

2) * Host sends a VOICE_ON event with a negative voice-id and params
velocity and note
* Host sends n VOICE_PARAM events for the negative voice-id
* Host calls plug->run()
* Plugin receives the VOICE_ON, allocates a voice-id, and and
sends a VOICE_ON event with the real voice-id and the negative
bogus one back to the host.
* Host sends a VOICE_OFF event to the plugin to end a voice,
plugin ACKs by sending it back.
* Plugin sends a VOICE_OFF event to the host if a voice is done
before the host sends VOICE_OFF (short sample, etc)
* Host keeps a count of voices, increment on VOICE_ON, decrement
on VOICE_OFF.

Modifications on those two:
a) Same as #1 but with no voice_off() method - use VOICE_OFF events
instead.

b) Modification on #1 and #2 - don't pass velocity and pitch params,
instead have them as controls
c) Same as b, except Velocity is not a continuous control - pass
velocity but keep pitch as a control (possibly pitch and
pitch-bend being different? not sure).


Preset and state save/restore
----
Should each plugin provide serialize() and deserialize() methods (thus
storing plugin state from the plugin itself in an arbitrary string) or
should the host be expected to write-out each control's data?

I'm leaning towards serialize/deserialize.

So, I'm looking for ideas and discussion!

Tim
Steve Harris
2002-12-07 12:27:01 UTC
Permalink
Post by Tim Hockin
This is what I am thinking: One of the flags on each control is whether it
is a CTRL_MASTER, or a CTRL_VOICE (or both). This allows the plugin to
define a ctrl labelled (for example) VELOCITY and the host can connect it to
any MIDI input, if it wants.
I'm not sure that making one control both is meaningful. Per voice and per
instrument controls should probably be kept seperate.
Post by Tim Hockin
1) * Host calls plug->note_on(timestamp, note, velocity), and
gets a voice-id.
I dont think I like the firstclass-ness of note and velocity, a we've
discussed they are meaningless for some instruments and could be
overloaded.
Post by Tim Hockin
* Host sends n VOICE_PARAM events to set up any params it wants
You could just send pitch and velocity this way?
Post by Tim Hockin
Should each plugin provide serialize() and deserialize() methods (thus
storing plugin state from the plugin itself in an arbitrary string) or
should the host be expected to write-out each control's data?
The host should do it. You dont want to have to write serialising code for
every single instrument. This also ensures that all the state can be
automated.

- Steve
David Olofson
2002-12-07 16:50:01 UTC
Permalink
Post by Steve Harris
Post by Tim Hockin
This is what I am thinking: One of the flags on each control is
whether it is a CTRL_MASTER, or a CTRL_VOICE (or both). This
allows the plugin to define a ctrl labelled (for example)
VELOCITY and the host can connect it to any MIDI input, if it
wants.
I'm not sure that making one control both is meaningful. Per voice
and per instrument controls should probably be kept seperate.
I'm still not sure I understand the terminology here... Considering
some recent posts, I assume that the following applies:

Instrument:
Monophonic or polyphonic plugin, capable of
maintaining one set of "patch data" (or "one
instrument sound") at a time. In popular MIDI
synth terminology, an Instrument would
correspond to "one part".

Voice:
An abstract object inside an Instrument,
representing "one note". A polyphonic Instrument
may maintain several Voices at a time. Physically,
a Voice may be anything from a single oscillator
to a network of oscillators, filters and other
objects.

Is that what you mean?

Personally, I'd rather not use the term "Instrument", but rather just
Plugin. In my terminology, an Instrument (as defined above) would be
a Channel, and Plugins may or may not be allowed to have multiple
Channels. As to the control protocol, if you want to do away with the
"channel" field, you could just allow synths to have more than one
event input port. One might think that that means more event decoding
overhead, but given that you'll most probably process one channel at
a time anyway, it would actually be an *advantage*. Oh, and you
wouldn't have to use separate negotiation schemes for
instrument/plugin controls and voice/note controls - you can just use
separate ports.

<audiality_internals>
BTW, in Audiality, there's no real plugin API for the synth engine
yet (it's used only for insert FX), but just objects with event
ports. As of now, there is one event port for each Channel, and one
event port for each physical Voice. "Virtual Voices" (ie Voices as
defined above, which I would rather call "Notes" or, just that;
Virtual Voices) are not defined in the "API", and may not exist in
some patches - the Patch Plugin (which is what converts Channel
Events into... anything, basically) decides how do deal with Channels
and Voice events.
</audiality_internals>
Post by Steve Harris
Post by Tim Hockin
1) * Host calls plug->note_on(timestamp, note, velocity), and
gets a voice-id.
I dont think I like the firstclass-ness of note and velocity, a
we've discussed they are meaningless for some instruments and could
be overloaded.
I agree. When I implemented the simple "2D positional" sound FX API
for the Audiality engine, I quickly realized that this <pitch,
velocity> tuple is just an arbitrary set which has been hammered into
our minds through the years. It's a sensible compromize to save
bandwidth, and it works well for most keyboard controlled
instruments, but that's about it.

I think it's time we stop thinking of two arbitrary parameters as an
obvious part of any "note on" message.


As to the "gets a voice ID" thing, no, I think this is a bad idea.
The host should *pick* a voice ID from a previously allocated pool of
Virtual Voice IDs. That way, you eliminate the host-plugin or
plugin-plugin (this is where it starts getting interesting)
roundtrip, which means you can, amond other things:

* send controls *before* you start a voice, if you like
* implement smarter and tighter voice allocation
* use a timestamped event system
* pipe events over cables, somewhat like MIDI
Post by Steve Harris
Post by Tim Hockin
* Host sends n VOICE_PARAM events to set up any params it wants
You could just send pitch and velocity this way?
Yes, I think so. There's slightly more overhead, obviously, but the
alternative is to maintain a performance hack in the API, all plugins
and all hosts... Also note that event size (which has to be fixed)
increases if you must squeeze in multiple parameters in some events.
Post by Steve Harris
Post by Tim Hockin
Should each plugin provide serialize() and deserialize() methods
(thus storing plugin state from the plugin itself in an arbitrary
string) or should the host be expected to write-out each
control's data?
The host should do it. You dont want to have to write serialising
code for every single instrument. This also ensures that all the
state can be automated.
++host_does_serialising;


//David Olofson - Programmer, Composer, Open Source Advocate

.- The Return of Audiality! --------------------------------.
| Free/Open Source Audio Engine for use in Games or Studio. |
| RT and off-line synth. Scripting. Sample accurate timing. |
`---------------------------> http://olofson.net/audiality -'
.- M A I A -------------------------------------------------.
| The Multimedia Application Integration Architecture |
`----------------------------> http://www.linuxdj.com/maia -'
--- http://olofson.net --- http://www.reologica.se ---
Steve Harris
2002-12-07 17:39:00 UTC
Permalink
Post by David Olofson
I'm still not sure I understand the terminology here... Considering
Yes, more or less. I was using instrument as in VSTi instrument, ie an
instance of a thing instantiated from a .so file. The word plugin is far
too overloaded.

The intention is that these things would (on the whole) be sound
generators, right? To me plugin implies inline processing.
Post by David Olofson
As to the "gets a voice ID" thing, no, I think this is a bad idea.
The host should *pick* a voice ID from a previously allocated pool of
Virtual Voice IDs. That way, you eliminate the host-plugin or
plugin-plugin (this is where it starts getting interesting)
Good idea.

- Steve
David Olofson
2002-12-07 21:32:00 UTC
Permalink
Post by Steve Harris
Post by David Olofson
I'm still not sure I understand the terminology here...
Considering some recent posts, I assume that the following
Yes, more or less. I was using instrument as in VSTi instrument, ie
an instance of a thing instantiated from a .so file. The word
plugin is far too overloaded.
Ok. (But VSTis can be multichanne/multipart, AFAIK.)
Post by Steve Harris
The intention is that these things would (on the whole) be sound
generators, right? To me plugin implies inline processing.
The word "plugin", AFAIK, simply refers to something you plug into an
application - it's not limited to DSP plugins or generators, and
doesn't even have to have anything to do with audio. It's just the
music industry that has hijacked the word. :-)


//David Olofson - Programmer, Composer, Open Source Advocate

.- The Return of Audiality! --------------------------------.
| Free/Open Source Audio Engine for use in Games or Studio. |
| RT and off-line synth. Scripting. Sample accurate timing. |
`---------------------------> http://olofson.net/audiality -'
.- M A I A -------------------------------------------------.
| The Multimedia Application Integration Architecture |
`----------------------------> http://www.linuxdj.com/maia -'
--- http://olofson.net --- http://www.reologica.se ---
Steve Harris
2002-12-08 01:09:00 UTC
Permalink
Post by David Olofson
Post by Steve Harris
Yes, more or less. I was using instrument as in VSTi instrument, ie
an instance of a thing instantiated from a .so file. The word
plugin is far too overloaded.
Ok. (But VSTis can be multichanne/multipart, AFAIK.)
OK, but thier still instruments :)
Post by David Olofson
The word "plugin", AFAIK, simply refers to something you plug into an
application - it's not limited to DSP plugins or generators, and
doesn't even have to have anything to do with audio. It's just the
music industry that has hijacked the word. :-)
Sure, sure. But it has been hijacked, theres not much point trying to
un-hijack it.

- Steve
David Olofson
2002-12-08 01:32:01 UTC
Permalink
Post by Steve Harris
Post by David Olofson
Post by Steve Harris
Yes, more or less. I was using instrument as in VSTi
instrument, ie an instance of a thing instantiated from a .so
file. The word plugin is far too overloaded.
Ok. (But VSTis can be multichanne/multipart, AFAIK.)
OK, but thier still instruments :)
Yes - but there's only one instance for N channels. :-)
Post by Steve Harris
Post by David Olofson
The word "plugin", AFAIK, simply refers to something you plug
into an application - it's not limited to DSP plugins or
generators, and doesn't even have to have anything to do with
audio. It's just the music industry that has hijacked the word.
:-)
Sure, sure. But it has been hijacked, theres not much point trying
to un-hijack it.
Right. I still think instruments are a *subset* of plugins, but I
guess we'll have to think of a new word if we want something that
people in general actually interpret that way.

Units? Which would actually be more correct, since that would include
JACK clients (running as separate processes) and the like, which are
*not* plugins, IMHO.


//David Olofson - Programmer, Composer, Open Source Advocate

.- The Return of Audiality! --------------------------------.
| Free/Open Source Audio Engine for use in Games or Studio. |
| RT and off-line synth. Scripting. Sample accurate timing. |
`---------------------------> http://olofson.net/audiality -'
.- M A I A -------------------------------------------------.
| The Multimedia Application Integration Architecture |
`----------------------------> http://www.linuxdj.com/maia -'
--- http://olofson.net --- http://www.reologica.se ---
Steve Harris
2002-12-08 12:56:01 UTC
Permalink
Post by David Olofson
Post by Steve Harris
Post by David Olofson
Ok. (But VSTis can be multichanne/multipart, AFAIK.)
OK, but thier still instruments :)
Yes - but there's only one instance for N channels. :-)
Right, but isn't that what were planning?

- Steve
Tim Hockin
2002-12-07 20:18:01 UTC
Permalink
Post by Steve Harris
Post by Tim Hockin
This is what I am thinking: One of the flags on each control is whether it
is a CTRL_MASTER, or a CTRL_VOICE (or both). This allows the plugin to
define a ctrl labelled (for example) VELOCITY and the host can connect it to
any MIDI input, if it wants.
I'm not sure that making one control both is meaningful. Per voice and per
instrument controls should probably be kept seperate.
Whether they are defined together or seperately, there are certainly
controls that apply to each voice and to each (instrument, channel (midi),
plugin). For example volume.

What advantage do you see to defining them seperately? Certainly the
structures are analogous, if not identical.

So no one has offered me any suggestions on how we handle the clash between
Master and per-voice controls. To re-ask:

<RE-ASK rephrased="true">
Let's assume a plugin provides a control named VOLUME which is both VOICE and
MASTER. Let's assume the current state of VOLUME is the value 5.0. Let's
assume the sequencer triggers a voice and sets VOLUME for that voice to 8.0.
The user then turns the master VOLUME down to 0. What happens to the value of
the voice's VOLUME.
a) goes to 0
b) ends up at 3.0
c) stays at 8.0

Maybe controls that are both MASTER and VOICE should be absolute values for
MASTER and scalars against the MASTER value per-voice?
</RE-ASK>
Post by Steve Harris
Post by Tim Hockin
1) * Host calls plug->note_on(timestamp, note, velocity), and
gets a voice-id.
I dont think I like the firstclass-ness of note and velocity, a we've
I agree, mostly. I threw this in as a bit of a red-herring, to see what
people were thinking.
Post by Steve Harris
Post by Tim Hockin
* Host sends n VOICE_PARAM events to set up any params it wants
You could just send pitch and velocity this way?
Absolutely. HOWEVER, I have one design issue with it: Velocity is not a
continuous control. You can't adjust the velocity halfway through a long
note. You can adjust the pitch. You can adjust portamento time. Velocity
relates SPECIFICALLY to the attack and release force of the musician.
Unless we all agree that velocity == loudness, which will be tough, since I
think _I_ disagree.

Could I get you to accept voice_on(velocity); and pass all the rest as
events?
Post by Steve Harris
Post by Tim Hockin
Should each plugin provide serialize() and deserialize() methods (thus
The host should do it. You dont want to have to write serialising code for
every single instrument. This also ensures that all the state can be
automated.
Two votes for this - I'll consider it decided.
Post by Steve Harris
I'm still not sure I understand the terminology here... Considering
Monophonic or polyphonic plugin, capable of
maintaining one set of "patch data" (or "one
Correct.
Post by Steve Harris
An abstract object inside an Instrument,
representing "one note". A polyphonic Instrument
Correct.
Post by Steve Harris
a Channel, and Plugins may or may not be allowed to have multiple
Channels. As to the control protocol, if you want to do away with the
"channel" field, you could just allow synths to have more than one
event input port. One might think that that means more event decoding
So I haven't come to this note in my TODO list yet. So we want to support
the idea of (MIDI word) multi-timbral plugins? Given that it is software,
we can say 'just load a new instance for the new channel'. It prevents
anyone from doing an exact software mockup of a bit of hardware, but I'm
inclined not to care that much..
Post by Steve Harris
ports. As of now, there is one event port for each Channel, and one
event port for each physical Voice. "Virtual Voices" (ie Voices as
Are you using multiple event ports, then, or just having one port per
instrument-plugin and sending EV_VOICE_FOO and EV_MASTER_FOO (modulo naming)
events?
Post by Steve Harris
bandwidth, and it works well for most keyboard controlled
instruments, but that's about it.
Other than Organs, which you've mentioned, what kinds of instruments don't
have some concept of velocity (whether they ignore it or not..). As I've
said above I have a hard-time reconciling velocity with any timed event
other than on/off.
Post by Steve Harris
As to the "gets a voice ID" thing, no, I think this is a bad idea.
The host should *pick* a voice ID from a previously allocated pool of
Virtual Voice IDs. That way, you eliminate the host-plugin or
plugin-plugin (this is where it starts getting interesting)
I'm still not clear on this. What plugin would trigger another plugin? Do
you envision that both the host and a plugin would be controlling this
plugin? If so, how do you reconcile that they will each have a pool of
VVIDs - I suppose they can get VVIDs from the host, but we're adding a fair
bit of complexity now.
Post by Steve Harris
* send controls *before* you start a voice, if you like
you can do this already - any voice-control event with a timestamp before
(or equal to) the note_on timestamp can be assumed to be a startup param.
Post by Steve Harris
* implement smarter and tighter voice allocation
I don't see what you mean by this, or how it matters. I see voices as being
the sole property of the instrument. All the host knows about them is that
they have some (int) id that is unique per-instrument.
Post by Steve Harris
* use a timestamped event system
umm, I thik this is not solely dependant on vvids - I think timestamps will
work just fine as proposed. It's a matter of taste, unity, and presentation
we're discussing.
Post by Steve Harris
* pipe events over cables, somewhat like MIDI
Ahh, now HERE is something interesting. I'd always assumed the plugin would
return something, self-allocating a voice. This is what you've called the
round-trip problem, I guess. But it seems to me that even if you were piping
something over a wire, you have some sort of local plugin handling it. THAT
plugin allocates the voice-id (arguing my model). The question I was asking
was: is voice-id allocation synchronous (call plug->voice_on(timestamp) and
get a valid voice-id right now), or is it async (send a note-on event and
wait for it to come back).

This raises another question for me - the host sends events to the plugins.
Do the plugins send events back? It seems useful. The host has to handle
syncronization issues (one recv event-queue per thread, or it has to be
plugin -> host callback, or something), but that's OK. Do timestamps matter
to the host, except as bookkeeping?
Post by Steve Harris
and all hosts... Also note that event size (which has to be fixed)
increases if you must squeeze in multiple parameters in some events.
Why does it have to be fixed-size? It doesn't strictly HAVE to be. On one
hand I HATE when an API says 'you have three available params' foo->param1,
foo->param2, foo->param3. If you need more, too bad. On the other hand,
there are performance advantages to having events pre-allocated, and
thereby, fixed sized, or at least bounded.
Post by Steve Harris
The intention is that these things would (on the whole) be sound
generators, right? To me plugin implies inline processing.
This API is not purely instrumental. It can certainly be for effects and
sinks, too. That said, we're spending a lot of time on the instrumental
part because it's the new ground wrt LADSPA.


Tim
Steve Harris
2002-12-07 21:23:00 UTC
Permalink
Post by Tim Hockin
What advantage do you see to defining them seperately? Certainly the
structures are analogous, if not identical.
Not defining them seperatly, but making a distinction, I dont think one
control should be applied per voice and per instrument.
Post by Tim Hockin
So no one has offered me any suggestions on how we handle the clash between
Ban it :)
Post by Tim Hockin
Post by Steve Harris
Post by Tim Hockin
* Host sends n VOICE_PARAM events to set up any params it wants
You could just send pitch and velocity this way?
Absolutely. HOWEVER, I have one design issue with it: Velocity is not a
continuous control. You can't adjust the velocity halfway through a long
note. You can adjust the pitch. You can adjust portamento time. Velocity
relates SPECIFICALLY to the attack and release force of the musician.
Unless we all agree that velocity == loudness, which will be tough, since I
think _I_ disagree.
No, velocity != loudless.

However, even in midi there can be more than one velocity value per note,
there is seperate release volcity IIRC. I expect the only reason attack
velocity is package up with the note start in MIDI is because otherwise
they couldn't guarantee it would arrive at the same time, and it saved a
few bytes.

Retriggering controlers (like that giant ribbon controller) could provide
multiple velovity changes within one "note" too.
Post by Tim Hockin
we can say 'just load a new instance for the new channel'. It prevents
anyone from doing an exact software mockup of a bit of hardware, but I'm
inclined not to care that much..
There are probably other reasons why you /might/ want this, but I cant
think of anyoffhand. I suspect one is OK and it makes things nice and
simple.
Post by Tim Hockin
Other than Organs, which you've mentioned, what kinds of instruments don't
have some concept of velocity (whether they ignore it or not..). As I've
said above I have a hard-time reconciling velocity with any timed event
other than on/off.
Analogue synths generally dont, some have channel pressure, but that is
slightly different and variable through the duration.
Post by Tim Hockin
I'm still not clear on this. What plugin would trigger another plugin? Do
you envision that both the host and a plugin would be controlling this
plugin? If so, how do you reconcile that they will each have a pool of
VVIDs - I suppose they can get VVIDs from the host, but we're adding a fair
bit of complexity now.
Not really the host can just allocate them from a pool of 32bit ints (for
example) and pass them in by the new_voice call.
Post by Tim Hockin
This raises another question for me - the host sends events to the plugins.
Do the plugins send events back? It seems useful. The host has to handle
syncronization issues (one recv event-queue per thread, or it has to be
plugin -> host callback, or something), but that's OK. Do timestamps matter
to the host, except as bookkeeping?
I'm not sure what the instrument will send back to the host that needs to
be timestamped.
Post by Tim Hockin
Why does it have to be fixed-size? It doesn't strictly HAVE to be. On one
hand I HATE when an API says 'you have three available params' foo->param1,
foo->param2, foo->param3. If you need more, too bad. On the other hand,
there are performance advantages to having events pre-allocated, and
thereby, fixed sized, or at least bounded.
Well, we were talking about floats, strings or opaque data blocks before,
so presumanbly the event would be either a float, a pointer to a string or
a pointer to a data block (+ a size).

If you need to send it over a wire you will have to serialise the strings
and blocks in variabley size chunks, but thats unavoiadable.
Post by Tim Hockin
Post by Steve Harris
The intention is that these things would (on the whole) be sound
generators, right? To me plugin implies inline processing.
This API is not purely instrumental. It can certainly be for effects and
sinks, too. That said, we're spending a lot of time on the instrumental
part because it's the new ground wrt LADSPA.
Its not purely aimed at instruments, but its heavily focused that way, it
probably wouldn't make sense to use this API for something LADSPA can do
well (eg. audio+control -> audio+control). If you need timestamps or
polyphony then this would be the way to go, but realisticly that means
intruments and a few other odd cases.

- Steve
David Olofson
2002-12-08 00:09:01 UTC
Permalink
Post by Steve Harris
Post by Tim Hockin
Post by Steve Harris
The intention is that these things would (on the whole) be
sound generators, right? To me plugin implies inline
processing.
This API is not purely instrumental. It can certainly be for
effects and sinks, too. That said, we're spending a lot of time
on the instrumental part because it's the new ground wrt LADSPA.
Its not purely aimed at instruments, but its heavily focused that
way, it probably wouldn't make sense to use this API for something
LADSPA can do well (eg. audio+control -> audio+control). If you
need timestamps or polyphony then this would be the way to go, but
realisticly that means intruments and a few other odd cases.
I disagree. I still think the VST folks are right when they say it's
a bad think of effects and instruments as two different kinds of
plugins. There are just too many technical similarities to motivate a
separation, and there are too many "weird things" plugin authors want
to do that needs features from both "classes".

Are we going to have LADSPA, and soon OAPI (or whatever it'll be) -
and in a year or two, another API, effectively merging LADSPA and
OAPI?

Why? What is it that LADSPA does that would be so complicated that an
instrument API must not support it?


//David Olofson - Programmer, Composer, Open Source Advocate

.- The Return of Audiality! --------------------------------.
| Free/Open Source Audio Engine for use in Games or Studio. |
| RT and off-line synth. Scripting. Sample accurate timing. |
`---------------------------> http://olofson.net/audiality -'
.- M A I A -------------------------------------------------.
| The Multimedia Application Integration Architecture |
`----------------------------> http://www.linuxdj.com/maia -'
--- http://olofson.net --- http://www.reologica.se ---
Steve Harris
2002-12-08 00:58:01 UTC
Permalink
Post by David Olofson
Are we going to have LADSPA, and soon OAPI (or whatever it'll be) -
and in a year or two, another API, effectively merging LADSPA and
OAPI?
Why? What is it that LADSPA does that would be so complicated that an
instrument API must not support it?
Nothing, OAPI or whatever will be a superset I imagine, but that implies
that LADSPA will still be simpler.

- Steve
David Olofson
2002-12-08 01:12:01 UTC
Permalink
Post by Steve Harris
Post by David Olofson
Are we going to have LADSPA, and soon OAPI (or whatever it'll be)
- and in a year or two, another API, effectively merging LADSPA
and OAPI?
Why? What is it that LADSPA does that would be so complicated
that an instrument API must not support it?
Nothing, OAPI or whatever will be a superset I imagine, but that
implies that LADSPA will still be simpler.
Well, I don't think that can be avoided... There aren't enough
"expendable" features in LADSPA that it would make up for the new
features, by far.


//David Olofson - Programmer, Composer, Open Source Advocate

.- The Return of Audiality! --------------------------------.
| Free/Open Source Audio Engine for use in Games or Studio. |
| RT and off-line synth. Scripting. Sample accurate timing. |
`---------------------------> http://olofson.net/audiality -'
.- M A I A -------------------------------------------------.
| The Multimedia Application Integration Architecture |
`----------------------------> http://www.linuxdj.com/maia -'
--- http://olofson.net --- http://www.reologica.se ---
David Olofson
2002-12-08 00:00:01 UTC
Permalink
Post by Tim Hockin
Post by Steve Harris
Post by Tim Hockin
This is what I am thinking: One of the flags on each control
is whether it is a CTRL_MASTER, or a CTRL_VOICE (or both).
This allows the plugin to define a ctrl labelled (for example)
VELOCITY and the host can connect it to any MIDI input, if it
wants.
I'm not sure that making one control both is meaningful. Per
voice and per instrument controls should probably be kept
seperate.
Whether they are defined together or seperately, there are
certainly controls that apply to each voice and to each
(instrument, channel (midi), plugin). For example volume.
What advantage do you see to defining them seperately? Certainly
the structures are analogous, if not identical.
So no one has offered me any suggestions on how we handle the clash
I think we're actually talking about *three* kinds of controls, while
everyone is still talking about their own set of two kinds of
controls. What I'm thinking is:

Master Controls:
Whether you have multipart/multichannel
instruments/plugins or not, these always
address the instrument/plugin instance
as a whole.

Ex: Master output volume of a
multipart sampler.

MIDI: System commands, some SysEx.

Channel/Part Controls:
These address a specific Channel/Part of
an instrument/plugin. If the instrument/
plugin has only one Channel/Part, these
messages can be considered equivalent to
Master Controls.

Ex: Channel dry send volume.

MIDI: All events with a channel field.

Voice/Note Controls
These address specific Notes or Voices,
to control them individually while they
are playing. Whether or not a Note or
Voice is a single oscillator or a network
of objects is irrelevant to the API -
this is an interface to an abstract object.

Ex: Note Aftertouch.

MIDI: NoteOn, NoteOff, Poly Pressure,
various Universal SysEx messages.
Post by Tim Hockin
<RE-ASK rephrased="true">
Let's assume a plugin provides a control named VOLUME which is both
VOICE and MASTER. Let's assume the current state of VOLUME is the
value 5.0. Let's assume the sequencer triggers a voice and sets
VOLUME for that voice to 8.0. The user then turns the master VOLUME
down to 0. What happens to the value of the voice's VOLUME.
a) goes to 0
b) ends up at 3.0
c) stays at 8.0
Maybe controls that are both MASTER and VOICE should be absolute
values for MASTER and scalars against the MASTER value per-voice?
</RE-ASK>
I don't think this has much to do with the API - and I don't see the
possibility of a clash. If you take any two controls, they may have
any relation whatsoever, *defined by the synth*, and the host should
assume nothing about that relation. Whether the two controls are both
VOICE, both MASTER or one of each, should not matter - the synth
still decide what their physical relation is.

So, in your example, you simply have two controls; MASTER::VOLUME and
VOICE::VOLUME. The synth may multiply them internally, or apply
MASTER::VOLUME in the output mixing loop, or whatever. Two controls,
two values - the two controls just happen to have the same hint:
VOLUME.
Post by Tim Hockin
Post by Steve Harris
Post by Tim Hockin
1) * Host calls plug->note_on(timestamp, note, velocity), and
gets a voice-id.
I dont think I like the firstclass-ness of note and velocity, a we've
I agree, mostly. I threw this in as a bit of a red-herring, to see
what people were thinking.
Post by Steve Harris
Post by Tim Hockin
* Host sends n VOICE_PARAM events to set up any params it wants
You could just send pitch and velocity this way?
Absolutely. HOWEVER, I have one design issue with it: Velocity is
not a continuous control. You can't adjust the velocity halfway
through a long note. You can adjust the pitch. You can adjust
portamento time. Velocity relates SPECIFICALLY to the attack and
release force of the musician.
You're thinking entirely in keyboardist terms here. :-) You're
forgetting that some instruments really don't have anything like
attack and release in real life, but rather "bow speed", "air speed"
and that kind of stuff.

I think I've suggested before that you may replace the velocity
"control" with a continous control like "bow speed" (what was that
generic name I came up with?) - which, if you think in keybordist
terms would correspond to key "position". Derive that at the point of
contact, and you have your velocity. You could then indeed think of
velocity as a continous control - it's just that your average
keyboard controller only changes it when you press or release a key.

So, given that controllers and instruments are different, it might be
a good idea to support *both* "velocity" and "position" (pick one for
each patch/sound!) - and both would be continous, of course.
Post by Tim Hockin
Unless we all agree that velocity ==
loudness, which will be tough, since I think _I_ disagree.
Well, I agree with you - velocity is *not* loudness. (There are other
controls for that kind of stuff; just look through the "standard"
MIDI CC list.)
Post by Tim Hockin
Could I get you to accept voice_on(velocity); and pass all the rest
as events?
Well, you won't get *me* to accept it that easy. ;-)


[...]
Post by Tim Hockin
Post by Steve Harris
a Channel, and Plugins may or may not be allowed to have multiple
Channels. As to the control protocol, if you want to do away with
the "channel" field, you could just allow synths to have more
than one event input port. One might think that that means more
event decoding
So I haven't come to this note in my TODO list yet. So we want to
support the idea of (MIDI word) multi-timbral plugins?
I think so, but I'm not 100% determined. (Audiality is still mostly a
monolith, and I'm not completely sure how to best split it up into
real plugins.)
Post by Tim Hockin
Given that
it is software, we can say 'just load a new instance for the new
channel'.
Yeah, but what if you want non-trivial interaction between the
parts/channels? (Drum kits or something...)

And either way, you may be able to optimize things better if you're
able to run the loops over parts/channels where appropriate, which is
something you cannot do if the parts/channels have to be separate
instances.
Post by Tim Hockin
It prevents anyone from doing an exact software mockup
of a bit of hardware, but I'm inclined not to care that much..
If that is prevented, I think VSTi and DXi will kill us before we
even get off the ground. Many users *like* plugins that act and look
like real hardware devices, and I don't think there's much point in
trying to reeducate them. Remember that lots of people actually come
from the hardware world.

This is not to say we should hold back inovation and proper software
design in favor of emulating hardware. I just believe that preventing
plugins from being multipart doesn't make much sense. Just reserve
one control/event port for the whole instrument/plugin, and then
allow the instrument/plugin to have as many other control/event ports
as it likes - just like audio in and out ports.
Post by Tim Hockin
Post by Steve Harris
ports. As of now, there is one event port for each Channel, and
one event port for each physical Voice. "Virtual Voices" (ie
Voices as
Are you using multiple event ports, then, or just having one port
per instrument-plugin and sending EV_VOICE_FOO and EV_MASTER_FOO
(modulo naming) events?
Well, there actually *are* things very similar to "instrument
plugins" in Audiality (although they don't process audio; only
events): Patch Plugins. Each Patch Plugin instance has an input event
port.

One argument of all events sent to Patch Plugins (or rather, to
Channels, which is where Patch Plugins are plugged in) is used as a
"tag" field, which polyphonic Patch Plugins let you use to reference
individual voices. The MIDI->Audiality mapper just throws the MIDI
pitch into this field, and -1 ("all") when there is no pitch.

As to master events, there are none of those at this point, since the
FX insert plugins - for some reason - don't use the event system for
control yet.
Post by Tim Hockin
Post by Steve Harris
bandwidth, and it works well for most keyboard controlled
instruments, but that's about it.
Other than Organs, which you've mentioned, what kinds of
instruments don't have some concept of velocity (whether they
ignore it or not..).
The organ doesn't have velocity at all - but the violin, and all wind
instruments I can think of, would be examples of instruments that
indeed *have* "some concept of velocity" - but one that is very far
from on/off with a velocity argument.
Post by Tim Hockin
Post by Steve Harris
As to the "gets a voice ID" thing, no, I think this is a bad
idea. The host should *pick* a voice ID from a previously
allocated pool of Virtual Voice IDs. That way, you eliminate the
host-plugin or plugin-plugin (this is where it starts getting
I'm still not clear on this. What plugin would trigger another
plugin?
Any plugin that processes events rather than (or in addition to)
audio. They're usually called "MIDI plugins".
Post by Tim Hockin
Do you envision that both the host and a plugin would be
controlling this plugin?
In some cases, that might make sense, yes.
Post by Tim Hockin
If so, how do you reconcile that they
will each have a pool of VVIDs - I suppose they can get VVIDs from
the host, but we're adding a fair bit of complexity now.
Where? They have to get the VVIDs from somewhere anyway. It's
probably not a great idea to just assume that plugins can accept any
integer number as a valid VVID, since that complicates/slows down
voice lookup.

In fact, Audiality *does* take whatever "voice tags" you give it -
but as a result, it has to search all voices owned by the channel to
find the voices. *heh* Could be optimized, but why not just use an
array of "Virtual Voice Info" structs; one for each valid VVID?

Anyway, dividing a range of integer numbers seems pretty trivial to
me - but it should still go into the host SDK lib, of course. :-)
Post by Tim Hockin
Post by Steve Harris
* send controls *before* you start a voice, if you like
you can do this already - any voice-control event with a timestamp
before (or equal to) the note_on timestamp can be assumed to be a
startup param.
So you want to start sorting *everything*, just so people can write
*slightly* simpler event sending code in some special cases?

I'd much rather require that events sent to a port are sent in
timestamp order, so the host can just merge sorted event lists. In
cases where you have multiple connections to the same event port, you
just use "shadow ports", so the host gets a number of ordered event
lists that it merges into one just before running the plugin that
owns the real input port.
Post by Tim Hockin
Post by Steve Harris
* implement smarter and tighter voice allocation
I don't see what you mean by this, or how it matters. I see voices
as being the sole property of the instrument. All the host knows
about them is that they have some (int) id that is unique
per-instrument.
Problem is that if that ID is the actual number of a real voice, you
can't steal a voice in the middle of a buffer. You have to wait until
the next buffer (or longer, if you're running in different theads, or
over wire), so the event sender has a chance hearing that "hey, that
voice now belongs to some other note!"

If the ID is decopled from physical voices, you (as a sender) can
just go on with your business, and you don't *have* to know whether
or not your IDs are still valid before sending events. The synth will
keep track of the *whole* voice allocation matter; not just part of
it.
Post by Tim Hockin
Post by Steve Harris
* use a timestamped event system
umm, I thik this is not solely dependant on vvids - I think
timestamps will work just fine as proposed.
Yes, but see above... This problem is real, and I still have it in
the lowest levels of Audiality, where it - fortunately - doesn't seem
to matter all that much. (Internal buffer size is restricted, so it
voice allocation granularity isn't all that bad.)

But consider what happens if you have voice related IDs on the
instrument API level, and try to play a fast series of short notes.
Since the plugin can't know for sure how long these notes will need
their voices, it simply has to give you a now voice for each note.
While, if you use VVIDs, you'd just pick a new VVID for each note,
and let the synth manage the allocation of actual voices. If the
notes are short enough, voices can be reused within a single buffer
or roundtrip cycle.
Post by Tim Hockin
It's a matter of
taste, unity, and presentation we're discussing.
No. It's a matter of whether or not to depend on roundtrips and
buffer splitting to avoid waste of resources.
Post by Tim Hockin
Post by Steve Harris
* pipe events over cables, somewhat like MIDI
Ahh, now HERE is something interesting. I'd always assumed the
plugin would return something, self-allocating a voice. This is
what you've called the round-trip problem, I guess. But it seems to
me that even if you were piping something over a wire, you have
some sort of local plugin handling it. THAT plugin allocates the
voice-id (arguing my model).
Why complicate things? With a protocol that doesn't entirely depend
on short round-trip latency for proper operation, you can just tunnel
the events directly to the receiver, possibly translating the
timestamps if you're not in sample sync. (Which you should be if
you're serious. Get proper hardware. :-)
Post by Tim Hockin
The question I was asking was: is
voice-id allocation synchronous (call plug->voice_on(timestamp) and
get a valid voice-id right now), or is it async (send a note-on
event and wait for it to come back).
Allocation is just a matter of preparing the plugin for receiving and
handling a specific number of VVIDs. You could just say that there
are always 65536 VVIDs, but that sounds both wasteful and restrictive
at the same time to me.

As to the implementation; if your plugin does what Audiality does
right now, the plugin may ignore the VVID allocation requests
entirely and just say "ok, whatever" - since anything that fits in
the VVID field is ok. (Audiality just throws the VVID into the voice
struct of the voice it actually allocates, and then looks for it when
it receives events talking about it.)
Post by Tim Hockin
This raises another question for me - the host sends events to the
plugins. Do the plugins send events back? It seems useful.
Definitely. In fact, applications that don't support plugins that
*send* events are frequently referred to as broken these days.
Post by Tim Hockin
The
host has to handle syncronization issues (one recv event-queue per
thread, or it has to be plugin -> host callback, or something), but
that's OK.
Well, if your threads are not in sample sync, you're in trouble...
But indeed, the host can deal with that as well, if for example,
you're crazy enough to want to use dual, unsynchronized or
"incompatible" sample rates. :-)
Post by Tim Hockin
Do timestamps matter to the host, except as
bookkeeping?
Timestamps matter a lot if you're going to do anything much at all
with the events. (Record them or convert them to MIDI, for example.)

That said, replies to requests (such as VVID allocation) should
"bypass" the normal events. (Not that it would make much of a
difference if you process them right after the plugin has returned
from process() or at the end of the engine cycle. It may matter if
you're piping events back and forth over wire, though.)
Post by Tim Hockin
Post by Steve Harris
and all hosts... Also note that event size (which has to be
fixed) increases if you must squeeze in multiple parameters in
some events.
Why does it have to be fixed-size? It doesn't strictly HAVE to be.
It doesn't *have* to be - but if you look at the MAIA or Audiality
event systems, you'll realize why I strongly prefer fixed size
events. It's all about performance and complexity.

Note that you still *can* pass around data blocks that won't fit in
an event. Just pass data blocks by reference. (I've designed
protocols for that for MAIA, if you're interested in the details.)
Post by Tim Hockin
On one hand I HATE when an API says 'you have three available
params' foo->param1, foo->param2, foo->param3. If you need more,
too bad.
What do you need them for?

I have multiple parameters in Audiality events, but it seems like
I'll need *less* parameters once I've cleaned out some silly
MIDIisms. (Such as having pitch and velocity in "start" events.)
Post by Tim Hockin
On the other hand, there are performance advantages to
having events pre-allocated, and thereby, fixed sized, or at least
bounded.
Exactly. Especially considering that you'll send *multiple* events
for every "voice_start", and that there may be a lot of control
changes while playing real music, performance matters. We can
probably forget about anything like traditional dynamic memory
allocation.

You *could* a few different event sizes - but then you'd have to
specify size when allocating them, and the deallocation code would
have to check some hidden field or something, so see how big every
event is, in order to put it back in the right pool.


(This was actually written by Steve Harris, AFAIK. :-)
Post by Tim Hockin
Post by Steve Harris
The intention is that these things would (on the whole) be sound
generators, right? To me plugin implies inline processing.
This API is not purely instrumental. It can certainly be for
effects and sinks, too. That said, we're spending a lot of time on
the instrumental part because it's the new ground wrt LADSPA.
Yes. Let's just don't forget that although we may need to design
something new from scratch to get a nice and clean API, we do *not*
automatically have to throw away the feature set of LADSPA.

We should basically have LADSPA functionality, but without LADSPA
style control ports, and with event ports + "abstract controls"
instead. Some of the "negotiation" calls may be replaced by events.
(MAIA philosophy again.)


//David Olofson - Programmer, Composer, Open Source Advocate

.- The Return of Audiality! --------------------------------.
| Free/Open Source Audio Engine for use in Games or Studio. |
| RT and off-line synth. Scripting. Sample accurate timing. |
`---------------------------> http://olofson.net/audiality -'
.- M A I A -------------------------------------------------.
| The Multimedia Application Integration Architecture |
`----------------------------> http://www.linuxdj.com/maia -'
--- http://olofson.net --- http://www.reologica.se ---
Tim Hockin
2002-12-07 22:19:00 UTC
Permalink
Post by Steve Harris
Not defining them seperatly, but making a distinction, I dont think one
control should be applied per voice and per instrument.
So one should not be able to manipulate a per-voice volume and a master
volume?
Post by Steve Harris
Post by Tim Hockin
So no one has offered me any suggestions on how we handle the clash between
Ban it :)
I'm not sure we can reasonably do that. We can say that no one control
applies to both master and voice, but then we end up always (or rather,
often) defining two volume controls, two pitch controls, two filter controls
etc and making them slightly different and behave differently in each plug.
Are we better to define the behavior, or at least offer a suggestion?
Post by Steve Harris
No, velocity != loudless.
ok, we agree on that.
Post by Steve Harris
However, even in midi there can be more than one velocity value per note,
there is seperate release volcity IIRC. I expect the only reason attack
velocity is package up with the note start in MIDI is because otherwise
they couldn't guarantee it would arrive at the same time, and it saved a
few bytes.
MIDI has 2 velocities - note_on and note_off. I see velocity as specific to
those two events. Isn't a new velocity event really a new strike or attack
of the instrument, or, in other words a new voice (even if it cancels the
previous voice)? That said, I hate that Velocity is special. I could be
convinced. I could also stand and argue. Not sure I know which I prefer...
Post by Steve Harris
Post by Tim Hockin
we can say 'just load a new instance for the new channel'. It prevents
There are probably other reasons why you /might/ want this, but I cant
think of anyoffhand. I suspect one is OK and it makes things nice and
simple.
I agree, and simplicity is better, I think. Still needs more noodling. If
it is obvious and easy, then I don't really mind it.
Post by Steve Harris
Analogue synths generally dont, some have channel pressure, but that is
slightly different and variable through the duration.
channel pressure would be a separate control, just as it is in MIDI.
Post by Steve Harris
Not really the host can just allocate them from a pool of 32bit ints (for
example) and pass them in by the new_voice call.
but if a plugin controls another plugin (as suggested by David) the
'pseudo-master' needs to either be a full-fledged host, or needs to get a
pool of VVIDs from the host via a host->get_vvid() callback. It's not TOO
bad, I guess.. But I don't see it as being a big-win on any front as
compared to a synchronous vid = plug->voice_on() mechanism.
Post by Steve Harris
Post by Tim Hockin
This raises another question for me - the host sends events to the plugins.
Do the plugins send events back? It seems useful. The host has to handle
syncronization issues (one recv event-queue per thread, or it has to be
plugin -> host callback, or something), but that's OK. Do timestamps matter
to the host, except as bookkeeping?
I'm not sure what the instrument will send back to the host that needs to
be timestamped.
Me neither, but it is becoming morer apparent that a plugin DOES need to
send events back. Arguments?
Post by Steve Harris
Well, we were talking about floats, strings or opaque data blocks before,
so presumanbly the event would be either a float, a pointer to a string or
a pointer to a data block (+ a size).
Assuming that control changes are the only types of events that flow.

some events, off the top of my head..
VOICE_ON (plugin to host) ?
VOICE_OFF (plugin to host and host to plugin)
VOICE_CTRL (host to plugin)
MASTER_CTRL (host to plugin)
SILENT (plugin to host - sent when reverb tails or whatnot have died..)
Post by Steve Harris
If you need to send it over a wire you will have to serialise the strings
and blocks in variabley size chunks, but thats unavoiadable.
yes, and let's not make this a wire protocol :) Seperate project.
Post by Steve Harris
Its not purely aimed at instruments, but its heavily focused that way, it
probably wouldn't make sense to use this API for something LADSPA can do
well (eg. audio+control -> audio+control). If you need timestamps or
polyphony then this would be the way to go, but realisticly that means
intruments and a few other odd cases.
Well, assuming a LADSPA wrapper in this API, a host can transparently use
LADSPA plugins in a multi-channel way. Stereo reverb can be built out of
two LADSPA mono reverbs. Just an idea.
David Olofson
2002-12-08 00:35:01 UTC
Permalink
On Saturday 07 December 2002 22.13, Tim Hockin wrote:
[...]
Post by Tim Hockin
Post by Steve Harris
Post by Tim Hockin
So no one has offered me any suggestions on how we handle the
Ban it :)
I'm not sure we can reasonably do that. We can say that no one
control applies to both master and voice, but then we end up always
(or rather, often) defining two volume controls, two pitch
controls, two filter controls etc and making them slightly
different and behave differently in each plug. Are we better to
define the behavior, or at least offer a suggestion?
2D "address space":
Dimension 1: Channel
Dimension 2: Control

One channel would be reserved for Master Events, and the others would
(obviously) deal with Channel Events. For each Channel, you can ask
for a list of Controls, and these should be named and hinted
appropriately.


[..]
Post by Tim Hockin
MIDI has 2 velocities - note_on and note_off. I see velocity as
specific to those two events. Isn't a new velocity event really a
new strike or attack of the instrument, or, in other words a new
voice (even if it cancels the previous voice)?
Nope. An a violin, you have no "velocity" event at all (unless you
want to use it for the subtle sound of the bow impacting or leaving
the strings), but rather a "bow speed" event - which is continous.
Post by Tim Hockin
That said, I hate
that Velocity is special. I could be convinced. I could also
stand and argue. Not sure I know which I prefer...
Well, keep arguing, and I'll try to figure out more reasons why
velocity is not special, nor universal. ;-)
Post by Tim Hockin
Post by Steve Harris
Post by Tim Hockin
we can say 'just load a new instance for the new channel'. It prevents
There are probably other reasons why you /might/ want this, but I
cant think of anyoffhand. I suspect one is OK and it makes things
nice and simple.
I agree, and simplicity is better, I think. Still needs more
noodling. If it is obvious and easy, then I don't really mind it.
If you want to keep your plugin simple, just say you support min 1
and max 1 channels, and that's it - you can hack away just as if the
API supported only one channel.

As to the host, a multichannel plugin would correspond to an array of
single channel plugins with *one* exception: In the case of single
channel plugins, you'd have to copy all Master Events to each
instance, while in the multichannel case, you just send them to that
single Master Event port.
Post by Tim Hockin
Post by Steve Harris
Analogue synths generally dont, some have channel pressure, but
that is slightly different and variable through the duration.
channel pressure would be a separate control, just as it is in
MIDI.
What if you need *only* pressure, and don't care about attack or
release velocity? Why not just make velocity optional as well? :-)
Post by Tim Hockin
Post by Steve Harris
Not really the host can just allocate them from a pool of 32bit
ints (for example) and pass them in by the new_voice call.
but if a plugin controls another plugin (as suggested by David) the
'pseudo-master' needs to either be a full-fledged host, or needs to
get a pool of VVIDs from the host via a host->get_vvid() callback.
I think the plugin should just ask the host for the number of VVIDs
it wants for each event output port, and let the host deal with it,
since the host knows where those outputs are connected.
Post by Tim Hockin
It's not TOO bad, I guess.. But I don't see it as being a big-win
on any front as compared to a synchronous vid = plug->voice_on()
mechanism.
(See previous posts.)
Post by Tim Hockin
Post by Steve Harris
Post by Tim Hockin
This raises another question for me - the host sends events to
the plugins. Do the plugins send events back? It seems useful.
The host has to handle syncronization issues (one recv
event-queue per thread, or it has to be plugin -> host
callback, or something), but that's OK. Do timestamps matter
to the host, except as bookkeeping?
I'm not sure what the instrument will send back to the host that
needs to be timestamped.
Me neither, but it is becoming morer apparent that a plugin DOES
need to send events back. Arguments?
A plugin can't really send events *back*, as it can't know more than
which port the events arrived on, unless there is a "sender" field in
all or some events. I think that in most cases, events that need
replies will come from the host - so you can just send the replies to
a "fixed" host event port. Either way, this stuff probably won't need
timestamps, because it won't even happen when the plugin is in the
processing net.

However, it might be handy for the host to be able to ask for the
values of specific controllers. It may or may not be useful to do
that multiple times per buffer - but if it is, you'll definitely want
the timestamps of the "reply events" to match those of your request
events, or things will get hairy...

Either way, this is really easy: Whenever you reply to an event, just
copy the timestamp field from the request event.
Post by Tim Hockin
Post by Steve Harris
Well, we were talking about floats, strings or opaque data blocks
before, so presumanbly the event would be either a float, a
pointer to a string or a pointer to a data block (+ a size).
Assuming that control changes are the only types of events that flow.
some events, off the top of my head..
VOICE_ON (plugin to host) ?
VOICE_OFF (plugin to host and host to plugin)
VOICE_CTRL (host to plugin)
MASTER_CTRL (host to plugin)
Sounds ok.
Post by Tim Hockin
SILENT (plugin to host - sent when reverb tails or whatnot have died..)
Great idea. Sample accurate end-of-tail notification. :-) (In
Audiality, I do that by fiddling with the plugin state, which is
rather ugly and not sample accurate.)
Post by Tim Hockin
Post by Steve Harris
If you need to send it over a wire you will have to serialise the
strings and blocks in variabley size chunks, but thats
unavoiadable.
yes, and let's not make this a wire protocol :) Seperate project.
Let's not *prevent* it from ever being a wire protocol, unless
strictly nescessary. :-)


//David Olofson - Programmer, Composer, Open Source Advocate

.- The Return of Audiality! --------------------------------.
| Free/Open Source Audio Engine for use in Games or Studio. |
| RT and off-line synth. Scripting. Sample accurate timing. |
`---------------------------> http://olofson.net/audiality -'
.- M A I A -------------------------------------------------.
| The Multimedia Application Integration Architecture |
`----------------------------> http://www.linuxdj.com/maia -'
--- http://olofson.net --- http://www.reologica.se ---
Steve Harris
2002-12-08 01:06:01 UTC
Permalink
Post by David Olofson
Post by Tim Hockin
SILENT (plugin to host - sent when reverb tails or whatnot have died..)
Great idea. Sample accurate end-of-tail notification. :-) (In
Audiality, I do that by fiddling with the plugin state, which is
rather ugly and not sample accurate.)
As long as its not required, you can't allways tell (eg. gong again).

- Steve
David Olofson
2002-12-08 01:20:01 UTC
Permalink
Post by Steve Harris
Post by David Olofson
Post by Tim Hockin
SILENT (plugin to host - sent when reverb tails or whatnot have died..)
Great idea. Sample accurate end-of-tail notification. :-) (In
Audiality, I do that by fiddling with the plugin state, which is
rather ugly and not sample accurate.)
As long as its not required, you can't allways tell (eg. gong
again).
Of course. There are three cases, I think:

1) Plugins that can't/won't tell when the tail has ended.
2) Plugins that don't have any tail at all.
3) Plugins that will give you SILENT events.

I think you'd have to tell the host where you belong, so the host
knows why it's not getting any SILENT events... (There's no point in
sending them for case 2), IMHO - but it's still ok to shut the plugin
off, or whatever, without analyzing the output or anything.)

Plain delay effects are one example of plugins that definitely
*should* be in 3) - since you can't be sure by just looking at the
output. It may be dead silent for a good while, and still have data
in the internal buffers.


Now, what's the definition of "silent"? :-)

IMHO, even when the plugin does the work, the *host* should decide
what the maximum signal level for "silent" is, since this differs
depending on the target data format. With 24 bits, you'll have to
wait quite a bit longer than with 16 bits, before you can "safely"
cut the tail...


//David Olofson - Programmer, Composer, Open Source Advocate

.- The Return of Audiality! --------------------------------.
| Free/Open Source Audio Engine for use in Games or Studio. |
| RT and off-line synth. Scripting. Sample accurate timing. |
`---------------------------> http://olofson.net/audiality -'
.- M A I A -------------------------------------------------.
| The Multimedia Application Integration Architecture |
`----------------------------> http://www.linuxdj.com/maia -'
--- http://olofson.net --- http://www.reologica.se ---
Nathaniel Virgo
2002-12-08 01:29:01 UTC
Permalink
Post by David Olofson
[..]
Post by Tim Hockin
MIDI has 2 velocities - note_on and note_off. I see velocity as
specific to those two events. Isn't a new velocity event really a
new strike or attack of the instrument, or, in other words a new
voice (even if it cancels the previous voice)?
Nope. An a violin, you have no "velocity" event at all (unless you
want to use it for the subtle sound of the bow impacting or leaving
the strings), but rather a "bow speed" event - which is continous.
Post by Tim Hockin
That said, I hate
that Velocity is special. I could be convinced. I could also
stand and argue. Not sure I know which I prefer...
Well, keep arguing, and I'll try to figure out more reasons why
velocity is not special, nor universal. ;-)
I think velocity is special because it is often needed by the synthesis
algorithm only at the beginning of the note. For instance, if you have a
physical model of a bell then the velocity will determine the shape and
magnitude of the impulse function you use to represent the striker. Changing
it midway through the note, after the bell has been hit, is meaningless.

That said, it would be cool if you could support multiple velocity-like
parameters, so that it would be possible to control the position of the
striker and the strength of the strike independently for each note. Multiple
poly-aftertouch-like controllers would also be cool, but I think they are a
different thing.
David Olofson
2002-12-08 01:51:01 UTC
Permalink
[...velocity or not...]
Post by Nathaniel Virgo
Post by David Olofson
Well, keep arguing, and I'll try to figure out more reasons why
velocity is not special, nor universal. ;-)
I think velocity is special because it is often needed by the
synthesis algorithm only at the beginning of the note. For
instance, if you have a physical model of a bell then the velocity
will determine the shape and magnitude of the impulse function you
use to represent the striker. Changing it midway through the note,
after the bell has been hit, is meaningless.
Of course. (Waving the club around in the air in front of the gong
will probably affect the sound slightly, but you may not be
interested in emulating that. :-)

However, how does velocity making sense for *some* instruments make
velocity special? I bet there are *at least* as many instruments
where velocity does *not* make sense, but "pressure/speed" does. So,
we might as well decide that "pressure/speed" is the normal (or
"special") way to control instruments, and velocity is the special
case, right?

I think both are special cases. In some cases, you don't need either
of them. In most cases, you'll need one of them, and sometimes, you
may even want to use both.
Post by Nathaniel Virgo
That said, it would be cool if you could support multiple
velocity-like parameters, so that it would be possible to control
the position of the striker and the strength of the strike
independently for each note.
Yes indeed. I have that kind of controls in Audiality (used only for
positional 2D audio so far) - but those are actually continous
controllers.
Post by Nathaniel Virgo
Multiple poly-aftertouch-like
controllers would also be cool, but I think they are a different
thing.
I'm not so sure... What happens if you hit a snare drum, and then let
the stick rest on the surface? ;-)

What I'm saying is that if you thing about what you're actually
trying to simulate, it becomes clear that continous controllers will
fit practically any application.

Want positional info on drums? Think of X and Y as the "aim point",
and just make sure you're aiming at the right spot before you kick
the velocity control to the desired value - which, with this
instrument, instantly triggers the sound. Setting velocity back to 0
would correspond to NoteOff - although at least 90% of all drum
patches will completely ignore that anyway.

Want to play the trumpet? Well, just set up the right pitch first,
and then "stroke" the breath control as required. Whenever breath is
non-zero, you have a note.


The point I think you're missing is that a "control change" event is
*exactly* the same thing as a "voice start" event on the bits and
bytes level. Both carry some data. Both will have the plugin run some
code at the specified time. It's just a matter of how the plugin
interprets the data - so you may decide to have the breath control
trigger your synths internal voice on/off events by detecting
transitions from and to 0.

It's nothing like LADSPA, where you have to *poll* control ports to
do this kind of stuff.


//David Olofson - Programmer, Composer, Open Source Advocate

.- The Return of Audiality! --------------------------------.
| Free/Open Source Audio Engine for use in Games or Studio. |
| RT and off-line synth. Scripting. Sample accurate timing. |
`---------------------------> http://olofson.net/audiality -'
.- M A I A -------------------------------------------------.
| The Multimedia Application Integration Architecture |
`----------------------------> http://www.linuxdj.com/maia -'
--- http://olofson.net --- http://www.reologica.se ---
Steve Harris
2002-12-08 11:41:00 UTC
Permalink
Post by Nathaniel Virgo
I think velocity is special because it is often needed by the synthesis
algorithm only at the beginning of the note. For instance, if you have a
physical model of a bell then the velocity will determine the shape and
magnitude of the impulse function you use to represent the striker. Changing
it midway through the note, after the bell has been hit, is meaningless.
A Bell is a good example of something where you want multiple velocity
events, you could call voice_on() or whatever to initialise the physical
model instance, then you send simultaneous velocity and position events to
indicate that the bell has been struck.

Its not true for a impulse derived bell but a weaveguide model of a bell
struck twice has a different sound to a pair of identical ones each struck
once at the respective times.

- Steve
Frank van de Pol
2002-12-08 18:37:00 UTC
Permalink
Post by Steve Harris
Post by Nathaniel Virgo
I think velocity is special because it is often needed by the synthesis
algorithm only at the beginning of the note. For instance, if you have a
physical model of a bell then the velocity will determine the shape and
magnitude of the impulse function you use to represent the striker. Changing
it midway through the note, after the bell has been hit, is meaningless.
A Bell is a good example of something where you want multiple velocity
events, you could call voice_on() or whatever to initialise the physical
model instance, then you send simultaneous velocity and position events to
indicate that the bell has been struck.
Its not true for a impulse derived bell but a weaveguide model of a bell
struck twice has a different sound to a pair of identical ones each struck
once at the respective times.
Nice examlple. To me it clearly illustrates that velocity (ie. force of note
on/off) is indeed a property of the note event. The handling of velocity for
said bell depends on the implementation of the synth; a physical modelling
synth could when retriggering with the same pitch adjust the sound of the
bell. Simple sample playback synths would most likely not have this level op
sophistication.

The violin example using the bow speed property is something different. In
fact, the bow speed is indeed a sort of continious controller, while the
strings could be hit by the bow with different velocity. How could one
otherwise achive a pizzicato effect? (I might use the wrong phrasing, but I
mean hitting like slapping a bass guitar the violin's strings using the
bow.)

Though not in the scope for the instrument plugin discussion, I'd like to
see compatibility with MIDI or other real-life protocols be considered. In
the end our sequencer applications would need to transparently support both
flavours.


Frank.
--
+---- --- -- - - - -
| Frank van de Pol -o) A-L-S-A
| ***@home.nl /\\ Sounds good!
| http://www.alsa-project.org _\_v
| Linux - Why use Windows if we have doors available?
Mark Knecht
2002-12-08 19:17:00 UTC
Permalink
Frank,
I agree with your comments. I'd also like to see the developers at
least understand and consider these sorts of protocols also. In the end
something good will come out of it.

Actually, the use of MIDI velocity information is a strong selling
point of GigaStudio. In GigaStudio you can map up to 16 different
samples for a given MIDI note. The most common usage in the better
libraries is to use 4 velocity levels to trigger different samples to be
played back. Inside of that velocity range the actual velocity number
serves as a volume scaling mechanism, but the overall MIDI velocity
allows the piano to be played with different samples.

After the sample is chosen, and scaled as per the specific velocity
info, it is then scaled again with MIDI volume.

There may be other ways to do stuff like this, but this is pretty
standard in GSt and other Win/Mac based sample players.

Cheers,
Mark
Post by Frank van de Pol
Nice examlple. To me it clearly illustrates that velocity (ie. force of note
on/off) is indeed a property of the note event. The handling of velocity for
said bell depends on the implementation of the synth; a physical modelling
synth could when retriggering with the same pitch adjust the sound of the
bell. Simple sample playback synths would most likely not have this level op
sophistication.
The violin example using the bow speed property is something different. In
fact, the bow speed is indeed a sort of continious controller, while the
strings could be hit by the bow with different velocity. How could one
otherwise achive a pizzicato effect? (I might use the wrong phrasing, but I
mean hitting like slapping a bass guitar the violin's strings using the
bow.)
Though not in the scope for the instrument plugin discussion, I'd like to
see compatibility with MIDI or other real-life protocols be considered. In
the end our sequencer applications would need to transparently support both
flavours.
Frank.
Tim Hockin
2002-12-08 01:08:01 UTC
Permalink
Post by Steve Harris
Post by David Olofson
Why? What is it that LADSPA does that would be so complicated that an
instrument API must not support it?
Nothing, OAPI or whatever will be a superset I imagine, but that implies
that LADSPA will still be simpler.
yeah, and I want to make it easy for an OAPI (I HATE that name - open to
ideas..) host to use LADSPA plugins. That is very important.

Tim
David Olofson
2002-12-08 01:25:01 UTC
Permalink
Post by Tim Hockin
Post by Steve Harris
Post by David Olofson
Why? What is it that LADSPA does that would be so complicated
that an instrument API must not support it?
Nothing, OAPI or whatever will be a superset I imagine, but that
implies that LADSPA will still be simpler.
yeah, and I want to make it easy for an OAPI (I HATE that name -
open to ideas..) host to use LADSPA plugins. That is very
important.
Of course. And there should be nothing preventing that. Unless I'm
missing something, the only really significant difference is that
LADSPA plugins don't have event ports, while "New API" plugins will.
You could probably even build a simple event->control semi-wrapper
into the host, to make LADSPA plugins fit in with minimal impact on
the rest of the host code. LADSPA plugins will then act just like
"New API" plugins that quantize timestamps to buffer boundaries. Or
you can have the semi-wrapper do full or quantized buffer splitting...


//David Olofson - Programmer, Composer, Open Source Advocate

.- The Return of Audiality! --------------------------------.
| Free/Open Source Audio Engine for use in Games or Studio. |
| RT and off-line synth. Scripting. Sample accurate timing. |
`---------------------------> http://olofson.net/audiality -'
.- M A I A -------------------------------------------------.
| The Multimedia Application Integration Architecture |
`----------------------------> http://www.linuxdj.com/maia -'
--- http://olofson.net --- http://www.reologica.se ---
Joern Nettingsmeier
2002-12-08 11:02:00 UTC
Permalink
Post by Tim Hockin
Post by Steve Harris
Post by David Olofson
Why? What is it that LADSPA does that would be so complicated that an
instrument API must not support it?
Nothing, OAPI or whatever will be a superset I imagine, but that implies
that LADSPA will still be simpler.
yeah, and I want to make it easy for an OAPI (I HATE that name - open to
ideas..) host to use LADSPA plugins. That is very important.
Linux Audio Developers' More Sophisticated Plugin API (=LADMSPA) ? :-D
--
Jörn Nettingsmeier
Kurfürstenstr 49, 45138 Essen, Germany
http://spunk.dnsalias.org (my server)
http://www.linuxdj.com/audio/lad/ (Linux Audio Developers)
Steve Harris
2002-12-08 11:51:00 UTC
Permalink
Post by Tim Hockin
Post by Steve Harris
Post by David Olofson
Why? What is it that LADSPA does that would be so complicated that an
instrument API must not support it?
Nothing, OAPI or whatever will be a superset I imagine, but that implies
that LADSPA will still be simpler.
yeah, and I want to make it easy for an OAPI (I HATE that name - open to
ideas..) host to use LADSPA plugins. That is very important.
Me too, not a very good name.

Your more immediate problem is making it easy for a LADSPA host to host
[insert name here] plugins. There are a lot of LADSPA hosts, and zero
[...] hosts.

- Steve
John Lazzaro
2002-12-08 02:31:01 UTC
Permalink
Post by David Olofson
The point I think you're missing is that a "control change" event is
*exactly* the same thing as a "voice start" event on the bits and
bytes level.
Lossy MIDI filters will prune away two MIDI Control Change commands
in a row for the same controller value with the same data value,
apart from controller numbers (like All Notes Off) whose semantics
have meaning in this case. And the assumption underlying the behavior
of these filters are present in subtle ways in other MIDI gear and
usages too. For example, a programming language that presents an
array with 128 members, holding the last-received (or default) value
of each MIDI controller, presents an API that implicitly does this
filtering, no matter how frequently the program samples the array.

-------------------------------------------------------------------------
John Lazzaro -- Research Specialist -- CS Division -- EECS -- UC Berkeley
lazzaro [at] cs [dot] berkeley [dot] edu www.cs.berkeley.edu/~lazzaro
-------------------------------------------------------------------------
David Olofson
2002-12-08 02:50:01 UTC
Permalink
Post by John Lazzaro
Post by David Olofson
The point I think you're missing is that a "control change" event
is *exactly* the same thing as a "voice start" event on the bits
and bytes level.
Lossy MIDI filters will prune away two MIDI Control Change commands
in a row for the same controller value with the same data value,
apart from controller numbers (like All Notes Off) whose semantics
have meaning in this case. And the assumption underlying the
behavior of these filters are present in subtle ways in other MIDI
gear and usages too. For example, a programming language that
presents an array with 128 members, holding the last-received (or
default) value of each MIDI controller, presents an API that
implicitly does this filtering, no matter how frequently the
program samples the array.
Yes - but we're not talking about MIDI here. We *may* require that
events are never lost, and even that it's not legal to send two
identical events in line to an event port, unless you really mean
something else than to set a controller to the same value twice.

Anyway, the examples I gave in one of the other posts don't rely on
this at all. They still go by the controller *values* - for example
changes from and to 0.

The point I was trying to make was that with an event system (or a
callback system, for that matter) you don't have to constantly poll a
number of control ports to see these transitions. You only have to
check when you actually get a "change" event, since that's the only
way a control can change it's value.


//David Olofson - Programmer, Composer, Open Source Advocate

.- The Return of Audiality! --------------------------------.
| Free/Open Source Audio Engine for use in Games or Studio. |
| RT and off-line synth. Scripting. Sample accurate timing. |
`---------------------------> http://olofson.net/audiality -'
.- M A I A -------------------------------------------------.
| The Multimedia Application Integration Architecture |
`----------------------------> http://www.linuxdj.com/maia -'
--- http://olofson.net --- http://www.reologica.se ---
John Lazzaro
2002-12-08 03:33:01 UTC
Permalink
Post by David Olofson
Yes - but we're not talking about MIDI here. We *may* require that
events are never lost, and even that it's not legal to send two
identical events in line to an event port, unless you really mean
something else than to set a controller to the same value twice.
Think ahead about how these sorts of requirements will be enforced:
will they be a "law of nature" (code checks to see if an app broke
the law, and takes action, like nature takes action when you try
to change the current flowing through an inductor :-), or will
it be unchecked by code? If its the latter, you can get into
this mode where everyone has extra checking and work-around code,
to handle impolite API users who aren't obeying the requirements.

-------------------------------------------------------------------------
John Lazzaro -- Research Specialist -- CS Division -- EECS -- UC Berkeley
lazzaro [at] cs [dot] berkeley [dot] edu www.cs.berkeley.edu/~lazzaro
-------------------------------------------------------------------------
David Olofson
2002-12-08 04:27:00 UTC
Permalink
Post by John Lazzaro
Post by David Olofson
Yes - but we're not talking about MIDI here. We *may* require
that events are never lost, and even that it's not legal to send
two identical events in line to an event port, unless you really
mean something else than to set a controller to the same value
twice.
will they be a "law of nature" (code checks to see if an app broke
the law, and takes action, like nature takes action when you try
to change the current flowing through an inductor :-), or will
it be unchecked by code? If its the latter, you can get into
this mode where everyone has extra checking and work-around code,
to handle impolite API users who aren't obeying the requirements.
Well, I wasn't seriously suggesting that we have that kind of silly
rules. Still, they would not be totally unreasonable, given that you
can never have two event sources manipulate the same control in a
useful way; it would just result in the value jumping around. So, who
would send the same value twice?

If someone frequently tries to set a control to the value it already
is set to, that's bad manners.

If a plugin actually starts recalculating filter coefficients and
stuff as a result of receiving such a control event, that could alse
be considered bad manners - BUT, avoiding it would mean an extra
conditional. Why have a conditional to avoid doing something
relatively harmless, that can only happen as a result of someone else
being nasty?

That said, if hosts want to filter event streams from poorly written
event generators, fine - but IMHO, it's a waste of resources,
shouldn't be needed at all, and can only prevents something that's
relatively harmless.


Anyway, there might be cases where we can use "trigger" controls,
where the value is an argument, and the plugin is supposed to
actually *do* something as a response, every time, regardless of the
value of the argument.

If we want those, they should be explicitly marked as special - just
like non-RT events. Or just like the few MIDI CCs that filters may
not remove. The only difference is that we don't have a fixed set of
controls with implicit rules, but use use some hints and flags
instead.


//David Olofson - Programmer, Composer, Open Source Advocate

.- The Return of Audiality! --------------------------------.
| Free/Open Source Audio Engine for use in Games or Studio. |
| RT and off-line synth. Scripting. Sample accurate timing. |
`---------------------------> http://olofson.net/audiality -'
.- M A I A -------------------------------------------------.
| The Multimedia Application Integration Architecture |
`----------------------------> http://www.linuxdj.com/maia -'
--- http://olofson.net --- http://www.reologica.se ---
Tim Hockin
2002-12-08 06:06:02 UTC
Permalink
Post by David Olofson
I think we're actually talking about *three* kinds of controls, while
everyone is still talking about their own set of two kinds of
Voice/Note Controls
So if we support multi-timbral (multi-part, multi-channel - too many words)
instruments, then yes, this is true.

Aside: I propose the following for naming stuff:

* Plugin: a chunk of code, loaded or not, that implements this API (a .so file
or a running instance).
* Instrument: an instance of a plugin that supports the instrument API bits
(analogous to a chunk of hardware from Roland).
* Channel: a 'part', 'channel' or 'instrument' from MIDI. Instruments have
1 or more Channels.
* Control: a knob, button, slider, or virtual control withing a Channel or
Instrument. Controls can be master (e.g. master volume), per-Channel (e.g.
channel pressure) or per-Voice (e.g. aftertouch).
* Preset: a stored or loaded set of values for all Controls in a Channel
* Voice: a playing sound within a Channel. Channels may have multiple voices.
* Port: an input/output connection on a Plugin. Audio data goes over ports.
Post by David Olofson
I don't think this has much to do with the API - and I don't see the
possibility of a clash. If you take any two controls, they may have
any relation whatsoever, *defined by the synth*, and the host should
I agree with that IF we say you have to have a separate control struct for
per-voice and per-part controls. If we define a control and say it has a
flag which identifies it as MASTER, PART, VOICE, then we do need to suggest
a relationship. I like defining a control once, but it's becoming more
obvious to NOT do that. They range of valid values may be wildly different.

Do we want to provide a suggested range of values for MIDI compatible
controls. For example Controls that are labelled CONTROL_AFTERTOUCH should
expect values between 0 and 1.0, and the host can map MIDI onto that? Or 0
and 127, like MIDI? or 0 and MAX_INT and let the synth blow up when you
pass a VELOCITY of 10000, and it expects 127?
Post by David Olofson
You're thinking entirely in keyboardist terms here. :-) You're
forgetting that some instruments really don't have anything like
attack and release in real life, but rather "bow speed", "air speed"
and that kind of stuff.
umm, speed == velocity, no? But I get your point, and I concede this point.
Velocity is an event.
Post by David Olofson
I think so, but I'm not 100% determined. (Audiality is still mostly a
monolith, and I'm not completely sure how to best split it up into
real plugins.)
plugins from being multipart doesn't make much sense. Just reserve
one control/event port for the whole instrument/plugin, and then
allow the instrument/plugin to have as many other control/event ports
as it likes - just like audio in and out ports.
I had been assuming a single event-queue per instance. Two questions: why
would a plugin (assuming not multi-timbral for now) have more than one
event-queue? Assuming we do support multi-timbral synths, is there an
advantage to having events per-channel event-queues? Do you envision each
Channel getting the ->run() method seperately, or does ->run() loop? If it
is looping anyway, is there an advantage to multiple event-queues?
Post by David Olofson
Post by Tim Hockin
Post by David Olofson
ports. As of now, there is one event port for each Channel, and
one event port for each physical Voice. "Virtual Voices" (ie
Voices as
I know you've done some exploring - do we really want one event-queue per
voice + one for each channel + one for master? or is one per instance
simpler?
Post by David Olofson
Post by Tim Hockin
Post by David Olofson
As to the "gets a voice ID" thing, no, I think this is a bad
idea. The host should *pick* a voice ID from a previously
allocated pool of Virtual Voice IDs. That way, you eliminate the
host-plugin or plugin-plugin (this is where it starts getting
I'm still not clear on this. What plugin would trigger another
plugin?
Any plugin that processes events rather than (or in addition to)
audio. They're usually called "MIDI plugins".
I can see controller plugins (It's another extension to this API, perhaps)
which deal with sending events to other plugins. And I guess an arppegiator
plugin could be a controller..
Post by David Olofson
Post by Tim Hockin
If so, how do you reconcile that they
will each have a pool of VVIDs - I suppose they can get VVIDs from
the host, but we're adding a fair bit of complexity now.
Where? They have to get the VVIDs from somewhere anyway. It's
probably not a great idea to just assume that plugins can accept any
integer number as a valid VVID, since that complicates/slows down
voice lookup.
you don't send just ANY integer, you get the integer to id a voice FROM the
plug. But if VOICE_ON is an event, it gets much uglier. Do let me see if I
get this right:

Host allocates the VVID pool (perhaps a bitmask or some other way of
indicating usage)
Host wants to send a VOICE_ON:
allocates an unused VVID
sends VOICE_ON with that VVID
when it sends a VOICE_OFF or receives a VOICE_OFF it can mark that
VVID as unused again.
Plugin wants to send a VOICE_ON:
calls host->get_vvids(32)
host allocates it 32 VVIDs
sends VOICE_ON with any of those, and must manage them itself
(alternatively, management is simpler if it just allocates one VVID
at a time and frees it when it is done)

Or does the HOST ask the INSTRUMENT for VVIDs? If so, why? Explain? :)
Post by David Olofson
I'd much rather require that events sent to a port are sent in
timestamp order, so the host can just merge sorted event lists. In
cases where you have multiple connections to the same event port, you
just use "shadow ports", so the host gets a number of ordered event
lists that it merges into one just before running the plugin that
owns the real input port.
I agree - this is the host's job, not the plugin's
Post by David Olofson
Post by Tim Hockin
host has to handle syncronization issues (one recv event-queue per
thread, or it has to be plugin -> host callback, or something), but
that's OK.
Well, if your threads are not in sample sync, you're in trouble...
I meant threading sync issues. Host can't just pass one event-queue to all
plugins in a multi-thread environment. But again, that is the HOSTs
problem. So the host can pass the address of it's event-queue during
activate(), or the plugin can make a host->send_event() callback.
Steve Harris
2002-12-08 12:04:01 UTC
Permalink
Post by Tim Hockin
Do we want to provide a suggested range of values for MIDI compatible
controls. For example Controls that are labelled CONTROL_AFTERTOUCH should
expect values between 0 and 1.0, and the host can map MIDI onto that? Or 0
and 127, like MIDI? or 0 and MAX_INT and let the synth blow up when you
pass a VELOCITY of 10000, and it expects 127?
I think they should be normalised to [0,1], theres nothing special about
MIDI.
Post by Tim Hockin
I had been assuming a single event-queue per instance. Two questions: why
would a plugin (assuming not multi-timbral for now) have more than one
event-queue? Assuming we do support multi-timbral synths, is there an
advantage to having events per-channel event-queues? Do you envision each
Channel getting the ->run() method seperately, or does ->run() loop? If it
is looping anyway, is there an advantage to multiple event-queues?
If we support multi timbrality a the API level then there is no point
having multiple queues (I think), you can just have "timbre" and "voice"
values in the struct (wow, multi-timbral really makes no sense does it ;)

- Steve
David Olofson
2002-12-08 23:12:01 UTC
Permalink
On Sunday 08 December 2002 06.00, Tim Hockin wrote:
[...]
Post by Tim Hockin
* Plugin: a chunk of code, loaded or not, that implements this API
(a .so file or a running instance).
* Instrument: an instance of a plugin that supports the instrument
API bits (analogous to a chunk of hardware from Roland).
* Channel: a 'part', 'channel' or 'instrument' from MIDI.
Instruments have 1 or more Channels.
* Control: a knob, button, slider, or virtual control withing a
Channel or Instrument. Controls can be master (e.g. master
volume), per-Channel (e.g. channel pressure) or per-Voice (e.g.
aftertouch).
* Preset: a stored or loaded set of values for all Controls in a Channel
* Voice: a playing sound within a Channel. Channels may have
multiple voices.
* Port: an input/output connection on a Plugin. Audio data goes over ports.
I totally agree with this. I'd just like to add that, depending on
the API design, Ports might come in four kinds total:
* Audio Input Ports
* Audio Output Ports
* Event Input Ports
* Event Output Ports
Post by Tim Hockin
Post by David Olofson
I don't think this has much to do with the API - and I don't see
the possibility of a clash. If you take any two controls, they
may have any relation whatsoever, *defined by the synth*, and the
host should
I agree with that IF we say you have to have a separate control
struct for per-voice and per-part controls. If we define a control
and say it has a flag which identifies it as MASTER, PART, VOICE,
then we do need to suggest a relationship. I like defining a
control once, but it's becoming more obvious to NOT do that. They
range of valid values may be wildly different.
Of course. MASTER, PART and VOICE controls are completely separate
things when it comes to numbers, valid ranges, internal handling and
all that. You could say that's the "relationship" between them.

As to "control struct", I'm not sure what you mean. The way Audiality
is designed, there is only one struct - the event struct - which is
used for all communication through the event system. Inside that
"protocol", where only the linked list and timestamp stuff is
strictly defined, various parts of the engine define their own,
higher level protocols, such as;

* Low level voices (ie "oscillators") with these events:
VE_START Start playing waveform <arg1>.
VE_STOP Stop and free voice.
VE_SET Set control <index> to <arg1>.
VE_ISET Set interpolated control <index> to <arg1>.
VE_IRAMP Ramp interpolated control <index> to <arg1>
over <arg2> frames.

VE_SET has one set of controls...
VC_WAVE Waveform index (latched on loop)
VC_LOOP looping on/off
VC_PITCH linear frequency
VC_RETRIG, retrig after N samples
VC_RANDTRIG random mod. to RETRIG
VC_PRIM_BUS target bus for ACC_VOLUME
VC_SEND_BUS target bus for ACC_SEND
VC_RESAMPLE Resampling mode

while VE_ISET and VE_IRAMP have their own set:
VIC_LVOL Left volume
VIC_RVOL Right volume
VIC_LSEND Left send
VIC_RSEND Right send

There's not conflict and no extra conditionals; just two sets of
controls, each with their own event(s). (BTW, note that VC_WAVE and
VIC_LVOL share the same actual enum value - but that's irrelevant, as
they're never used as arguments to the same event type.)

Now, this is hardwired stuff, but it's no different with a "real"
API. Just have one set of events for each class of controls.
Obviously, that applies to the "negotiation" API as well; you need
separate calls, or an extra argument to tell the control classes
apart.
Post by Tim Hockin
Do we want to provide a suggested range of values for MIDI
compatible controls. For example Controls that are labelled
CONTROL_AFTERTOUCH should expect values between 0 and 1.0, and the
host can map MIDI onto that?
Something like that. I'd like to use an enum for "control kind"
(corresponding to those "standard" MIDI CCs), and some hints and
stuff to tell the host about range, linearity, absolute limits,
sensible default/"no effect" value and that sort of stuff.

Or we could use strings, but I *definitely* prefer enums, since those
can be checked at compile time. You can still have two-way binary
compatibility. Just have a default case "I don't know what this is"
in the host.
Post by Tim Hockin
Or 0 and 127, like MIDI?
No, no MIDIisms on that level, please. It's easy enough to throw a
suitable factor into the int->float conversion.
Post by Tim Hockin
or 0 and
MAX_INT and let the synth blow up when you pass a VELOCITY of
10000, and it expects 127?
Just like with LADSPA, the plugin will have to tell the host whether
or not it has absolute upper and/or lower limits. In VST, the range
is [0,1], period, but I think that's too restrictive. Scaling that
range to whatever you need is fine, but that's not the point: Some
controls just don't *have* sensible absolute limits - so why force
plugin designers to "invent" some?
Post by Tim Hockin
Post by David Olofson
You're thinking entirely in keyboardist terms here. :-) You're
forgetting that some instruments really don't have anything like
attack and release in real life, but rather "bow speed", "air
speed" and that kind of stuff.
umm, speed == velocity, no?
Yes, that's what I've been thinking every now and then all the time.
However, it seems that the term "velocity" is generally thought of as
"instantaneous velocity at the time of impact". MIDI brainwashing
again; that's why I tend to think of "velocity" as a parameter rather
than anything that could be a continous control...

Another reason would be that if you accelerate a drum stick up to a
certain velocity, it'll bounce at impact - and that doesn't map all
that well to one-way communication and "don't rely on short roundtrip
latency!", does it? Force feedback MIDI controllers? :-)

Normally, plugins just do what you tell them. They don't change their
own input controls. (Well, they *could*, but that would be pretty
hairy, I think... Or maybe not. Anyone dare to try it? ;-)
Post by Tim Hockin
But I get your point, and I concede
this point. Velocity is an event.
Continuous control...? ;-)
Post by Tim Hockin
Post by David Olofson
I think so, but I'm not 100% determined. (Audiality is still
mostly a monolith, and I'm not completely sure how to best split
it up into real plugins.)
plugins from being multipart doesn't make much sense. Just
reserve one control/event port for the whole instrument/plugin,
and then allow the instrument/plugin to have as many other
control/event ports as it likes - just like audio in and out
ports.
I had been assuming a single event-queue per instance.
So have I (when designing MAIA) - until I actually implemented
Audiality. There *should* be a single per-instance event port - the
Master Event Port - but you may optionally have one event port for
each channel as well, if you have any use for per-channel events. You
may well just use the Master Event Port for everything in a
multichannel plugin, but then you're on your own when it comes to
addressing of channels and stuff. (That would require more dimensions
of addressing in each event, which means a bigger event struct, more
decoding overhead etc.)
Post by Tim Hockin
Two
questions: why would a plugin (assuming not multi-timbral for now)
have more than one event-queue?
It wouldn't have to. If there's only one Channel, Master and Channel
events are effectively equivalent - so you won't need to tell them
apart, right? Consequently, you can just have a single Master Event
Port, and register your full range of events on that.

You *could* have a Channel Event Port for your single channel (if you
even have something like that internally), but there's not much
point, unless you really have two loops in your plugin, and want
events for them delivered separately.
Post by Tim Hockin
Assuming we do support
multi-timbral synths, is there an advantage to having events
per-channel event-queues?
Yes, definitely. As it is in Audiality (which made me realize this)
you could view each Channel as a plugin instance - and they do indeed
have one event port (which implies one queue) per instance. Each
Channel has a typical "MAIA style" loop:

while(frames_left_to_process)
{
while(next_event->timestamp == now)
{
process_and_remove(next_event);
}
for(samples_left_to_next_event)
{
process_audio();
}
}

It just sits there for one buffer at a time, worrying only about it's
*own* events and it's own audio, and that's it.

Now, if you would merge multiple Channels into one plugin instance,
it would not make much sense to squeeze all events through the same
port. You would only have to split them up internally in the plugin,
or you would get a very inefficient loop structure, that effectively
splits buffers for the whole synth for every single event.
Post by Tim Hockin
Do you envision each Channel getting the
->run() method seperately, or does ->run() loop? If it is looping
anyway, is there an advantage to multiple event-queues?
run() (or process(), as I prefer to call it) should be called only
once per buffer, and there should be only one per instance. That's
the one major point with supporting multichannel at all; the *plugin*
gets to decide how and when things are done for the whole set of
Channels. If processing all channels in parallel is better in some
places; fine, do so.

Besides, multiple run()/process() calls would mean more overhead than
a single call + internal looping. You can probably disregard the
actual function call overhead in most cases, but plugins may have a
whole lot of work to do in each call before they can actually enter
the loop.

Another thing to keep in mind is that a multichannel plugin may have
a highly optimized master mixing/routing section hooked up to the
Master Event Port. Where would you run that if you have one
run()/process() call for each Channel? You would effectively end up
with at the very least N+1 plugins for N Channels.

If the Master section is also involved before, or in the middle of
the signal frow of Channels, you'll soon have countless little
plugins, communicating through Audio and Event Ports, instead of one,
potentially highly optimized "monolith" plugin.
Post by Tim Hockin
Post by David Olofson
Post by Tim Hockin
Post by David Olofson
ports. As of now, there is one event port for each Channel,
and one event port for each physical Voice. "Virtual Voices"
(ie Voices as
I know you've done some exploring - do we really want one
event-queue per voice + one for each channel + one for master? or
is one per instance simpler?
No, not one per *voice*. Voices have way too short lifetime, so it
just wouldn't make sense to try and track them with actual objects on
the API level. They should be considered Abstract Objects, that are
only addessed through a field in Voice events that are sent to the
Channel Event Port.

(Note that I actually do have one event port for each voice in
Audiality - but that's just because it happens to be the simplest and
most efficient way of controlling them. Implementation stuff.)


[...]
Post by Tim Hockin
Post by David Olofson
Post by Tim Hockin
I'm still not clear on this. What plugin would trigger another
plugin?
Any plugin that processes events rather than (or in addition to)
audio. They're usually called "MIDI plugins".
I can see controller plugins (It's another extension to this API, perhaps)
No. Let's not repeat Steinberg's mistake here as well.
Post by Tim Hockin
which deal with sending events to other plugins. And I
guess an arppegiator plugin could be a controller..
Yes. It could also listen to an audio track, extracting pitch
information from it.
Post by Tim Hockin
Post by David Olofson
Post by Tim Hockin
If so, how do you reconcile that they
will each have a pool of VVIDs - I suppose they can get VVIDs
from the host, but we're adding a fair bit of complexity now.
Where? They have to get the VVIDs from somewhere anyway. It's
probably not a great idea to just assume that plugins can accept
any integer number as a valid VVID, since that complicates/slows
down voice lookup.
you don't send just ANY integer, you get the integer to id a voice
FROM the plug.
You get a *range* of valid VVIDs from the host (which has to ask the
plugin at some point), by asking for it, saying how many VVIDs you
need. Plugins that want to reallocate internal tables to deal with
the VVIDs will generally have to do this outside the RT thread, but
plugins that use the tag/search approach (like Audiality as of now)
don't even have to be asked; the host could maintain the VVIDs all by
itself.
Post by Tim Hockin
But if VOICE_ON is an event, it gets much uglier.
Host allocates the VVID pool (perhaps a bitmask or some other way
of indicating usage)
allocates an unused VVID
sends VOICE_ON with that VVID
when it sends a VOICE_OFF or receives a VOICE_OFF it can mark that
VVID as unused again.
It seems like this would be a host global matter... What would be the
point in that? All you need is a way to address Voices within *one
Channel* - you don't have to worry about other Plugins, or even other
Channels on the same Plugin. (For table based implementations,
plugins have one table per Channel, and for tag/search
implementations, you just tag Voices with VVIDs *and* Channel
indices.)
Post by Tim Hockin
calls host->get_vvids(32)
host allocates it 32 VVIDs
You do this when the host tells you to connect to an event port, and
what you do is actually:

first_id = host->get_vvids(target_port, number_of_VVIDs);
if(first_id < 0)
give_up;

Note that the first_id thing is actually *only* needed when the host
connects multiple output ports to the same input port. Normally, it
will be 0.
Post by Tim Hockin
sends VOICE_ON with any of those, and must manage them itself
(alternatively, management is simpler if it just allocates one
VVID at a time and frees it when it is done)
Definitely not. That would mean at least two function calls for every
VOICE_ON event... *heh* (Meanwhile, Audiality event allocation,
sending and handling macros will only ever make a function call when
the Event Pool is empty - and that's normally a critical error. Each
common event operation is only a matter of a few CPU instructions.)
Post by Tim Hockin
Or does the HOST ask the INSTRUMENT for VVIDs? If so, why?
Explain? :)
The host will at some point ask the plugin, or rather *tell* the
plugin how many VVIDs are needed for each Event Input port it wants
to connect. Plugins may say

"I don't have Voices!"
(No need for VVIDs...)

or

"Anything that fits makes a fine VVID for me."
(Full 32 bit range, no actual allocation needed.

or

"I need internal tables. How many VVIDs do you need?"
(Host says N, and plugin allocates tables for N
VVIDs, addressed as 0..N-1.)

The last case would be the most interesting one, since it requires
that plugins allocate memory whenever the host needs more VVIDs. That
is, if plugins do this themselves using malloc() or something, you
simply cannot connect to Event Ports without taking the plugin out of
the RT engine - *unless* you decide on a number of VVIDs to allocate
for each Channel of every plugin, right when they're instantiated.

A third solution would be to have the *host* allocate memory for the
tables needed, and just pass it to the plugin for initialization. Or
even better (so you don't have to run the table init code in RT
context), just have a Channel flag that tells the host that calling
plugin->allocate_vvids() is not RT safe, but *is* thread safe. The
host can safely call plugin->allocate_vvids() for a Channel that is
not in use, while the plugin is running in the RT thread.


[...]
Post by Tim Hockin
Post by David Olofson
Post by Tim Hockin
host has to handle syncronization issues (one recv event-queue
per thread, or it has to be plugin -> host callback, or
something), but that's OK.
Well, if your threads are not in sample sync, you're in
trouble...
I meant threading sync issues. Host can't just pass one
event-queue to all plugins in a multi-thread environment. But
again, that is the HOSTs problem.
Yes. If the event pool is part of the host struct (which I don't even
have one in Audiality - yet! *heh*), you're safe. All you have to do
is put proxy ports in between when connecting plugins accross the
engine threads, so you never touch ports that might be read by
someone else at the same time.

Getting that to work without events arriving late is less trivial,
though! :-) (When you have more than one thread, plugin execution
order is no longer guaranteed, obviously - unless you enforce it by
having blocking or spinning synchronization at critical points in the
net or something.)
Post by Tim Hockin
So the host can pass the address
of it's event-queue during activate(), or the plugin can make a
host->send_event() callback.
Well, the host's event queue is the smallest problem. Getting plugins
to run properly in more than one thread is the big deal - but if that
works, reply events to the host is trivial. Just put the repyl event
port in the host struct as well, and have one host struct for each
engine thread.


//David Olofson - Programmer, Composer, Open Source Advocate

.- The Return of Audiality! --------------------------------.
| Free/Open Source Audio Engine for use in Games or Studio. |
| RT and off-line synth. Scripting. Sample accurate timing. |
`---------------------------> http://olofson.net/audiality -'
.- M A I A -------------------------------------------------.
| The Multimedia Application Integration Architecture |
`----------------------------> http://www.linuxdj.com/maia -'
--- http://olofson.net --- http://www.reologica.se ---
Tim Hockin
2002-12-08 06:58:01 UTC
Permalink
Post by David Olofson
Post by Tim Hockin
controls, two filter controls etc and making them slightly
different and behave differently in each plug. Are we better to
define the behavior, or at least offer a suggestion?
Dimension 1: Channel
Dimension 2: Control
Do we really want to have different controls for different channels? I can
see how it might be neat, but since multiple channels is convenient
for hardware people, shouldn't we leave the controls the same, too? :)

Simpler perhaps:

nchannels
master_controls[]
channel_controls[]
voice_controls[]

or, if we agree flags for this are best:
controls[] (each is flagged MASTER/CHANNEL/VOICE)
Post by David Olofson
What if you need *only* pressure, and don't care about attack or
release velocity? Why not just make velocity optional as well? :-)
Even though I have conceded this point - JUST IGNORE IT? Whatever, it's old
news :)
Post by David Olofson
However, it might be handy for the host to be able to ask for the
values of specific controllers. It may or may not be useful to do
that multiple times per buffer - but if it is, you'll definitely want
the timestamps of the "reply events" to match those of your request
events, or things will get hairy...
Uggh, can we keep the get() of control values simpler than events? My
previos proposal had a control->get() method and a ctrl->set() method.
Obviously, the set() is superceded by events. Is the get, too?
Post by David Olofson
Post by Tim Hockin
SILENT (plugin to host - sent when reverb tails or whatnot have
died..)
Great idea. Sample accurate end-of-tail notification. :-) (In
Audiality, I do that by fiddling with the plugin state, which is
rather ugly and not sample accurate.)
I noodled on your state-model and realized that a state-change is just an
event. :)
Steve Harris
2002-12-08 11:48:00 UTC
Permalink
Post by Tim Hockin
Do we really want to have different controls for different channels? I can
see how it might be neat, but since multiple channels is convenient
for hardware people, shouldn't we leave the controls the same, too? :)
Sounds sensible.
Post by Tim Hockin
Post by David Olofson
However, it might be handy for the host to be able to ask for the
values of specific controllers. It may or may not be useful to do
that multiple times per buffer - but if it is, you'll definitely want
the timestamps of the "reply events" to match those of your request
events, or things will get hairy...
Uggh, can we keep the get() of control values simpler than events? My
previos proposal had a control->get() method and a ctrl->set() method.
Obviously, the set() is superceded by events. Is the get, too?
What is this for? If you want to implement this then they need to be
events (well, sample accurate), but they sound like a pain to implement.

- Steve
Tim Hockin
2002-12-08 12:12:01 UTC
Permalink
Post by Joern Nettingsmeier
Post by Tim Hockin
yeah, and I want to make it easy for an OAPI (I HATE that name - open to
ideas..) host to use LADSPA plugins. That is very important.
Linux Audio Developers' More Sophisticated Plugin API (=LADMSPA) ? :-D
umm, veto?
Steve Harris
2002-12-08 13:03:12 UTC
Permalink
Post by Tim Hockin
Post by Joern Nettingsmeier
Post by Tim Hockin
yeah, and I want to make it easy for an OAPI (I HATE that name - open to
ideas..) host to use LADSPA plugins. That is very important.
Linux Audio Developers' More Sophisticated Plugin API (=LADMSPA) ? :-D
umm, veto?
I jokingly suggested XLP in a previous thread on a similar subject. It
could be... eXtended Linux Plugins?

Maybe XLI would be good here.

- Steve
Tim Hockin
2002-12-08 12:14:01 UTC
Permalink
Post by Steve Harris
A Bell is a good example of something where you want multiple velocity
events, you could call voice_on() or whatever to initialise the physical
model instance, then you send simultaneous velocity and position events to
indicate that the bell has been struck.
I think it has been argued well enough that velocity be a continuous
controller.

I'll concede that one.
Tim Hockin
2002-12-08 12:17:00 UTC
Permalink
Post by Steve Harris
Post by Tim Hockin
Uggh, can we keep the get() of control values simpler than events? My
previos proposal had a control->get() method and a ctrl->set() method.
Obviously, the set() is superceded by events. Is the get, too?
What is this for? If you want to implement this then they need to be
events (well, sample accurate), but they sound like a pain to implement.
Well, my first thought is that we don;t want to send a CTRL_CHANGE event to
the host every time a control changes, Do we want the host to have to send
CTRL_GET events and wait for CTRL_VALUE events, or can we say 'one tick
granularity is all the host can get' wrt CTRL_GET?
Steve Harris
2002-12-08 12:53:01 UTC
Permalink
Post by Tim Hockin
Post by Steve Harris
Post by Tim Hockin
Uggh, can we keep the get() of control values simpler than events? My
previos proposal had a control->get() method and a ctrl->set() method.
Obviously, the set() is superceded by events. Is the get, too?
What is this for? If you want to implement this then they need to be
events (well, sample accurate), but they sound like a pain to implement.
Well, my first thought is that we don;t want to send a CTRL_CHANGE event to
the host every time a control changes, Do we want the host to have to send
CTRL_GET events and wait for CTRL_VALUE events, or can we say 'one tick
granularity is all the host can get' wrt CTRL_GET?
The problem is that the dsp code doesn't always know the real value of a
controller: if its expensive to calcuate the coefficients you normally
just calcuate them once for every N samples then interpolate the
coefficients, which bear no simple relation to the controller value.

If the host is marshaling the events then it should know what the last
controller value it sent to the instrument was.

OT: I think you do want to handle note_on as an event, as you may want to
process it with an event processing only plugin, eg. something to may midi
style keyboard events to a gong striker (eg. map pitch to position and
keep velocity). Or even an arpegiator.

- Steve
Tim Hockin
2002-12-08 20:34:01 UTC
Permalink
Post by Mark Knecht
I agree with your comments. I'd also like to see the developers at
least understand and consider these sorts of protocols also. In the end
something good will come out of it.
We're not discussing discarding velocity. We're discussing whether it is a
property of the note on/off events, or whether it can be unified with the
rest of the event model and become a simple event delivered at note_on time.

If a plugin wants to handle continuous velocity changes (a violin?) it can.
It CAN, but it doesn't HAVE TO. That is the markings of a good API, in my
mind, and that is why I have conceded this point :)

Tim
Mark Knecht
2002-12-08 21:14:01 UTC
Permalink
Serves me right I suppose...

have a nice day...
Post by Tim Hockin
Post by Mark Knecht
I agree with your comments. I'd also like to see the developers at
least understand and consider these sorts of protocols also. In the end
something good will come out of it.
We're not discussing discarding velocity. We're discussing whether it is a
property of the note on/off events, or whether it can be unified with the
rest of the event model and become a simple event delivered at note_on time.
If a plugin wants to handle continuous velocity changes (a violin?) it can.
It CAN, but it doesn't HAVE TO. That is the markings of a good API, in my
mind, and that is why I have conceded this point :)
Tim
Tim Hockin
2002-12-08 21:25:00 UTC
Permalink
Post by Steve Harris
Post by Tim Hockin
Well, my first thought is that we don;t want to send a CTRL_CHANGE event to
the host every time a control changes, Do we want the host to have to send
CTRL_GET events and wait for CTRL_VALUE events, or can we say 'one tick
granularity is all the host can get' wrt CTRL_GET?
If the host is marshaling the events then it should know what the last
controller value it sent to the instrument was.
I'll buy that. Once the host reads the initial values, it should refrain
from bothering the plugin.

Aside: Do we see a need for plugins to spontaneously change a control? What
of a plugin that has a velocity-capped control (you turn a knob fast, and it
eventually gets there) or something that crossfades between two values
automatically. Do we need to send CTRL_CHANGE events to the host, or should
we do a 'watcher' callback. Watchers is how AudioUnits works. I don't
really like it.

Also to think about - what if the host sends a bad value, and a control
wants to reject it? Ahh, asyncronicity. I guess for this case the plugin
sends back some FAIL event or simpler sends back a CTRL_CHANGE with the old
value (or the min/max if the host has gone too far).
Post by Steve Harris
OT: I think you do want to handle note_on as an event, as you may want to
process it with an event processing only plugin, eg. something to may midi
style keyboard events to a gong striker (eg. map pitch to position and
keep velocity). Or even an arpegiator.
Arpegiator is what finally convinced me of this, yesterday evening
(California time).

Just out of curiosity, where are you located physically, in the real world?
Our timezones are certainly not in sync.

Tim
Steve Harris
2002-12-08 23:05:01 UTC
Permalink
Post by Tim Hockin
I'll buy that. Once the host reads the initial values, it should refrain
from bothering the plugin.
I think the host should be able to give the instrument its initial values,
read for deafults, or a settings file etc.
Post by Tim Hockin
Aside: Do we see a need for plugins to spontaneously change a control? What
of a plugin that has a velocity-capped control (you turn a knob fast, and it
eventually gets there) or something that crossfades between two values
automatically. Do we need to send CTRL_CHANGE events to the host, or should
we do a 'watcher' callback. Watchers is how AudioUnits works. I don't
really like it.
I'm afraid I'm not familar with the AudioUnits API, but I dont/think/ this
situation requires the instrument to inform the host, it should just be
able to damp (slew rate limit) the control, shouldn't it? The result from
playing back automation data will be the same, so no harm done.
Post by Tim Hockin
Also to think about - what if the host sends a bad value, and a control
wants to reject it? Ahh, asyncronicity. I guess for this case the plugin
sends back some FAIL event or simpler sends back a CTRL_CHANGE with the old
value (or the min/max if the host has gone too far).
Bad value? LADSPA has the concept that there are no bad values, the plugin
has to accept anything. It can choose to ignore it, but it mustn't
segfault or whatever.

This is not necceserily the right thing, but it does make some things
simpler. I'm not aware of any situations where this has required the
plugin to tell the user "that was a crap value", usually the wideband
distortion is hint enough ;)

In practice most hosts only send values within the range hints.
Post by Tim Hockin
Just out of curiosity, where are you located physically, in the real world?
Our timezones are certainly not in sync.
England, UK. GMT+1. I'm not a mornings person though ;)

- Steve
Tim Hockin
2002-12-08 21:31:01 UTC
Permalink
Post by Steve Harris
Post by Tim Hockin
Post by Joern Nettingsmeier
Linux Audio Developers' More Sophisticated Plugin API (=LADMSPA) ? :-D
umm, veto?
I jokingly suggested XLP in a previous thread on a similar subject. It
could be... eXtended Linux Plugins?
Maybe XLI would be good here.
I once suggested OPI, but was stringly suggested to include 'audio' or
'sound' or something in it. Problem is that 'Plugin Interface' or 'Plugin
Architecture' takes up two letters, and we all know good acronyms are THREE
letters. Also 'Plugin Interface' (which I prefer to 'architecture') happens
to be the last part of the already well defined 'API' acronym. Maybe we can
leave that bit out?

As for 'Linux' I prefer it be OS neutral. And please don't say GNU.

I've converted it to PAPI for now, since it is easier to speak, and I
'speak' in my head as I write. {Portable, Public, Puerile} Audio Plugin
Interface. or 'PAPI Audio Plugin Interface'?

My personal preference is for something that doesn't have to be spelled to
be spoken. I know most people say VSTi as 'visty'.

Tim
Steve Harris
2002-12-08 23:15:59 UTC
Permalink
Post by Tim Hockin
I've converted it to PAPI for now, since it is easier to speak, and I
'speak' in my head as I write. {Portable, Public, Puerile} Audio Plugin
Interface. or 'PAPI Audio Plugin Interface'?
In parts of the UK "pappy" is equivalent to shite, ie. redolent of
excrement ;)
Post by Tim Hockin
My personal preference is for something that doesn't have to be spelled to
be spoken. I know most people say VSTi as 'visty'.
Agreed, our German brethren seem to want to spell out
El-Ah-Dee-Es-Pee-Ah, which is pretty inconvenient.

OTOH I think Joern was amused when I insisted on pronouncing luss, as in
"can you just ls that directory".

- Steve

Loading...