JACK Synthesizer Manager Proposal

Audun Halland and I have been thinking about a set of related problems in the realm of Linux audio. The first result is the following proposal, meant to gather feedback from the community via the LAD and LAU mailing lists.

JSM, the JACK Synthesizer Manager

We propose a programm that acts as a proxy between sequencing software and both software and hardware sythesizers. Among the goals are unified patch selection and making projects more portable.

If we get the impression that the JSM is something that both developers and users will find handy and use, then development might start real soon.

In this text, we avoid going into technical details to foster free thought and discussion.

Use Cases

Patch selection

Goal: Choose patches from all available hardware and software synthesizers.

Giorgio uses a single means to select a patch among all patches of all of his software and hardware synthesizers. He uses meta-data to find the right patch. The right connections are made automatically.

Computer as syntheszier

Goal: Use the computer as a compound synthesizer in a live performance.

Hiromi has her keyboard connected to her laptop live on stage. She uses several soft-synths via keyboard-split and layering. A few selected parameters are bound to the wheels of the keyboard. After each song, she switches from one setup to the next with least effort.

Collaboration

Goal: Exchange projects without having to change settings back and forth.

Alice and Bob take turns on working on a project. They use different hardware but don’t have to manually change connections and choose patches on each turn because of an abstraction layer.

MIDI Interface Ports

The problem with MIDI interface ports is that the hardware on the other side and its setup might change. Or be entirely different if people exchange projects. An abstraction layer can make this more comfortable to handle.

The JSM takes care of the mapping between software ports and MIDI interface ports. It can work on a per MIDI channel level.

Patches and Instrument Definitions

Patches and controllers are chosen by name; the user doesn’t have to deal with cryptic numbers. For kit-programms, name mappings are given (e.g. bass drum on C1).

Patch selection happens by a single means, offering all available patches (JACK apps, plugins, hardware). Making the required MIDI and audio connections is automated as far as possible.

Categorization

Categories help to find the right patch among many. When exchanging projects, they help to replace unavailable patches with similar ones.

Virtual/Compound Synthesizers

From the outside, the computer can be dealt with like a single compound synthesizer. Different synthesizers can be triggered from ranges on a single keyboard (key splits). Synthesizers can be layered. The whole setup can be switched with programm changes.

JACK to ALSA Bridge

JSM could be the de facto JACK MIDI to ALSA MIDI bridge. No Jack “SYSTEM” midi ports, the jack world only sees the devices offered by JSM.

—
Audun Halland and Thorsten Wilms

Ardour MIDI Editing 4

So there’s the question how rectangular selection shall work: selecting notes that are touched by the rectangle vs only selecting notes that are fully within the rectangle. MusE and Nautilus use the first approach. Several other sequencers and file managers likely, too. Inkscape and likely other 2d graphics tools use the later approach. I find that I can’t remember details like this for many applications I used.
Here’s an example pattern in MusE:

Selection widths in pixels for the first 3 chords. Minimum and maximum to select the one chord and no other note:

	Touch	Cover
Minimum	0, 0 , 0	48, 16, 48
Maximum	48, 16, 48	64, 94, 78
Range	48, 16, 48	16, 78, 30

I think the numbers clearly support that cover-select has a strong advantage over touch-select if it’s about selecting short notes, but makes selecting longer notes next to short notes harder . If a note extends beyond the canvas, it can only be selected via covering with the additinal rule that in such a case, covering the visible extents of a note is enough.

Ardour MIDI Editing 3

Previously:
Ardour MIDI Editing 1
Ardour MIDI Editing 2

Underlay

Another MIDI region as underlay. Making use of colour perspective: blue steps into the background. For the same reason my notes are orangish, to come to the front (and be warm, inviting). The short grooves mark grid steps and are meant to look distinct from notes in a row.

Smart dragging

Dragging from empty space does rubber band selection. If the selection rectangle doesn’t fully enclose any note, the starting row is filled with a note (indicated by an outline while dragging). In the very odd case of someone wanting a note that completely overlaps another note on the same row, it can still be done by creating a shorter note and then extending it, or creating the note on another row and moving it.

Rubber banding selecting all touched (not necessarily encased) notes could be had with a modifier key.

The encased note is selected. I will likely have to create more plain graphics, but for now I’m having a bit of fun showing what could be done 😉

Initially I thought of showing the 3 target areas on mouse-over plus highlighting the active one. But sectioned highlighting does both jobs.

2 variations on indicating whether a note is exactly on a grid position or not (bottom note is not).

Ardour MIDI Editing 2

Following up on the abstract Ardour MIDI Editing 1, some pretty pictures this time 😉
Didn’t pay attention to integration with Ardour’s look yet. There’s a light theme in development and I might target it once it stabilizes.

The piano bar is meant to become part of the track headers and could be used for vertical scrolling and maybe zooming. The 1 of each bar is marked with a blue background, each other odd beat with a lighter blue background. The big numbers are just an idea to help with orientation, as the time rulers above the track canvas might be far away. Might be too busy, though.

The notes are meant to have 3 distinct target areas for moving the start, whole note or end. To make short notes in a row easier to read, it might be better to show the handles only on mouse-over.

Showing velocity as bars on the note canvas. Chords are obviously a problem.

The first note shows how indicating note on and off velocity could look like. Followed by another approach for stacked notes.

Another style, 2 different approaches for stacked notes.

A whole set of ways to indicate velocity right on the notes. Last 2 columns on and off velocity. Indicating off velocity should be optional, as it is seldomly used, I think.

A row below the note area with bars like in so many sequencers is still a good idea.

Ardour MIDI Editing

Dry stuff I know, but even if no one bothers to read it, it helped me to organize my thoughts 😉

This post as PDF

Table of Contents

• 1 Introduction
• 2 Why MIDI in Ardour?
• 3 Note Event Operations
  • 3.1 Random Access Operations
    • 3.1.1 Single Note Events
    • 3.1.2 Single or Multiple Events
    • 3.1.3 Multiple Events
  • 3.2 Real-time and Sequential Access Operations
  • 3.3 Frequency of Use
• 4 Representation
  • 4.1 Velocity
  • 4.2 Drums vs Piano
• 5 Mouse Control
  • 5.1 Target Areas
  • 5.2 Conventions
  • 5.3 Modality
  • 5.4 Overload
  • 5.5 Strategies
    • 5.5.1 One tool per operation
    • 5.5.2 Quasimodes
    • 5.5.3 Selection, Action
    • 5.5.4 Keys instead of clicks
    • 5.5.5 Smart Dragging
• 6 Keyboard Control
  • 6.1 MIDI Cursor
  • 6.2 On and Off
  • 6.3 Navigation
  • 6.4 Add Notes
  • 6.5 Clipboard overlay

1 Introduction

Ardour is as of now an audio-only sequencer, but this years Google Summer of Code project by Dave Robillard will add MIDI sequencing capabilities. He prepared the infrastructure in a GSoC 2006
project.

Regarding interaction and interface design, graphical editing of MIDI is the most interesting and crucial aspect. For now I will address editing of note events, exploring the problem and solutions
space.

The central question is how to make the operations necessary to edit note events available in the most usable way.

2 Why MIDI in Ardour?

Some people stated a preference for Ardour staying audio-only. They are concerned about added complexity, the required development effort and either don’t need MIDI themselves or think it can be
handled with separate applications.

It’s common in modern music production to combine audio like drum-loops with MIDI sequenced lines. In such a case, having 2 time-lines in 2 separate windows and applications that need to be synced
is not a solution, but a problem. Being able to edit one part while seeing how it relates to another is fundamental. This is also an argument for on-canvas MIDI editing. Another is avoiding
additional windows the user has to deal with.

The existing open-source audio/MIDI sequencers are all no match for Ardour’s audio capabilities and none of them has on-canvas MIDI editing, so there’s a gap to fill.

3 Note Event Operations

Note events can be manipulated in a number ways depending on selection and the mode of access.

3.1 Random Access Operations

Operations on note events that can and have to happen independently of transport state / playback:

• Add
• Edit
  • Move (time)
  • Transpose
  • Adjust duration
  • Adjust velocity (for both note ons and offs)
• Delete
• Audition

3.1.1 Single Note Events

Operations applicable only to single note events, listed with parameters and possible sources:

• Add
  • Location
    • Mouse location
    • Edit-cursor location
  • Duration
    • Drag (with or without modifier, free or with snapping)
    • Holding a key for the duration (mouse, keyboard or MIDI input)
    • Default value
  • Note number
    • Mouse cursor location
    • Edit-cursor position (assuming a cursor that has a note axis position, not time-line only)
    • MIDI input
    • Alphanumeric input (note number, name and octave, tracker style keymapping)
    • Default value
  • Velocity (note on / off)
    • Default value
    • Drag (with or without modifier, from or in a special area)
    • MIDI input
    • Alphanumeric input

3.1.2 Single or Multiple Events

Operations applicable to single or several selected events:

• Delete
• Move
• Adjust duration (can be seen as moving note ons or offs)
• Transpose (absolute or in a scale)
• Cut (copy to clipboard and clear selection)
• Copy
• Paste
• Cut span (notes and empty space)
• Copy span (notes and empty space)
• Paste span (notes and empty space)
• Delete span (move following to leave no gap)
• Trim (only keep selection)
• Split notes at location
• Clear span (notes partly in will be cut off)
• Divide n-times

3.1.3 Multiple Events

Operations applicable only to multiple events:

• Time-stretching
• Shrink / stretch note range
• Mirror on time axis
• Mirror on pitch axis

3.2 Real-time and Sequential Access Operations

Operations that could be applied to events as they are played back or stepped through:

• Delete
• Cut out
• Delay (until the trigger is released)
• Overdub / Replace
• Insert (combined record and delay)
• Overwrite single parameter (example: note number)
• Manipulate single parameter (example: transpose notes)

3.3 Frequency of Use

Depending on the current task, the different operations are more or less frequently needed. Most frequently listed first, based on my own experience.

If not all basic operations can be made available in a straightforward way, it will have to be considered to model modes based on these sets.

Editing real-time recorded MIDI, following the question of what kind or errors need to be fixed most often:

• Move
• Adjust duration
• Adjust velocity
• Delete
• Add

Programming MIDI. Add is clearly most important, as without it, there’s nothing. Otherwise the order is

• Add
• Move
• Delete
• Adjust duration
• Adjust velocity

Tweaking MIDI:

• Adjust duration
• Adjust velocity
• Move

4 Representation

The horizontal time-line is a given an only the matrix / piano-roll approach with bars representing note length allows to put audio and MIDI into relation.

4.1 Velocity

Sensible representations of velocity values:

• On notes
  • Numeric value
  • Colour (hue or brightness)
• Overlay: bars on the same canvas area. Dealing with polyphony is tricky.
• Special area. The common row below the note area. Comparable to automation tracks.

4.2 Drums vs Piano

Many sequencers have special drum tracks / views.

A drum editor can do away with having rows represent notes in continuation and instead allow reordering and assigning ports and channels per row.

As note length tends to be of no concern for percussive instruments, symbols that denote a location but no duration are used, often a rhomb.

Drum and piano style editing could be combined by always allowing rows to be reordered, having optional per row port and channel settings and using both note bars and rhombs (or similar symbol,
understood as alias for notes of an adjustable equal length).

5 Mouse Control

5.1 Target Areas

Besides moving notes, it must also be possible to resize them, which can be though of as moving either note ons or offs.

Short notes on common zoom levels are likely too small to offer target areas (handles) for 2 (whole note, note off) or even 3 operations (whole, on, off).

Approaches:

1. The user can always zoom in if he has to edit short notes
2. Resizing handles that extend to outside the actual note. But how to avoid overlap with other notes and handles?
3. Separate tools / modes for moving whole notes vs note ons and offs.

5.2 Conventions

Conventions found across several platforms applications do not just set user expectations, but encourage habituation. Habits are hard to change.

Conventions that should be followed:

• Left-clicking an item toggles its selection state (and deselects other items)
• Ctrl-left-clicking toggles item selection state without deselecting other items (items can be added to the selection)
• Shift-left-clicking allows the selection of a span of items
• Left-click-dragging from an empty space does rubber band selection
• Right-clicking opens a context menu

5.3 Modality

To make the operations necessary for editing MIDI available via the mouse, usually a number of modes is introduced in the form of tools.

Modes add context the user needs to keep in mind, as the same actions will not always lead to the same result anymore. Mode switching also leads to additional steps to be taken.

Therefore it is desirable to minimize modality.

Jeff Raskin proposed the use of quasimodes, where a mode is kept in place only as long as a key is held down. This way the user can hardly forget about it. Holding a key for a longer duration is
tiresome, though.

Quasimodes should be considered for short tasks, but avoided for long tasks.

5.4 Overload

The most efficient way to add notes via mouse would be to just click where they shall appear. Or click-drag to also define their length. Just clicking on a note or dragging from empty space are
expected to work for selecting. Moving notes would be most simple by just dragging.

5.5 Strategies

5.5.1 One tool per operation

A set of tools like it can be found in some sequencers and many graphics applications.

Modifier keys could be used to temporarily switch to other tools, or to switch to a sub-mode.

Straightforward, but having to switch tools is costly and icons representing the tools take space, but should be present for discoverability.

5.5.2 Quasimodes

Use modifiers and mouse buttons to have one main mode and everything else as quasimodes. One could even differentiate between modifier keys being pressed down before or after a drag is initiated,
as done in GIMP.

Could be very efficient after training, but hard to learn / memorize. Depends on documentation and users reading it . A way out could be status bar messages, but there’s not even a status bar and
all space is precious.

5.5.3 Selection, Action

Use the mouse to select single notes or to draw selection rectangles only. All other actions work on the selection and could be triggered by shortcuts, context menu or GUI buttons. Adding a note
would happen by filling the selection (requires that white space is selectable).

Simple and fast selection in all cases (rubber-banding can start on notes), but not being able to directly draw notes or to drag them around is quite a drawback.

5.5.4 Keys instead of clicks

Use mouse for position and trigger actions with keys. Example: A to add notes.

Hard to learn. Difficult to find enough free and matching keys. Requires a one hand on mouse, one on keyboard approach for everything.

5.5.5 Smart Dragging

Use dragging from empty space for rubber band selection. Fill the selection with a note if no other note is completely covered. Use dragging on a note for
moving it (or all selected notes).

This would make drawing notes that overlap other notes completely a little complicated: draw elsewhere, move. I think it’s save to assume such overlaps are very rarely (if ever) wanted, so this
wouldn’t be a problem.

6 Keyboard Control

Just some rough ideas. Needs to be evaluated in relation to mouse control.

6.1 MIDI Cursor

As the term edit cursor is already taken for a special location mark, I will call this concept a MIDI cursor. The underlying idea is handling MIDI more like text in an editor. It marks a location
on the time-line and a note row.

6.2 On and Off

Treat notes ons and offs as separately selectable items. A note is selected if both on and off are selected.

6.3 Navigation

Setting vertical location absolute: Shift-A, B, C, D, E, F, G, with # as toggle. Setting vertical location relative: Cursor keys up / down. Moving
horizontally: Cursor-keys left / right.

Ctrl for bigger steps, Alt for fractions Shift for selection span

6.4 Add Notes

Add note at current position: Enter. Add note by name: A, B, C, D, E, F, G, with # as toggle.

Edit Notes

6.5 Clipboard overlay

Copy/Cut results in an overlay of the notes attached to the edit cursor.