Class TutorialOne_05_GesturePlayground
- All Implemented Interfaces:
processing.core.PConstants
QUICK START
- Launch the sketch. A display window and a palette of Graphical User Interface (GUI) controls appears.
The display window has an audio file preloaded. The grayscale values in the image are transcoded audio
samples. An overlaid rainbow spectrum traces the Signal Path, which maps an audio signal to the image
pixels. The Signal Path is created by the PixelMapGen
multigenand managed by the PixelAudioMappermapper. The Signal Path starts in the upper left corner and ends in the lower right corner. - Drawing is already turned on, so go ahead and drag the mouse to draw a line. As in TutorialOne_03_Drawing, a brushstroke appears when you release the mouse. TutorialOne_03_Drawing gave you limited control over the attributes of the brushstroke and its associated audio parameters. In GesturePlayground, you can control nearly all the available parameters with the control palette.
- At the top of the control palette, you'll find Path Source radio buttons and sliders for setting the geometry of the brush curve. When the curve is set to Reduced Points or Curve Points, the epsilon slider will allow you to visualize changes in the curve. For the curve points representation of the curve, the Curve Points slider will add or subtract subdivision points.
- The GUI palette displays controls for the current audio synthesis instrument. Press the 't' key to change the instrument. The control palette will reflect the changes. The control palette provides three play modes: one for editing granular synthesis parameters, another for the sampler synthesizer, and a "play only" mode where you can play both instruments but don't have editing enabled.
- The controls for the Sampler are fairly simple. You can change the number of points in the curve with the geometry controls. You can also change the duration of the gesture and the number of points in it with the Resample and Duration sliders. Finally, there's a Sampler Envelope menu that will change the ADSR envelope of each sampler event point.
- The Granular Synth has all the controls of the Sampler synth except for the envelopes, plus many controls for granular synthesis:
- The Hop Mode radio buttons determine if the duration of the granular event is determined by the gesture timing data in the brushstroke's PACurveMaker instance, or by the Grain Length and Hop Length sliders.
- Burst Count sets the number of linear grains at each event point. Its effect is to expand the sound of each grain.
- Grain Length and Hop Length sliders control the spacing of the grains. Hop Length is only used for Fixed Hop Mode. Grain and Hop durations are in milliseconds.
- The Warp radio buttons and slider control non-linear timing changes to the gesture. Experiment to find out how they work.
- There are many key commands too, including the 'o' command to load a new audio files. Some commands are particularly useful with granular synthesis:
- The 'q' command key will calculate the optimal number of grains in a gesture (usually in GESTURE Path Mode) and update the control palette. This can provide smooth granular synthesis even as it preserves the timing characteristic of the gesture.
- The 'c' command key will print configuration data to the console.
- The 'x' command key deletes the brush you are hovering over, if it is editable.
- The 'w' command key swaps the instrument type of the brush you are hovering over and changes edit mode to match.
About GesturePlayground
GesturePlayground uses a GUI to provide a tour of the usage and properties of the
AudioBrush subclasses GranularBrush and SamplerBrush,
the GestureSchedule class, and the Sampler and Granular audio synthesis
instruments PASamplerInstrumentPool and PAGranularInstrumentDirector.
An AudioBrush combines a PACurveMaker and a GestureGranularConfig.Builder.
PACurveMaker models gestures, one of the core concepts of PixelAudio. In its simplest
encoded form, the PAGesture interface, a gesture consists of an array of points
and an array of times. The times array records the times when something as-yet-unspecified
will happen at the corresponding point in the points array. The times array and the points
array must be the same size.
In my demos for PixelAudio, what happens at a point is typically an audio event and an
animation event. A very specific sound happens at every point in the bitmap image because
bitmap locations map onto audio buffer indices with a one-to-one correspondence. The mapping
is generated by a PixelMapGen instance and managed by a PixelMapper instance. Gestures over
the 2D space of an image become paths through audio buffers. The audio buffer is accessed
either by a granular synthesis engine or by a sampling synthesizer. For the granular synth, a
gesture corresponds to a non-linear traversal of an audio buffer, potentially as a continuous
sequence of overlapping grains with a single envelope. The sampling synthesizer treats each
point as a discrete event with its own envelope. Depending on how gestures and schedules are
structured, the two synthesizers can sound very similar, but there are possibilities in each
that the other cannot realize. As you might expect, GranularBrush implements granular synth
events and SamplerBrush implements sampler synth events. Both rely on PACUrveMaker which, in
addition to capturing the raw gesture of drawing a line, provides methods to reduce points
and times, create Bezier paths, and generate event schedules. PACurveMaker scheduling data
can be modified by changing duration, interpolating samples, or by non-linear time warping.
GesturePlayground uses GestureScheduleBuilder to interpolate and warp time and
point lists.
The parameters for gesture modeling, granular and sampling synthesis, time and sample
interpolation, and audio events are modeled in the GUI, which uses
GestureGranularConfig.Builder gConfig to track its current state. A
GestureGranularConfig instance is associated with each AudioBrush. When you click on an
AudioBrush and activate it, its configuration data is loaded to the GUI and you can edit it.
It will be saved to the brush when you select another brush or change the edit mode. When a
brush is activated with a click, the schedule is built from its PACurveMaker and
GestureGranularConfig.Builder instance variables:
GestureSchedule schedule = scheduleBuilder.build(gb.curve(), cfg.build(), audioOut.sampleRate());
Code Details
The calling chain for a GranularBrush:
mouseClicked() calls scheduleGranularBrushClick(gb, x, y);.
In scheduleGranularBrushClick(...) we get a reference to the audio buffer
buf and then use the PACurveMaker object gb.curve() and
gb.snapshot() to build a GestureSchedule, sched.
sched gets timing and location information for the gesture from
gb.curve() and modifies it with the settings from the control palette which are
stored gb.snapshot().
We port the granular synthesis parameters from the brush to a
GestureGranularParams object, and then call playGranularGesture(buf,
sched, gParams) to play the granular synth. We also call
storeGranularCurveTL(...), which sets up UI animation events to track the
grains.
Parameter buf is the audio signal that is the source of our grains, parameter
sched provides the points and times for grains and parameter params
provides the core parameters for granular synthesis.
playGranularGesture() builds arrays for buffer position and pan for each
individual grain and then calls gDir.playGestureNow(buf, sched, params, startIndices,
panPerGrain) to play the PAGranularInstrumentDirector granular synth. The 'p' command
key can toggle per-grain pitch jitter, which calls playGestureNow()in a slightly
different way. See playGranularGesture() for details.
PAGranularInstrumentDirector its own calling chain that goes all the way down to
the individual sample level using the Minim library's UGen interface. If you just want to
play music, you'll probably never have to deal with the hierarchy of classes directly, but
comments PAGranularInstrumentDirector may be useful.
Part of the calling chain for a SamplerBrush:
mouseClicked() calls scheduleSamplerBrushClick(sb, x, y).
In scheduleSamplerBrushClick() we get array of points on the curve with
getPathPoints(sb) and then use sb.snapshot() and
scheduleBuilder.build() to build a GestureSchedule
Finally, we pass the schedule and a small time offset to storeSamplerCurveTL(),
an array of TimedLocation objects that is checked at every pass through the
draw() loop and posts both Sampler instrument triggers and animation events.
Unlike the Granular instrument, which requires very accurate timing, the Sampler synth
requires less precision, so we can handle it through the UI frames. Sample-accurate timing is
a topic for another as-yet-unreleased example sketch.
The runSamplerBrushEvents() method executes the UI brushstroke animation and the
Sampler audio events. Sampler events all pass through pool.playSample(samplePos,
samplelen, amplitude, env, pitch, pan).
Press ' ' to spacebar triggers a brush if we're hovering over a brush, otherwise it triggers a point event. Press 'c' or 'C' to print the current configuration status to the console. Press 't' to switch between Granular and Sampler editing and playing. Press 'z' to change the drawing mode of the hover brush. Press 'd' to toggle doPlayOnDraw to play when a drawing gesture ends or not. Press 'p' to jitter the pitch of granular gestures. Press 'k' to apply the hue and saturation in the colors array to mapImage (not to baseImage). Press 'K' to apply hue and saturation in colors to baseImage and mapImage. Press 'l' or 'L' to toggle loading data to both image and audio buffers when you open either an image or an audio file. Press 'f' or 'F' to toggle verbose output to the console. Press 'o' to open an audio file. Press 'r' or 'R' to reset synths to defaults -- TODO may be dropped. Press 'q' to automatically set an active GRANULAR brush to have an optimized number of samples. Press 'x' to delete the current active brush shape or the oldest brush shape. Press 'X' to delete the most recent brush shape. Press 'h' or 'H' to print help.See also: Bagatelle.java for a performance-oriented application with a GUI, presets, and networking.