public class TutorialOne_06_WindowBufferextends processing.core.PApplet
TutorialOne_06_WindowBuffer
A continuation of TutorialOne_03_Drawing that keeps the drawing, brush,
sampler, granular, bounds, and envelope code from TutorialOne_03_Drawing,
but changes the audio source model to load large audio files into a
windowed buffer:
anthemSignal / anthemBuffer hold the entire loaded audio file,
windowBuff is a moving window over anthemSignal,
audioSignal / playBuffer / mapImage hold only the current visible window,
Display coordinates map first to the visible window, then to the backing
full-file source by adding windowBuff.getIndex().
The new features are fit on top of Tutorial_03_Drawing variables and methods.
Support for WindowedBuffer is flagged with "// *** WindowedBuffer support *** //".
Look for the flag to how TutorialOne_03_Drawing was adapted.
See Tutorial_03_Drawing for information about drawing, audio synthesis,
audio events and animation. Here we'll document just the new features.
This sketch adds a windowed audio buffer to the drawing, interaction and
audio synthesis tools of TutorialOne_03_Drawing. A windowed audio buffer combines an audio buffer with a
method that positions a "window" within a larger source or "backing" buffer and
reads data from it to a separate data structure. TutorialOne_06_WindowBuffer uses
PixelAudio's WindowedBuffer class to store a backing buffer read
from an audio file. It uses the backing buffer as the source array for the
smaller audio buffers audioSignal and playBuffer. We
can use the window function to select which portion of the backing buffer we want
to see and interact with. Data in audioSignal is transcoded as pixel values in
mapImage, an image used in this sketch to provide a visual
representation of the data in the audio buffer. A PixelAudioMapper instance,
mapper, handles the mapping of data between image and audio
structures, and also maps mouse interactions on the display image to sample
indices in the audio buffer.
The Granular and Sampler instruments both use the backing audio buffer directly
as their audio source, though they can also use the smaller audio buffers derived
from the window function. The window that moves through the backing buffer is
exactly the same size as the display image and the smaller audio buffers. It can
be positioned anywhere within the backing buffer. It also can be automatically
moved across the backing buffer at a specified rate, animating the display image.
Interaction with the sketch using the mouse and keyboard takes into account the
indexing of the windowed audio buffer. A mouse click on the display image will
trigger a sound from the sampler or granular instrument by requesting a sample
index from mapper and adding the current window index in the backing buffer to
it, thus obtaining the correct sample index in the backing buffer to forward to
the instrument. It is possible to animate mapImage by pixel-shifting, which
shifts the audio index with respect to the number of pixels shifted. Probably you
won't want to run pixel-shifting and automatic window advancing together, but is
is possible. As with the window index, the pixel shift index will be used to
determine the audio sample index.
QUICK START
Run the sketch. It will load the audio file specified in daPath and daFile, and write
it to the display image as colors overlaid on a visualization of the audio values along
the signal path. You can change the file with the 'o' key command.
The drawing and audio synthesis features are the same as in TutorialOne_03_Drawing,
but now mouse interactions will trigger audio events based on the current position of
the windowed buffer within the backing buffer. You can trigger audio events at different
locations in the audio file by moving the window and clicking on the display image.
Press spacebar or click to trigger audio events at audio index corresponding to the
mouse location. Draw brushstrokes for both Sampler and Granular events.
Press 'T' to toggle automatic window advancement. Press '{' or '}' to jump back/forward
one full window. Press '(' or ')' to jump back/forward one half window. Press 'R' to
rewind the WindowBuffer.
Press 'Y' to toggle raindrop point events while windowing.
Experiment with drawing brushstrokes and triggering audio events while moving through
the buffer.
Here are the key commands for this sketch:
Press UP ARROW to increase audio gain by 3 dB.
Press DOWN ARROW to decrease audio gain by 3 dB.
Press RIGHT ARROW to increase Sampler audio gain by 3 dB.
Press LEFT ARROW to decrease Sampler audio gain by 3 dB.
Press SHIFT-RIGHT ARROW to increase Granular audio gain by 3 dB.
Press SHIFT-LEFT ARROW to decrease Granular audio gain by 3 dB.
Press ' ' (spacebar) to trigger a brush if we're hovering over a brush, otherwise trigger a point event.
Press '1' to set brushstroke under cursor to PathMode ALL_POINTS.
Press '2' to set brushstroke under cursor to PathMode REDUCED_POINTS.
Press '3' to set brushstroke under cursor to PathMode CURVE_POINTS.
Press 't' to set brushstroke under cursor to use Sampler or Granular synth.
Press 'f' to set brushstroke under cursor to FIXED hop mode for granular events.
Press 'g' to set brushstroke under cursor to GESTURE hop mode for granular events.
Press 'y' to select granular or sampler synth for click response.
Press 'r' to select long or short grain duration .
Press 'b' to toggle between long burst grains and short burst grains.
Press 'p' to jitter the pitch of granular gestures.
Press 'P' to adjust pitch of current brushstroke.
Press '>' or '.' to increment epsilon value of current brush (reduced points decrease).
Press '<' or ',' to decrement epsilon value of current brush (reduced points increase).
Press ']' to increment curve steps value of current brush.
Press '[' to decrement curve steps value of current brush.
Press 'm' to change the Sampler noise reduction profile.
Press 'e' to change the envelope we're using .
Press 'E' to toggle whether we adjust envelope duration in relation to gesture duration.
Press 'a' to start or stop animation (bitmap rotation along the signal path).
Press 'd' to toggle play audio on new brush .
Press 'c' to apply color from image file to display image.
Press 'k' to apply the hue and saturation in the colors array to mapImage (not to baseImage).
Press 'K' to apply hue and saturation in colors to baseImage and mapImage.
Press 'j' to turn audio and image blending on or off.
Press 'n' to normalize the audio buffer to -6.0dB.
Press 'L' or 'l' to determine whether to load a new file to both image and audio or not.
Press 'o' or 'O' to open an audio or image file and load to image or audio buffer or both.
Press 's' to save display image to a PNG file.
Press 'S' to save audio buffer to a .wav file.
Press 'w' to write the image HSB Brightness channel to the audio buffer as transcoded sample values .
Press 'W' to write the audio buffer samples to the image as color values.
Press 'x' to delete the current active brush shape or the oldest brush shape.
Press 'X' to delete the most recent brush shape.
Press 'u' to mute audio.
Press 'V' to record a video.
Press 'h' or 'H' to show help message in the console.
WindowBuffer additions:
Press 'T' to toggle automatic WindowBuffer traversal.
Press '{' or '}' to jump back/forward one full window.
Press '(' or ')' to jump back/forward one half window.
Press 'R' to rewind the WindowBuffer.
Press 'Y' to toggle raindrop point events while windowing.
Utility method for applying hue and saturation values from a source array of RGB values
to the brightness values in a target array of RGB values, using a lookup table to redirect indexing.
Apply color map hue and saturation to mapImage or baseImage.
int[]
applyColorShifted(int[] colorSource,
int[] graySource,
int[] lut,
int shift)
Utility method for applying hue and saturation values from a source array of RGB values
to the brightness values in a target array of RGB values, using a lookup table to redirect indexing,
taking into account any pixels that were shifted.
Calculates the display image coordinates corresponding to a specified audio sample index,
use this version when the position is used to calculate local display coordinates.
Initializes a PACurveMaker instance with allPoints as an argument to the factory method
PACurveMaker.buildCurveMaker() and then fills in PACurveMaker instance variables from
variables in the calling class (TutorialOneDrawing, here).
Initializes global variables gParamsGesture and gParamsFixed, which provide basic
settings for granular synthesis the follows gesture timing or fixed hop timing
between grains.
Attempts to load image data from a selected file into mapImage, then calls writeImageToAudio()
to transcode HSB brightness channel to audio and writes it to playBuffer and audioSignal.
Renders a frame of animation: moving along the signal path, copies baseImage pixels to
mapImage pixels, adjusting the index position of the copy using totalShift --
i.e.
Generates an array of rainbow colors using the HSB color space.
Parameters:
size - the number of entries in the colors array
Returns:
an array of RGB colors ordered by hue
initImages
publicvoidinitImages()
Initializes mapImage with the colors array.
MapImage handles the color data for mapper and also serves as our display image.
BaseImage is intended as a reference image that typically only changes when you load a new image.
initDrawing
publicvoidinitDrawing()
Initializes drawing and drawing interaction variables.
Preload an audio file using a file path and a filename.
Parameters:
path - the fully qualified path to the file's directory, ending with a '/'
filename - the name of the file
draw
publicvoiddraw()
Overrides:
draw in class processing.core.PApplet
animate
publicvoidanimate()
stepAnimation
publicvoidstepAnimation()
Step through the animation, called by the draw() method.
Will also record a frame of video, if we're recording.
renderFrame
publicvoidrenderFrame(int step)
Renders a frame of animation: moving along the signal path, copies baseImage pixels to
mapImage pixels, adjusting the index position of the copy using totalShift --
i.e. we don't actually rotate the pixels, we just shift the position they're copied to.
Parameters:
step - current animation step
handleDrawing
publicvoidhandleDrawing()
Handles user's drawing actions: draws previously recorded brushstrokes,
tracks and generates interactive animation and audio events.
a reference to the brushstroke the mouse is over, or null if there's no brushstroke.
updateHover
voidupdateHover()
Update the hoverBrush and hoverIndex global variables.
writeToScreen
publicvoidwriteToScreen(String msg,
int x,
int y,
int weight,
boolean isWhite)
Displays a line of text to the screen, usually in the draw loop. Handy for debugging.
typical call: writeToScreen("When does the mind stop and the world begin?", 64, 1000, 24, true);
Parameters:
msg - message to write
x - x coordinate
y - y coordinate
weight - font weight
isWhite - if true, white text, otherwise, black text
raindrops
publicvoidraindrops(int count)
Random point events along the upper part of the current window, useful for
hearing time-structured files while the window is advancing.
mousePressed
publicvoidmousePressed()
The built-in mousePressed handler for Processing, clicks are handled in mouseClicked().
Overrides:
mousePressed in class processing.core.PApplet
mouseDragged
publicvoidmouseDragged()
Overrides:
mouseDragged in class processing.core.PApplet
mouseReleased
publicvoidmouseReleased()
Overrides:
mouseReleased in class processing.core.PApplet
mouseClicked
publicvoidmouseClicked()
Overrides:
mouseClicked in class processing.core.PApplet
keyPressed
publicvoidkeyPressed()
Built-in keyPressed handler, forwards events to parseKey.
Overrides:
keyPressed in class processing.core.PApplet
keyReleased
publicvoidkeyReleased()
Overrides:
keyReleased in class processing.core.PApplet
parseKey
publicvoidparseKey(char key,
int keyCode)
Handles key press events passed on by the built-in keyPressed method.
By moving key event handling outside the built-in keyPressed method,
we make it possible to post key commands without an actual key event.
Methods and interfaces and even other threads can call parseKey().
This opens up many possibilities and a some risks, too.
Parameters:
key - the key that was pressed, as a char
keyCode - keyCode for the key that was pressed
parseKeyWB
publicvoidparseKeyWB(char key,
int keyCode)
showHelp
publicvoidshowHelp()
to generate help output, run RegEx search/replace on parseKey case lines with:
// case ('.'): // (.+)
// println(" * Press $1 to $2.");
Utility method for applying hue and saturation values from a source array of RGB values
to the brightness values in a target array of RGB values, using a lookup table to redirect indexing.
Parameters:
colorSource - a source array of RGB data from which to obtain hue and saturation values
graySource - an target array of RGB data from which to obtain brightness values
lut - a lookup table, must be the same size as colorSource and graySource
Returns:
the graySource array of RGB values, with hue and saturation values changed
publicint[]applyColorShifted(int[] colorSource,
int[] graySource,
int[] lut,
int shift)
Utility method for applying hue and saturation values from a source array of RGB values
to the brightness values in a target array of RGB values, using a lookup table to redirect indexing,
taking into account any pixels that were shifted.
Parameters:
colorSource - a source array of RGB data from which to obtain hue and saturation values
graySource - an target array of RGB data from which to obtain brightness values
lut - a lookup table, must be the same size as colorSource and graySource
shift - number of pixels/array indices to shift
Returns:
the graySource array of RGB values, with hue and saturation values changed
callback method for chooseFile(), handles standard audio and image formats for Processing.
If a file has been successfully selected, continues with a call to loadAudioFile() or loadImageFile().
Attempts to load audio data from a selected file into playBuffer, then calls
writeAudioToImage() to transcode audio data and write it to mapImage.
If you want to load the image file and audio file separately, comment out writeAudioToImage().
Attempts to load image data from a selected file into mapImage, then calls writeImageToAudio()
to transcode HSB brightness channel to audio and writes it to playBuffer and audioSignal.
Blends audio data from buffer "src" into buffer "dest" in place.
The formula per sample is:
dest[i] = weight * src[i] + (1 - weight) * dest[i]
Parameters:
dest - Destination buffer (will be modified)
src - Source buffer to blend into dest
weight - Blend ratio (0.0 = keep dest, 1.0 = replace with src)
normalize
public staticvoidnormalize(float[] signal,
float targetPeakDB)
Normalizes a single-channel signal array to a target RMS level in dBFS (decibels relative to full scale).
0 is the maximum digital amplitude. -6.0 dB is 50% of the maximum level.
Transcodes audio data in sig[] and writes it to color channel chan of img
using the lookup tables in mapper to redirect indexing. Calls mapper.mapSigToImgShifted(),
which will throw an IllegalArgumentException if sig.length != img.pixels.length
or sig.length != mapper.getSize().
Parameters:
sig - an array of float, should be audio data in the range [-1.0, 1.0]
mapper - a PixelAudioMapper
img - a PImage
chan - a color channel
shift - the number of indices to shift when writing audio
setAlphaWithBlack
publicintsetAlphaWithBlack(int argb,
int alpha)
Sets the alpha channel of an RGBA color, conditionally setting alpha = 0 if all other channels = 0.
This method writes a color channel from an image to playBuffer, fulfilling a
central concept of the PixelAudio library: image is sound. Calls mapper.mapImgToSig(),
which will throw an IllegalArgumentException if img.pixels.length != sig.length or
img.width * img.height != mapper.getWidth() * mapper.getHeight().
Sets totalShift = 0 on completion: the image and audio are now in sync.
Parameters:
img - a PImage, a source of data
mapper - a PixelAudioMapper, handles mapping between image and audio signal
Installs a full-file backing source for WindowedBuffer and instruments.
Sets the window size to mapper.getSize()
refreshWindowFromBacking
voidrefreshWindowFromBacking(boolean advance)
Copies the current window into audioSignal and playBuffer. The Sampler and
Granular instruments still read from anthemSignal / anthemBuffer.
ensureSamplerReady
voidensureSamplerReady()
Ensures that all resources and variable necessary for the Sampler synth are ready to go.
ensureGranularReady
voidensureGranularReady()
Ensures that all resources and variable necessary for the Granular synth are ready to go.
initTimedEventLists
voidinitTimedEventLists()
initGranularParams
publicvoidinitGranularParams()
Initializes global variables gParamsGesture and gParamsFixed, which provide basic
settings for granular synthesis the follows gesture timing or fixed hop timing
between grains.
audioMouseClick
publicvoidaudioMouseClick(int x,
int y)
Handles mouse clicks that happen outside a brushstroke.
Parameters:
x - x-coordinate of mouse click
y - y-coordinate of mouse click
handleClickOutsideBrush
publicinthandleClickOutsideBrush(int x,
int y)
Point clicks are mouse location, but the initial samplePos is the
backing source index. Long granular bursts still create
display points in the current window; playGranularGesture(...) remaps
them to backing indices.
Parameters:
x -
y -
backingGranularSignal
float[]backingGranularSignal()
getSamplePos
publicintgetSamplePos(int x,
int y)
Calculates the index of the image pixel within the signal path,
taking the shifting of pixels and audioSignal into account.
See MusicBoxBuffer for use of a windowed buffer in this calculation.
Parameters:
x - an x coordinate within mapImage and display bounds
y - a y coordinate within mapImage and display bounds
Returns:
the index of the sample corresponding to (x,y) on the signal path
Calculates the display image coordinates corresponding to a specified audio sample index,
use this version when the position is used to calculate local display coordinates.
Parameters:
pos - an index into an audio signal, must be between 0 and width * height - 1.
Bottleneck "commit" method for audio state.
This is the ONLY method that should mutate the global audio signal state.
Modified for use with WindowedBuffer and a backing buffer, installBackingSource()
and refreshWindowFromBacking() set the various buffers. The ensureSamplerReady()
and ensureGranularReady() reset instruments.
In PixelAudio examples, the signal is typically loaded from a file, but
it could also be signal cached in memory, a signal generated by code, audio
captured live, etc.
Parameters:
sig - an audio signal
sourceSampleRate - audio sample rate for sig,
usually obtained when reading from an audio file
totalMs - desired duration of the envelope in milliseconds
Returns:
an ADSRParams envelope
initAllPoints
publicvoidinitAllPoints()
Initializes allPoints and adds the current mouse location to it.
addDrawingPoint
publicvoidaddDrawingPoint(int x,
int y)
While user is dragging the mouses and isDrawMode == true, accumulates new points
to allPoints and event times to allTimes. Sets sampleX, sampleY and samplePos variables.
We constrain points outside the bounds of the display window. An alternative approach
is be to ignore them, which may give a more "natural" appearance for fast drawing.
Parameters:
x - x-coordinate of point to add to allPoints
y - y-coordinate of point to add to allPoints
clipToWidth
publicintclipToWidth(int i)
Clips parameter i to the interval (0..width-1)
Parameters:
i - integer to clip to width
Returns:
value within the range 0..width-1
clipToHeight
publicintclipToHeight(int i)
Clips parameter i to the interval (0..width-1)
Parameters:
i - integer to clip to height
Returns:
value within the range 0..height-1
jitterCoord
publicprocessing.core.PVectorjitterCoord(int x,
int y,
int deviationPx)
Parameters:
x - x-coordinate
y - y-coordinate
deviationPx - distance deviation from mean
Returns:
a PVector with coordinates shifted by a Gaussing variable
Initializes a PACurveMaker instance with allPoints as an argument to the factory method
PACurveMaker.buildCurveMaker() and then fills in PACurveMaker instance variables from
variables in the calling class (TutorialOneDrawing, here).