Package net.paulhertz.pixelaudio

Class PixelAudioMapper

java.lang.Object

net.paulhertz.pixelaudio.PixelAudioMapper

public class PixelAudioMapper
extends Object

PixelAudioMapper maps between 1D "signal" arrays of audio samples formatted as floating point values in the range [-1, 1] and 2D "image" arrays formatted as RGBA integer pixel data. This class is designed to handle one-to-one mappings between signal and image arrays. The mappings are managed by lookup tables (LUTs) created by a separate mapping generator class, PixelMapGen. The values in the LUTs are index numbers of pixels in a bitmap or of samples in a signal. If you think of the signal as a path that visits each pixel in the image, one lookup table, signalToImageLUT, lists the index numbers of each pixel the path visits in the image, in the order that it traverses them. There is a similar lookup table for the image, imageToSignalLUT, that lets you look up the signal value corresponding to each pixel in the image. For example, when you load audio samples to the signal array, signalToImageLUT lets you find the corresponding pixels for each sample and update them to visualize the signal as a 2D image. You can save the image to a file and later load it to a bitmap for display. The pixel values can then be written to an audio buffer using imageToSignalLUT.

Some typical uses for this class include:

• Reading an audio file or audio stream into the signal array and then writing its values, transcoded to RGB integer values, to the image array for display as a visualization.
• Using interaction with an image to trigger audio events at precise locations in a signal.
• Running audio filters on an image-as-signal and writing the results to the image.
• Running image algorithms on a signal-as-image and writing the results back to the signal.
• Synthesizing image data and audio and then animating the data while interactively triggering audio events.

DATA REPRESENTATION

PixelAudioMapper requires image arrays to contain standard 24- or 32-bit RGB or RGBA pixel data, in row major order, with (0,0) at upper left corner. It requires signal arrays to contain values in the range [-1.0, 1.0], a standard format for audio samples.

For the sake of generality, the enclosing classes for image and audio data remain external to PixelAudioMapper, which just works with the arrays of audio samples or image pixel data that they provide. In Processing, PImage wraps image data. You could also use Java's BufferedImage class. I have been using the minim library for audio, (https://code.compartmental.net/minim/). The built-in audio in Processing 4 is definitely also an option. PImage and BufferedImage typically store color pixel data in an array of RGB or RGBA integer formatted values -- exactly what we need. Audio classes use a variety of formats, particularly when reading from files, and provide methods for setting and changing the format of audio sample data.

Image

         Width w, Height h
         Index values {0..(w * h - 1)} point into the pixel array.
         Index to coordinate conversion for row major order with index i, width w, height h:
                i = x + w * y;
                x = i % w; y = i/w   // using integer math;
         Default data format: 24-bit RGB or 32-bit RGBA, for display from a bitmap to a computer monitor.
         RGBA includes an alpha channel A.
 

Signal

         Array with same cardinality as image data array {0..(w * h - 1)}
         Default data format: floating point values in the range  [-1.0, 1.0]
 

LOOKUP TABLES

At their most general, lookup tables or LUTs set up a one-to-one correspondence between two arrays of the same cardinality, independent of the format of their data values. Every element in one array corresponds to exactly one element in the other array. Starting from array A, for an element at index A[i] we find the index of the corresponding element in array B at aToBLUT[i]. An element j in array B has the index of its counterpart in array A at bToALUT[j].

In PixelAudioMapper, we employ two LUTs, signalToImageLUT and imageToSignalLUT, to map elements in signal or image to the corresponding position in image or signal.

        signalToImageLUT: integer values over {0..(h * w - 1)} map a signal array index to a pixel array index
        imageToSignalLUT: integer values over (0..(h * w - 1)} map an image array index to a signal array index
 

In signalToImageLUT, we can get the pixel index in the image for any index in the signal. In imageToSignalLUT, we can get index in the signal for any pixel index in the image.

Each array is the inverse of the other: for an array index i:

        signalToImageLUT[imageToSignalLUT[i]] == i;
        imageToSignalLUT[signalToImageLUT[i]] == i;
 

Image data is always in row major order for PImage, our image data class. Signal values can be mapped to locations in the image in any arbitrary order, as long their coordinates traverse the entire image. A typical reordering might be a zigzag from upper left to lower right of an image, or a space-filling fractal, or even a randomly shuffled order. The coordinates of each pixel in the image are stored as indices (i = x + w * y) in signalToImageLUT.

Once we know the “pixel index” for each value in the signal and have initialized signalToImageLUT, we can initialize imageToSignalLUT:

        for (int i = 0; i < w * h - 1; i++) {
                imageToSignalLUT[signalToImageLUT[i]] = i;
        }
  

The LUTs are generated by a subclass of PixelMapGen that is passed as an argument to the PixelAudioMapper constructor. Each PixelMapGen subclass generates: 1. a set of coordinates for the path traced by the signal over the image, 2. pixelMap for mapping from signal to image (signalToImageLUT in PixelAudioMapper), and 3. sampleMap (imageToSignalLUT in PixelAudioMapper), for mapping from image to signal PixelAudioMapper works with copies of the two LUTs, and can access or obtain a copy of the coordinates if needed. This strategy allows the generator classes to be compact and reusable, while the host class, PixelAudioMapper, can handle exchanges between audio and pixel data using its copies of the LUTs. Note the the pixel array and the signal array length must equal the image size = width * height.

To work with PixelAudioMapper, first create a PixMapGen instance with the width and height of the image you are addressing. The PixMapGen instance will generate the LUTs for its particular mapping for you. You can then pass it to the PixelAudioMapper constructor, which will initialize its variables from copies of the PixMapGen LUTs. Some of the logic behind this process is explained in my notes to the PixMapGen abstract class.

MAPPING AND TRANSCODING

We typically use the LUTs whenever we change the data in the signal or the image and want to write the new values to its counterpart, updating the appearance of the image or the sound of the audio signal. If the values in the arrays are in different formats, we will need to transcode the values from one format to the other. We have two methods, in pseudocode here:

        mapSigToImg             map signal values to the image: img[i] = transcode(sig[imgLUT[i]]);
        mapImgToSig             map image values to the signal: sig[i] = transcode(img[sigLUT[i]]);
 

The img variable in the pseudocode corresponds to an array of RGB data from a bitmap class. The sig variable corresponds to an array of floating point samples from an audio class.

In addition, we can write image or signal values directly, without using the LUTs. This operation transforms the order of the pixel or signal values.

        writeImgToSig   write image values directly to the signal: sig[i] = transcode(img[i]);
        writeSigToImg   write signal values directly to the image: img[i] = transcode(sig[i]);
 

READING AND WRITING SUBARRAYS

// TODO rewrite this section

When we want to work with subarrays of data from the signal or the image, it can be ordered either by the signal or image array order or by mapping with the corresponding LUT. In the case of images, we also have standard methods of reading and writing rectangular selections. We can define some methods to read and write data either in the order determined by the signal or by rectangular areas in the image. We’ll call the signal order methods pluck (read) and plant (write), and the pixel order methods peel (read) and stamp (write).

Arguments to mapping and writing methods are written so that source precedes target. Using this convention, most methods have a unique signature that also indicates how they function. Where there are ambiguities or a need for clarification, I have renamed the function, as in pluckPixelsAsFloat, pluckSamplesAsInt, peelPixelsAsFloat, and peelSamplesAsInt.

ARRAY SHIFTING<?h2>

Standard operations we can perform with the signal array:

   shiftLeft()          an array rotation where index values decrease and wrap around at the beginning
   shiftRight()         an array rotation where index values increase and wrap around at the end
 

Shifting has proved so useful for animation that I am including it in the class. The shift methods also demonstrate how to update the signal and pixel arrays. The WaveSynth class provides other methods for animation.

OTHER OPERATIONS

The following are suggestions for methods that could be implemented using PixelArrayMapper.

• audio synthesis (the WaveSynth algorithm used in the animation for Campos | Temporales)
• pattern generation (the Argosy pattern algorithm for Campos | Temporales, https://vimeo.com/856300250)
• phase shifting, amplitude modulation, etc.
• FFT operations on both image and signal data
• pixel sorting, typically on image data
• blur, sharpen, etc.
• blending images
• mixing signals

UPDATING AUDIO AND IMAGE

As a rule, operations on the signal should be followed by writing to the image, and operations on the image should be followed by writing to the signal. This will keep the values synchronized, even though they have different numerical formats.

In most of the examples that accompany this library, audio data uses the Lightness channel of an HSL representation of the image's RGB data, but this is by no means the only way of doing things. Using the Lightness channel restricts audio data to 8 bits, apt for glitch esthetics, but noisy. It's also possible to maintain high resolution data in the signal by processing image and audio data separately, and writing audio data to the image but not in the other direction.

Finally, it bears mentioning that the image can be treated as simply an interface into an audio buffer, where events such as mouse clicks or drawing and animation trigger audio events but do not modify the audio buffer. Library examples will provide some suggestions for this strategy.

Note: Representing ranges

I am following the convention from mathematical notation of representing closed ranges with [] and open ranges with (). I occasionally lapse into the alternate notations {0..(h * w - 1)} or [0, 255]. When values are integers, it should be understood that the range covers only integer values. Floating point values are continuous, to the limits of digital computation.

See Also:

• PixelMapGen
• WaveSynth

Nested Class Summary

Nested Classes

Modifier and Type	Class	Description
static enum	PixelAudioMapper.ChannelNames	List of available color channels, "L" for lightness, since "B" for brightness is taken

Constructor Summary

Constructors

Constructor	Description
PixelAudioMapper(PixelMapGen gen)	Basic constructor for PixelAudio, sets up all variables.

Method Summary

Modifier and Type	Method	Description
static final int	alphaComponent(int argb)	Returns alpha channel value of a color.
static int	applyAll(float sample, int rgb)
static int	applyAll(int rgbSource, int rgb)
static int	applyAlpha(float sample, int rgb)
static final int	applyAlpha(int rgba, int rgb)	Takes the alpha channel from one color, rgba, and applies it to another color, rgb.
static int	applyBlue(float sample, int rgb)
static int	applyBlue(int rgbSource, int rgb)
static int	applyBrightness(float sample, int rgb)
static int	applyBrightness(int rgbSource, int rgb)
static int	applyGreen(float sample, int rgb)
static int	applyGreen(int rgbSource, int rgb)
static int	applyHue(float sample, int rgb)
static int	applyHue(int rgbSource, int rgb)
static int	applyRed(float sample, int rgb)
static int	applyRed(int rgbSource, int rgb)
static int	applySaturation(float sample, int rgb)
static int	applySaturation(int rgbSource, int rgb)
static float	brightness(int rgb)	Extracts the brightness component from an RGB value.
static float	brightness(int rgb, float[] hsbPixel)	Extracts the brightness component from an RGB value.
static final String	colorString(int argb)
static final int	composeColor(int[] comp)	Creates a Processing ARGB color from r, g, b, values in an array.
static final int	composeColor(int r, int g, int b)	Creates an opaque Processing RGB color from r, g, b values.
static final int	composeColor(int r, int g, int b, int a)	Creates a Processing ARGB color from r, g, b, and alpha channel values.
PixelMapGen	getGenerator()
ArrayList<int[]>	getGeneratorCoordinatesCopy()
String	getGeneratorDescription()
static final int	getGrayscale(int rgb)
int	getHeight()
int[]	getImageToSignalLUT()
int[]	getInverseArray(int[] src)	Call only on arrays containing the values 0..array.length-1, which can be used to reorder bitmaps.
int[]	getSignalToImageLUT()
int	getSize()
int	getWidth()
static float	hue(int rgb)	Extracts the hue component from an RGB value.
static float	hue(int rgb, float[] hsbPixel)	Extracts the hue component from an RGB value.
int[]	lookupCoordinate(int signalPos)
int	lookupPixel(int signalPos)
int	lookupSample(int imagePos)
int	lookupSample(int x, int y)
void	mapImgToSig(int[] img, float[] sig)	Map current image pixel values to the signal, updating the signal array.
void	mapImgToSig(int[] img, float[] sig, PixelAudioMapper.ChannelNames fromChannel)	Map current image pixel values to the signal, updating the signal array, deriving a value from specified color channel of the image.
void	mapSigToImg(float[] sig, int[] img)	Map signal values to the image using all channels (effectively, grayscale).
void	mapSigToImg(float[] sig, int[] img, PixelAudioMapper.ChannelNames toChannel)	Map signal values to a specified channel in the image.
int[]	peelPixels(int[] img, int x, int y, int w, int h)	Copy a rectangular area of pixels in image (row major) order and return it as an array of RGB values.
float[]	peelPixelsAsAudio(int[] img, int x, int y, int w, int h)	Copy a rectangular area of pixels in image (row major) order and return it as an array of audio values (-1.0f..1.0f).
float[]	peelSamples(float[] sig, int x, int y, int w, int h)	Follow the coordinates of rectangle defined by x, y, w, h and return the corresponding signal values.
int[]	peelSamplesAsRGB(float[] sig, int x, int y, int w, int h)
void	plantPixels(float[] sprout, int[] img, int signalPos, int length, PixelAudioMapper.ChannelNames toChannel)	Starting at signalPos, writes length transcoded values from audio sample array sprout into a specified channel of RGB array img, in signal order.
void	plantPixels(int[] sprout, int[] img, int signalPos, int length)	Starting at signalPos, writes length values from RGB array sprout into RGB array img, in signal order.
void	plantPixels(int[] sprout, int[] img, int signalPos, int length, PixelAudioMapper.ChannelNames toChannel)	Starting at signalPos, writes length values from RGB array sprout into a specified channel of RGB array img, in signal order.
void	plantSamples(float[] sprout, float[] sig, int signalPos, int length)	Starting at signalPos, insert length audio samples from source array sprout into target array of audio samples sig.
void	plantSamples(int[] sprout, float[] sig, int signalPos, int length)	Starting at signalPos, insert length transcoded RGB samples from source array sprout into target array of audio samples sig.
void	plantSamples(int[] sprout, float[] sig, int signalPos, int length, PixelAudioMapper.ChannelNames fromChannel)	Starting at signalPos, insert length transcoded RGB samples from channel fromChannel of source array sprout into target array of audio samples sig.
int[]	pluckPixels(int[] img, int signalPos, int length)	Starting at signalPos, reads length values from pixel array img in signal order using signalToImageLUT to redirect indexing and then returns them as an array of RGB pixel values in signal order.
float[]	pluckPixelsAsAudio(int[] img, int signalPos, int length, PixelAudioMapper.ChannelNames fromChannel)	Starting at signalPos, reads length values from pixel array img in signal order using signalToImageLUT to redirect indexing and then returns them as an array of transcoded float values.
float[]	pluckSamples(float[] sig, int signalPos, int length)	Starting at signalPos, reads length values from float array sig and returns a new array of audio values in signal order.
int[]	pluckSamplesAsRGB(float[] sig, int signalPos, int length)	Starting at signalPos, reads length values from float array sig and transcodes and returns them as an RGB array in signal order.
static float[]	pullPixelAudio(int[] rgbPixels, float[] samples, PixelAudioMapper.ChannelNames chan)	Converts an array of pixel channel values to an array of audio sample values, mapping sample values to the interval [-1.0, 1.0], with no remapping of array order.
static float[]	pullPixelAudio(int[] rgbPixels, int[] lut, float[] samples, PixelAudioMapper.ChannelNames chan)	Converts an array of pixel channel values to an array of audio sample values, mapping sample values to the interval [-1.0, 1.0], using a lookup table to change the order of resulting array.
static float	pullPixelAudio(int rgb, PixelAudioMapper.ChannelNames chan)	Converts a pixel channel value to an audio sample value, mapping the result to [-1.0, 1.0].
static float[]	pullRawChannel(int[] rgbPixels, PixelAudioMapper.ChannelNames chan)	Extracts a selected channel from an array of rgb values.
static int[]	pushAudioPixel(float[] buf, int[] rgbPixels, int[] lut, PixelAudioMapper.ChannelNames chan)	Replaces a specified channel in an array of pixel values, rgbPixels, with a value derived from an array of floats, buf, that represent audio samples.
static int[]	pushAudioPixel(float[] buf, int[] rgbPixels, PixelAudioMapper.ChannelNames chan)	Replaces a specified channel in an array of pixel values, rgbPixels, with a value derived from an array of floats, buf, that represent audio samples.
static int	pushAudioPixel(float sample, int rgb, PixelAudioMapper.ChannelNames chan)	Applies an audio value (source) to an RGB color (target) in a selected channel.
static int[]	pushAudioPixel(int[] colors, int[] rgbPixels, int[] lut, PixelAudioMapper.ChannelNames chan)	Replaces a specified channel in an array of pixel values, rgbPixels, with a value derived from an array of RGB values, colors.
static int[]	pushAudioPixel(int[] colors, int[] rgbPixels, PixelAudioMapper.ChannelNames chan)	Replaces a specified channel in an array of pixel values, rgbPixels, with a value derived from another array of RGB values, colors.
static int	pushAudioPixel(int rgbSource, int rgb, PixelAudioMapper.ChannelNames chan)	Applies an RGB color value (source) to an RGB color (target) in a selected channel.
void	regenerate()	Calls generator.generate() to recreate coordinates and LUTs.
int[]	remapPixels(int[] img, int[] lut)
float[]	remapSamples(float[] sig, int[] lut)
static final void	reverseArray(float[] arr, int l, int r)	Reverses an arbitrary subset of an array of floats.
static final void	reverseArray(int[] arr, int l, int r)	Reverses an arbitrary subset of an array of ints.
static final int[]	rgbaComponents(int argb)	Breaks a Processing color into R, G, B and A values in an array.
static final int[]	rgbComponents(int rgb)	Breaks a Processing color into R, G and B values in an array.
static final void	rotateLeft(float[] arr, int d)	Rotates an array of floats left by d values.
static final void	rotateLeft(int[] arr, int d)	Rotates an array of ints left by d values.
static final void	rotateRight(float[] arr, int d)
static final void	rotateRight(int[] arr, int d)
static float	saturation(int rgb)	Extracts the saturation component from an RGB value.
static float	saturation(int rgb, float[] hsbPixel)	Extracts the saturation component from an RGB value.
static int	setAlpha(int argb, int alpha)
void	setGenerator(PixelMapGen newGen)
void	stampPixels(float[] stamp, int[] img, int x, int y, int w, int h, PixelAudioMapper.ChannelNames toChannel)
void	stampPixels(int[] stamp, int[] img, int x, int y, int w, int h)
void	stampPixels(int[] stamp, int[] img, int x, int y, int w, int h, PixelAudioMapper.ChannelNames toChannel)
void	stampSamples(float[] stamp, float[] sig, int x, int y, int w, int h)
void	stampSamples(int[] stamp, float[] sig, int x, int y, int w, int h)
String	toString()
int	transcode(float val)	Converts a float value in the range [-1.0, 1.0] to an int value in the range [0..255].
float	transcode(int val)	Converts an int value in the range [0..255] to a float value in the range [-1.0, 1.0].
void	writeImgToSig(int[] img, float[] sig)	Writes transcoded pixel values directly to the signal, without using a LUT to redirect.
void	writeImgToSig(int[] img, float[] sig, PixelAudioMapper.ChannelNames fromChannel)
void	writeSigToImg(float[] sig, int[] img)
void	writeSigToImg(float[] sig, int[] img, PixelAudioMapper.ChannelNames toChannel)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Details

PixelAudioMapper

public PixelAudioMapper(PixelMapGen gen)

Basic constructor for PixelAudio, sets up all variables.

Parameters:

gen - A PixelMapGenINF instance -- should be initialized already.

Method Details

getWidth

public int getWidth()

Returns:

the width of the image

getHeight

public int getHeight()

Returns:

the height of the image

getSize

public int getSize()

Returns:

the length of the signal array (== length of image pixel array and the LUTs)

toString

public String toString()

Overrides:

toString in class Object

Returns:

a string representation of our data, possibly partial

getSignalToImageLUT

public int[] getSignalToImageLUT()

Returns:

the lookup table that maps an index in the signal to the corresponding pixel index in the image.

getImageToSignalLUT

public int[] getImageToSignalLUT()

Returns:

the lookup table that maps pixel values in the image to the corresponding entry in the signal.

getGeneratorCoordinatesCopy

public ArrayList<int[]> getGeneratorCoordinatesCopy()

getGeneratorDescription

public String getGeneratorDescription()

getGenerator

public PixelMapGen getGenerator()

setGenerator

public void setGenerator(PixelMapGen newGen)

regenerate

public void regenerate()

Calls generator.generate() to recreate coordinates and LUTs.

getInverseArray

public int[] getInverseArray(int[] src)

Call only on arrays containing the values 0..array.length-1, which can be used to reorder bitmaps. The array returned will restore the order.

remapPixels

public int[] remapPixels(int[] img,
 int[] lut)

remapSamples

public float[] remapSamples(float[] sig,
 int[] lut)

mapSigToImg

public void mapSigToImg(float[] sig,
 int[] img)

Map signal values to the image using all channels (effectively, grayscale). On completion, img[] contains new values. The img array and the sig array must be the same size.

Parameters:

sig - source array of floats in the audio range [-1.0, 1.0]

img - target array of RGB pixel values

mapSigToImg

public void mapSigToImg(float[] sig,
 int[] img,
 PixelAudioMapper.ChannelNames toChannel)

Map signal values to a specified channel in the image. On completion, img[] contains new values. The img array and the sig array must be the same size.

Parameters:

sig - an array of floats in the audio range [-1.0, 1.0]

img - an array of RGB pixel values

toChannel - the channel to write transcoded values to

mapImgToSig

public void mapImgToSig(int[] img,
 float[] sig)

Map current image pixel values to the signal, updating the signal array. There are several ways to do this derive a value we want from the image: we use the brightness channel in the HSB color space. On completion, sig[] contains new values. The img array and the sig array must be the same size.

Parameters:

sig - an array of floats in the audio range [-1.0, 1.0]

img - an array of RGB pixel values

mapImgToSig

public void mapImgToSig(int[] img,
 float[] sig,
 PixelAudioMapper.ChannelNames fromChannel)

Map current image pixel values to the signal, updating the signal array, deriving a value from specified color channel of the image. On completion, sig[] contains new values. The img array and the sig array must be the same size.

Parameters:

sig - an array of floats in the audio range [-1.0, 1.0]

img - an array of RGB pixel values

fromChannel - the color channel to get a value from

writeImgToSig

public void writeImgToSig(int[] img,
 float[] sig)

Writes transcoded pixel values directly to the signal, without using a LUT to redirect. V Values are calculated with the standard luminosity equation, gray = 0.3 * red + 0.59 * green + 0.11 * blue.

Parameters:

img - source array of RGB pixel values

sig - target array of audio samples in the range [-1.0, 1.0]

writeImgToSig

public void writeImgToSig(int[] img,
 float[] sig,
 PixelAudioMapper.ChannelNames fromChannel)

Parameters:

img - an array of RGB pixel values, source

sig - an array of audio samples in the range [-1.0, 1.0], target

fromChannel - channel in RGB or HSB color space, from ChannelNames enum

writeSigToImg

public void writeSigToImg(float[] sig,
 int[] img)

Parameters:

sig - an array of audio samples in the range [-1.0, 1.0], source

img - an array of RGB pixel values, target

writeSigToImg

public void writeSigToImg(float[] sig,
 int[] img,
 PixelAudioMapper.ChannelNames toChannel)

Parameters:

sig - an array of audio samples in the range [-1.0, 1.0], source

img - an array of RGB pixel values, target

toChannel - channel in RGB or HSB color space, from ChannelNames enum

transcode

public int transcode(float val)

Converts a float value in the range [-1.0, 1.0] to an int value in the range [0..255].

Parameters:

val - a float value in the range [-1.0, 1.0]

Returns:

an int mapped to the range [0..255]

transcode

public float transcode(int val)

Converts an int value in the range [0..255] to a float value in the range [-1.0, 1.0].

Parameters:

val - an int int he range [0..255]

Returns:

a float mapped to the range [-1.0, 1.0]

lookupSample

public int lookupSample(int x,
 int y)

lookupSample

public int lookupSample(int imagePos)

lookupPixel

public int lookupPixel(int signalPos)

lookupCoordinate

public int[] lookupCoordinate(int signalPos)

pluckPixels

public int[] pluckPixels(int[] img,
 int signalPos,
 int length)

Starting at signalPos, reads length values from pixel array img in signal order using signalToImageLUT to redirect indexing and then returns them as an array of RGB pixel values in signal order. Note that signalPos = this.imageToSignalLUT[x + y * this.width] or

Parameters:

img - array of RGB values, the pixels array from a bitmap image with the same width and height as PixelAudioMapper

signalPos - position in the signal at which to start reading pixel values from the image, following the signal path

length - length of the subarray to pluck from img, reading pixel values while following the signal path

Returns:

a new array of pixel values in signal order

pluckPixelsAsAudio

public float[] pluckPixelsAsAudio(int[] img,
 int signalPos,
 int length,
 PixelAudioMapper.ChannelNames fromChannel)

Starting at signalPos, reads length values from pixel array img in signal order using signalToImageLUT to redirect indexing and then returns them as an array of transcoded float values. Note that signalPos = this.imageToSignalLUT[x + y * this.width] or this.lookupSample(x, y). Starting at image coordinates (x, y), reads values from pixel array img using imageToSignalLUT to redirect indexing and returns them as an array of transcoded audio values in signal order.

Parameters:

img - source array of RGB pixel values, typically from the bitmap image you are using with PixelAudioMapper

signalPos - position in the signal at which to start reading pixel values from the image, following the signal path

length - length of the subarray to pluck from img

fromChannel - the color channel from which to read pixel values

Returns:

a new array of audio values in signal order

pluckSamples

public float[] pluckSamples(float[] sig,
 int signalPos,
 int length)

Starting at signalPos, reads length values from float array sig and returns a new array of audio values in signal order. No redirection is needed when reading from the signal. Note that signalPos = this.imageToSignalLUT[x + y * this.width] or this.lookupSample(x, y).

Parameters:

sig - source array of audio values

signalPos - a position in the sig array

length - number of values to read from sig array

Returns:

a new array with the audio values we read

pluckSamplesAsRGB

public int[] pluckSamplesAsRGB(float[] sig,
 int signalPos,
 int length)

Starting at signalPos, reads length values from float array sig and transcodes and returns them as an RGB array in signal order. Note that signalPos = this.imageToSignalLUT[x + y * this.width] or this.lookupSample(x, y).

Parameters:

sig - source array of audio values (-1.0f..1.0f)

signalPos - entry point in the sig array

length - number of values to read from the sig array

Returns:

an array of RGB values where r == g == b, derived from the sig values

plantPixels

public void plantPixels(int[] sprout,
 int[] img,
 int signalPos,
 int length)

Starting at signalPos, writes length values from RGB array sprout into RGB array img, in signal order. img is typically the pixel array from a PImage with the same width and height as PixemAudioMapper. Note that signalPos = this.imageToSignalLUT[x + y * this.width] or this.lookupSample(x, y).

Parameters:

sprout - source array of RGB values to insert into target array img, in signal order

img - target array of RGB values. in image (row major) order

signalPos - position in the signal at which to start writing pixel values to the image, following the signal path

length - number of values from sprout to insert into img array

plantPixels

public void plantPixels(int[] sprout,
 int[] img,
 int signalPos,
 int length,
 PixelAudioMapper.ChannelNames toChannel)

Starting at signalPos, writes length values from RGB array sprout into a specified channel of RGB array img, in signal order. Note that signalPos = this.imageToSignalLUT[x + y * this.width] or this.lookupSample(x, y).

Parameters:

sprout - source array of RGB values to insert into target array img, in signal order

img - target array of RGB values, in image order

signalPos - position in the signal at which to start writing pixel values to the image, following the signal path

length - number of values from sprout to insert into img array

toChannel - color channel to write the sprout values to

plantPixels

public void plantPixels(float[] sprout,
 int[] img,
 int signalPos,
 int length,
 PixelAudioMapper.ChannelNames toChannel)

Starting at signalPos, writes length transcoded values from audio sample array sprout into a specified channel of RGB array img, in signal order. Note that signalPos = this.imageToSignalLUT[x + y * this.width] or this.lookupSample(x, y). Inserts elements from a source array of audio values (-1.0f..1.0f) into a specified color channel in a target array of RGB pixel values following the signal path.

Parameters:

sprout - source array of RGB values to insert into target array img, in signal order

img - target array of RGB values, in image order

signalPos - position in the signal at which to start writing pixel values to the image, following the signal path

length - number of values from sprout to insert into img array

toChannel - color channel to write the sprout values to

plantSamples

public void plantSamples(float[] sprout,
 float[] sig,
 int signalPos,
 int length)

Starting at signalPos, insert length audio samples from source array sprout into target array of audio samples sig. No redirection by lookup tables is used.

Parameters:

sprout - source array of audio values (-1.0f..1.0f)

sig - target array of signal values, in signal order

signalPos - start point in sig array

length - number of values to copy from sprout array to sig array

plantSamples

public void plantSamples(int[] sprout,
 float[] sig,
 int signalPos,
 int length)

Starting at signalPos, insert length transcoded RGB samples from source array sprout into target array of audio samples sig. No redirection by lookup tables is is used.

Parameters:

sprout -

sig -

signalPos -

length -

plantSamples

public void plantSamples(int[] sprout,
 float[] sig,
 int signalPos,
 int length,
 PixelAudioMapper.ChannelNames fromChannel)

Starting at signalPos, insert length transcoded RGB samples from channel fromChannel of source array sprout into target array of audio samples sig. No redirection by lookup tables is is used.

Parameters:

sprout -

sig -

signalPos -

length -

fromChannel -

peelPixels

public int[] peelPixels(int[] img,
 int x,
 int y,
 int w,
 int h)

Copy a rectangular area of pixels in image (row major) order and return it as an array of RGB values. This is a standard image method, for example, public PImage get(int x, int y, int w, int h) in Processing. TODO How much error checking do we want in the pluck/plant/peel/stamp methods? If we check, do we fall through or post an error message?

Parameters:

img -

x -

y -

w -

h -

Returns:

peelPixelsAsAudio

public float[] peelPixelsAsAudio(int[] img,
 int x,
 int y,
 int w,
 int h)

Copy a rectangular area of pixels in image (row major) order and return it as an array of audio values (-1.0f..1.0f).

Parameters:

img -

x -

y -

w -

h -

Returns:

peelSamples

public float[] peelSamples(float[] sig,
 int x,
 int y,
 int w,
 int h)

Follow the coordinates of rectangle defined by x, y, w, h and return the corresponding signal values.

Parameters:

sig -

x -

y -

w -

h -

Returns:

peelSamplesAsRGB

public int[] peelSamplesAsRGB(float[] sig,
 int x,
 int y,
 int w,
 int h)

stampPixels

public void stampPixels(int[] stamp,
 int[] img,
 int x,
 int y,
 int w,
 int h)

stampPixels

public void stampPixels(int[] stamp,
 int[] img,
 int x,
 int y,
 int w,
 int h,
 PixelAudioMapper.ChannelNames toChannel)

stampPixels

public void stampPixels(float[] stamp,
 int[] img,
 int x,
 int y,
 int w,
 int h,
 PixelAudioMapper.ChannelNames toChannel)

stampSamples

public void stampSamples(float[] stamp,
 float[] sig,
 int x,
 int y,
 int w,
 int h)

stampSamples

public void stampSamples(int[] stamp,
 float[] sig,
 int x,
 int y,
 int w,
 int h)

rotateLeft

public static final void rotateLeft(int[] arr,
 int d)

Rotates an array of ints left by d values. Uses efficient "Three Rotation" algorithm.

Parameters:

arr - array of ints to rotate

d - number of elements to shift, positive for shift left, negative for shift right

rotateRight

public static final void rotateRight(int[] arr,
 int d)

rotateLeft

public static final void rotateLeft(float[] arr,
 int d)

Rotates an array of floats left by d values. Uses efficient "Three Rotation" algorithm.

Parameters:

arr - array of floats to rotate

d - number of elements to shift, positive for shift left, negative for shift right

rotateRight

public static final void rotateRight(float[] arr,
 int d)

reverseArray

public static final void reverseArray(int[] arr,
 int l,
 int r)

Reverses an arbitrary subset of an array of ints.

Parameters:

arr - array to modify

l - left bound of subset to reverse

r - right bound of subset to reverse

reverseArray

public static final void reverseArray(float[] arr,
 int l,
 int r)

Reverses an arbitrary subset of an array of floats.

Parameters:

arr - array to modify

l - left bound of subset to reverse

r - right bound of subset to reverse

rgbComponents

public static final int[] rgbComponents(int rgb)

Breaks a Processing color into R, G and B values in an array.

Parameters:

rgb - a Processing color as a 32-bit integer

Returns:

an array of integers in the range [0, 255] for 3 primary color components: {R, G, B}

rgbaComponents

public static final int[] rgbaComponents(int argb)

Breaks a Processing color into R, G, B and A values in an array.

Parameters:

argb - a Processing color as a 32-bit integer

Returns:

an array of integers in the intRange [0, 255] for 3 primary color components: {R, G, B} plus alpha

alphaComponent

public static final int alphaComponent(int argb)

Returns alpha channel value of a color.

Parameters:

argb - a Processing color as a 32-bit integer

Returns:

an int for alpha channel

applyAlpha

public static final int applyAlpha(int rgba,
 int rgb)

Takes the alpha channel from one color, rgba, and applies it to another color, rgb.

Parameters:

rgba - The source color from which we get the alpha channel (a mask, for instance)

rgb - The target color we want to change

Returns:

A color with the RGB values from rgb and the A value from rgba

setAlpha

public static int setAlpha(int argb,
 int alpha)

composeColor

public static final int composeColor(int r,
 int g,
 int b,
 int a)

Creates a Processing ARGB color from r, g, b, and alpha channel values. Note the order of arguments, the same as the Processing color(value1, value2, value3, alpha) method.

Parameters:

r - red component [0, 255]

g - green component [0, 255]

b - blue component [0, 255]

a - alpha component [0, 255]

Returns:

a 32-bit integer with bytes in Processing format ARGB.

composeColor

public static final int composeColor(int r,
 int g,
 int b)

Creates an opaque Processing RGB color from r, g, b values. Note the order of arguments, the same as the Processing color(value1, value2, value3) method.

Parameters:

r - red component [0, 255]

g - green component [0, 255]

b - blue component [0, 255]

Returns:

a 32-bit integer with bytes in Processing format ARGB.

composeColor

public static final int composeColor(int[] comp)

Creates a Processing ARGB color from r, g, b, values in an array.

Parameters:

comp - array of 3 integers in range [0, 255], for red, green and blue components of color alpha value is assumed to be 255

Returns:

a 32-bit integer with bytes in Processing format ARGB.

getGrayscale

public static final int getGrayscale(int rgb)

Parameters:

rgb - an RGB color value

Returns:

a number in the range [0, 255] equivalent to the luminosity value rgb

colorString

public static final String colorString(int argb)

Parameters:

argb - an RGB color

Returns:

a String equivalent to a Processing color(r, g, b, a) call, such as "color(233, 144, 89, 255)"

hue

public static float hue(int rgb,
 float[] hsbPixel)

Extracts the hue component from an RGB value. The result is in the range (0, 1).

Parameters:

rgb - The RGB color from which we will obtain the hue component in the HSB color model.

hsbPixel - array of float values filled in by this method

Returns:

A floating point number in the range (0..1) that can be multiplied by 360 to get the hue angle.

hue

public static float hue(int rgb)

Extracts the hue component from an RGB value. The result is in the range (0, 1).

Parameters:

rgb - The RGB color from which we will obtain the hue component in the HSB color model.

Returns:

A floating point number in the range (0..1) that can be multiplied by 360 to get the hue angle.

saturation

public static float saturation(int rgb,
 float[] hsbPixel)

Extracts the saturation component from an RGB value. The result is in the range (0, 1).

Parameters:

rgb - The RGB color from which we will obtain the hue component in the HSB color model.

hsbPixel - array of float values filled in by this method

Returns:

A floating point number in the range (0, 1) representing the saturation component of an HSB color.

saturation

public static float saturation(int rgb)

Extracts the saturation component from an RGB value. The result is in the range (0, 1).

Parameters:

rgb - The RGB color from which we will obtain the hue component in the HSB color model.

Returns:

A floating point number in the range (0, 1) representing the saturation component of an HSB color.

brightness

public static float brightness(int rgb,
 float[] hsbPixel)

Extracts the brightness component from an RGB value. The result is in the range (0, 1).

Parameters:

rgb - The RGB color from which we will obtain the hue component in the HSB color model.

hsbPixel - array of float values filled in by this method

Returns:

A floating point number in the range (0, 1) representing the brightness component of an HSB color.

brightness

public static float brightness(int rgb)

Extracts the brightness component from an RGB value. The result is in the range (0, 1).

Parameters:

rgb - The RGB color from which we will obtain the hue component in the HSB color model.

Returns:

A floating point number in the range (0, 1) representing the brightness component of an HSB color.

applyBrightness

public static int applyBrightness(float sample,
 int rgb)

applyHue

public static int applyHue(float sample,
 int rgb)

applySaturation

public static int applySaturation(float sample,
 int rgb)

applyRed

public static int applyRed(float sample,
 int rgb)

applyGreen

public static int applyGreen(float sample,
 int rgb)

applyBlue

public static int applyBlue(float sample,
 int rgb)

applyAlpha

public static int applyAlpha(float sample,
 int rgb)

applyAll

public static int applyAll(float sample,
 int rgb)

applyBrightness

public static int applyBrightness(int rgbSource,
 int rgb)

applyHue

public static int applyHue(int rgbSource,
 int rgb)

applySaturation

public static int applySaturation(int rgbSource,
 int rgb)

applyRed

public static int applyRed(int rgbSource,
 int rgb)

applyGreen

public static int applyGreen(int rgbSource,
 int rgb)

applyBlue

public static int applyBlue(int rgbSource,
 int rgb)

applyAll

public static int applyAll(int rgbSource,
 int rgb)

pullPixelAudio

public static float pullPixelAudio(int rgb,
 PixelAudioMapper.ChannelNames chan)

Converts a pixel channel value to an audio sample value, mapping the result to [-1.0, 1.0].

Parameters:

rgb - an RGB pixel value

chan - channel to extract from the RGB pixel value

Returns:

pullPixelAudio

public static float[] pullPixelAudio(int[] rgbPixels,
 float[] samples,
 PixelAudioMapper.ChannelNames chan)

Converts an array of pixel channel values to an array of audio sample values, mapping sample values to the interval [-1.0, 1.0], with no remapping of array order.

Parameters:

rgbPixels - an array of RGB pixel values

samples - an array of audio samples, which may be null, whose values will be set from rgbPixels

chan - channel to extract from the RGB pixel values Will be initialized and returned if null

Returns:

a array of floats mapped to the audio range, assigned to samples

pullPixelAudio

public static float[] pullPixelAudio(int[] rgbPixels,
 int[] lut,
 float[] samples,
 PixelAudioMapper.ChannelNames chan)

Converts an array of pixel channel values to an array of audio sample values, mapping sample values to the interval [-1.0, 1.0], using a lookup table to change the order of resulting array.

Parameters:

rgbPixels - an array of RGB pixel values

lut - a lookup table for redirecting rgbPixels indexing, typically imageToSignalLUT

chan - channel to extract from the RGB pixel values

samples - an array of audio samples, which may be null, whose values will be set from rgbPixels. Will be initialized and returned if null.

Returns:

a array of floats mapped to the audio range, identical to samples

pullRawChannel

public static float[] pullRawChannel(int[] rgbPixels,
 PixelAudioMapper.ChannelNames chan)

Extracts a selected channel from an array of rgb values. The values returned are within the ranges expected for the channel requested: (0..1) for HSB and [0, 255] for RGB. If you want to use RGB channels as signal values, you'll need to map their range to [-1.0, 1.0]. If you want to use them as integer values, you'll need to typecast them, for examnple: int redValue = (int) floatArray[i];

Parameters:

rgbPixels - rgb values in an array of int

chan - the channel to extract, a value from the ChannelNames enum

Returns:

the extracted channel values as an array of floats

pushAudioPixel

public static int pushAudioPixel(float sample,
 int rgb,
 PixelAudioMapper.ChannelNames chan)

Applies an audio value (source) to an RGB color (target) in a selected channel. The audio is mapped from the interval (-1.0, 1.0) to (0..255).

Parameters:

sample - an audio sample

rgb - an RGB color

chan - a ChannelNames value selecting the channel to use

Returns:

an RGB color with the selected channel modified by the sample value

pushAudioPixel

public static int[] pushAudioPixel(float[] buf,
 int[] rgbPixels,
 PixelAudioMapper.ChannelNames chan)

Replaces a specified channel in an array of pixel values, rgbPixels, with a value derived from an array of floats, buf, that represent audio samples. Upon completion, the pixel array rgbPixels contains the new values, always in the RGB color space.

Both arrays, rgbPixels and buf, must be the same size.

In the HSB color space, values are assumed to be floats in the range (0..1), so the values from buf need to be mapped to the correct ranges for HSB or RGB [0, 255]. We do some minimal limiting of values derived from buf[], but it is the caller's responsibility to constrain them to the audio range [-1.0, 1.0].

Parameters:

rgbPixels - an array of pixel values

buf - an array of floats in the range (-1.0, 1.0)

chan - the channel to replace

Returns:

rgbPixels with the selected channel modified by the buf values

pushAudioPixel

public static int[] pushAudioPixel(float[] buf,
 int[] rgbPixels,
 int[] lut,
 PixelAudioMapper.ChannelNames chan)

Replaces a specified channel in an array of pixel values, rgbPixels, with a value derived from an array of floats, buf, that represent audio samples. The supplied lookup table, lut, is intended to redirect the indexing of rgbPixels following the signal path. We are stepping through the buf array (the signal), so rgbPixels employs imageToSignalLUT to find where each index i into buf is pointing in the image pixels array, which is rgbPixels. Upon completion, the pixel array rgbPixels contains the new values, always in the RGB color space.

All three arrays, rgbPixels, buf, and lut must be the same size.

In the HSB color space, values are assumed to be floats in the range (0..1), so the values from buf need to be mapped to the correct ranges for HSB or RGB [0, 255]. We do some minimal limiting of values derived from buf[], but it is the caller's responsibility to constrain them to the audio range [-1.0, 1.0].

Parameters:

rgbPixels - an array of pixel values

buf - an array of floats in the range [-1.0, 1.0]

lut - a lookup table to redirect the indexing of the buf, typically imageToPixelsLUT

chan - the channel to replace

Returns:

rgbPixels with selected channel values modified by the buf values

pushAudioPixel

public static int pushAudioPixel(int rgbSource,
 int rgb,
 PixelAudioMapper.ChannelNames chan)

Applies an RGB color value (source) to an RGB color (target) in a selected channel.

Parameters:

rgbSource - an RGB color sample

rgb - an RGB color

chan - a ChannelNames value selecting the channel to use

Returns:

an RGB color with the selected channel modified by the sample value

pushAudioPixel

public static int[] pushAudioPixel(int[] colors,
 int[] rgbPixels,
 PixelAudioMapper.ChannelNames chan)

Replaces a specified channel in an array of pixel values, rgbPixels, with a value derived from another array of RGB values, colors. Upon completion, the pixel array rgbPixels contains the new values, always in the RGB color space.

Both arrays, rgbPixels and colors, must be the same size.

For operations in the HSB color space, HSB values will be extracted from the RGB colors and mapped to the correct ranges for HSB or RGB [0, 255].

Parameters:

colors - an array of RGB colors

rgbPixels - an array of pixel values

chan - the channel to replace

Returns:

rgbPixels with the selected channel modified by the colors array values

pushAudioPixel

public static int[] pushAudioPixel(int[] colors,
 int[] rgbPixels,
 int[] lut,
 PixelAudioMapper.ChannelNames chan)

Replaces a specified channel in an array of pixel values, rgbPixels, with a value derived from an array of RGB values, colors. The supplied lookup table, lut, is intended to redirect the indexing of rgbPixels following the signal path. We are stepping through the color array (the RGB signal), so rgbPixels employs imageToSignalLUT to find where each index i into colors is pointing in the image pixels array, rgbPixels. Upon completion, the pixel array rgbPixels contains the new values, always in the RGB color space.

All three arrays, rgbPixels, colors, and lut must be the same size.

In the HSB color space, HSB values will be extracted from the RGB colors and mapped to the correct ranges for HSB or RGB [0, 255].

Parameters:

colors - an array of RGB values

rgbPixels - an array of pixel values

lut - a lookup table to redirect indexing, typically imageToPixelsLUT

chan - the channel to replace

Returns:

rgbPixels with selected channel values modified by the buf values