Package net.paulhertz.pixelaudio.example

Class LoadAudioToImage

java.lang.Object
processing.core.PApplet
net.paulhertz.pixelaudio.example.LoadAudioToImage
All Implemented Interfaces:
processing.core.PConstants
public class LoadAudioToImage extends processing.core.PApplet
LoadAudioToImage shows how you can open audio files and transcode their image data to an image. Once the image is loaded, you can click on it to play back the audio using a PASamplerInstrument. Hovering over the image and pressing the spacebar will also trigger an audio event. The hightlightSample() method highlights the pixels that correspond to the audio signal that is played. The highlight changes the pixels and can change the audio, too: just press the 'w' key to transcode the image to an audio signal and write it to the PASamplerInstrument. You can also load audio to individual RGB or HSB Hue and Brightness channels. To hear the results of loading to different channels, write the image to the audio signal ('w' key) and click in the image.

This sketch also includes some common methods for adjusting the brightness and contrast of an image: gamma adjustment changes contrast up of down and histogram equalization stretches the range of brightness values in an image. These operations will change the audio, too, if you write the image to the audio signal. The gamma operations both make the audio quieter, while the histogram stretch usually makes the audio louder. 

 Press 'o' or 'O' to open an audio file and load it to the signal and all channels of the image.
 Press 'r' to open an audio file and load it to the signal and the RED channel of the image.
 Press 'g' to open an audio file and load it to the signal and the GREEN channel of the image.
 Press 'b' to open an audio file and load it to the signal and the BLUE channel of the image.
 Press 'l' to open an audio file and load it to the signal and the HSB Brightness channel of the image.
 Press 'h' to open an audio file and load it to the signal and the HSB Hue channel of the image.
 Press 'k' to overlay the rainbow signal path on the image.
 Press 'm' to equalize the histogram of the image.
 Press '+' or '=' to apply gamma value  "gammaLighter" to the image.
 Press '-' or '_' to apply gamma value  "gammaDarker" to the image.
 Press 'w' to transcode the image HSB Brightness channel and write it to the signal.
 Press 's' to save the current display image as a PNG file.
 Press '?' to show Help Message in the console.

  • Field Summary

    Fields
    Modifier and Type
    Field
    Description
    (package private) ADSRParams
    adsr
     
    (package private) float
    attackTime
     
    (package private) File
    audioFile
     
    (package private) String
    audioFileName
     
    (package private) String
    audioFilePath
     
    (package private) String
    audioFileTag
     
    (package private) int
    audioLength
     
    (package private) ddf.minim.AudioOutput
    audioOut
     
    (package private) float[]
    audioSignal
     
    (package private) int
    blau
     
    (package private) int
    blendAlpha
     
    (package private) PixelAudioMapper.ChannelNames
    chan
     
    (package private) int[]
    colors
     
    (package private) float
    decayTime
     
    (package private) int
    duration
     
    (package private) float
    gammaDarker
     
    (package private) float
    gammaLighter
     
    (package private) int[]
    gammaTable
     
    (package private) int
    groc
     
    (package private) HilbertGen
    hGen
     
    (package private) int
    histoHigh
     
    (package private) int
    histoLow
     
    (package private) processing.core.PImage
    mapImage
     
    (package private) PixelAudioMapper
    mapper
     
    (package private) int
    mapSize
     
    (package private) float
    maxAmplitude
     
    (package private) ddf.minim.Minim
    minim
    Minim audio library
    (package private) PixelAudio
    pixelaudio
    PixelAudio library
    (package private) int
    pixelPos
     
    (package private) ddf.minim.MultiChannelBuffer
    playBuffer
     
    int[]
    randColors
     
    (package private) Random
    rando
     
    (package private) float
    releaseTime
     
    (package private) int[]
    rgbSignal
     
    (package private) int
    roig
     
    (package private) int
    samplePos
     
    (package private) float
    sampleRate
     
    (package private) float
    sustainLevel
     
    (package private) PASamplerInstrument
    synth
     
    (package private) int
    taronja
     
    (package private) int
    vert
     
    (package private) int
    violet
     

    Fields inherited from class processing.core.PApplet

    args, ARGS_BGCOLOR, ARGS_DISABLE_AWT, ARGS_DISPLAY, ARGS_EDITOR_LOCATION, ARGS_EXTERNAL, ARGS_FULL_SCREEN, ARGS_HIDE_STOP, ARGS_LOCATION, ARGS_PRESENT, ARGS_SKETCH_FOLDER, ARGS_STOP_COLOR, ARGS_UI_SCALE, ARGS_WINDOW_COLOR, DEFAULT_HEIGHT, DEFAULT_WIDTH, disableAWT, displayHeight, displayWidth, dmouseX, dmouseY, emouseX, emouseY, exitCalled, EXTERNAL_MOVE, EXTERNAL_STOP, finished, firstMouse, focused, frameCount, frameRate, frameRateLastNanos, g, height, insideDraw, javaPlatform, javaVersion, javaVersionName, key, keyCode, keyEvent, keyPressed, keyRepeatEnabled, looping, matchPatterns, mouseButton, mouseEvent, mousePressed, mouseX, mouseY, pixelDensity, pixelHeight, pixels, pixelWidth, platform, pmouseX, pmouseY, ratioLeft, ratioScale, ratioTop, recorder, redraw, rheight, rmouseX, rmouseY, rwidth, surface, useNativeSelect, width, windowX, windowY

    Fields inherited from interface processing.core.PConstants

    ADD, ALPHA, ALT, AMBIENT, ARC, ARGB, ARROW, BACKSPACE, BASELINE, BEVEL, BEZIER_VERTEX, BLEND, BLUR, BOTTOM, BOX, BREAK, BURN, CENTER, CHATTER, CHORD, CLAMP, CLOSE, CODED, COMPLAINT, CONTROL, CORNER, CORNERS, CROSS, CURVE_VERTEX, CUSTOM, DARKEST, DEG_TO_RAD, DELETE, DIAMETER, DIFFERENCE, DILATE, DIRECTIONAL, DISABLE_ASYNC_SAVEFRAME, DISABLE_BUFFER_READING, DISABLE_DEPTH_MASK, DISABLE_DEPTH_SORT, DISABLE_DEPTH_TEST, DISABLE_KEY_REPEAT, DISABLE_NATIVE_FONTS, DISABLE_OPENGL_ERRORS, DISABLE_OPTIMIZED_STROKE, DISABLE_STROKE_PERSPECTIVE, DISABLE_STROKE_PURE, DISABLE_TEXTURE_MIPMAPS, DODGE, DOWN, DXF, ELLIPSE, ENABLE_ASYNC_SAVEFRAME, ENABLE_BUFFER_READING, ENABLE_DEPTH_MASK, ENABLE_DEPTH_SORT, ENABLE_DEPTH_TEST, ENABLE_KEY_REPEAT, ENABLE_NATIVE_FONTS, ENABLE_OPENGL_ERRORS, ENABLE_OPTIMIZED_STROKE, ENABLE_STROKE_PERSPECTIVE, ENABLE_STROKE_PURE, ENABLE_TEXTURE_MIPMAPS, ENTER, EPSILON, ERODE, ESC, EXCLUSION, FX2D, GIF, GRAY, GROUP, HALF_PI, HAND, HARD_LIGHT, HINT_COUNT, HSB, IMAGE, INVERT, JAVA2D, JPEG, LANDSCAPE, LEFT, LIGHTEST, LINE, LINE_LOOP, LINE_STRIP, LINES, LINUX, MACOS, MACOSX, MAX_FLOAT, MAX_INT, MIN_FLOAT, MIN_INT, MITER, MODEL, MODELVIEW, MOVE, MULTIPLY, NORMAL, OPAQUE, OPEN, OPENGL, ORTHOGRAPHIC, OTHER, OVERLAY, P2D, P3D, PATH, PDF, PERSPECTIVE, PI, PIE, platformNames, POINT, POINTS, POLYGON, PORTRAIT, POSTERIZE, PROBLEM, PROJECT, PROJECTION, QUAD, QUAD_BEZIER_VERTEX, QUAD_STRIP, QUADRATIC_VERTEX, QUADS, QUARTER_PI, RAD_TO_DEG, RADIUS, RECT, REPEAT, REPLACE, RETURN, RGB, RIGHT, ROUND, SCREEN, SHAPE, SHIFT, SOFT_LIGHT, SPAN, SPHERE, SPOT, SQUARE, SUBTRACT, SVG, TAB, TARGA, TAU, TEXT, THIRD_PI, THRESHOLD, TIFF, TOP, TRIANGLE, TRIANGLE_FAN, TRIANGLE_STRIP, TRIANGLES, TWO_PI, UP, VERTEX, WAIT, WHITESPACE, WINDOWS, X, Y, Z

  • Constructor Summary

    Constructors
    Constructor
    Description
    LoadAudioToImage()
     

  • Method Summary

    Modifier and Type
    Method
    Description
    int[]
    adjustGamma(int[] source)
     
    void
    applyColor(int[] colorSource, int[] graySource, int[] lut)
     
    void
    audioMouseClick(int x, int y)
    Respond to a mouse click with an audio event.
    int
    calcSampleLen(int durationMS)
     
    void
    chooseFile()
     
    int
    clipToHeight(int y)
     
    int
    clipToWidth(int x)
     
    void
    draw()
     
    void
    fileSelected(File selectedFile)
     
    double
    gauss(double mean, double variance)
    Returns a Gaussian variable using a Java library call to Random.nextGaussian.
    int[]
    getColors()
     
    int[]
    getHistoBounds(int[] source)
     
    void
    hightlightSample(int pos, int length)
     
    int[]
    histoStretch(int[] source, int low, int high)
     
    void
    initAudio()
     
    void
    initMapper()
     
    void
    keyPressed()
     
    void
    loadAudioFile(File audioFile)
    Attempts to load audio data from a selected file into playBuffer, then calls writeAudioToImage() to transcode audio data and write it to mapImage
    static void
    main(String[] args)
     
    void
    mouseClicked()
     
    int
    playSample(int samplePos, int sampleCount, float amplitude)
    Plays an audio sample with PASamplerInstrument and default ADSR.
    void
    setGamma(float gamma)
     
    void
    settings()
     
    void
    setup()
     
    void
    showHelp()
     
    void
    shuffle(int[] intArray)
    Shuffles an array of integers into random order.
    void
    writeAudioToImage(float[] sig, PixelAudioMapper mapper, processing.core.PImage img, PixelAudioMapper.ChannelNames chan)
    Transcodes audio data in sig[] and writes it to color channel chan of mapImage using the lookup tables in mapper to redirect indexing.
    void
    writeImageToAudio()
     

    Methods inherited from class processing.core.PApplet

    abs, abs, acos, alpha, ambient, ambient, ambient, ambientLight, ambientLight, append, append, append, append, append, append, applyMatrix, applyMatrix, applyMatrix, applyMatrix, applyMatrix, arc, arc, arraycopy, arraycopy, arraycopy, arrayCopy, arrayCopy, arrayCopy, asin, atan, atan2, attrib, attrib, attrib, attribColor, attribNormal, attribPosition, background, background, background, background, background, background, background, beginCamera, beginContour, beginPGL, beginRaw, beginRaw, beginRecord, beginRecord, beginShape, beginShape, bezier, bezier, bezierDetail, bezierPoint, bezierTangent, bezierVertex, bezierVertex, binary, binary, binary, binary, blend, blend, blendColor, blendMode, blue, box, box, brightness, calcSketchPath, camera, camera, ceil, checkAlpha, checkExtension, choice, choice, circle, clear, clip, color, color, color, color, color, color, color, color, colorMode, colorMode, colorMode, colorMode, concat, concat, concat, concat, concat, concat, concat, constrain, constrain, copy, copy, copy, cos, createFont, createFont, createFont, createGraphics, createGraphics, createGraphics, createImage, createInput, createInput, createInputRaw, createOutput, createOutput, createPath, createPath, createPrimaryGraphics, createReader, createReader, createReader, createShape, createShape, createShape, createWriter, createWriter, createWriter, cursor, cursor, cursor, cursor, curve, curve, curveDetail, curvePoint, curveTangent, curveTightness, curveVertex, curveVertex, dataFile, dataPath, day, debug, degrees, delay, dequeueEvents, desktopFile, desktopPath, die, die, directionalLight, displayDensity, displayDensity, dispose, dist, dist, edge, ellipse, ellipseMode, emissive, emissive, emissive, endCamera, endContour, endPGL, endRaw, endRecord, endShape, endShape, exec, exec, exit, exitActual, exitCalled, exp, expand, expand, expand, expand, expand, expand, expand, expand, expand, expand, expand, expand, expand, expand, expand, expand, expand, expand, fill, fill, fill, fill, fill, fill, filter, filter, filter, floor, flush, focusGained, focusLost, frameMoved, frameRate, frustum, fullScreen, fullScreen, fullScreen, fullScreen, get, get, get, getExtension, getGraphics, getMatrix, getMatrix, getMatrix, getSurface, green, handleDraw, handleKeyEvent, handleMethods, handleMouseEvent, hex, hex, hex, hex, hideMenuBar, hint, hour, hue, image, image, image, imageMode, initSurface, insertFrame, isLooping, join, join, keyPressed, keyReleased, keyReleased, keyTyped, keyTyped, launch, lerp, lerpColor, lerpColor, lightFalloff, lights, lightSpecular, line, line, link, listFiles, listFiles, listPaths, loadBytes, loadBytes, loadBytes, loadFont, loadImage, loadImage, loadJSONArray, loadJSONArray, loadJSONObject, loadJSONObject, loadPixels, loadShader, loadShader, loadShape, loadShape, loadStrings, loadStrings, loadStrings, loadStrings, loadTable, loadTable, loadXML, loadXML, log, loop, mag, mag, main, main, main, makeGraphics, map, mask, match, matchAll, max, max, max, max, max, max, method, millis, min, min, min, min, min, min, minute, modelX, modelY, modelZ, month, mouseClicked, mouseDragged, mouseDragged, mouseEntered, mouseEntered, mouseExited, mouseExited, mouseMoved, mouseMoved, mousePressed, mousePressed, mouseReleased, mouseReleased, mouseWheel, mouseWheel, nf, nf, nf, nf, nf, nf, nfc, nfc, nfc, nfc, nfp, nfp, nfp, nfp, nfs, nfs, nfs, nfs, noClip, noCursor, noFill, noise, noise, noise, noiseDetail, noiseDetail, noiseSeed, noLights, noLoop, norm, normal, noSmooth, noStroke, noTexture, noTint, orientation, ortho, ortho, ortho, parseBoolean, parseBoolean, parseBoolean, parseBoolean, parseByte, parseByte, parseByte, parseByte, parseByte, parseByte, parseByte, parseByte, parseChar, parseChar, parseChar, parseChar, parseFloat, parseFloat, parseFloat, parseFloat, parseFloat, parseFloat, parseFloat, parseInt, parseInt, parseInt, parseInt, parseInt, parseInt, parseInt, parseInt, parseInt, parseInt, parseInt, parseInt, parseJSONArray, parseJSONObject, parseXML, parseXML, pause, perspective, perspective, pixelDensity, point, point, pointLight, pop, popMatrix, popStyle, postEvent, postWindowMoved, postWindowResized, pow, print, print, print, print, print, print, print, print, print, printArray, printCamera, println, println, println, println, println, println, println, println, println, println, println, printMatrix, printProjection, printStackTrace, push, pushMatrix, pushStyle, quad, quadraticVertex, quadraticVertex, radians, random, random, randomGaussian, randomSeed, rect, rect, rect, rectMode, red, redraw, registerMethod, requestImage, requestImage, resetMatrix, resetShader, resetShader, resume, reverse, reverse, reverse, reverse, reverse, reverse, reverse, rotate, rotate, rotateX, rotateY, rotateZ, round, runSketch, runSketch, runSketch, saturation, save, saveBytes, saveBytes, saveBytes, saveFile, saveFrame, saveFrame, saveJSONArray, saveJSONArray, saveJSONObject, saveJSONObject, savePath, saveStream, saveStream, saveStream, saveStream, saveStream, saveStrings, saveStrings, saveStrings, saveTable, saveTable, saveXML, saveXML, scale, scale, scale, screenX, screenX, screenY, screenY, screenZ, second, selectCallback, selectFolder, selectFolder, selectFolder, selectInput, selectInput, selectInput, selectOutput, selectOutput, selectOutput, set, set, setMatrix, setMatrix, setMatrix, setSize, shader, shader, shape, shape, shape, shapeMode, shearX, shearY, shell, shininess, shorten, shorten, shorten, shorten, shorten, shorten, shorten, showDepthWarning, showDepthWarningXYZ, showMethodWarning, showMissingWarning, showSurface, showVariationWarning, sin, size, size, size, sketchDisplay, sketchFile, sketchFullScreen, sketchHeight, sketchOutputPath, sketchOutputStream, sketchPath, sketchPath, sketchPixelDensity, sketchRenderer, sketchSmooth, sketchWidth, sketchWindowColor, smooth, smooth, sort, sort, sort, sort, sort, sort, sort, sort, sort, sort, specular, specular, specular, sphere, sphereDetail, sphereDetail, splice, splice, splice, splice, splice, splice, splice, splice, splice, splice, splice, splice, splice, split, split, splitTokens, splitTokens, spotLight, sq, sqrt, square, start, startSurface, stop, str, str, str, str, str, str, str, str, str, str, stroke, stroke, stroke, stroke, stroke, stroke, strokeCap, strokeJoin, strokeWeight, style, subset, subset, subset, subset, subset, subset, subset, subset, subset, subset, subset, subset, subset, subset, subset, subset, subset, subset, tan, text, text, text, text, text, text, text, text, text, text, text, textAlign, textAlign, textAscent, textDescent, textFont, textFont, textLeading, textMode, textSize, texture, textureMode, textureWrap, textWidth, textWidth, textWidth, thread, tint, tint, tint, tint, tint, tint, translate, translate, triangle, trim, trim, unbinary, unhex, unregisterMethod, updatePixels, updatePixels, urlDecode, urlEncode, vertex, vertex, vertex, vertex, vertex, windowMove, windowMoved, windowRatio, windowResizable, windowResize, windowResized, windowTitle, year

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Field Details

    • pixelaudio

      PixelAudio pixelaudio
      PixelAudio library

    • hGen

      HilbertGen hGen

    • mapper

      PixelAudioMapper mapper

    • mapSize

      int mapSize

    • mapImage

      processing.core.PImage mapImage

    • colors

      int[] colors

    • chan

      PixelAudioMapper.ChannelNames chan

    • minim

      ddf.minim.Minim minim
      Minim audio library

    • audioOut

      ddf.minim.AudioOutput audioOut

    • playBuffer

      ddf.minim.MultiChannelBuffer playBuffer

    • sampleRate

      float sampleRate

    • audioSignal

      float[] audioSignal

    • rgbSignal

      int[] rgbSignal

    • audioLength

      int audioLength

    • duration

      int duration

    • synth

      PASamplerInstrument synth

    • adsr

      ADSRParams adsr

    • maxAmplitude

      float maxAmplitude

    • attackTime

      float attackTime

    • decayTime

      float decayTime

    • sustainLevel

      float sustainLevel

    • releaseTime

      float releaseTime

    • audioFile

      File audioFile

    • audioFilePath

      String audioFilePath

    • audioFileName

      String audioFileName

    • audioFileTag

      String audioFileTag

    • rando

      Random rando

    • roig

      int roig

    • groc

      int groc

    • blau

      int blau

    • vert

      int vert

    • taronja

      int taronja

    • violet

      int violet

    • randColors

      public int[] randColors

    • pixelPos

      int pixelPos

    • samplePos

      int samplePos

    • blendAlpha

      int blendAlpha

    • histoHigh

      int histoHigh

    • histoLow

      int histoLow

    • gammaLighter

      float gammaLighter

    • gammaDarker

      float gammaDarker

    • gammaTable

      int[] gammaTable

  • Constructor Details

    • LoadAudioToImage

      public LoadAudioToImage()

  • Method Details

    • main

      public static void main(String[] args)

    • settings

      public void settings()
      Overrides:
      settings in class processing.core.PApplet

    • setup

      public void setup()
      Overrides:
      setup in class processing.core.PApplet

    • initMapper

      public void initMapper()

    • initAudio

      public void initAudio()

    • getColors

      public int[] getColors()

    • draw

      public void draw()
      Overrides:
      draw in class processing.core.PApplet

    • keyPressed

      public void keyPressed()
      Overrides:
      keyPressed in class processing.core.PApplet

    • showHelp

      public void showHelp()

    • applyColor

      public void applyColor(int[] colorSource, int[] graySource, int[] lut)

    • mouseClicked

      public void mouseClicked()
      Overrides:
      mouseClicked in class processing.core.PApplet

    • audioMouseClick

      public void audioMouseClick(int x, int y)
      Respond to a mouse click with an audio event.

    • clipToWidth

      public int clipToWidth(int x)
      Parameters:
      x - a value to constrain to the current window width
      Returns:
      the constrained value

    • clipToHeight

      public int clipToHeight(int y)
      Parameters:
      y - a value to constrain to the current window height
      Returns:
      the constrained value

    • calcSampleLen

      public int calcSampleLen(int durationMS)
      Returns:
      a length in samples with some Gaussian variation

    • chooseFile

      public void chooseFile()

    • fileSelected

      public void fileSelected(File selectedFile)

    • loadAudioFile

      public void loadAudioFile(File audioFile)
      Attempts to load audio data from a selected file into playBuffer, then calls writeAudioToImage() to transcode audio data and write it to mapImage
      Parameters:
      audioFile - an audio file

    • writeAudioToImage

      public void writeAudioToImage(float[] sig, PixelAudioMapper mapper, processing.core.PImage img, PixelAudioMapper.ChannelNames chan)
      Transcodes audio data in sig[] and writes it to color channel chan of mapImage using the lookup tables in mapper to redirect indexing. Calls mapper.mapSigToImg(), which will throw an IllegalArgumentException if sig.length != img.pixels.length or sig.length != mapper.getSize().
      Parameters:
      sig - an source array of float, should be audio data in the range [-1.0, 1.0]
      mapper - a PixelAudioMapper
      img - a target PImage, modified by audio data in sig
      chan - a color channel

    • playSample

      public int playSample(int samplePos, int sampleCount, float amplitude)
      Plays an audio sample with PASamplerInstrument and default ADSR.
      Parameters:
      samplePos - position of the sample in the audio buffer
      sampleCount - number of samples to play
      amplitude - amplitude of the samples on playback
      Returns:
      the calculated sample length in samples

    • hightlightSample

      public void hightlightSample(int pos, int length)

    • writeImageToAudio

      public void writeImageToAudio()

    • gauss

      public double gauss(double mean, double variance)
      Returns a Gaussian variable using a Java library call to Random.nextGaussian.
      Parameters:
      mean -
      variance -
      Returns:
      a Gaussian-distributed random number with mean mean and variance variance

    • shuffle

      public void shuffle(int[] intArray)
      Shuffles an array of integers into random order. Implements Richard Durstenfeld's version of the Fisher-Yates algorithm, popularized by Donald Knuth. see http://en.wikipedia.org/wiki/Fisher-Yates_shuffle
      Parameters:
      intArray - an array of ints, changed on exit

    • getHistoBounds

      public int[] getHistoBounds(int[] source)

    • histoStretch

      public int[] histoStretch(int[] source, int low, int high)

    • setGamma

      public void setGamma(float gamma)

    • adjustGamma

      public int[] adjustGamma(int[] source)