Image Filtering

This article will cover a basic image filter using both javascript and webgl shaders.

Javascript Filter

We can first do the filter implementation using p5 and javascript.

Displaying the image

Preload your image from a file (png, jpg, etc).

sketch.js

let img;

function preload() {
    img = loadImage("./loic.png");
}

Set the size of the canvas using the same size as your image.

sketch.js

function setup() {
    createCanvas(img.width, img.height);
}

Draw your image by changing the canvas pixels color value for each pixel to the same value as the image.

Start by loading both of the pixels arrays.

sketch.js

function setup() {
    // ...

    img.loadPixels();
    loadPixels();
}

Loop through the pixels array based on the width and height of the canvas.

sketch.js

function setup() {
    // ...

    for (let x = 0; x < width; x++) {
        for (let y = 0; y < height; y++) {
            // ...
        }
    }
}

Find the index of the pixel using the density of the screen/canvas.

sketch.js

function setup() {
    // ...
    let d = pixelDensity();

    for (let x = 0; x < width; x++) {
        for (let y = 0; y < height; y++) {
            let index = 4 * ((d + y) * width * d + (d + x));
            // ...
        }
    }
}

Get the r, g, b and a value of the pixel of the image.

sketch.js

function setup() {
    // ...
    let d = pixelDensity();

    for (let x = 0; x < width; x++) {
        for (let y = 0; y < height; y++) {
            let index = 4 * ((d + y) * width * d + (d + x));
            let r = img.pixels[index];
            let g = img.pixels[index + 1];
            let b = img.pixels[index + 2];
            let a = img.pixels[index + 3];
            // ...
        }
    }
}

Change the color of the pixel of the canvas.

sketch.js

function setup() {
    // ...
    let d = pixelDensity();

    for (let x = 0; x < width; x++) {
        for (let y = 0; y < height; y++) {
            let index = 4 * ((d + y) * width * d + (d + x));
            let r = img.pixels[index];
            let g = img.pixels[index + 1];
            let b = img.pixels[index + 2];
            let a = img.pixels[index + 3];

            pixels[index] = r;
            pixels[index + 1] = g;
            pixels[index + 2] = b;
            pixels[index + 3] = a;
        }
    }
}

Finally, update the canvas pixels.

sketch.js

function setup() {
    // ...
    let d = pixelDensity();

    for (let x = 0; x < width; x++) {
        for (let y = 0; y < height; y++) {
            let index = 4 * ((d + y) * width * d + (d + x));
            let r = img.pixels[index];
            let g = img.pixels[index + 1];
            let b = img.pixels[index + 2];
            let a = img.pixels[index + 3];

            pixels[index] = r;
            pixels[index + 1] = g;
            pixels[index + 2] = b;
            pixels[index + 3] = a;
        }
    }

    updatePixels();
}

loic

Creating a chromatic filter

A chromatic image is basically an image without color, or in other words, the only colors are shades of grey.

Create a variable to store the chromatic value. We calculate it by using the brightness of the pixels, so to result 1 value from 3 (r, g, b) we must calculate the average of the 3 values.

const cumulative = (r + g + b) / 3;

Let's update the existing code within the loops to use the new chromatic value for each of the color channel (r, g, b).

sketch.js

function setup() {
    // ...
    for (let x = 0; x < width; x++) {
        for (let y = 0; y < height; y++) {
            // ...

            pixels[index] = cumulative;
            pixels[index + 1] = cumulative;
            pixels[index + 2] = cumulative;
            pixels[index + 3] = a;
        }
    }
    // ...
}

loic

We can also set a threshold where which the color of the pixel will either be black or white.

// 0 = black
// 255 = white

const threshold = 150;
cumulative = cumulative > threshold ? 255 : 0;

loic

Smoothen the bright part of the threshold by mapping its original values to a given range.

const threshold = 150;
cumulative = cumulative > threshold ? map(cumulative, 150, 255, 50, 255) : 0;

Which will result in a more detailed image.

loic

We can also try creating a different filter. We will be making a glitched effect.

Now instead of setting a value to each canvas pixels directly, lets create points and draw those with the proper color.

sketch.js

function setup() {
    // ...

    loadPixels(); // line removed

    for (let x = 0; x < width; x++) {
        for (let y = 0; y < height; y++) {
            // ...

            stroke(cumulative, a);
            point(x, y);

            pixels[index] = cumulative; // line removed
            pixels[index + 1] = cumulative; // line removed
            pixels[index + 2] = cumulative; // line removed
            pixels[index + 3] = a; // line removed
        }
    }
    
    updatePixels(); // line removed
}

Add a bit of randomness to the size of the points.

sketch.js

function setup() {
    // ...

    for (let x = 0; x < width; x++) {
        for (let y = 0; y < height; y++) {
            // ...

            strokeWeight(random(3, 5));
            stroke(cumulative, a);
            point(x, y);
        }
    }
}

Change the points to lines and set the second x to the current x + a random amount.

sketch.js

const lineHalfMaxLength = 100;

function setup() {
    // ...
    
    for (let x = 0; x < width; x++) {
        for (let y = 0; y < height; y++) {
            // ...

            strokeWeight(random(3, 5));
            stroke(cumulative, a);
            line(x, y, x + random(-lineHalfMaxLength, lineHalfMaxLength), y);
            point(i, j); // line removed
        }
    }
}

loic

Now we can try adding back the color.

sketch.js

const lineHalfMaxLength = 100;

function setup() {
    // ...
    
    for (let x = 0; x < width; x++) {
        for (let y = 0; y < height; y++) {
            // ...

            const cumulative = (r + g + b) / 3;
            strokeWeight(random(3, 5));
            if (threshold < cumulative) {
                stroke(r, g, b, a);
            } else {
                stroke(0, a);
            }
            line(x, y, x + random(-lineHalfMaxLength, lineHalfMaxLength), y);
        }
    }
}

loic

We can try a different variation for the filter.

Let's use the color of the image and only apply it to the pixels where the chromatic value is higher than the threshold.

sketch.js

const threshold = 150;
let img;

function preload() {
    img = loadImage("loic.jpeg");
}

function setup() {
    createCanvas(img.width, img.height);

    img.loadPixels();
    loadPixels();

    let d = pixelDensity();

    for (let x = 0; x < width; x++) {
        for (let y = 0; y < height; y++) {
            const index = 4 * ((d + y) * width * d + (d + x));
            const r = img.pixels[index];
            const g = img.pixels[index + 1];
            const b = img.pixels[index + 2];
            const a = img.pixels[index + 3];

            const cumulative = (r + g + b) / 3;

            if (cumulative > threshold) {
                pixels[index] = img.pixels[index];
                pixels[index + 1] = img.pixels[index + 1];
                pixels[index + 2] = img.pixels[index + 2];
            } else {
                pixels[index] = 0;
                pixels[index + 1] = 0;
                pixels[index + 2] = 0;
            }
            pixels[index + 3] = a;
        }
    }

    updatePixels();
}

loic

We could also map the colored pixels values.

sketch.js

const threshold = 165;

// ...

function setup() {
    // ...

    for (let x = 0; x < width; x++) {
        for (let y = 0; y < height; y++) {
            // ...

            if (cumulative > threshold) {
                pixels[index] = map(img.pixels[index], threshold, 255, 0, 255);
                pixels[index + 1] = map(img.pixels[index + 1], threshold, 255, 0, 255);
                pixels[index + 2] = map(img.pixels[index + 2], threshold, 255, 0, 255);
            } else {
                // ...
            }
            // ...
        }
    }

    // ...
}

loic

WebGL Shader

Why? Well currently our filter is running on the CPU, which is really powerful, but can't do a lot of processes/calculations all at once. So instead we can convert our current filter to be run on the GPU which is not as powerful as the CPU, but allows for a lot more processes to be run at once.

Displaying the image

We first need to create a shader file, which will be used to run our filter on the GPU.

Create a shader.vert file (vertex shader) that will contain a main function.

shader.vert

void main() {
    // ...
}

The vertex shader will be used to define the position of the pixels on the canvas that we want to update the color for.

Set the position in the vertex shader

shader.vert

attribute vec3 aPosition;

void main() {
    gl_Position = vec4(aPosition, 1.0);
}

We have now introduced a new variable aPosition which will be used to set the position of the pixels on the canvas. Note that this variable is provided by p5.js when using the shader() function.

Create a shader.frag file (fragment shader) that will also contain a main function.

shader.frag

void main() {
    // ...
}

The fragment shader will be used to define the color of the pixels on the canvas that we want to update.

Set the color in the fragment shader.

shader.frag

void main() {
    gl_FragColor = vec4(1.0, 0.0, 0.0, 1.0);
}

We have now set the color of the pixels to red for each pixel.

Now let's update our sketch to use those new shaders.

sketch.js

let img;
let shd;

function preload() {
  img = loadImage("./loic.png");
  shd = loadShader('shader.vert', 'shader.frag');
}

function setup() {
  createCanvas(img.width, img.height, WEBGL);
}

function draw() {
  shader(shd);
  rect(-(width / 2), -(height / 2), width, height);
}

As we can see we have changed the rendering mode to WEBGL, which will allows us to use shaders.

createCanvas(img.width, img.height, WEBGL);

By the same fact this as set the origin of our canvas to the center.

We then specify which shader to use in the draw loop.

shader(shd);

And finally we draw a rectangle that will be used to render the shader on.

rect(-(width / 2), -(height / 2), width, height);

Looking at the result we can see that the shader is applied to the canvas, but only a quarter of it is red...

This is because the vertex shader start at the center of the canvas, so we need to update the position of the vertex to start in the bottom left corner of the canvas.

shader.vert

attribute vec3 aPosition;

void main() {
    vec4 positionVec4 = vec4(aPosition, 1.0);
    // * 2 is to use the full canvas size
    // - 1 is to place it back to a new origin (from the center to the bottom left)
    positionVec4.xy = (positionVec4.xy * 2.0) - 1.0;
    gl_Position = positionVec4;
}

Looking at the result we can see that the shader is applied to the entire canvas.

Now that we have the canvas setup to use the shader let's add back our image using the shader.

First define a new uniform called u_texture that will be used to pass the image to the shader.

sketch.js

function draw() {
    shader(shd);
    shd.setUniform('u_resolution', [width, height]);
    shd.setUniform('u_texture', img);
    rect(-(width / 2), -(height / 2), width, height);
}

Notice that we also defined a resolution for the shader, which will be used to calculate the position of the pixels in the canvas.

After which we need to update the fragment shader to use the image.

shader.frag

uniform vec2 u_resolution;
uniform sampler2D u_texture;

void main() {
    vec2 uv = gl_FragCoord.xy / u_resolution.xy;
    vec4 color = texture2D(u_texture, uv);
    gl_FragColor = color;
}

Looking at the result again we can see that the shader is rendering our image, but...

One obvious issue is that the image is flipped. This is because the origin of the canvas is now in the bottom left corner instead of the top left corner.

We can correct that by flipping the y axis.

shader.frag

uniform vec2 u_resolution;
uniform sampler2D u_texture;

void main() {
    vec2 uv = gl_FragCoord.xy / u_resolution.xy;
    uv.y = 1.0 - uv.y; // flip the y axis
    vec4 color = texture2D(u_texture, uv);
    gl_FragColor = color;
}

We are pretty much back to where we were at the start.

Creating a chromatic filter

Let's add back the chromatic effect.

shader.frag

uniform vec2 u_resolution;
uniform sampler2D u_texture;

void main() {
    vec2 uv = gl_FragCoord.xy / u_resolution.xy;
    uv.y = 1.0 - uv.y; // flip the y axis
    vec4 color = texture2D(u_texture, uv);

    float cumulative = (color.r + color.g + color.b) / 3.0;

    gl_FragColor = vec4(cumulative, cumulative, cumulative, 1.0);
}

Now let's add back the threshold.

shader.frag

uniform vec2 u_resolution;
uniform sampler2D u_texture;

void main() {
    vec2 uv = gl_FragCoord.xy / u_resolution.xy;
    uv.y = 1.0 - uv.y; // flip the y axis
    vec4 color = texture2D(u_texture, uv);

    float cumulative = (color.r + color.g + color.b) / 3.0;

    float threshold = 0.5;
    cumulative = cumulative > threshold ? 1.0 : 0.0;

    gl_FragColor = vec4(cumulative, cumulative, cumulative, 1.0);
}

Let's add back the smoothening of the bright part of the threshold.

shader.frag

uniform vec2 u_resolution;
uniform sampler2D u_texture;

void main() {
    vec2 uv = gl_FragCoord.xy / u_resolution.xy;
    uv.y = 1.0 - uv.y; // flip the y axis
    vec4 color = texture2D(u_texture, uv);

    float cumulative = (color.r + color.g + color.b) / 3.0;

    float threshold = 0.5;
    cumulative = cumulative > threshold ? smoothstep(threshold, 1.0, cumulative) : 0.0;

    gl_FragColor = vec4(cumulative, cumulative, cumulative, 1.0);
}

I also want the threshold to be dynamic, so let's add a uniform for it.

sketch.js

function draw() {
    // ...
    shd.setUniform('u_threshold', 0.5);
    // ...
}

shader.frag

uniform vec2 u_resolution;
uniform sampler2D u_texture;
uniform float u_threshold;

void main() {
    vec2 uv = gl_FragCoord.xy / u_resolution.xy;
    uv.y = 1.0 - uv.y; // flip the y axis
    vec4 color = texture2D(u_texture, uv);

    float cumulative = (color.r + color.g + color.b) / 3.0;

    cumulative = cumulative > u_threshold ? smoothstep(u_threshold, 1.0, cumulative) : 0.0;

    gl_FragColor = vec4(cumulative, cumulative, cumulative, 1.0);
}

You can also add the color back with some change.

shader.frag

precision mediump float;

uniform vec2 u_resolution;
uniform sampler2D u_texture; // Input texture
uniform float u_threshold;

float map(float value, float min1, float max1, float min2, float max2) {
    return min2 + (value - min1) * (max2 - min2) / (max1 - min1);
}

void main() {
    vec2 uv = gl_FragCoord.xy / u_resolution;
    
    // Flip the Y-axis
    uv.y = 1.0 - uv.y;
    
    // Sample color from the texture
    vec4 texColor = texture2D(u_texture, uv);
    
    float cum = (texColor.r + texColor.g + texColor.b) / 3.0;
    
    if (cum > u_threshold) {
        texColor.r = map(texColor.r, u_threshold, 1.0, 0.0, 1.0);
        texColor.g = map(texColor.g, u_threshold, 1.0, 0.0, 1.0);
        texColor.b = map(texColor.b, u_threshold, 1.0, 0.0, 1.0);
    } else {
        texColor.rgb *= 0.0;
    }
    
    gl_FragColor = texColor;
}

But keep in mind that using if statement in shaders is not recommended as it is a lot more intensive than pure math.

In our case we could update the code to use the step function which receive 2 float value, and return 0.0 if the first value is smaller than the second and 1.0 otherwise.

Knowing this we can create a new float variable called mult which will either be 0.0 or 1.0, which we can then use it to multiple our color values with.

shader.frag

// ...
void main() {
    // ...

    float cum = (texColor.r + texColor.g + texColor.b) / 3.0;
    float mult = step(u_threshold, cum);
    
    texColor.r = map(texColor.r, u_threshold, 1.0, 0.0, 1.0);
    texColor.g = map(texColor.g, u_threshold, 1.0, 0.0, 1.0);
    texColor.b = map(texColor.b, u_threshold, 1.0, 0.0, 1.0);
    texColor.rgb *= mult;
    
    // ...
}

See this post on the usage of the step function and other alternatives to using if conditions.

See this post for a more comprehensive on branching in shader code.

If you want to play around with the code, you can find it here.