android - Why is there an offset when I render this overlay?


Keywords:android 


Question: 

I use Vuforia SDK to render the video stream of my phone's camera on the screen.

Phone's camera video stream on screen

So the texture is generated by the Vuforia library, not me.

The shaders used to render this background are:

// Vertex Shader
attribute vec4 a_position;
attribute vec2 a_textureCoords;
varying vec2 v_textureCoords;
uniform mat4 u_projectionMatrix;

void main()
{
    gl_Position = u_projectionMatrix * a_position;
    v_textureCoords = a_textureCoords;
}


// Fragment Shader
varying highp vec2 v_textureCoords;
uniform sampler2D u_currentTexture;

void main()
{
    vec4 currentColor = texture2D(u_currentTexture, v_textureCoords);
    gl_FragColor = currentColor;
}

Now, I want an overlay in the upper-left corner of the screen:

Video stream with overlay in the upper-left corner

I don't want this overlay to display only a pink texture, but rather a multiply blend of the pink texture and the background texture. Note that the textures do not have the same coordinates.

But for now, let's forget about the blending and let's just render the background texture in the shader program of the pink texture. So in the end, yes, one should see no difference between the background-only version and the bacground with overlay version.

Overlay with offset

As you can see (look at the painting and the top of the chair), there is a small offset...

The shaders used to render the overlay are:

// Vertex Shader
attribute vec4 a_position;
attribute vec2 a_currentTextureCoords;
varying vec2 v_currentTextureCoords;

void main()
{
    gl_Position = a_position;
    v_currentTextureCoords = a_currentTextureCoords;
}


// Fragment Shader
varying highp vec2 v_currentTextureCoords;
uniform sampler2D u_currentTexture;
uniform sampler2D u_backgroundTexture;

void main()
{
    vec2 screenSize = vec2(1080.0, 1920.0);
    vec2 cameraResolution = vec2(720.0, 1280.0);
    vec2 texelSize = vec2(1.0 / screenSize.x, 1.0 / screenSize.y);
    vec2 scaleFactor = vec2(cameraResolution.x / screenSize.x, cameraResolution.y / screenSize.y);
    vec2 uv = gl_FragCoord.xy * texelSize * scaleFactor;
    uv = vec2(scaleFactor.y - uv.y, scaleFactor.x - uv.x);
    vec4 backgroundColor = texture2D(u_backgroundTexture, uv);

    gl_FragColor = backgroundColor;
}

Are my calculations wrong?


1 Answer: 

Why do you need this line?

uv = vec2(scaleFactor.y - uv.y, scaleFactor.x - uv.x);

Not sure what arithmetic relationship the absolute texture coordinates have with the scale factor which needs an addition or a subtraction ...

P.S. it's not related to your question, but your shaders will be shorter and easier to read if you just use the vector operations in the language. For example, replace:

vec2 scaleFactor = vec2(cameraResolution.x / screenSize.x, cameraResolution.y / screenSize.y);

... with ...

vec2 scaleFactor = cameraResolution / screenSize;

As long as the vector types are the same length, it will do exactly what you expect with a lot less typing ...