Pixel perfect touch or how to determine if a pixel of a sprite is transparent

[twistapple]

Hello

I am developing using cocos2d-x ver 3.10

I am trying to find a way to determine if a pixel in a spite is transparent or not.
I want to know if the user has touched the transparent or the non-transparent part of a sprite.

I have read many examples but none solves the problem.
The example that is closest to what I am trying to do is a function provided by @grimfate which you can find here: Detect touch in sprite

The critical part is:

var pixel = new Uint8Array(4);

var renderTexture = cc.RenderTexture.create(5, 5);

renderTexture.beginWithClear(0, 0, 0, 0);
node.visit();
gl.readPixels(0, 0, 1, 1, gl.RGBA, gl.UNSIGNED_BYTE, pixel);
renderTexture.end();

The pixel array should contain the colors of the pixel at 0,0 point of the canvas.

Unfortunately it does not work for me. Pixel array always contains zeroes.
I read that gl.readPixels(…) does not work the same way in version 3.x.

Is there a way that I can detect the transparency of pixels in the sprite ?

[stevetranby]

As for your issue I believe your problem is trying to read the pixels immediately instead of waiting the required 1 frame. The node has not actually rendered its pixels into the renderTexture until the frame commits on the GPU. I believe there’s a way to set up a callback into RenderTexture where you can call the readPixels function.

Have you thought about storing the texture image data ( CCImage or uint8_t buffer[] ) alongside the texture and run collisions on the CPU? This way you’ll get pixel perfect touch detection, without having to go through the GPU, but at the cost of at most 2x memory - you could pack collision info into a single bit per pixel (so 32-bit image could store collision data in 1/32 the memory size).

Have you thought about using tessellated polygons (PolyInfo sprites) if you don’t need pixel-perfect, but rather pixel-close detection? The reason I ask this is that touch detection is inherently not precise (unless using a precision touch system like trackpad or apple’s pencil).

[twistapple]

Thank you very much @stevetranby for your immediate answer.

As for your 1st proposal (1st paragraph), I have read on multiple posts that it’s actually a matter of timing (as you say) that was introduced in version 3.x of Cocos2d. My problem is that, since I am new to cocos development (and I have no experience with OpenGL), I have no idea on how to implement it.

To tell you the truth, I was a little bit surprised that Cocos2d, with a so rich API, did not supply in its library, an easy way to accomplish this task - to test if a pixel in the sprite is transparent or not. It is one of the first problems I had to deal with.

As for your second and third proposal, I have, again, no idea on how to do it. The shape of the sprite (image) I am dealing with is a circle. A polygon could be a shape that could go close to a circle shape, but it sounds a little bit too much for the task.

I would really appreciate a code sample that works on cocos2d-x v.3.x. It would also be a plus, if the solution could also work on iOS and Android.

[stevetranby]

It looks like you can also force a render in the same single update. Here’s discussion.
Synchronous glReadPixel in cocos2d-x-v3.

In that discussion here’s one implementation:

Cocos2d-x 3.x Pixel Perfect Hit Detection

You’ll find that the key is creating a callback command (which is the implementation of my 1st paragraph suggestion).

void HitDetectHelper::HitDetectHelperNode::draw(Renderer *renderer, const Mat4& transform, uint32_t flags)
{
    _customCommand.init(_globalZOrder);
    _customCommand.func = CC_CALLBACK_0(HitDetectHelper::HitDetectHelperNode::readPixel, this);
    renderer->addCommand(&_customCommand);
}
void HitDetectHelper::HitDetectHelperNode::readPixel() {
    glReadPixels(_pixelPoint.x, _pixelPoint.y, 1, 1, GL_RGBA, GL_UNSIGNED_BYTE, _pixelBuffer);
}

Here’s another discussion linked in the first that discusses an implementation.
Pixel collision detection for v3.x.

Honestly if you are new to OpenGL and Cocos2d this is unfortunately a topic that requires a bit of knowledge on how rendering works. We’re no longer living in a world of 2D bitmap memory based video graphics where you can read/write to the video buffer directly. We’re now in a world of 3D where you send a bunch of data and commands that the GPU uses to generate a screen full of pixels every frame.

[twistapple]

Thanks again @stevetranby, you have been very helpful.

I have already seen the Synchronous glReadPixel in cocos2d-x-v3 discussion but I do not quite understand it. I am not sure how this code could be converted to JavaScript.

As for the Alexander’s Wong implementation, it is very interesting but again it is written in C++. From what I understand, draw() creates a custom command that executes a callback function, that executes the glReadPixels() function, which in turn stores the bytes in the _pixelBuffer, when the command is executed. Then we need getPixelValue() to get the actual value of the pixel. When is the correct time to execute getPixelValue(), or how do we know that the command has finished executing ?

Is there any implementation for JavaScript that uses this custom command callback ?

It would be perfect if there was a JavaScript solution that could then work through JSB…

[stevetranby]

Ah, yes sorry I didn’t notice this was Javascript topic. You can hack this for now, and it may be the answer but I would investigate further if it were my project.

var renderTexture = cc.RenderTexture.create(5, 5);
renderTexture.beginWithClear(255, 128, 0, 0);
node.visit();
renderTexture.end();

// need to wait a frame before reading (should see 255,128,0,0 in output)
this.scheduleOnce( function() {
  var pixels = new Uint8Array(4);
  gl.readPixels(0,0,1,1, gl.RGBA, gl.UNSIGNED_BYTE, pixels);
  for(pixel in pixels) {
    cc.log("ret[" + pixel + "] = " + pixels[pixel]);
  }
} );

[twistapple]

Thank you for the sample @stevetranby

What you are suggesting in your example is that I schedule the action of reading a pixel so that it will be executed a little bit later.

I tried to modify your example so that it can work with the initial function isTransparent(), proposed by @grimfate, but with no luck. It did not work for me.

• If the sprite it clicked on a “solid” pixel It returns the RGB colors of the pixel and a 255 at the end.
• If the sprite is clicked in the transparent area I get [0,0,0,255] as a result. Is this the color of the canvas ? I don’t know ! Again I cannot get transparency.

There is also a complexity in synchronizing the sprite relocation and returning the boolean result of the test.

• The sprite should be moved to position 0,0 in such a way that the touched pixel is in coordinates 0,0 of the canvas.
• Then the gl.readPixels(…) should be executed (from the scheduler)
• Then the sprite should be moved back to its original location. But this means that the relocation of the sprite back to its original position should also be done by the scheduler. So there is a latency here that the human eye can easily detect.

[stevetranby]

You said “touch” in previous points. So the first question is a precision input device being used always with your game? If not then pixel-perfect is overkill.

Are your sprites that you want to check regular shapes? If so they could flag which shape type they are and do circle, rect, polygon based collision checking either using simple regular shapes (parameterized shapes) where you can create the shape with {center, radius} or {center, width, height} or similar. Then testing w/circle, for example, just do a radial distance check to the click.

If you still think you need pixel-perfect (due to shapes with many pixel holes and irregular boundaries) then you might want to look into color-ID rendering for something you can search about with the phrase “opengl color picking glreadpixels” or similar where answers will explain you should render a color per entity, or two colors if you only render a single thing and only need to know a pixel is present or not.

This worked for me. You’re mileage may vary (YMMV)
I prob can’t help much beyond this. So if you still can’t get it working or have questions, probably smart to create a new topic for it (guessing no one else will reply to this one).

 cc.log("constructing!");

 cc.director.setClearColor(cc.color(0, 0, 128, 64));

 if ('opengl' in cc.sys.capabilities) {
   var x = winSize.width;
   var y = winSize.height;

   var blue = new cc.LayerColor(cc.color(0, 0, 255, 128));
   var red = new cc.LayerColor(cc.color(255, 0, 0, 128));
   var green = new cc.LayerColor(cc.color(0, 255, 0, 128));
   var white = new cc.LayerColor(cc.color(255, 255, 255, 128));

   cc.log("x,y = " + x + ", " + y);

   blue.scale = 0.5;
   blue.x = -x / 4;
   blue.y = -y / 4;

   red.scale = 0.5;
   red.x = x / 4;
   red.y = -y / 4;

   green.scale = 0.5;
   green.x = -x / 4;
   green.y = y / 4;

   white.scale = 0.5;
   white.x = x / 4;
   white.y = y / 4;

   this.addChild(blue, 10);
   this.addChild(white, 11);
   this.addChild(green, 12);
   this.addChild(red, 13);


   var node = cc.Sprite.create("res/texture512x512.png");
   node.setNormalizedPosition(cc.p(0.5, 0.5));
   //node.setTextureRect(cc.rect(0,0,20,20));
   node.setColor(cc.color(255, 128, 0));
   node.setOpacity(128);

   this.addChild(node);

   //var renderTexture = cc.RenderTexture.create(5, 5);
   //renderTexture.beginWithClear(255, 128, 0, 0);
   //node.visit();
   //renderTexture.end();
   //this.addChild(renderTexture);//.getTexture());

   // need to wait a frame before reading (should see 255,128,0,0 in output)
   this.scheduleOnce(function() {
     var ws = cc.director.getWinSize();
     var w = ws.width;
     var h = ws.height;
     var pixels = new Uint8Array(4);
     gl.readPixels(0, 0, 1, 1, gl.RGBA, gl.UNSIGNED_BYTE, pixels);
     for (pixel in pixels) {
       cc.log("ret[" + pixel + "] = " + pixels[pixel]);
     }
     gl.readPixels(w, 0, 1, 1, gl.RGBA, gl.UNSIGNED_BYTE, pixels);
     for (pixel in pixels) {
       cc.log("ret[" + pixel + "] = " + pixels[pixel]);
     }
     gl.readPixels(0, h, 1, 1, gl.RGBA, gl.UNSIGNED_BYTE, pixels);
     for (pixel in pixels) {
       cc.log("ret[" + pixel + "] = " + pixels[pixel]);
     }
     gl.readPixels(w, h, 1, 1, gl.RGBA, gl.UNSIGNED_BYTE, pixels);
     for (pixel in pixels) {
       cc.log("ret[" + pixel + "] = " + pixels[pixel]);
     }
   });

output:

JS: ret[0] = 0
JS: ret[1] = 0
JS: ret[2] = 192
JS: ret[3] = 96
JS: ret[0] = 128
JS: ret[1] = 0
JS: ret[2] = 64
JS: ret[3] = 96
JS: ret[0] = 0
JS: ret[1] = 128
JS: ret[2] = 64
JS: ret[3] = 96
JS: ret[0] = 128
JS: ret[1] = 128
JS: ret[2] = 192
JS: ret[3] = 96

[stevetranby]

Maybe you’re trying to do pixel-perfect collision testing between two objects on screen (not related to input) and if this is the case you can search on that, but it’s similar to “color-ID picking” where you color each sprite in question a different color (prob do a bbox collision first to weed out unecessary gpu/shader tests).
e.g. https://stackoverflow.com/questions/18546066/pixel-perfect-collision-detection-in-cocos2dx

Also there are likely other solutions like using the stencil buffer. Most solutions will probably utilize the shader pipeline, however, unless as mentioned previously you store collision information separate from the texture data.

I’m interested more in the picking part of this topic and we’ve implemented something like it in Unity and OpenGL for testing. Maybe I’ll eventually write about cocos2d-x/js sample of pixel-perfect picking and sprite collision in the future. If you arrive at a solution and are able and willing to post your code or the relevant portions, please consider it.

[twistapple]

Thank you very much @stevetranby for your answers once again.

I will check everything that you proposed, but the question, although it seems simple, it remains unanswered. This is probably due to the fact that I did not express myself correctly from the beginning.

To put in in a simple way:

I want to create a JavaScript function that will work in cocos2d-x v.3.x like the following:

function isTransparent(node, pos) {
         ...
         ...
         return isPixelTransparent;
}

that returns true if a pixel at the specified position of a sprite (node) is transparent.

I doesn’t have to use OpenGL. The only thing that is important is that it is a function that will work through JSB.

Probably I will open a new thread, as you suggested, and I will try to be more clear this time.

Thanks again for your help.

[stevetranby]

Ah, what you are trying to do is unsupported using cocos2d-x directly, afaik.

You will have to write your own code either completely in JSB or write it in C++ (if you’re only submitting to native app stores) and then auto-generate or otherwise write the JSB bindings yourself.

You can try to use the graphics system (GPU/OpenGL or Canvas) you’ll have to do what we’ve discussed with glReadPixels but only render the single sprite. This will have lag, synchronous blocking issues unless you async it, require render pipeline to flush, and everything we’ve discussed thus far whether you use color-picking or direct single node render-test out of band GPU commands.

This will be much easier if you only use HTML5 Canvas, however, and would be more similar to the discussed option I’ll mention again.

This option is that you manage the transparent pixels in each node/sprite yourself. This would likely only be relevant for Sprite’s with image textures. Store a map from texture/sprite ID to a bit-array where each bit represents an image pixel being transparent or not.

Anyway, if you really need this functionality I believe your going to have to go down a path of custom research and writing custom code.

Unless someone else has an “out of box”, ready to go, “no custom code” required solution that I just can’t see myself.

Good luck.

[stevetranby]

And to be clear this custom code would probably be grimfate’s solution, slightly modified with components of my code samples. That was the reason for the sample I gave; how to get readpixels to work so you can combine it with grimfate’s isTransparent solution.

This is an straight-forward problem, just requires some work. There’s just no isPixelTransparent(sprite, pixelCoordinate) function out-of-box in cocos2d, afaik.

[stevetranby]

Here’s an example blog post on with the same idea I’ve mentioned about generating a pixel bit mask and using that instead of glReadPixels (which to repeat as a broken record is almost always not performant, unless you write it with perfect asyncrony):

Fast Pixel-Perfect Collision Detection for Cocos2D with Example Code (1/2)

Fast Pixel-Perfect Collision Detection for Cocos2D with Example Code (2/2)

Similar Posts/Topics for my reference in case I decide to write a blog post about this topic:

• http://alexanderwong.me/post/108850552513/cocos2d-x-3x-pixel-perfect-hit-detection
• Pixel Perfect Collision Detection cocos2d-x v3
• HELP - Converting some C++ Logic to Javascript
• Pixel collision detection for v3.x
• Pixel Perfect Collision Detection

HTML5-only version
demo - http://heartbitgames.com/proto/NewTransparencyTest/
code - http://pastebin.com/raw/y77ZftdH

[piotrros]

@stevetranby do you have some simple working example of checking pixel color from a particular point in Node (in c++)?
So doesn’t matter if it’s sprite, spine skeleton or something else it’ll return me what color does have particular point (x, y).

[stevetranby]

I’m not sure. I may have a test project still around in my hard drive somewhere, but I don’t have it lying around at the moment. Everything laid out in this forum topic should get you to the check single pixel color … so not sure where you’re having issue?

The color does have to exist in the render buffer at the time of glReadPixel request, but yeah you could have a 1000 different sprites/spine/etc and you’d get the pixel color displayed at the time of read.

Hope you already got it working.

[piotrros]

I’ve got a solution to the problem from @grimfate

Touch detection on non-transparent pixels

Wow, you rock! Thanks for help! Yes, I understood the explanation perfectly. So the way how it works is pretty messy. How do you think will it be hard to change Sprite to Node? I mean I can have a node with several children or have something else like sprite skeleton. In order to make any sense of using it in my app is to be able to support all of these features. In the nutshell the idea is to make it as universal as possible. I have a container of something and it doesn’t matter what’s insid…