CommandBufferGL related crash

pccsoares · May 13, 2020, 11:18am

I’m using Firebase Crashlytics on game World Soccer Challenge and surprised to see that I only get 99.95% crash free users. This is mostly due to the same crash callstack, around a call to cocos2d::backend::CommandBufferGL::drawElements.
This is the call stack:

Crashed: Thread: SIGSEGV 0x0000000000000000
libc.so at (Missing)()
…
at cocos2d::backend::CommandBufferGL::drawElements(cocos2d::backend::PrimitiveType, cocos2d::backend::IndexFormat, unsigned int, unsigned int) + 352(CommandBufferGL.cpp:352)
at cocos2d::Renderer::drawCustomCommand(cocos2d::RenderCommand*) + 678(CCRenderer.cpp:678)
at (Missing)()
at cocos2d::Renderer::doVisitRenderQueue(std::__ndk1::vector<cocos2d::RenderCommand*, std::__ndk1::allocatorcocos2d::RenderCommand* > const&) + 354(CCRenderer.cpp:354)
at (Missing)()
at cocos2d::Renderer::captureScreen(cocos2d::RenderCommand*)(functional)
at cocos2d::Renderer::visitRenderQueue(cocos2d::RenderQueue&) + 341(CCRenderer.cpp:341)
at cocos2d::Renderer::render() + 371(CCRenderer.cpp:371)
at cocos2d::Node::transform(cocos2d::Mat4 const&) + 1266(CCNode.cpp:1266)
at cocos2d::Scene::render(cocos2d::Renderer*, cocos2d::Mat4 const&, cocos2d::Mat4 const*) + 218(CCScene.cpp:218)
at cocos2d::Director::drawScene() + 292(CCDirector.cpp:292)
at cocos2d::Director::mainLoop() + 1392(CCDirector.cpp:1392)

On this game I’m still using using cocos2d-x 3.15.

Is this a know issue? Or is it something related to my code?
Should I migrate to v4.0 to get this fixed?

Darren · May 14, 2020, 4:20am

Hi,
the info that you provide is kind of confusing…
this class and namespace is definitely from v4.0, but you mention using cocos2d-x 3.15?
cocos2d::backend::CommandBufferGL
It just doesn’t tally…

My suggestion is to run your game on debugger and then perform the in-game capture screen feature to see what is causing the crash.

You can reference the cpp tests for how screen capture is done if that helps

pccsoares · May 14, 2020, 8:34am

duhhh… I had already migrated this game to cocos2d-x-4.0.
So it’s a 4.0 callstack that we are looking at. Sorry.

Anyway, I don’t have any capture screen feature implemented so I really don’t see where this comes from.
The closer feature that I have is a part of the game that I save a sprite to PNG and change its colors for later reuse. Can this be the place?

RenderTexture* renderTexture = RenderTexture::create(width, height);
renderTexture->begin();
sprite->visit();
renderTexture->end();
renderTexture->saveToFile(imagePath, Image::Format::PNG, true, [&](RenderTexture* rt, const std::string& path) ...

And this crash only affects 0.04% of the users, so I guess it would be really hard to reproduce on a test environment.

R101 · May 14, 2020, 11:00am

Just so you are aware, there is a memory leak in that code that was fixed only recently, but not sure if it’s been merged in yet. It may not be the cause of your issue, but you should fix it in your own code since you’re using that specific method. You can read about it here: https://github.com/cocos2d/cocos2d-x/issues/20408.

pccsoares · May 14, 2020, 11:08am

Thanks!!! that was probably it.

Darren · May 15, 2020, 3:05am

@pccsoares umm… according the callstack in your post, capture screen is being called

you can do a search in your project to check where you are calling this function utils::captureScreen OR using this class CaptureScreenCallbackCommand any where.
And then perform the steps that will trigger that code.

And thank you @R101 for highlighting that issue! didn’t know about that memory leak

pccsoares · June 13, 2020, 10:33pm

I would like to revisit this.
Just released my new cocos game World Soccer Champs and the same callstack appears on the top of the crashes. 0.02% of all sessions end up with this:

  at cocos2d::backend::CommandBufferGL::drawElements(cocos2d::backend::PrimitiveType, cocos2d::backend::IndexFormat, unsigned int, unsigned int) + 352(CommandBufferGL.cpp:352)
       at cocos2d::Renderer::drawCustomCommand(cocos2d::RenderCommand*) + 678(CCRenderer.cpp:678)
       at (Missing)()
       at cocos2d::Renderer::doVisitRenderQueue(std::__ndk1::vector<cocos2d::RenderCommand*, std::__ndk1::allocator<cocos2d::RenderCommand*> > const&) + 354(CCRenderer.cpp:354)
       at cocos2d::Renderer::captureScreen(cocos2d::RenderCommand*)(functional)
       at cocos2d::Renderer::visitRenderQueue(cocos2d::RenderQueue&) + 341(CCRenderer.cpp:341)
       at cocos2d::Renderer::render() + 371(CCRenderer.cpp:371)
       at cocos2d::Node::transform(cocos2d::Mat4 const&) + 1266(CCNode.cpp:1266)
       at cocos2d::Scene::render(cocos2d::Renderer*, cocos2d::Mat4 const&, cocos2d::Mat4 const*) + 218(CCScene.cpp:218)

I have no captureScreen being called anywhere on the code.
Also, just to be sure it was not the memory leak mentioned above, I added the fix to my cocos v4 source code.

R101 · June 14, 2020, 1:34am

Are you using any extra libraries that may be calling Cocos2d-x code (like SDKBOX)? I’m not saying that SDKBOX would be calling such code, but it does integrate with Cocos2d-x, so if you have any such modules you should check if they’re calling captureScreen.

Other than that, if you are absolutely certain captureScreen is not being called directly, then this may be a bigger problem, something like a memory overwrite, which is changing the path of the code at some stage. Before making this assumption though, have you looked into what your app is doing just prior to this event occurring?

pccsoares · June 14, 2020, 11:55am

I’m using SDKBOX, but just with Review, IAP and SdkboxPlay. I don’t see why they should be calling captureScreen. As far as I’m aware, these plugins are not open source. How can I check if they are calling captureScreen?

Regarding the memory overwrite possibility, if the code path would have been changed, wouldn’t the callstack be different in every crash? In these crashes, the callstack is exactly the same on all crashes.

Thanks for the help.

R101 · June 14, 2020, 3:00pm

captureScreen is called from several sections of code that aren’t directly related to it. Search for it in the source and you’ll see in the render code the switch/case section where it’s called. If something strange is going on with the conditional value, then that is only likely location where it would be called. I’m simply guessing here.

If you have the debug version of those libraries you’re using, you should be able to check if that function is referenced in there. You can use a utility like objdump to find that out.

pccsoares · June 23, 2020, 11:27pm

By the nature of the crashes I tend to believe that captureScreen may not be called at all by any sdkboxplugin.
I get many other similar crashes with a similar callstack and without captureScreen:

at cocos2d::backend::CommandBufferGL::drawElements(cocos2d::backend::PrimitiveType, cocos2d::backend::IndexFormat, unsigned long, unsigned long) + 352(CommandBufferGL.cpp:352)
       at cocos2d::Renderer::drawCustomCommand(cocos2d::RenderCommand*) + 670(CCRenderer.cpp:670)
       at cocos2d::Renderer::processRenderCommand(cocos2d::RenderCommand*) + 249(CCRenderer.cpp:249)
       at cocos2d::Renderer::doVisitRenderQueue(std::__ndk1::vector<cocos2d::RenderCommand*, std::__ndk1::allocator<cocos2d::RenderCommand*> > const&) + 354(CCRenderer.cpp:354)
       at cocos2d::Renderer::render() + 371(CCRenderer.cpp:371)
       at cocos2d::Scene::render(cocos2d::Renderer*, cocos2d::Mat4 const&, cocos2d::Mat4 const*) + 218(CCScene.cpp:218)
       at cocos2d::Director::drawScene() + 292(CCDirector.cpp:292)
       at cocos2d::Director::mainLoop() + 1392(CCDirector.cpp:1392)

all these crashes (including the callstack that shows the captureScreen) come right after an Appodeal video ad being displayed with success, and the display coming back to the game. I’m using JNI to directly call Appodeal.
It’s is driving me crazy. Despite affecting only 0.01% of the users, I see this exception for 2000+ users in less than 2 weeks.

R101 · June 24, 2020, 1:18am

OK, so that is interesting. Check this recent change that was made to address an unitialised variable issue in the CommandBufferGL.cpp file, if you haven’t already come across it:

https://github.com/cocos2d/cocos2d-x/commit/121f8e90bb579e455e86d6856251b0d9067c4c89

The crash could very well be the result of whatever random value that uninitialised member variable has. Fix it in your code, and see if it helps at all.

pccsoares · June 24, 2020, 8:22am

Very promissing. Thanks.
Publishing a new version of the game with the fix and will let you know if it works.

pccsoares · June 25, 2020, 10:20am

Unfortunately it was not that.

it’s weird that the callstacks slightly differ. All of them segmentation faults, and always after the video ad being displayed.

There are 4 different groups of callstacks, with hundreds of crashes on the four of them:

	at cocos2d::backend::CommandBufferGL::drawElements(cocos2d::backend::PrimitiveType, cocos2d::backend::IndexFormat, unsigned int, unsigned int) + 352(CommandBufferGL.cpp:352)
   at cocos2d::Renderer::drawCustomCommand(cocos2d::RenderCommand*) + 678(CCRenderer.cpp:678)
   at cocos2d::Renderer::doVisitRenderQueue(std::__ndk1::vector<cocos2d::RenderCommand*, std::__ndk1::allocator<cocos2d::RenderCommand*> > const&) + 354(CCRenderer.cpp:354)
   at cocos2d::Renderer::captureScreen(cocos2d::RenderCommand*)(functional)

at cocos2d::backend::CommandBufferGL::setUniforms(cocos2d::backend::ProgramGL*) const + 453(CommandBufferGL.cpp:453)
   at cocos2d::backend::CommandBufferGL::prepareDrawing() const + 383(CommandBufferGL.cpp:383)
   at cocos2d::backend::CommandBufferGL::drawElements(cocos2d::backend::PrimitiveType, cocos2d::backend::IndexFormat, unsigned long, unsigned long) + 350(CommandBufferGL.cpp:350)
   at cocos2d::Renderer::drawCustomCommand(cocos2d::RenderCommand*) + 670(CCRenderer.cpp:670)
   at cocos2d::Renderer::processRenderCommand(cocos2d::RenderCommand*) + 249(CCRenderer.cpp:249)
   at cocos2d::Renderer::doVisitRenderQueue(std::__ndk1::vector<cocos2d::RenderCommand*, std::__ndk1::allocator<cocos2d::RenderCommand*> > const&) + 354(CCRenderer.cpp:354)
   at cocos2d::Renderer::render() + 371(CCRenderer.cpp:371)

at cocos2d::backend::CommandBufferGL::drawElements(cocos2d::backend::PrimitiveType, cocos2d::backend::IndexFormat, unsigned long, unsigned long) + 352(CommandBufferGL.cpp:352)
   at cocos2d::Renderer::drawCustomCommand(cocos2d::RenderCommand*) + 670(CCRenderer.cpp:670)
   at cocos2d::Renderer::processRenderCommand(cocos2d::RenderCommand*) + 249(CCRenderer.cpp:249)
   at cocos2d::Renderer::doVisitRenderQueue(std::__ndk1::vector<cocos2d::RenderCommand*, std::__ndk1::allocator<cocos2d::RenderCommand*> > const&) + 354(CCRenderer.cpp:354)
   at cocos2d::Renderer::render() + 371(CCRenderer.cpp:371)

   at cocos2d::backend::ProgramState::getProgram() const + 96(ProgramState.h:96)
       at cocos2d::Renderer::setRenderPipeline(cocos2d::PipelineDescriptor const&, cocos2d::backend::RenderPassDescriptor const&) + 753(CCRenderer.cpp:753)
       at cocos2d::Renderer::setRenderPipeline(cocos2d::PipelineDescriptor const&, cocos2d::backend::RenderPassDescriptor const&) + 753(CCRenderer.cpp:753)
       at cocos2d::Renderer::drawCustomCommand(cocos2d::RenderCommand*) + 661(CCRenderer.cpp:661)
       at cocos2d::Renderer::processRenderCommand(cocos2d::RenderCommand*) + 249(CCRenderer.cpp:249)
       at cocos2d::Renderer::doVisitRenderQueue(std::__ndk1::vector<cocos2d::RenderCommand*, std::__ndk1::allocator<cocos2d::RenderCommand*> > const&) + 354(CCRenderer.cpp:354)
       at cocos2d::Renderer::render() + 371(CCRenderer.cpp:371)

I find it also strange that I’m the only one refering this.
Any other tips, ideas? Thanks a lot for the help.

smitpatel88 · June 25, 2020, 11:41am

Are you creating/deleting any Node at that time?
Make sure you are doing in CocosThread. This could be the issue of crash.

pccsoares · June 25, 2020, 12:03pm

I am creating and deleting nodes at that time.

When the video ends there is a callback that is triggered from the Ad SDK, that then calls a method on my Scene to change the contents.
These callbacks may be parallel threads.
I will try to use performFunctionInCocosThread and see if it helps.

pccsoares · June 27, 2020, 1:30am

That was it!!! Running all the event callbacks inside performFunctionInCocosThread did the trick.
I’m super happy I just solved this bug in my games that was haunting me for years.
Thanks so much to everyone who helped me on this.

smitpatel88 · June 27, 2020, 5:05am

For years…!!! Wow.
This is very common for cocos environment.