Label render crash

kalika06 · May 8, 2020, 7:15pm

Hi,
I have a label which gets its text updated with a new string every second (I’m using its parent Node’s schedule method for that). It works fine.
But then, if I add to the parent Node another child Label with the same font as the first one, after a few ticks with successful string change, the first label’s rendering code fails with the following stack trace:

Exception Type: EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note: EXC_CORPSE_NOTIFY
Triggered by Thread: 0

Thread 0 name: Dispatch Thread 0 Crashed:
0 libsystem_kernel.dylib 1 libsystem_pthread.dylib 2 libsystem_c.dylib 3 libsystem_c.dylib 4 SteveAndMaggieFoodGame 5 SteveAndMaggieFoodGame 6 SteveAndMaggieFoodGame 7 SteveAndMaggieFoodGame 8 SteveAndMaggieFoodGame 9 SteveAndMaggieFoodGame 10 SteveAndMaggieFoodGame 11 SteveAndMaggieFoodGame 12 SteveAndMaggieFoodGame 13 SteveAndMaggieFoodGame 14 SteveAndMaggieFoodGame 15 SteveAndMaggieFoodGame 16 SteveAndMaggieFoodGame 17 SteveAndMaggieFoodGame 18 SteveAndMaggieFoodGame 19 QuartzCore 20 IOKit 21 CoreFoundation 22 CoreFoundation 23 CoreFoundation 24 CoreFoundation 25 CoreFoundation 26 GraphicsServices 27 UIKitCore 28 SteveAndMaggieFoodGame 29 libdyld.dylib queue: com.apple.main-thread
0x00000001953c7ec4 __pthread_kill + 8
0x00000001952e31d8 pthread_kill$VARIANT$mp + 136
0x0000000195237844 abort + 100
0x0000000195236c64 err + 0
0x000000010113bdb0 cocos2d::Ref::release() + 2981296 (CCRef.cpp:99)
0x0000000100ff68a0 cocos2d::updateBatchCommand(cocos2d::BatchCommand&) + 1648800 (CCLabel.cpp:709)
0x0000000100ff6bac cocos2d::updateShaderProgram() + 1649580 (CCLabel.cpp:694)
0x0000000100ffb268 cocos2d::draw(cocos2d::Renderer*, cocos2d::Mat4 const&, unsigned int) + 1667688 (CCLabel.cpp:1779)
0x0000000100ffbe20 cocos2d::drawSelf(bool, cocos2d::Renderer*, unsigned int) + 1670688 (CCLabel.cpp:0)
0x0000000100ffbd10 cocos2d::visit(cocos2d::Renderer*, cocos2d::Mat4 const&, unsigned int) + 1670416 (CCLabel.cpp:0)
0x00000001010360dc cocos2d::Node::visit(cocos2d::Renderer*, cocos2d::Mat4 const&, unsigned int) + 1908956 (CCNode.cpp:1248)
0x00000001010360dc cocos2d::Node::visit(cocos2d::Renderer*, cocos2d::Mat4 const&, unsigned int) + 1908956 (CCNode.cpp:1248)
0x00000001010360dc cocos2d::Node::visit(cocos2d::Renderer*, cocos2d::Mat4 const&, unsigned int) + 1908956 (CCNode.cpp:1248)
0x0000000101063684 cocos2d::Scene::render(cocos2d::Renderer*, cocos2d::Mat4 const&, cocos2d::Mat4 const*) + 2094724 (CCScene.cpp:210)
0x00000001011f7cf0 cocos2d::GLView::renderScene(cocos2d::Scene*, cocos2d::Renderer*) + 3751152 (CCGLView.cpp:495)
0x00000001010f8288 cocos2d::Director::drawScene() + 2704008 (CCDirector.cpp:0)
0x00000001010fad8c cocos2d::Director::mainLoop() + 2715020 (CCDirector.cpp:1395)
0x00000001010faddc cocos2d::Director::mainLoop(float) + 2715100 (CCDirector.cpp:1404)
0x0000000101211064 -[CCDirectorCaller doCaller:] + 3854436 (CCDirectorCaller-ios.mm:139)
0x000000019bf15130 CA::Display::DisplayLink::dispatch_items+ 74032 (unsigned long long, unsigned long long, unsigned long long) + 632
0x0000000196519930 IODispatchCalloutFromCFMessage + 488
0x00000001955258ac __CFMachPortPerform + 172
0x000000019554f07c CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE1_PERFORM_FUNCTION + 56
0x000000019554e7a8 __CFRunLoopDoSource1 + 444
0x000000019554967c __CFRunLoopRun + 2168
0x0000000195548adc CFRunLoopRunSpecific + 464
0x000000019f4e9328 GSEventRunModal + 104
0x000000019965663c UIApplicationMain + 1936
0x0000000100fa6388 main + 1319816 (main.m:5)
0x00000001953d2360 start + 4

Additional remarks:

if I set the label’s string to the same string every scheduled tick, no crash
if the second label’s font is different, no crash!
happens on both Andy and iOS

I have no experience whatsoever on batching and know little of rendering. If anyone has any idea what might be going on, please tell me.

edit: I’m on V.4

R101 · May 9, 2020, 3:02am

Are you by any chance doing the string updates inside a thread other than the main thread? You really should show the relevant bits of code related to this issue.

smitpatel88 · May 9, 2020, 4:36am

Just a tip: Make sure your ttf file name doesnt contain any space and it should be short name. We faced few issues due to this in past.

kalika06 · May 9, 2020, 9:57am

@R101 Update the UI from another thread…?! < winces > I would never do such a horrible thing! But just to be sure, I’ve confirmed that my scheduler works on the main thread.

The code’s more complicated than that, but there you go, a hastily-prepared min working (crashing) example:

Scene’s onEnter:

_timeLabel = cocos2d::Label::createWithTTF("Example label", "fonts/ComicSansMSBold.ttf", 120*ScalingUtils::getScaleForFont());
    addChild(_timeLabel);

`
(…)

schedule(CC_SCHEDULE_SELECTOR(SubGameSceneShoot::onSecondElapsed), 1, CC_REPEAT_FOREVER, 0);

`

Selector:

void SubGameSceneShoot::onSecondElapsed(float dt){
    --shootGameState.currentTimeSeconds;
    _timeLabel->setString(MiscUtils::clockMinSecTimeString(shootGameState.currentTimeSeconds));
}

On button click (which causes a crash):

bool SubGameSceneShoot::onFastForwardButtonClicked()
{
    auto anotherLabel = cocos2d::Label::createWithTTF("Another example label", "fonts/ComicSansMSBold.ttf", 120*ScalingUtils::getScaleForFont());
    addChild(anotherLabel);
    return true;
}

Misc utils just return a string formatted for the time, nothing special, if I change it to std::to_string(schootGameState.currentTimeSeconds), I also get a crash.

R101 · May 9, 2020, 10:30am

Can you try something out quickly and let me know if it works or not:

bool SubGameSceneShoot::onFastForwardButtonClicked()
{
    this->scheduleOnce([this](float)
    {
        auto* anotherLabel = cocos2d::Label::createWithTTF("Another example label", "fonts/ComicSansMSBold.ttf", 120*ScalingUtils::getScaleForFont());
        addChild(anotherLabel);
    }, 0.f, "FastForward");
    return true;
}

Have you tried to create a minimal sample project that reproduces this issue, just to eliminate any other possible causes of the problem? Also, have you tried it with v3.17.2 of cocos2d-x?

kalika06 · May 9, 2020, 10:51am

Crashes just as nicely
Yep, I was thinking of creating the minimal sample project. I’ll try to find time for that. <3 weeks past deadline already>

kalika06 · May 9, 2020, 5:19pm

A clean project with the minimal example does not crash. Dammit! I’ll get back to you when I’ve found the reason. This is intriguing! Thanks for the help .

R101 · May 10, 2020, 2:54am

That is good to know! Now it’s just a matter of narrowing down the possible causes. Good luck!

kalika06 · May 10, 2020, 6:57pm

I’m sorry. Bad news. This crash seems a bit random so it got me fooled at first.
Actually, this is a minimal crashing example, on a clean project.
Remarks:

at first, it didn’t crash on my iPad mini 4, I had to add and remove some more labels to make it crash.
so after a few reinstalls and changes, it started crashing. I then cut the code to the following minimal example, and it still crashes (boo).
I installed the app on my iPad Pro to see whether it will crash at once, or later. It crashed after the first install.
I ran it on my Huawei tab running Android 8.0. It does not crash, but some of the numbers disappear (!!!).
It crashes in the same place as before, in the void Label::updateBatchCommand(Label::BatchCommand &batch) method, line CC_SAFE_RELEASE_NULL(pipelineDescriptor.programState); (doesn’t exactly look like safe releasing, does it? )
Doesn’t matter whether the font is Marker Felt or ComicSansBold.
Maybe I’ve done something illegal, using setString like this? Or maybe I should report a bug to the engine guys (how do I do that?)?

bool HelloWorld::init()
{
    if ( !Scene::init() )
    {
        return false;
    }

    Vec2 origin = Director::getInstance()->getWinSize()/2;

    _crashingLabel = Label::createWithTTF("0", "fonts/Marker Felt.ttf", 120);
    _crashingLabel->setPosition(origin);
    _crashingLabel->setColor(cocos2d::Color3B(200 ,200, 200));
    addChild(_crashingLabel, 1);
    _crashingLabel->setWidth(origin.x);
    
    this->schedule([this](float)
                   {
                        ++_i;
                       _crashingLabel->setString(std::to_string(_i));
                   }, 1, CC_REPEAT_FOREVER, 0, "repeat");
    
    return true;
}

UPDATE:
My amateur debugging shows that the crash happens when, in the Label::draw method, once in a while the size of batchNodes is 2 and batchCommands is 1 and this block is entered. (In non-crashing cases both are equal to 1, except for the first draw).

if (_batchCommands.size() != _batchNodes.size())
{
   _batchCommands.resize(_batchNodes.size());
   updateShaderProgram();
}

kalika06 · May 10, 2020, 9:07pm

I think we are over-releasing the commands’ PipelineDescriptor’s programState, which has been cloned but not retained.

Ah wait. Do the batch commands correspond to the characters in the label? In that case, it’s strange since the crash always happens at 7.

Adding CC_SAFE_RETAIN(cp); in the *ProgramState::clone() method prevents the crash, but makes some of the labels disappear (like on Android. I’m testing on iOS all the time). 7,8 and 9 disappear, but 10 shows up. Seems extending the batch commands vector happens too soon (?).
And I don’t know yet about the potential leaks.

But I forgot that creating the object sets the reference count to 1. So additional retaining is not the solution. But then, why is the reference count wrong?

Gee, I really could use some course on lower-level stuff like that.

kalika06 · May 10, 2020, 10:01pm

Nope. With retaining in the clone() method it also crashes randomly with the following stack trace:

Thread 1 Queue : com.apple.main-thread (serial)
#0	0x0000000191963e40 in nanov2_allocate_from_block$VARIANT$mp.cold.1 ()
#1	0x000000019194ad14 in nanov2_allocate_from_block$VARIANT$mp ()
#2	0x0000000191949f98 in nanov2_allocate$VARIANT$mp ()
#3	0x000000019194a480 in nanov2_calloc$VARIANT$mp ()
#4	0x0000000191958924 in malloc_zone_calloc ()
#5	0x0000000191959208 in calloc ()
#6	0x000000019199c97c in _objc_rootAllocWithZone ()
#7	0x00000001bf970fcc in -[MTLToolsBuffer initWithBaseObject:parent:] ()
#8	0x00000001bf9aa2b0 in -[MTLDebugBuffer initWithBuffer:device:options:] ()
#9	0x00000001bf9cb94c in -[MTLDebugDevice newBufferWithLength:options:] ()
#10	0x0000000101b83600 in ___lldb_unnamed_symbol2203$$libMTLCapture.dylib ()
#11	0x000000010057154c in cocos2d::backend::BufferMTL::BufferMTL(id<MTLDevice>, unsigned long, cocos2d::backend::BufferType, cocos2d::backend::BufferUsage) at /Users/kasia/Documents/misc_projects/NiceCleanCocosProject/NiceCleanCococProject/cocos2d/cocos/renderer/backend/metal/BufferMTL.mm:40
#12	0x000000010057172c in cocos2d::backend::BufferMTL::BufferMTL(id<MTLDevice>, unsigned long, cocos2d::backend::BufferType, cocos2d::backend::BufferUsage) at /Users/kasia/Documents/misc_projects/NiceCleanCocosProject/NiceCleanCococProject/cocos2d/cocos/renderer/backend/metal/BufferMTL.mm:33
#13	0x000000010057ae0c in cocos2d::backend::DeviceMTL::newBuffer(unsigned long, cocos2d::backend::BufferType, cocos2d::backend::BufferUsage) at /Users/kasia/Documents/misc_projects/NiceCleanCocosProject/NiceCleanCococProject/cocos2d/cocos/renderer/backend/metal/DeviceMTL.mm:95
#14	0x000000010052379c in cocos2d::CustomCommand::createIndexBuffer(cocos2d::backend::IndexFormat, unsigned long, cocos2d::backend::BufferUsage) at /Users/kasia/Documents/misc_projects/NiceCleanCocosProject/NiceCleanCococProject/cocos2d/cocos/renderer/CCCustomCommand.cpp:85
#15	0x000000010034b5d8 in cocos2d::Label::updateBuffer(cocos2d::TextureAtlas*, cocos2d::CustomCommand&) at /Users/kasia/Documents/misc_projects/NiceCleanCocosProject/NiceCleanCococProject/cocos2d/cocos/2d/CCLabel.cpp:1607
#16	0x000000010034b73c in cocos2d::Label::updateEffectUniforms(cocos2d::Label::BatchCommand&, cocos2d::TextureAtlas*, cocos2d::Renderer*, cocos2d::Mat4 const&) at /Users/kasia/Documents/misc_projects/NiceCleanCocosProject/NiceCleanCococProject/cocos2d/cocos/2d/CCLabel.cpp:1616
#17	0x000000010034c5dc in cocos2d::Label::draw(cocos2d::Renderer*, cocos2d::Mat4 const&, unsigned int) at /Users/kasia/Documents/misc_projects/NiceCleanCocosProject/NiceCleanCococProject/cocos2d/cocos/2d/CCLabel.cpp:1806
#18	0x000000010034ce84 in cocos2d::Label::drawSelf(bool, cocos2d::Renderer*, unsigned int) at /Users/kasia/Documents/misc_projects/NiceCleanCocosProject/NiceCleanCococProject/cocos2d/cocos/2d/CCLabel.cpp:1910
#19	0x000000010034cd74 in cocos2d::Label::visit(cocos2d::Renderer*, cocos2d::Mat4 const&, unsigned int) at /Users/kasia/Documents/misc_projects/NiceCleanCocosProject/NiceCleanCococProject/cocos2d/cocos/2d/CCLabel.cpp:1892
#20	0x0000000100377fe8 in cocos2d::Node::visit(cocos2d::Renderer*, cocos2d::Mat4 const&, unsigned int) at /Users/kasia/Documents/misc_projects/NiceCleanCocosProject/NiceCleanCococProject/cocos2d/cocos/2d/CCNode.cpp:1249
#21	0x00000001003a601c in cocos2d::Scene::render(cocos2d::Renderer*, cocos2d::Mat4 const&, cocos2d::Mat4 const*) at /Users/kasia/Documents/misc_projects/NiceCleanCocosProject/NiceCleanCococProject/cocos2d/cocos/2d/CCScene.cpp:210
#22	0x0000000100503cd8 in cocos2d::GLView::renderScene(cocos2d::Scene*, cocos2d::Renderer*) at /Users/kasia/Documents/misc_projects/NiceCleanCocosProject/NiceCleanCococProject/cocos2d/cocos/platform/CCGLView.cpp:494
#23	0x000000010040b924 in cocos2d::Director::drawScene() at /Users/kasia/Documents/misc_projects/NiceCleanCocosProject/NiceCleanCococProject/cocos2d/cocos/base/CCDirector.cpp:292
#24	0x000000010040e220 in cocos2d::Director::mainLoop() at /Users/kasia/Documents/misc_projects/NiceCleanCocosProject/NiceCleanCococProject/cocos2d/cocos/base/CCDirector.cpp:1392
#25	0x000000010040e270 in cocos2d::Director::mainLoop(float) at /Users/kasia/Documents/misc_projects/NiceCleanCocosProject/NiceCleanCococProject/cocos2d/cocos/base/CCDirector.cpp:1403
#26	0x000000010051dfd8 in -[CCDirectorCaller doCaller:] at /Users/kasia/Documents/misc_projects/NiceCleanCocosProject/NiceCleanCococProject/cocos2d/cocos/platform/ios/CCDirectorCaller-ios.mm:137
#27	0x0000000101da55f8 in -[DYDisplayLinkInterposer forwardDisplayLinkCallback:] ()

I’m about to give up and wait for some hints, boo.

tdebock · May 11, 2020, 12:57am

this->schedule([this](float)
                   {
                        ++_i;
                       _crashingLabel->setString(std::to_string(_i));
                   }, 1, CC_REPEAT_FOREVER, 0, "repeat");

On this example, can you change to:

this->schedule([this](float)
                   {
                        ++_i;
                       std::string str = std::to_string(_i);
                       _crashingLabel->setString(str);
                   }, 1, CC_REPEAT_FOREVER, 0, "repeat");

Check if there is anything weird going on with the string object memory wise upon crash?
Just a random thought. I have had issues with string references and memory issues in the past.

R101 · May 11, 2020, 2:38am

This is a little concerning. Any chance you can upload the test project somewhere and link it in here? Best thing you can do is add all folders excluding the cocos2d folder and any build folders (meaning don’t include the 3-4GB Android build folders or Win32 build folders etc.). The final archive should be pretty small after that.

If you don’t already have a github.com account, then make one, and you can submit this as an issue in the cocos2dx repository: https://github.com/cocos2d/cocos2d-x/issues

Also, have you tried the latest code from the v4 github repository? This may have already been fixed in there. I personally haven’t had any issues at all with v4 on Windows or Android with labels (at least not yet), but I am using the latest source from the v4 github repo.

EDIT:
Looks like this exact issue may have already been reported multiple times along with a possible fix:
https://github.com/cocos2d/cocos2d-x/issues/20000
https://github.com/cocos2d/cocos2d-x/issues/20478

Apply the fix from there and see if it helps.

Darren · May 11, 2020, 3:48am

Not sure if this will help or not, i have changed this line of code https://github.com/cocos2d/cocos2d-x/blob/v4/cocos/renderer/backend/ProgramState.cpp#L231 to:
cp->_vertexLayout = std::make_shared<VertexLayout>(*_vertexLayout);

which fixes some 3d mesh issues that i had. But it causes label crashing issue on metal (apple devices).

for my fix i’ve changed the Label::updateBatchCommand function to be like so:

void Label::updateBatchCommand(Label::BatchCommand &batch)
{
    CCASSERT(_programState, "programState should be set!");

    auto& pipelineDescriptor = batch.textCommand.getPipelineDescriptor();
	setVertexLayout(pipelineDescriptor); // this will set vertex layout for _programState
	// after setting once, just clone from it
    CC_SAFE_RELEASE_NULL(pipelineDescriptor.programState);
    pipelineDescriptor.programState = _programState->clone();

    auto &pipelineShadow = batch.shadowCommand.getPipelineDescriptor();
    CC_SAFE_RELEASE_NULL(pipelineShadow.programState);
	pipelineShadow.programState = _programState->clone();

    auto &pipelineOutline = batch.outLineCommand.getPipelineDescriptor();
    CC_SAFE_RELEASE_NULL(pipelineOutline.programState);
    pipelineOutline.programState = _programState->clone();
}

you can try applying these 2 changes and see if it does anything for your problem

kalika06 · May 11, 2020, 9:28am

Thank you guys, it’s such a pleasure posting questions here.
@R101 thanks, next time the issues will be my place to start. However, those fixes do not work, neither of them.

Hmm, more debugging shows that this IS indeed vector resize issue. But that fix doesn’t seem to work. But obviously it worked for that guy. I’m sure the code’s the same. What am I missing? I’ll keep you posted. Probably some silly mistake of mine.

(Latest GitHub build of cocos does not help).

EDIT: my mistake - I only changed two of the files out of four. So the fix from the first link of @R101 (issue no 20000) prevents the crash on iOS, but makes the labels disappear sometimes, just like on Android. Something’s still missing.

I’ll keep investigating later when I find time.

kalika06 · May 11, 2020, 9:45am

Darren, thanks, looked promising, but… still crashing.

kalika06 · May 11, 2020, 9:52am

@tdebock, I’m sorry, I’m not sure I understand what you would have me check? (what can possibly go wrong with a local, created at runtime, string, that is referenced and copied before going out of scope?)

kalika06 · May 17, 2020, 6:57pm

I have opened an issue with this problem: https://github.com/cocos2d/cocos2d-x/issues/20523

ericb365 · May 28, 2020, 4:01am

I had the same issue, I think this bug might be related with font size. It crashed whenever I changed the string of labels that where with createWithTTF. Noticed my font size was 37.5 and when I rounded it, it fixed the problem. Argument does expect a float tho. Also noticed that going over 45 in font size would also cause the same issue. Results may vary using different fonts and label scaling.

cybergate · July 10, 2020, 4:48am

I also have this problem and fixed with these 2 updates.
For crash:

github.com/cocos2d/cocos2d-x

Label rendering may crash due to _batchCommands vector resize

opened 02:56PM - 16 Mar 20 UTC

cadaver

- cocos2d-x version: 4.0 https://github.com/cocos2d/cocos2d-x/blob/63d340836b…6e3ee117025829ae126993176a631c/cocos/2d/CCLabel.cpp#L1900 When number of Label's BatchCommands grows during Label::draw() execution, resizing the BatchCommand vector can cause the old CustomCommands (and their vertex buffers) within them to be destructed. Yet they will be tried to be updated into, as the _vertexCapacity fields of the CustomCommands are copied during the vector resize, and the commands are still thought to be valid. This causes a crash. To fix, I simply cleared the BatchCommand vector before resize, to ensure the CustomCommands within BatchCommands will be constructed to an initial state, and vertex buffers will be recreated correctly during updateBuffer() before rendering. Ie. _batchCommands.clear(); _batchCommands.resize(_batchNodes.size());

For missing character:

Cheers.