How to debug android native crash with anonymous and unknown backtrace?

cocos2d-x
#1

I use cocos2d-x and ndk-build to build app on arm64. But when i run it on 64bit device, the app crash randomly with error signal 11 (SIGSEGV), and the backtrace shows anonymous and unknown.

I use cocos2d-x 3.17.1, ndk 16, Android Studio 3.4.1, gradle tools 3.2.0 and gradle wrapper 4.6.
I tried ndk-stack but it didn’t show me more useful information.

This is the log in the Logcat.

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
Build fingerprint: 'Xiaomi/chiron/chiron:8.0.0/OPR1.170623.027/V10.3.1.0.ODECNXM:user/release-keys'
Revision: '0'
ABI: 'arm64'
pid: 17667, tid: 17711, name: GLThread 135726  >>> com.test.myapp <<<
signal 11 (SIGSEGV), code 2 (SEGV_ACCERR), fault addr 0x72adf0f460
     x0   00000072bc089378  x1   0000000000000000  x2   fffd8072bc08bc18  x3   fffd8072bb3f4950
     x4   00000000ee1763de  x5   fffd8072bc096c88  x6   ff687373604f6d64  x7   7f7f7f7f7f7f7f7f
     x8   00000072b09a9f08  x9   00000072b09a9f00  x10  fffffffffffffffb  x11  00000072a9bcc6c8
     x12  00000072aceade40  x13  0000000000000000  x14  0000000000697474  x15  00000072a9ac6c10
     x16  0000000000000001  x17  fffa0072a9ac58c8  x18  0000000000000012  x19  00000072b09a9e98
     x20  fffd8072a9bc31e0  x21  00000072a9bcfa00  x22  00000072bc0893d8  x23  00000072ac48ab40
     x24  00000072b162cca8  x25  00000072ade90978  x26  00000072a94d0c20  x27  00000072a99953e0
     x28  00000072a9995070  x29  00000072b0da8080  x30  fffd8072a9ac48a0
     sp   00000072b0da8060  pc   00000072adf0f460  pstate 0000000080000000
 backtrace:
     #00 pc 0000000000078460  <anonymous:00000072ade97000>
     #01 pc fffd8072a9ac489c  <unknown>

This is the log with ndk-stack

********** Crash dump: **********
Build fingerprint: 'Xiaomi/chiron/chiron:8.0.0/OPR1.170623.027/V10.3.1.0.ODECNXM:user/release-keys'
pid: 17667, tid: 17711, name: GLThread 135726  >>> com.test.myapp <<<
signal 11 (SIGSEGV), code 2 (SEGV_ACCERR), fault addr 0x72adf0f460
Stack frame #00 pc 0000000000078460  <anonymous:00000072ade97000>
Stack frame #01 pc fffd8072a9ac489c  <unknown>

I expect the backtrace and ndk-stack can show me where the problem is, but it shows only anonymous and unknown.

Is anybody released arm64 application on google play?
#2

@R101, @mars3142 @CrazyHappyGame and others here are awesome at the mysterious world of Android internals. Perhaps we can ask them to see what they think?

#3

Any chance you could switch to cmake builds, and make your life so much easier? :wink:

Anyhow, perhaps an obvious question, but have you checked the APK contents to make sure there are 64bit binaries in there? Specifically, arm64-v8a libraries.

Have you run the 32 bit armeabi-v7a version to ensure it actually works?

#4

Hey,
Stack traces indicate that crash happened in anonymous mem region which is most likely art generated code for some Java code that may belongs to cocos2dx or some third party Java/jar.
In this situations stack trace does not make much sense, you better put some logs and try to isolate faulting module first. Also make sure that your Android device does not become low at memory at runtime, sometime low memory can cause this kind of crash too. I hope you are not using any native crash handler if so please show me logs of your native crash handlers.

#5

If you can reproduce crash then I think you should debug it:
https://developer.android.com/studio/debug

#6

yes i use android studio to analyze the apk, and it contains both armeabi-v7a and arm64-v8a libraries.
when i build only armeabi-v7a version, it runs correctly on the same device.

I tried cmake today, and the 64 bit version still crashes.

#7

In that case, you need to do exactly what @CrazyHappyGame suggested, and connect your Android device to your PC, and debug the 64bit application.

#8

Hi, it crashes quite randomly, and I don’t know which part has problem.
I have tried debug with android studio. But when it crashed, the log didn’t help and I could not trace the stack.

#9

Hi, I think I didn’t use any native crash handler.

Since the problem maybe caused by third party jar and the app crashed randomly, how can I detect which part is wrong?

#10

I’m also facing something similar issues, and working with Bugsnag developers to figure out why. They can’t get the ndk crash data reliabliy , even on a fresh cocos2dx project.

#11

Is this happening on xiaomi phones only?
If not it’s not specific to device, can you share your debug build of app?
Looking again on your traces, I have come up with another potential cause, as crashed thread is glthread and thread must be created by native code as thread stack does not have art/dalvik symbols. This thread might be created by some third party rendering extensions or might be created by opengl driver on behalf of cocos2dx’s rendering thread(some driver work like that). In either case you can comment out scene/layer/node and other game rendering components one by one or do some binary search to find out offending module/function. You need to experiment with it till your app stop crashing. It’s tedious but should be effective to narrow down the issue.

#12

Not just xiaomi, it will phone to bugsnag to report crash

https://filedn.com/l3TGy7Y83c247u0RDYa9fkp/temp/cocos2dx/testcrashbugsnagcocos3.17.2.apk

#13

@overcocos
I am sorry above message was meant to @smirkking. Although I had downloaded your app and looked into it. Bugsnag swallowed stack trace which otherwise should be visible in logcat. I can confirm Bugsnag collected crash report regarding SIGFPE may be caused by Integer/float divide by zero and sent it to Bugsang server.

#14

Thanks, actually at bugsnap, we can’t get the correct symbols, only some parts. The bugsnag guys are still trying to figure out why.

#15

We add logs in our code and find that maybe jit cause the crash.
So we add code below to disable jit on android 64 bit device.

-- at tht start of main.lua
if cc.Application:getInstance():getTargetPlatform() == 3 and jit.arch == "arm64" then
    jit.off()
end

Finally it works in our case.
Thanks very much for your help @slackmoehrle @R101 @shivmsit @CrazyHappyGame @overcocos

#16

Some developer in Chinese forum said, it works after roll back to 3.16. So i thought if the luajit in 3.16 works because 3.17.1 updated luajit to 2.1.0-beta3.

#17

@PatriceJiang built latest luajit, it includes crash fix, you may have a try: https://github.com/PatriceJiang/cocos2d-x-3rd-party-libs-bin/tree/v3-latest-luajit/lua/luajit/prebuilt/android.

#18

Thanks. We tried the new library and it won’t crash on 64 bit device.
But there is new problem on “or” operator.
I just simplify the case.

local color = cc.c3b(0,  0, 0) or cc.c3b(255, 255, 255)
print(color.r, color.g, color.b)

Most of time, the result is right “0 0 0”. But sometimes, the result is “255 255 255”.

#19

Please correct me if I’m wrong, but isn’t the LUA or a logical operator, and not a bitwise operator? The line local color = cc.c3b(0, 0, 0) or cc.c3b(255, 255, 255) doesn’t make sense if that is in fact the case.

If it somehow is a bitwise operator in this specific statement, then shouldn’t the result always be 255, 255, 255, since 0 OR 255 = 255?

#20

https://www.lua.org/pil/3.3.html

In LUA, all logical operators consider false and nil as false and anything else as true.
So a or b equals to b only when a is false or a is nil.