Hard to find Memory or Threading issue

I realize Cocos2D-X is no longer supported, but I have a bit of a unique problem, and truly any direction would be very helpful.

My game uses a mostly custom game engine, but we started out using Cocos-2Dx 3.16 for the per-platform setup for things such as OpenGL context, sounds, and networking. This has worked great over the years and we actively publish on iOS and Android and then we debug on Mac.

Over the last 3-6 months, we have noticed that Android has destabilized. This is true on Android even if we comment out all of our custom code and simply let Android boot up to a black screen. The issue is that it is unstable in highly unpredictable and varying ways which makes me suspect some sort of heap corruption issue. Using HWASan does not seem to find anything. Additionally, it can not crash sometimes, but it will always crash on the boot screen after a few frames when attached to the debugger.

Ways I have seen it crash:

  • Fails to release graphics memory for some reason
  • OpenGL crash in random calls when allocating memory
  • At startup in the debugger, it tends to crash in the garbage collector or threading code within the Android Runtime

My best guess is this has occurred because this 7 year old Cocos code is somehow not compatible with the newer SDKs Google forces you to use now, or it has something to do with the newer versions of the Android operating system themself. I have done my best to cut out every sub system of cocos2d-x as possible to narrow down the problem, but so far, it always acts the same, which makes me think its got to be either in the JNI bridge, OpenGL setup, or the sound driver setup. Any ideas, insight or assistance would be deeply appreciated!