Noesis Integration with cocos2dx?

I have been working with cocos2dx for the past couple of months and have grown to like it a lot! But if there is one area where it could be lacking is its ability to build UI (preferably using some form of vector graphics).

To do this we have been looking at an engine called Noesis (that is based on XAML) that has the ability to build and render UI. It also claims that it can be used with any openGL pipeline and has been integrated with a few openGL based Game engines successfully. There is a GLKit integration with Noesis already!

The idea here is to subclasss “Node” and to create a draw function that uses Noesis’ renderer to perform the drawing commands. The draw function can be called by the scene’s draw function adding the custom node.

The recommended drawing flow (as guided by the author of Noesis) was the following.

// Offscreen
xamlRenderer->Render(commands.offscreenCommands.GetPtr());

// Draw scene with cocos2d

// Render GUI on top of cocos scene
xamlRenderer->Render(commands.commands.GetPtr());

//Invalidate cocos state
cocos2d::GL::invalidateStateCahche()

I got the Noesis Library (lib.a) and the header files and was able to successfully include and link it with my xcode project running cocos2dx. But im facing run time issues (crashes) when i try to render the xaml. The following code shows how i have set my draw functions for my XAML node and following that is the screenshot for the errors im getting!

Has anybody successfully tried this integration before? If not what am i doing wrong and what needs to be done to get this to work?

void MyXAMLNode::loadElement(){
Noesis::PtrNoesis::FrameworkElement xaml=Noesis::GUI::LoadXamlNoesis::FrameworkElement(“Tux.xaml”);
xamlRenderer = Noesis::GUI::CreateRenderer(xaml.GetPtr());

}
void MyXAMLNode::drawRenderCommands(){

xamlRenderer->SetSize(this->getContentSize().width , this->getContentSize().height);
Noesis::GUI::Tick();
static Noesis::Float64 t;
t += 1/60;
xamlRenderer->Update(t);
xamlRenderer->Render(xamlRenderer->WaitForUpdate().commands.GetPtr());
cocos2d::GL::invalidateStateCache();

}

void MyXAMLNode::drawOffScreenCommands(){
xamlRenderer->Render(xamlRenderer->WaitForUpdate().offscreenCommands.GetPtr());

xamlRenderer->SetSize(this->getContentSize().width , this->getContentSize().height);
Noesis::GUI::Tick();
static Noesis::Float64 t;
t += 1/60;
xamlRenderer->Update(t);

}

4 Likes

HI, i’m also very interested by the Noesis, it’s very good idea, i can’t help :), but i have very appreciated the idea, keep going please.

To isolate the problem I would disable the Offscreen pass. That way, things are easier. With that, what error are you getting? It seems that error are happening inside Cocos, anyone has a clue about why that could be happening?

Yes i did try doing this by omitting the offscreen render commands, i only included that here with the idea of full disclosure of the problem. I still encountered a run time crash when i tried to do that!

please, paste the exact crash when you are omitting the offscreen commands.

So, the rendering pipeline of cocos2d-x perhaps doesn’t offer all the needed features to make the integration smoother, but this is what you need to do:

  • Have a NeosisNode

I think you already have this. Just draw everything in a CustomCommand

  • Invoke Neosis render

From your CustomCommand you can all any OpenGL code, so you can call xamlRenderer->Render(), but after this you will need restore the previous GL state

  • Restoring the previous GL state

Calling cocos2d::GL::invalidateStateCache() is not enough. That function only invalidates a few GL states, but not all (will be fixed in v4)…
So you might need to call some GL code manually to restore to the previous GL state.

One way to do this is by using the OpenGL ES Frame Grabber that comes with Xcode 5. You can do inspect the GL calls that are being done by the Neosis renderer, in order to know which states need to be restored.

Regarding the crash, it is difficult to tell why it is crashing there. A working test case will be the easiest way to know why it is crashing, but before that, please make sure that you are following the steps that I mentioned above.

thanks.

1 Like

To manually restore the GL state I recommend the following code:

https://github.com/FourthQuarter/NoesisLeadwerksWrapper/blob/master/Integration/UI/OpenGLState.cpp

I think the crash is happening because calling invalidateStateCache resets the matrix stack, so you only have one matrix of each type. When exiting visit, the node pops its matrix off the stack, so it could conceivably run out and crash or assert.

So the solution seems to be not calling invalidateStateCache but instead save and restore with the code I posted.

1 Like

I took all the tips from this forum and seem to have made some progress regarding using Noesis with cocos2dx. I setup the draw inside of a custom command like Ricardo suggested and i am using some of the code from the openGLState class that was pointe out by Jesus.

This is how my XAMLNode looks right now

void WDXAMLNode::loadElement(){
Noesis::PtrNoesis::FrameworkElement xaml=Noesis::GUI::LoadXamlNoesis::FrameworkElement(“Tux.xaml”);

xamlRenderer = Noesis::GUI::CreateRenderer(xaml.GetPtr());
glState = OpenGLState();

}

void WDXAMLNode::drawAll(){
glState.Store();
//this->drawOffScreenCommands();
this->drawRenderCommands();
glState.Restore();

}

void WDXAMLNode::draw(Renderer* renderer, Mat4 &transform, uint32_t flags){
noesisCommand->init(1.0, transform, flags);
noesisCommand->func = CC_CALLBACK_0(WDXAMLNode::drawAll, this);
renderer->addCommand(noesisCommand);

}
void WDXAMLNode::drawRenderCommands(){

xamlRenderer->SetSize(this->getContentSize().width , this->getContentSize().height);
Noesis::GUI::Tick();
static Noesis::Float64 t;
t += 1/60;
xamlRenderer->Update(t);
xamlRenderer->Render(xamlRenderer->WaitForUpdate().commands.GetPtr());

}

void WDXAMLNode::drawOffScreenCommands(){
xamlRenderer->Render(xamlRenderer->WaitForUpdate().offscreenCommands.GetPtr());

xamlRenderer->SetSize(this->getContentSize().width , this->getContentSize().height);
Noesis::GUI::Tick();
static Noesis::Float64 t;
t += 1/60;
xamlRenderer->Update(t);

}

I am using the openGLState class to do the save and restore of the GPU state. I had to make some changes to the class to get it to work in my project.

I had to omit these symbols
> PFNGLBINDFRAMEBUFFERPROC glBindFramebuffer;

PFNGLBINDBUFFERPROC glBindBuffer;
PFNGLACTIVETEXTUREPROC glActiveTexture;
PFNGLUSEPROGRAMPROC glUseProgram;
PFNGLGETVERTEXATTRIBIVPROC glGetVertexAttribiv;
PFNGLENABLEVERTEXATTRIBARRAYARBPROC glEnableVertexAttribArray;
PFNGLDISABLEVERTEXATTRIBARRAYARBPROC glDisableVertexAttribArray;
PFNGLBLENDEQUATIONPROC glBlendEquation;
PFNGLBINDVERTEXARRAYPROC glBindVertexArray;

and the following lines of code in the openGLState constructor

> GL_IMPORT(PFNGLBINDFRAMEBUFFERPROC, glBindFramebuffer);

GL_IMPORT(PFNGLBINDBUFFERPROC, glBindBuffer);
GL_IMPORT(PFNGLACTIVETEXTUREPROC, glActiveTexture);
GL_IMPORT(PFNGLUSEPROGRAMPROC, glUseProgram);
GL_IMPORT(PFNGLGETVERTEXATTRIBIVPROC, glGetVertexAttribiv);
GL_IMPORT(PFNGLENABLEVERTEXATTRIBARRAYARBPROC, glEnableVertexAttribArray);
GL_IMPORT(PFNGLDISABLEVERTEXATTRIBARRAYARBPROC, glDisableVertexAttribArray);
GL_IMPORT(PFNGLBLENDEQUATIONPROC, glBlendEquation);
GL_IMPORT(PFNGLBINDVERTEXARRAYPROC, glBindVertexArray);

Im assuming these symbols are particular to the game engine noesis was being integrated with. Im not sure if i need to do replace these lines of code with something that is compliant with cocos2dx, but for now i have simply left them out.

I had to make some changes to some of the code in the save() and restore() methods as well to silence some warnings (which might be because of using openGLES).

I am trying to import a sample xaml file (Tux.xaml) while my run time crash is gone, i dont see the xaml element on screen, instead i see this. Im not sure if this is the intention of the trial version or if the xaml is not being drawn at all!

I also get a run time error log from cocos2dx (on every draw) which says

OpenGL error 0x0500 in /Users/master/Desktop/…/cocos2d/cocos/renderer/CCRenderer.cpp saveRenderState 144

Any ideas on what might be happening? Seems like im close to the solution (hopefully!)

1 Like

@alias1234
If you can see something on the screen, it means it is very promising :smile:

Things to take into account:

  • In your custom command callback, you will receive the transformation matrix: const Mat4& transform. That is the node-to-parent transform matrix. You should use it to render your Neosis node… perhaps you are already using it, but I couldn’t find it in the code that you posted.
1 Like

Isnt passing the transform matrix (Mat4 &transform) to the custom command enough? Like so?

void WDXAMLNode::draw(Renderer* renderer, Mat4 &transform, uint32_t flags){
noesisCommand->init(1.0, transform, flags);
noesisCommand->func = CC_CALLBACK_0(WDXAMLNode::drawAll, this);
renderer->addCommand(noesisCommand);

}

@alias1234
you have to use the matrix in your renderer code.

in opengl es 2.0 there is no stack of matrices, so we pass the “stacked” matrix as an argument. And you have to use that matrix in your code…

I mentioned on before that it was the Node-to-Parent matrix… but I was wrong, that matrix is the Node-to-World (the Model View matrix), so you can pass it to your shader directly, in case your shader is expecting a Model View matrix.

Good! Things are starting to appear on screen. I see two problems in the screenshot:

  • I would say that the trial expired. That is why you are getting a black screen. I am sending you a new one.
  • It seems that the Noesis Renderer is not configured with the proper size. The text “NoesisGUI Trial…” should appear smaller.

Awesome! Will use the new library files you send over.

Couldnt the size of the text have something to do with what @ricardo pointed out? That i need to pass the transform matrix to the noesis renderer? If so is there a way to do that?

No, it is not related to passing matrices. We do not accept matrices in our API. You just have to set correctly the dimensions of the surface where noesisGUI is being rendered to. Have you checked if those values are correct?

I was able to use the newer library files you sent over, it worked well and i can see the penguin on screen finally!

@jsantos i am setting the dimensions using the xamlRenderer->SetSize method. Since im using a container (subclassed node) i want the xamlRenderer to render to the size of the subclassed node. So i am trying to set the size of the xamlRenderer to the size of the subclassed node.

Is this what SetSize() expects? Or am i missing something here?

For the image you see i manually set the size of the xaml renderer to (400 , 400)

Also the openGL error still persists.

OpenGL error 0x0500 in /…/cocos2d/cocos/renderer/CCRenderer.cpp saveRenderState 144

This is the function that seems to be throwing this error, and it happens when glGetError() is called from within CHECK_GL_ERROR_DEBUG(). Not sure whats going on here exactly!

void RenderQueue::saveRenderState()
{
_isDepthEnabled = glIsEnabled(GL_DEPTH_TEST);
_isCullEnabled = glIsEnabled(GL_CULL_FACE);
glGetBooleanv(GL_DEPTH_WRITEMASK, &_isDepthWrite);

CHECK_GL_ERROR_DEBUG();

}

7 Likes

This is a game changer … Great work

2 Likes

SetSize() must be set to the dimensions of the target surface. If not set like that, you won’t get pixel perfect results.

@alias1234
Well done!
Not sure why you are getting the error though :frowning: