Cocos2dx VR API

With all this VR hype, I was commissioned to create a VR API for cocos2d-x.
The idea is that 3d games will have the option to be use VR. And our 2D games as well! why not, right? Instead of playing Super Mario on your TV, you play it in a VR headset. I think it be a very interesting feature.

Each VR platform out there has its own set of APIs and libraries. The idea is to create a VR abstraction API for cocos2d-x to make easier the porting to different VR platforms.

I’ve just started the research, and I’ll post my progress here.

The VR platforms that I’ll study are:

  • Gear-VR
  • Occulus
  • Cardboard
  • Deepoon

songchengjiang already created a cocos2d-x VR example which I’ll use as well: https://github.com/songchengjiang/cocosVR/

I’ll take a look at Unity VR API as well, since I think they already have an abstraction for VR.

Any other?

9 Likes

I’ve just found this project: https://github.com/rsanchezsaez/CardboardSDK-iOS which I think will save me at like 2 or 3 weeks of work.

it has all the needed distortions for the eyes.

1 Like

This might be helpful also:

1 Like

@ricardo I pushed a simple Cardboard VR app for android that I built with Unity. The most interesting part with Unity is the asset store where I was able to obtain complex meshes and materials. I will soon push an app where there are whales floating on top of the viewer.

Please ask away any questions, happy to share my project too, it’s pretty simple source code though.

ps my VR app - https://play.google.com/store/apps/details?id=net.humblestack.vr.ballgame

1 Like

more code from google: https://github.com/google/vrview
this time it is official source code (not de-compiled code) and with Apache license. Better use this code, than the non-official one to avoid possible legal issues.

The relationship between Director, GLview, Scene and Renderer is kind of coupled/confusing. It is not bad, but I think it could be improved a bit… but I don’t want to break backward compatibility, so that will be a v4.0 thing.

so, for v3.x I think the easiest way to support VR is by:

  • Have a new view called VRGLView (or something like that), responsible for implementing the stereo view + distortion
  • Director should delegate more responsibilities to the “view” and not be in charge of invoking the rendering
  • Scene::render(...) should be called twice from the view with the correct projections.

Something like this:

void Director::drawScene()
{
...
    if (_runningScene)
    {
        _runningScene->stepPhysicsAndNavigation(_deltaTime);
        _renderer->clearDrawStats();
        
        // OLD WAY
        // _runningScene->render(_renderer);  

       // NEW WAY:
       _view->renderScene(_runningScene);
        
        _eventDispatcher->dispatchEvent(_eventAfterVisit);
    }
...
}

// NEW argument: Projection
// which will either be the Left eye project, Right eye projection or the center projection
void Scene::render(Renderer* renderer, const Projection& projection)
{
...
}

and the only thing users should do in order to support VR is in AppDelegate.cpp:

    if(!glview) {
        // old way:
        // glview = GLViewImpl::create("Cpp Tests");
        // new way:
        glview = VRGLViewImpl::create("Cpp Tests");
        director->setOpenGLView(glview);
    }

3d music + head tracking will need to be added manually to the game, of course, but with that line change “2d games” could be played in VR.

At least in theory… now time to implement it. I know that I will find some issues with Camera since it has some stuff hardcoded.

5 Likes

May want to also look into http://mozvr.com/ and/or related vr on the web for cocos2d-js.

1 Like

I started this branch:

For the moment it doesn’t even compile. It is based on the Cardboard iOS and JS code. I spent some time in trying to understand how VR works internally. Hopefully next week I’ll have something working.
Basically what I’m doing is adding our own VR experience (compatible with Cardboard) which could be easily replaced by different VR libraries: oculus, cardboard, etc…

changing the strategy a bit:
Instead of adding the API on cocos2d-x, I’ll add it on the cocosVR example:

I’m implementing the generic VR rendering. This is my repo:

1 Like

Hi, Riq
We are make progressing. I heard that Songcheng.Jiang has nearly finished porting cocos2d-x games to cardboard. Before this, he has implemented occulus, gear and deepoon porting. He will propose good suggestions for VR API after he finished cardborad.

@dabingnn great! I’ve been working on our native VR solution… I’m working on the cocosVR repo. I’ll send a PR with the working VR this week.

@ricardo I just finished the integration of cardboard SDK, If you have already completed the design of VR API, could you share it with us? I think it can help me to design my VR API quickly.

@songchengjiang
good to know that you have finished cardboard integration!

But perhaps there was a misunderstanding. I thought that you were working on the VR API.
I was working on implementing our own VR renderer, which is almost finished.
I still have some issues with the some projections, but I think it is easy to fix it.
I’ll continue working it… probably I’ll have it working Ok tomorrow.

in my mind, the VR API should be part of cocos2d-x, and just need to do something like:
director->enableVR() or something like that.

Internally the VR API should have 2 main components:

  • input: which means head tracking
  • stereo rendering

and users should be able to specify a device/sdk somehow, eg:

  • builtin (the one that I’m working on)
  • occulus
  • deepoon
  • cardboard
  • etc…

Let’s put the ideas here and we can build the cocos2d VR API together!

1 Like

@ricardo OK, thanks for your suggestion!
In the next step, I will design the VR API in detail, I think that will take 2 or 3 days,
when I finish it, I will share with you.
I will request merging the PR of Cardboard to the master branch of cocosVR today, I hope it can do a little helping.

@songchengjiang ok. great! thanks.

yeah! :smile:
so, VR is kind of working… some variables are hardcoded and other stuff… but it is a nice proof of concept that we can have our own VR implementation. I think I’m done with this. There is no point in continuing with this proof-of-concept since it “works”.

PR: https://github.com/chukong/CocosVR/pull/15

So, the next step:

  • create the VR API in cocos2d-x.
  • and in the process of creating the cocos2d VR API, remove all the hardcoded values.

@songchengjiang I’ll start with the cocos2d-x VR API now.

4 Likes

Yeah, really great. Congratulations!

@ricardo Very glad to hear that! Let us designing cocos2d VR API together!

@songchengjiang

Ok… so, let’s start:

There two uses cases:

Use Case A: I want to “port” my existing game to VR

This is a quick & dirty way to have VR in your game.
Users will need to upgrade cocos2d-x v3.12 (3.13?) and then just enable VR rendering, and nothing more than that.
This won’t work for many games, since head-tracking won’t be enabled. So it will only be applicable for games that use a “remote control” or something like that (specially for games that only require just one button, like Flappy Bird). If games require a complex control, it won’t work I guess.

Use Case B: I want to design a new game with full VR experience (or change an existing game)

  • In this case, users will either create a new game from scratch
  • or migrate existing ones: They will need to add headtracking input support in their games, and adapt the game mechanics to VR.

The only different between Use Case A and Use Case B, from an API point of view, is that:

  • Use Case A: Only uses VR rendering
  • Use Case B: Uses full VR: VR rendering + VR headtracking

So, let’s talk about high level components:

VR Rendering:

Who is responsible for owning this?
CocosVR creates a Node with a custom rendering command, and two cameras.
And I think it was Ok as a proof of concept, but it is not suitable for a final feature. VR should work with the need to modify existing game code (Use Case A).

I think the VR rendering should be implemented in the CCGLView class. And also Director should deletegate more responsibilities to CCGLView, like the rendering.

Something like:

Director::drawScene()
{
    ...
     _glview->renderScene(_runningScene, _renderer);
    ...
}

CCGLView::renderScene(Scene* scene, Renderer* renderer)
{
   // if VR is enabled to the stereo rendering
   // else, do the mono-rendering.

  // delegate the rendering to the right VR rendering instance:
  // deepoon rendererer
  // cardboard renderer
  // ...
}

VR headtracking
We need a new class for the headtracking… in Unity it is called InputTracking, in Cardboard is called HeadTransform… not sure what is the name in the other devices.
But right now the name is not that important (as long is it is a good name).
This class should have at least these properties:

  • head position / forwardVector : returns a vec3 or vec4
  • head rotation: returns a quaternion

VR Misc:
A class that reports info about the VR device. Perhaps driver version (eg: deepoon v1.2), and other properties reported by the driver.

@songchengjiang
Thoughts? What do you think?

@songchengjiang
I started the VR fork for cocos2d-x.
it is here: https://github.com/ricardoquesada/cocos2d-x/tree/vr

it compiles… but it doesn’t work yet. I’m implementing the “Generic” rendering, but it would be great if you could add Cardboard support.

I added the VRProtocol which is based on what you said:
https://github.com/ricardoquesada/cocos2d-x/blob/vr/cocos/vr/CCVRProtocol.h

and I’m implementing the “generic” thing here: CCVRGeneric.cpp:
https://github.com/ricardoquesada/cocos2d-x/blob/vr/cocos/vr/CCVRGeneric.cpp

ideas? thoughts?