@ricardo I implemented a framework base on your ideas.
But the biggest difference between our implementations is my VRRenderer class inherit from Renderer class, not implemented in the CCGLView class.
The following is my consideration:
CCGLView is a abstraction of GL rendering environment. it doesn’t care how to render the scenes(stereo render or not), for it is just the primitives rendering to FRAMEBUFFER.
In order to satisfy the Use Case A, we should render all scenes twice(left eye and right eye), I think visiting render queue twice is OK, one for left eye, another for right eye, we can change the ViewProjectMatrix before visiting render queue.
Details see here: CCVRRenderer.cpp:
HeadTracking implementation is here CCHeadTracking.cpp:
In CCDirector.cpp, we should initialize the VRRenderer and HeadTracking base on VR platform(we can define those in Configuration class, see Director::setDefaultValues())
In order to implement above ideas, I added some interfaces to Renderer and did a little changes in Director::drawScene() and Scene::render(Renderer* renderer), see here:
in CCRenderer.h:
/** Renders into the GLView all the queued `RenderCommand` objects */
virtual void render();
virtual void startRender(){}
virtual void endRender(){}
I implemented a simple version of VRCardboard inherit from VRProtocol to explain my ideas, see here:
Base on the framework: Use Case A:
user just updating the cocos2d-x engine.
Use Case B:
user updating the cocos2d-x engine.
updating camera position and rotation base on HeadTracking per-frame in coding.
It is good to know that you added cardboard support! yeah!
Regarding the rendering, the VR code could be in the rendering class, or view, or somewhere else. And it will work.
But I didn’t understand why you put it as a subclass of Renderer.
The responsibility of the renderer, is to render things. You send it triangles, and it will render them.
It doesn’t know about stereo rendering, or mono rendering.
I think the VR logic should be outside the renderer… in a module that will tell the renderer to render things twice.
(And also, for performance reasons, the renderer should not have virtual functions)
If we keep the VR logic outside the Renderer it will keep the renderer code simpler, easier to maintain, and even, if in the future we want to replace the current renderer with something like bgfx, it will be easier.
The GLView, on the other hand, it is not just a GL abstraction. It sets the viewport, size, scissor… because it is the “view”.
In fact, regardless of the VR feature, I think the director should only care about scene management, and delegate all the rendering to the view. Unfortunately Director is too fat.
I saw above code, I thought you want to calling scene::render directly. Is that right?
But in scene::render, we can see for every camera, we need draw once, and need change the VP(ViewProjectMatrix), how to ensure that every camera has correct VP? As we know the VP of Left Eye is different with Right Eye’s, means:
LEFT EYE: FINAL_VP = VP * leftEyeOffsetMatrix;
RIGHT EYE: FINAL_VP = VP * rightEyeOffsetMatrix;
scene::render didn’t include all drawing elements, like FPS label, but we had to drawing that stats, right? see Director::drawScene().
partial codes of Director::drawScene() here:
if (_displayStats)
{
showStats();
}
_renderer->render();
Based on the above questions, those are why I want to implement stereo rendering in a subclass of Renderer, even if it could not be a suitable implementation.
I means, regardless of how to implement the VR rendering class, I think we should consider above questions. Is that right?
Any thoughts?
Regarding the stats, I think the view should be responsible for drawing/placing stuff in the “view”.
Director should only be a scene manager with other things little things like FPS stats, and that… but the view should be reponsible for drawing them.
So the view should ask the director “hey… give me the stats, because I need to place them in the view”.
what do you think? do you like it?
UPDATE:
I’ll try to remove the “defaultCamera” from Scene, and the one that is being passed to Scene::render(renderer, camera) is going to be used as the default one.
Currently it is kind of working that in my branch… but touches are other things are broken… debugging…
Sounds Great!
Regarding the VRProtocol, I think it just a abstraction of VR SDKs API(GearVR API, Cardboard API, etc.), it provides us with the required interfaces, not responsible for rendering, see following definition:
Then, we can have two classes, one for rendering(may be in GLView), another for headtracking, they have the same VRProtocol instance. renderer can use rendering API of VRProtocol for VR rendering, and headtracker can use tracking API of VRProtocol for headtracking.
If developers only want to render scene in VR mode, they just enable VR rendering, no more code.
If developers want to use VR rendering and headtracking, first, they enable VR rendering, second, In coding, they create or request a HeadTracking class and change the ViewMatrix of the cameras by themselves per-frame. The advantage of that is they can decide which camera need headtrakcing, which does not(like HUD camera).
So, I’m refactoring the GLView/Scene relationship and it kind of works, but I’m not receiving events… and I found out why: the EventDispatcher is iterating the Scene’s cameras. And I don’t understand why yet. Events should be propagated regardless of the camera.
In fact, the same event is going to be sent multiple times if there are more than one camera.
The code is here:
and I want to remove that logic from EventDispatcher, but I don’t want to break anything also.
Do you know who is yangws, and why he added that code?
@songchengjiang
Another question… what is the purpose of having a default camera?
I mean, I know that one camera is needed… and that is the default one. But does it have any special functionality?
Because I noticed that some internal code is calling getDefaultCamera() and they are doing some stuff based on that. But my plan was to have two default cameras… one for the left eye, and the other for the right eye.
And the default camera was passed to the scene at render time.
No additional cameras were used. Just two: left and right, and they are the default cameras.
The good thing about this approach is that it renders the scene only twice: once per eye
The bad thing is that there are a few places that don’t like this approach and I have to fix it. So, this approach won’t be 100% backward compatible. I guess it will be 99.9% backward compatible.
Also, Camera has a few bugs when used with an FBO + RenderTarget. Apparently there is only one example using it, in cpp-tests. And probably that’s the reason why I’m triggering some bugs.
In any case, I’ll fix those bugs so that Camera can be used with an FBO + RenderTarget in an easy way. No need to create FBO manually.
And after fixing the FBO bugs, I’ll be able to add distortion rendering .
@ricardo The event logic is added to framebuffer and FBO because the users may add multiple 3D and 2D objects, and add cameras to see them. So we can add camera1, to see 3d objects 1,2,3 and 2d object a, b. then add camera2 to see 3d objects 4,5,6,7 and 2d object c,d,e. They act like two groups, to make them do not affect each other, the two cameras are moved far away from the other one. If we want to make 3D and 2D object clickable, the click events should iterate cameras.
This mechanism added a flexible way to rendering 2D and 3D objects in the scene.
Thoughts for this? Should it be removed?
Actually I like the FBO+RenderTarget thing attached to the camera.
But I’m triggering some bugs right now. So, I don’t want to remove that functionality, just fix the bugs that I’m triggering.
By any chance, do you know the viewport is being multiplied by the FBO size… here, in this line:
@ricardo The reason why using multiple cameras is historical, for 2D and 3D rendering compatibility, like @dabingnn said.
Regarding default camera, I have two ideas:
a) like you said, use two default cameras, one for the left eye, another for the right eye.
b) use one default camera, and rendering twice, we change the VIEWPROJECTION Matrix per rendering.
The advantage of a) is we having clear code, we can render scene in separate camera, etc. But the shortcoming is obvious, we need user to create two user-cameras, for per eye. I think backward compatible is difficult.
The advantage of b) is just to make up the shortcomings of a), we have a good backward compatible, user just need to create one user-camera. for headtracking, user just set the center head view matrix to user-camera, and we multiply offset view matrix of per eye in rendering.
Users won’t create cameras. The cameras are going to be created by the VRProtocol implementation. So, from a user point of view a) and b) are almost the same.
Sorry, could you add more details. I didn’t understand option b)
Are you going to reuse the same camera? But you are going to change the eye position before rendering it?
If so, it could work as well, but as you said, the design won’t be as clean as a). I would prefer to go with a) It seems to be more flexible.
Any particular reason why the viewport is normalized when using FBO, but not normalized when not using it? Looks confusing for me.
I noticed that Cameras are “isolated”… and it makes sense. Each camera will “capture” part of the screen an render it. But there is no global camera… a camera that will capture the output of the other cameras. I thought that the “default” camera was a “global” one… but it is just another camera. So I can’t use it to capture the whole scene.
But I might need to introduce the concept of “global camera”… and that will fix the issue.
The code right now is super simple, I like it. But I need to fix this issue. Which seems, at least in theory, easy to fix.
Great！
For now, I think we have a clear framework of VR API.
So, if there are no other problems about VR API,in next step, I will doing the integration of cardboard SDK based on our design. could you give me a stable version even if it is not rendering correctly?
@ricardo I like the design to make our code simple.
some of my suggestions:
It seems that cocos2d-x need to support two working mode: MonoRendering and StereoRendering. I prefer to abstract it out.
How about rendering scene to a frameBuffer scene->render(renderer, framebuffer), then iterate cameras attached. because users may add their cameras, it may conflicts.
For normalised viewport, It seems that absolute viewport may behave differently if viewport size exceeds framebuffer size. Normalised one means it will never trigger this if we specify viewport in (0,1) section. However, viewport is a experimental class, I think we can change it to absolute mode if it is better.
sure… I think there is just one more thing to solve, and is the eye transform and user cameras.
The current design is:
VRProtocol::render(...) is calling Scene::render(renderer, camera)
But that is not enough. Because only one camera has the “stereo” property. In order to work correctly all the cameras must have have a “left” and “right” eye. Rendering twice with the “left” eye is not enough.
So, I think the API should be more or less like this:
Scene::render(Renderer* renderer, const Mat4& eyeView)
{
for(auto camera: getCameras())
{
camera.setEyeTransform(eyeView);
....and continue doing what it is doing...
}
}
if someone is using a camera manually (nobody is doing is, but just in case someone is using it), after calling Camera::apply() it should call Camera:: restore
Framebuffer::default is no longer used. The “default” one is the one from the context.
A few bugs here and there, but almost there.
UPDATE:
some fixes. works setting the eye on the camera.
works when a camera is present.
works when no cameras are present
TODO:
have “Headset” abstraction. We need “Headset view size”
have an “Eye” abstraction… needed to position them correctly
