Graphics backend for opengl and metal

Sorry for late reply, i was ill and just come back to work.

The repo is used to show the API design, and i am not sure if i will use that repo to do real coding. I will confirm it when i starting coding. And thanks for your help.

The design is for v4.x and want to make developer do as few changes as possible. And i don’t think it is a wrapper of OpenGL rendering, most of the concept comes from metal or vulkan.

The engine uses OpenGL codes every where, i don’t think just use #ifdef can port to metal.

Why you think it is a wrapper of OpenGL rendering? Even rewritten for Vulkan from scratch, but not all Android devices support Vulkan well, OpenGL ES backend is also needed. Then a graphic backend design is needed too.

If we have a solution for v3 that run on metal, then i think we can design for vulkan from scratch in v4 as you said.

1 Like

I’m just trying to offer suggestions for limiting the amount of work required :smiley:
It was a mostly immediate gut reaction, and therefore probably aren’t valid for the real v4 development.

I think my only point is that if developers “need” Metal support for v3 then the team (or community) might as well try to pursue the least amount of work necessary to support it.

v4, however, should either be written with care for each target (Vulkan, GLES2/3, Metal, DX12) or might as well use a wrapper (e.g. BGFX/SDL) and only write target-specific backends after the rest of the improvements and new features or streamlined API is designed and written.

I do understand that a single back-end renderer, say a simple Metal render written in a new bare game project that doesn’t have to integrate into cocos2d-x’s API would not be a ton of work to get running, there are likely many edge cases and issues integrating with shaders and assets that will inevitably require more work than expected upfront. This likely multiplies with additional renderers.

Any given game should either write their own back-end renderer, or if we’re honest, just use one of the wrappers or already written engines that have Metal/DX support already built-in.

This is just my thought process on how I’d approach this for any future game where c++ and part or all of the cocos2d-x engine were to be utilized.

I’m also mainly giving any thoughts or suggestions based on the limited amount of development that has been applied to cocos2d-x (for c++) over the last few years. I’ll actually be surprised to see a true v4 of cocos2d-x as I fully expect cocos2d-x-lite to become v4 and otherwise v3 will just be iterated on into the foreseeable future.

2c

+1 for all Vulkan, GLES2/3, Metal, DX12.

Why not use BGFX/SDL?
In case developing new renderer cocos developers will spent a lot of time for develop and then fix many bugs.
I think it’s much easier and faster to use polished library.

I answered it here. Graphic part is important to an engine, cocos should control it. BGFX is not just a graphic API wrapper, it is a renderer.

Yep, only support metal is not enough for v4. What i mean is that it will break compatibility a lot, and we should not do in v3.x.

Could you please share the experience or add some link here? We want to do as little work as possible to support metal.

The API provided is easy to expand to support vulkan, and we have no plan for DX12, we can use OpenGL or vulkan on windows.

Sounds good on all comments.
Your path forward seems to be good based on your feedback.

I think the only thing I’ll reiterate is that since you can use Vulkan on Windows, Android, and Linux then I would maybe suggest writing a cocos2d-x custom Vulkan renderer. Then use MoltenVK to support iOS/Mac since it is an open source thin wrapper around Metal. Keep the current OpenGL ES renderer and write any code necessary to have the old GL rendering code conform to the new cocos2d-x rendering API that would be chosen as a compiler/#define flag, or whatever.

https://github.com/KhronosGroup/MoltenVK .

Especially since I do know people had issues with OpenGL ES 3.0 (3.1 may have fixed issues) and therefore it probably also doesn’t make much sense to write a ES 3.0 renderer, instead just keep using 2.0 for those who need to build for older devices, and Vulkan for all others.

https://godotengine.org/article/abandoning-gles3-vulkan-and-gles2 .
(not sure if still valid, but I think it makes sense to go Vulkan + existing GLES2)

I guess I just don’t see a great reason for writing a Metal-specific back-end immediately, since OpenGL ES deprecation only means unsupported, but still available (at least through Sept. 2019) and Vulkan, for better or worse, seems to be the common API into the future (likely to see WebGL 3, or renamed WebGPU or whatever, take on a similar command/parallel API as Vulkan, instead of its currently being based on OpenGL ES).

And my final comment is that my main point is just to suggest getting v4 1.0 out the door by focusing on a single rendering backend (whether Vulkan or BGFX/SDL/etc). Afterward a Metal-specific and any other non-required code and features would then be appropriate.

As always, up to your team.

Your suggestion is good, if we do like this, then it may have problem. For example, cocos is written in vulkan in v4, apple delete OpenGL ES and not all android devices support vulkan. Then developers should do like:

  • use v3.x to release for Android devices that doesn’t support vulkan
  • use v4.x to release for other platforms

The problems is that, what APK to release for Android? Is it possible to release 2 APK and select correct one when installing? The other problem is we can’t promise that the API of v3.x and v4.x are the same, then how games can work both with v3.x and v4.x.

[sorry this is long, I’m just throwing out my thoughts, and a reminder that mostly I’m no longer using this engine, but I try to support it as a community and continue to hope for a continued successful future - so don’t take my word for being worth anything]

Yeah I guess I’m just promoting or suggesting work toward an engine for the future, and not current, devices. Therefore a majority of the devices that support Vulkan/Metal will be a reasonably high percentage.

I’m also sort of assuming based on various comments and commits and repositories that have been worked on and created over the past few years that v4.x is going to ultimately be v3.x with some features removed that were not used very much and replace the rendering layer to better support the command-style rendering GPU APIs (Vulkan/Metal/DX12).

Therefore a v3.x game may not need any game-specific code changes, or hopefully at worst only small code changes, to support building the game using the v4.x engine.

In this way there would only be breaking changes at the rendering and shader layer. If a game has written custom OpenGL code, overridden onDraw (et al), or use custom shaders then there would be some amount of code changes or new code/shader to be written for those games.

Possibly also a removal of certain features if they’re not used much, at least from the default library. Maybe also a more componentized build source set up such that one could more easily add, but also disable, a component like Physics support.

Again, it’s also possible that the OpenGL rendering support could still exist in this incremental v4.x engine.

v3.x used when building for older Android devices that don’t support Vulkan.
v4.x used when building for iOS/Mac and those supporting Vulkan.

This is probably my mistake using the terminology “v4.x” for this basically same engine with a new rendering backend, and therefore you could decide to name it v3.20 or whatever and keep it as an incremental build focused only on adding support for Vulkan or Metal.

I think my idea is to maybe clean up some code allowing for some breaking changes, adding the new rendering backend, calling it v4.x and then work on an entirely new game engine (essentially) for v5.x with ECS or whatever other higher-level changes you want to make. Maybe that one is basically cocos-lite focused mostly on the CocosCreator/Javascript engine going into the future.


If instead you plan on v4.x to be a entirely new engine, such as using an entirely different architectures like ECS, or changing the way actions are applied, how memory is managed, how nodes are managed within a scene, etc, then my advice is probably invalid or incomplete.

In that case I guess it probably makes sense to write a Metal rendering backend for v3.x and postponing v4.x engine. My advice could still apply to this v4.x entirely new engine by looking into using an already written graphics rendering library/framework (e.g. BGFX).

(it really just depends on how much time your team plans to put toward the various components and features of each version and programming language)

Another promising BGFX-like rendering HAL to take ideas from if you decide cocos2d-x will write custom Metal (et al) backends. It’s written in rust, but the ideas should be understandable, and its possible you could take ideas to flesh out your v4 rendering API.
https://github.com/gfx-rs/gfx .

And their explanation for their approach regarding Vulkan/Metal/MoltenVK.
https://gfx-rs.github.io/2018/04/09/vulkan-portability.html .

And their reference to an info post regarding vulkan portability.
https://www.khronos.org/blog/khronos-announces-the-vulkan-portability-initiative .

@stevetranby thanks for your information. I think you are right. For v3, we should just need a metal API, don’t have to consider vulkan. And for v4, we can just write for vulkan from scratch.

Have a metal API can implement in two ways:

  1. use metal API to implement OpenGL ES API, such as moltengl
  2. has a metal similar APl, and implement it in OpenGL ES.

1 may take much time, because it should support all OpenGL ES API. 2 is much easier.

But doing like this way, then we have to modify engine twice, every time will take a long time. So can not just metal API, should have a backend that consider metal/vulkan/GL. Thought the first version may not suitable for vulkan, but it can be easily modified to adapt vulkan in future.

3 Likes

I found some limitation of glsl-optimizer, it can not handle builtin functions. For example, when i use the shader as input

const char* frag = "\
    varying vec2 v_texCoord; \
    uniform mat4 CC_PMatrix; \
    uniform sampler2D u_texture; \
    void main() \
    { \
        gl_FragColor = texture2D(u_texture, v_texCoord); \
    }";

Then the output is

#include <metal_stdlib>
#pragma clang diagnostic ignored "-Wparentheses-equality"
using namespace metal;
struct xlatMtlShaderInput {
  float2 v_texCoord;
};
struct xlatMtlShaderOutput {
  float4 gl_FragColor;
};
struct xlatMtlShaderUniform {
  float4x4 CC_PMatrix;
};
fragment xlatMtlShaderOutput xlatMtlMain (xlatMtlShaderInput _mtl_i [[stage_in]], constant xlatMtlShaderUniform& _mtl_u [[buffer(0)]]
  ,   texture2d<float> u_texture [[texture(0)]], sampler _mtlsmp_u_texture [[sampler(0)]])
{
  xlatMtlShaderOutput _mtl_o;
  float4 tmpvar_1 = 0;
  tmpvar_1 = texture2D (u_texture, _mtl_i.v_texCoord);
  _mtl_o.gl_FragColor = tmpvar_1;
  return _mtl_o;
}

As you see, texture2D is not changed, which is invalid in metal shader. And i found the issue confirms it. So it seems it is impossible to not change shader when supports metal, but i wonder how moltengl achieve it.

They probably wrote their own translator or compiler or even just some find/replace scripts.

Others (don’t bother reading, just noting that this is a common practice):

Does cocos2d have that many shaders in the default engine? I realize that others will have to rewrite their own custom shaders in part or in full, but why not just either rewrite all of them by hand, or use this translator and then run a find/replace on the functions that don’t translate.

I’d run the glsl-optimizer, then find/replace, get working, use while the first working Metal backend is finished. Later you can cleanup the Metal shaders renaming structs/functions/etc if desired.

You should also bump that issue and add any other issues so that someone can ideally fix it by end of 2018.

The issues are:

  • we need only one shader for all platforms, so i don’t think your method work.
  • many developers will use their own shaders, it is better that all shaders can work without any moidification

From moltengl user guide i found the information

MoltenGL uses technology from the open-source GLSL optimizer and Mesa projects to perform GLSL to MSL shader conversion during Runtime Shader Conversion and in the MoltenGLShaderConverter tool.

So i think moltengl improve optimizer or use new version of Mesa which fixes the issues. I will take a look of Mesa to check it.

I will not create other issues since there is already an existing issue which the author says will not fix it.

Yeah I should’ve mentioned using another’s or basing their own off another’s (unity). I sort of was implying it, but wasn’t clear.

I’ll just have to agree to disagree and let you continue on with your desired approach. I’m probably discussing this in a manner that’s focused on getting v3.20 w/Metal support out the door soon, rather than later, and that it is fine if developers have to do a little work. It’s probably smart for you to focus on the longer-term approach of building out the necessary workflow, pipeline, and tools for the new backends and APIs to ideally allow for games to build without any changes.

Godspeed.

It is impossible that developers do not modify any changes. I just want to let them modify as less as possible. I think have only one version of shader is important, and it is better using exiting shaders. But i can not achieve it with existing tools, and it is impossible for me to write our own tools, it will take a long time. So i will change to find other way to achieve it: use only one version of shaders.

From the tutorial it seems it is easy to upgrade shader to 3.0 in runtime. So it is able to do:

  • convert gles 2.0 shader to 3.0 in runtime
  • compile shader to spir-v and then metal shader

I will do some experiments.

1 Like

glslang supports ESSL 3.1 and upper, which requires location for user input/output. I don’t know how to work with ESSL 1 which doesn’t support explicitly setting locations in shader. And it will affect the API design.

Can’t you still just query for the location on the c++ side?

Feel free to start a new post with same title more specific question(s) based on your experiments and findings, and rename (prefix) this one with “zArchive - OLD” or whatever if you want to get a fresh set of eyes on the topic. Sorry to clog it up with what amounts to a discussion that would’ve been best done over chat.

I think i don’t describe the question clearly. What i mean is that, in ESSL 3.1, the shader may look like

#version 310

layout(location = 0) in vec3 position;
layout(location = 4) in vec4 color;
...

In the shader, color's location is set to 4. How can we make sure color is bound to 4 in GLSL 1?