Graphics backend for opengl and metal

cocos2d-x C++
#1

As Apple deprecated OpenGL/OpenGL ES, i am doing research how to support metal for Cocos2d-x. So i am designing a new backend API to support at least OpenGL/OpenGL ES 2/metal. It may support vulkan in future, but is not part of the first version.

The API design may be finished in next week. The biggest challenge is how to support a cross platform shader and how to pass value to shader. About cross platform shader, i plan to use current shader for all platforms, which means don’t have to modify existing shaders. The design is that:

  • use glslang to compile glsl shader to SPIR-V
  • use SPIRV-Cross to convert SPIR-V to metal shader

I am not sure if it works, so i am doing some experiments these days. I will update progress here. You are appreciated if you can share your experience.

We can continue to discuss passing value to shaders after confirming the cross platform shader design.

8 Likes
Progress of new backend API
IOS and openGL?
#2
#3

I am trying to use shaderc to compile the shader, but meet error message:

myshader?: error: #version: Desktop shaders for Vulkan SPIR-V require version 140 or higher
myshader?:5: warning: attribute deprecated in version 130; may be removed in future release
myshader?:5: error: 'location' : SPIR-V requires location for user input/output
myshader?:6: warning: attribute deprecated in version 130; may be removed in future release
myshader?:6: error: 'location' : SPIR-V requires location for user input/output
myshader?:7: warning: varying deprecated in version 130; may be removed in future release
myshader?:7: error: 'location' : SPIR-V requires location for user input/output

The corresponding codes are:

void convertGLSLToSpirv(bool vert, const std::string& source)
{
    shaderc::Compiler compiler;
    shaderc::CompileOptions options;
    
    shaderc_shader_kind kind;
    if (vert)
        kind = shaderc_glsl_vertex_shader;
    else
        kind = shaderc_glsl_fragment_shader;
    
    auto result = compiler.CompileGlslToSpv(source.c_str(),
                                            source.length(),
                                            kind,
                                            "myshader?",
                                            options);
    if (result.GetCompilationStatus() != shaderc_compilation_status_success)
    {
        std::cerr << result.GetErrorMessage();
    }
    
    auto resultAsm = compiler.CompileGlslToSpvAssembly(source.c_str(),
                                                       source.length(),
                                                       kind,
                                                       "myshader?",
                                                       options);
    size_t sizeAsm = (resultAsm.cend() - resultAsm.cbegin());
    
    char* buffer = reinterpret_cast<char*>(malloc(sizeAsm + 1));
    memcpy(buffer, resultAsm.cbegin(), sizeAsm);
    buffer[sizeAsm] = '\0';
    printf("SPIRV ASSEMBLY DUMP START\n%s\nSPIRV ASSEMBLY DUMP END\n", buffer);
    free(buffer);
}

int main(int argc, const char * argv[]) {
    
    const char* vert = R"(
#ifdef GL_ES
    precision highp float;
#endif
    attribute vec2 a_position;
    attribute vec2 a_texCoord;
    varying vec2 v_texCoord;
    void main()
    {
        gl_Position = vec4(a_position, 0, 1);
        v_texCoord = a_texCoord;
    }
    )";
    
    convertGLSLToSpirv(true, vert);
    
    return 0;
}

So it seems can not use shaderc for simple usage, will try to use glslang directly.

1 Like
#4

Using glslang also meets error too:

A03101308207:bin minggo$ ./glslangValidator -H  ./test.vert
./test.vert
ERROR: #version: ES shaders for Vulkan SPIR-V require version 310 or higher
ERROR: ./test.vert:4: 'attribute' : Reserved word.
ERROR: ./test.vert:4: 'attribute' : no longer supported in es profile; removed in version 300
ERROR: ./test.vert:4: '' : compilation terminated
ERROR: 4 compilation errors.  No code generated.

It seems should modify the shader codes. I am not sure if GLES 2 supports ES shader version 310.

#6

There’s a few things that must get translated from GLSL version #version 100 to 310.

Maybe try Unity’s optimizer that outputs GL/Metal source?

Maybe to fix your issues you can also use macros to translate (e.g. is not exhaustive) into #version 310 glsl code

Either created into new shader files, e.g. shader_xx_v310.vert, or one could possibly #ifdef to include both version in same file.

// include in vertex shaders
#define attribute in
#define varying out

// include in fragment shaders
#define varying in

// translate function calls 
// **may be invalid, I think it takes same arguments as #version 1xx
#define texture2d texture
#define textureCube texture

Edit: Try this modification of your shader in that test. (more work would still be necessary)

    const char* vert = R"(
#version 310 es
#ifdef GL_ES
    precision highp float;
#endif
    in vec2 a_position;
    in vec2 a_texCoord;
    out vec2 v_texCoord;
    void main()
    {
        gl_Position = vec4(a_position, 0, 1);
        v_texCoord = a_texCoord;
    }
    )";
2 Likes
#7

Note there would be more required changes.

WebGL 2 is basically OpenGL ES 3.x.
https://webgl2fundamentals.org/webgl/lessons/webgl-shaders-and-glsl.html .

2.x to 3.x (#version 100 to #version 300 es)
https://webgl2fundamentals.org/webgl/lessons/webgl1-to-webgl2.html .

Specs:
https://www.khronos.org/files/opengles20-reference-card.pdf .
https://www.khronos.org/files/opengles31-quick-reference-card.pdf .

For example, the frag shader needs to define its output as user out variable instead of gl_FragColor.

// this only works without further specification and/or C++ API code because it's 
// the only out variable declared/defined.

out vec4 my_FragColor; 
//...
main
{
  my_FragColor = vec4(1,0,1,1);
}
3 Likes
#8

@stevetranby thanks for your reply, i just notice glsl-optimizer yesterday, i think this way is better than modify the shader manually currently. Because it can transfer to GLSL which is needed to support vulkan:

  • just pass the shader directly to GLES/GL backend
  • translate shader to metal shader then pass to metal backend
  • (in future) translate shader to GLSL, then use glslang to compile it to SPIR-V
1 Like
#9

I used glsl-optimizer to translate the vertex shader, the output is

#include <metal_stdlib>
#pragma clang diagnostic ignored "-Wparentheses-equality"
using namespace metal;
struct xlatMtlShaderInput {
  float2 a_position [[attribute(0)]];
  float2 a_texCoord [[attribute(1)]];
};
struct xlatMtlShaderOutput {
  float4 gl_Position [[position]];
  float2 v_texCoord;
};
struct xlatMtlShaderUniform {
};
vertex xlatMtlShaderOutput xlatMtlMain (xlatMtlShaderInput _mtl_i [[stage_in]], constant xlatMtlShaderUniform& _mtl_u [[buffer(0)]])
{
  xlatMtlShaderOutput _mtl_o;
  float4 tmpvar_1 = 0;
  tmpvar_1.zw = float2(0.0, 1.0);
  tmpvar_1.xy = _mtl_i.a_position.xy;
  _mtl_o.gl_Position = tmpvar_1;
  _mtl_o.v_texCoord = _mtl_i.a_texCoord;
  return _mtl_o;
}

_mtl_u is bindded on buffer 0, then what’s the buffer index of _mtl_i?

#10

Ok, i see. We can bind vertex in any vertex buffer, just make sure:

  • it is not bound for other usage, for example buffer 1 is used for uniforms here
  • the buffer index used in RenderEncoder.setVertexBuffer should match settings in vertexDescriptor

The code snipper looks like:

MTLRenderPipelineDescriptor *pipelineStateDescriptor = [[MTLRenderPipelineDescriptor alloc] init];
pipelineStateDescriptor.vertexDescriptor.layouts[2].stride = sizeof(float) * 4;  // index is 2
pipelineStateDescriptor.vertexDescriptor.layouts[2].stepFunction = MTLVertexStepFunctionPerVertex; // index is 2
pipelineStateDescriptor.vertexDescriptor.attributes[0].format = MTLVertexFormatFloat2; pipelineStateDescriptor.vertexDescriptor.attributes[0].bufferIndex = 2; // index is 2
pipelineStateDescriptor.vertexDescriptor.attributes[0].offset = 0;
pipelineStateDescriptor.vertexDescriptor.attributes[1].format = MTLVertexFormatFloat2;
pipelineStateDescriptor.vertexDescriptor.attributes[1].bufferIndex = 2; // index is 2
pipelineStateDescriptor.vertexDescriptor.attributes[1].offset = sizeof(float) * 2;

// index is 2
[renderEncoder setVertexBuffer:_vertices
                                offset:0
                              atIndex:2];

So we should know what buffer indexes are used in the vertex shader. We can use MTLRenderPipelineReflection to get needed information.

#11

The API design look like https://github.com/minggo/multi-backend/blob/master/multi-backend.md. It is not a complete design, but i think it includes most important contents.

You are appreciated if you have any feedback.

#12

Maybe there’s a setting which forces the optimizer to write out both as buffers?

xlatMtlShaderInput _mtl_i [[buffer(0)]], 
constant xlatMtlShaderUniform& _mtl_u [[buffer(1)]])

You’ll probably want to use the Metal specific size instead of using sizeof()

MemoryLayout<Float>.size

And yeah you should be able to get all info if you load from source with the reflection api.

#13

I had that comment opened for a while as I was confirming some things. I’ll look over your document and let you know if I have any feedback.

#14

A quick initial comment is based partly on “why not use BGFX (or similar)?” question. I think I understand that you want to try and add the back-end without any code changes to developers game code, or as few as possible, to adopt the new backend.

BGFX does have a semi-generic wrapper (as that is its purpose).

https://github.com/bkaradzic/bgfx/blob/909df17878091270012f7bdf9e5fcfc632b1a9e8/src/renderer_mtl.h .

https://github.com/bkaradzic/bgfx/blob/909df17878091270012f7bdf9e5fcfc632b1a9e8/src/renderer_mtl.mm .

NOTE: Don’t read these in their entirety, but they may be a helpful reference as you progress.

#15

I am not sure if there is a setting for it. But i think it is not a big deal since uniform buffer is using index 0, so i think it is safe to using index 1.

#16

Yeah, that’s probably true, and most (or all) shaders probably won’t have any additional buffers to worry about.

I’ll have to look through my custom OpenGL shaders tomorrow.

I’m curious also how it handles transformation of Samplers. It should be straight forward, but that optimizer shader output is quite mechanical.

Also, might want to throw in error handling while you’re building this.

do {
  pipelineState = try device.makeRenderPipelineState(descriptor: pipelineStateDescriptor)
} catch let error {
  fatalError(error.localizedDescription)
}

I am willing to test this out as you get further along. Let me know if you’re going to work in the repo you just shared. Or where it’d be if not.

I probably don’t have tons of time to actually help write the code, but if there were some easy things I would be willing to push up some PRs or write issues with small code changes embedded.

I’m not really that experienced with Metal, but I can try to help out where/when I am able.

2 Likes
#17

Before I sign off tonight, my main question is: Are you planning this as a drop-in invisible backend for cocos2d-x v3.x? or is this a proper new rendering API for cocos2d-x v4.x?

Those are two very different goals where I’d give entirely different feedback. Not that my opinions matter all that much.

Edit:

For v3 I would do what you’re doing, but don’t create a lot of architecture around the direct Metal API calls. Just either #ifdef in the same rendering class methods or write an alternative CCRenderMetal and similar.

For v4 I would re-consider writing Metal directly and either re-write for Vulkan from scratch (don’t “port” from OpenGL rendering since it’s quite a bit different) and just use MoltenVK. Or use a wrapper like BGFX or similar. Or if you feel like it write a front-end API similar to BGFX, but custom written as cocos2d-x wrapper API (like its other platform-independent APIs) that can swap various cocos2d-x rendering backends.

2c

1 Like
#18

Sorry for late reply, i was ill and just come back to work.

The repo is used to show the API design, and i am not sure if i will use that repo to do real coding. I will confirm it when i starting coding. And thanks for your help.

The design is for v4.x and want to make developer do as few changes as possible. And i don’t think it is a wrapper of OpenGL rendering, most of the concept comes from metal or vulkan.

The engine uses OpenGL codes every where, i don’t think just use #ifdef can port to metal.

Why you think it is a wrapper of OpenGL rendering? Even rewritten for Vulkan from scratch, but not all Android devices support Vulkan well, OpenGL ES backend is also needed. Then a graphic backend design is needed too.

#19

If we have a solution for v3 that run on metal, then i think we can design for vulkan from scratch in v4 as you said.

1 Like
Getting Started with Cocos2D
#23

I’m just trying to offer suggestions for limiting the amount of work required :smiley:
It was a mostly immediate gut reaction, and therefore probably aren’t valid for the real v4 development.

I think my only point is that if developers “need” Metal support for v3 then the team (or community) might as well try to pursue the least amount of work necessary to support it.

v4, however, should either be written with care for each target (Vulkan, GLES2/3, Metal, DX12) or might as well use a wrapper (e.g. BGFX/SDL) and only write target-specific backends after the rest of the improvements and new features or streamlined API is designed and written.

I do understand that a single back-end renderer, say a simple Metal render written in a new bare game project that doesn’t have to integrate into cocos2d-x’s API would not be a ton of work to get running, there are likely many edge cases and issues integrating with shaders and assets that will inevitably require more work than expected upfront. This likely multiplies with additional renderers.

Any given game should either write their own back-end renderer, or if we’re honest, just use one of the wrappers or already written engines that have Metal/DX support already built-in.

This is just my thought process on how I’d approach this for any future game where c++ and part or all of the cocos2d-x engine were to be utilized.

I’m also mainly giving any thoughts or suggestions based on the limited amount of development that has been applied to cocos2d-x (for c++) over the last few years. I’ll actually be surprised to see a true v4 of cocos2d-x as I fully expect cocos2d-x-lite to become v4 and otherwise v3 will just be iterated on into the foreseeable future.

2c

#24

+1 for all Vulkan, GLES2/3, Metal, DX12.