Building Your First VR Game With Cocos CreatorVR And Rokid Air

Introduction: This article is an entry for the Cocos Chinese community’s “Call for Papers.” The author used Cocos CreatorXR to port an existing project to Rokid Air.

Some time ago, Cocos released the XR content development tool Cocos CreatorXR, which supports one-click conversion of existing game projects to XR versions and is compatible with most major brand devices in the market. This allows developers to develop once for multi-device cross-platform deployment.

This time, we will take Rokid Air as an example to experience the process of converting games with Cocos CreatorXR and deploying them to AR/VR devices.

Some time ago, Cocos released the XR content development tool Cocos CreatorXR, which supports one-click conversion of existing game projects to XR versions and is compatible with most major brand devices in the market, allowing developers to develop once for multi-device cross-platform deployment.

This time, we will take Rokid Air as an example to experience the process of converting games with Cocos CreatorXR and deploying them to AR/VR devices.

Pre-development preparation

Hardware

Rokid Air eyewear.

Android phones that support access to AR mode.

https://air.rokid.com/userguide

If you want to enter AR mode:
Only applicable to Android phones with USB-C Display Port and OTG (Android10 and above), specific models:
Huawei Mate 10/10 Pro, Mate 20/20 Pro/20X, Mate 30E/30/30Pro, Mate 40/40 Pro, Mate X2, P30 Pro, 40 Pro, P50/50 Pro Samsung S10 Qualcomm, S20 FE Qualcomm/S20 +Qualcomm/S20U Qualcomm, S21+ Qualcomm/S21U Qualcomm,Note 20 Qualcomm/Note 20U Qualcomm, Galaxy Z Fold 3Qualcomm, W22
·Honor V20Magic 3/3 Pro OnePlus 7/7T/7T Pro,8/8T/8T Pro,9/9T/9T ProOPPO Find X2/X2 Pro, Find X3/X3 Pro
Black Shark 3, Black Shark 4

Software

Rokid Air App (I recommended downloading directly from the official website, there may be a lag in updates from major app markets).

https://air.rokid.com/download

Note: Some incompatibility issues exist on the Rokid Air app version 1.9.1. Please make sure not to use this version of the Rokid Air app for debugging and publishing. Previous versions and versions 1.9.2 or later will not be affected.

Cocos Creator 3.6.1

Install and configure the Android environment.

https://docs.cocos.com/creator/manual/en/editor/publish/setup-native-development.html

XR Plugin

https://store.cocos.com/app/detail/4117

Casting screen (optional)

adb tcpip + scrcpy

Rokid AIR screen casting solution (Win).

https://ivb750bbcz.feishu.cn/docs/doccniMDevcb6lgto0DzXPEn8Xe

Rokid AIR Screen Casting Solution (Mac):

https://ivb750bbcz.feishu.cn/docs/doccndTeu2iDmQt7a2fSJqnpLvg

Hello World

Create a new VR template.

Open the xr_main scene, and you can see that the camera is hanging under the node XR HMD.

The scene is empty and a bit lonely. Feel free to add some 3D objects, adjust them to your preferred angle and position, and ensure they are in the camera’s field of view.

Build Release

Select Project->Build Release in the menu bar, select XR Rokid Air for the release platform, and fill in the package name for the first release. The first build may be a bit slower, but later builds will be faster.

Open {in-person project}/build/xr-rokid/publish/release and install the package to your phone.

image

If there is an exception, it is generally an exception in the configuration related to the Android environment, and you can check the build log to locate the problem.

Effect Preview

After installing the app, it opens directly like this.

You need to connect AR glasses and enter AR mode in the Rokid Air App.

The package you just installed can be found in its application library.

image

When it opens, shake your head to see the objects we added earlier.

640 (5)

And the phone shows the gamepad.

Enter the event

Handle input events

To better control the game, you need to know the joystick events on the phone.

The code for listening to events in Cocos Creator is as follows:

input.on(Input.EventType.GAMEPAD_INPUT, this._gamepadEvent, this);

Get the handle input parameters as follows.

private _gamepadEvent(event: EventGamepad) {
    const gamepad = event.gamepad;
}

The members of the gamepad can be seen in cc.d.ts.

A simple script can be written to verify that the rocker parameters return.

import { _decorator, Component, input, Input, EventGamepad, Vec2, Label } from 'cc';
const { ccclass, property } = _decorator;

@ccclass('GamepadEventTest')
export class GamepadEventTest extends Component {
    @property(Label)
    lb_leftStickValue: Label = null!

    @property(Label)
    lb_rightStickValue: Label = null!

    start() {
        input.on(Input.EventType.GAMEPAD_INPUT, this._gamepadEvent, this);
    }

    private _gamepadEvent(event: EventGamepad) {
        const gamepad = event.gamepad;
        const leftStickValue = gamepad.leftStick.getValue() as Vec2;
        this.lb_leftStickValue.string = `${leftStickValue.toString()}`;

        const rightStickValue = gamepad.rightStick.getValue() as Vec2;
        this.lb_rightStickValue.string = `${rightStickValue.toString()}`;
    }
}

and add label node validation output to the scene.

Previewing the effect after repacking shows the output of the rocker.

Head-mounted display posture input events

This event gets some crucial parameters of the glasses, such as the rotation angle, which may be used when customizing the glasses interaction.

The code for listening to the event in Cocos Creator is as follows.

input.on(Input.EventType.HMD_POSE_INPUT, this._dispatchEventHMDPose, this);

Obtain the input parameters as follows.

private _dispatchEventHMDPose(eventHMD: EventHMD) {
    const hmdInputDevice = eventHMD.hmdInputDevice;
}

The related logic can be found in the code in extensions/xr-plugin/assets/xr/component/device/pose-tracker.ts .

simple-shooting

I am porting an existing game to Rokid Air glasses. This time I’m using an official Cocos sample game demo to demonstrate.

Sample Demo Download - GitHub.

Once you have downloaded the project, find the main scene and export the resource package.

After exporting the resource package, import it back into helloworld.

Switch to the shooting game scene, find the main camera and switch to XR HMD.

Build parameters to select the current new scene.

After connecting the glasses and opening the app, the effect is shown in the following image.

640 (6)

The remaining issues are tweaking the parameters, accessing the input events, and modifying the script assets/simple-shooting/scripts/ShootingPlayerController.ts, with the following core reference code.

    //input.on(Input.EventType.GAMEPAD_INPUT, this._gamepadEvent, this);
        //input.on(Input.EventType.HMD_POSE_INPUT, this._dispatchEventHMDPose, this);

    private _dispatchEventHMDPose(eventHMD: EventHMD) {
        const hmdInputDevice = eventHMD.hmdInputDevice;
        
        const _quatPose:Quat = hmdInputDevice.headMiddleOrientation.getValue();
        _quatPose.getEulerAngles(tempVec3_a);

        const horizontalRot = this.node.getRotation();
        horizontalRot.getEulerAngles(tempVec3_b);
        tempVec3_b.y = tempVec3_a.y;
        this.node.setRotationFromEuler(tempVec3_b);

        if (-tempVec3_a.x > this.viewDownAngle && -tempVec3_a.x < this.viewUpAngle) {
            const verticalRot = this.verticalViewNode.getRotation();
            verticalRot.getEulerAngles(tempVec3_b);
            tempVec3_b.x = -tempVec3_a.x;
            this.verticalViewNode.setRotationFromEuler(tempVec3_b);
        }
    }

    aValue = false;
    bValue = false;
    private _gamepadEvent(event: EventGamepad) {
        const gamepad = event.gamepad;
        const leftStickValue = gamepad.leftStick.getValue() as Vec2;
        if(this._velocity.z !=leftStickValue.y && this._velocity.x!=-leftStickValue.x){
            this._velocity.z = leftStickValue.y;
            this._velocity.x = -leftStickValue.x;
            if(leftStickValue.lengthSqr()>0){
                this.changeToAnimState(PlayerAnimState.Running);
            }else{
                this.checkToIdle();
            }
        }
       
        if (gamepad.buttonSouth.getValue() === 1 && !this.aValue) {
            this.aValue = true;
            this.doAction(ActionType.Shoot,true)
        } else if (gamepad.buttonSouth.getValue() === 0) {
            this.aValue = false;
        }
        if (gamepad.buttonEast.getValue() === 1 && !this.bValue) {
            this.bValue = true;
            this.doAction(ActionType.ThrowGrenade,true)
        } else if (gamepad.buttonEast.getValue() === 0) {
            this.bValue = false;
        }
    }

Resource Links

Demo Download

https://forum.rokid.com/person/file/fileUrl?ossId=open-platform/forums/ff8c77709481483187755d66c5739d171667977039066.zip

The download links to the resource packages used in this article are attached to the forum post.

Feel free to visit the forum and discuss it together!

Click to learn more about Cocos CreatorXR’s capabilities, usage, and feedback is welcome! This version was released as an official plugin in v3.6.1. Developers can download and install it via the plugin or enable it via Dashboard’s templates/cases.

Cocos CreatorXR Plugin Download Link

https://store.cocos.com/app/detail/4117

Plugin Forum Post