Proposal for v3.x: Improved sound system

cocos2d-x C++
#1

The current sound system for cocos2d-x v2.1, is good enough, but I think we could add a better, more advanced sound system.

So, if we are going to add new sound system, which features would you like to have in it:

So far, these are the features proposed in this discussion: https://groups.google.com/forum/#!topic/cocos2d-js-devel/DxH8QJcU03A

  • 3D positional audio
    * Integration with Node position
  • MIDI
  • Play background music: loop, queue, volume,
  • Play sounds effects: play, loop, queue, many channels, pitch, pan, gain
  • Hardware accelerated play whenever available (eg: iOS)
  • Formats: mp3, wav, ogg?
  • Async loading
  • seekToCurrentTime
  • Transitions: cross-fade between queue items

What else ?

PS: This is not an extension of SimpleAudioEngine, but rather new sound engine written from scratch. Once it is done, SimpleAudioEngine could be just a thing wrapper of this new engine.

PS2: If we have time, it will be part of v3.0. If not, it will be part of v3.1

What features do you wish Cocos2d-x provided?
#2

Perfect!
Would love to see these also into the HTML5.

BTW, if we could have like platform specific imports would be amazing, otherwise some of my projects have a lot of data just for double music(in MP3 and OGG).

#3

Instead of building on your own, it is better to leave this to experts.
i.e, writing an adapter to work with existing libraries

#4

The ability to get direct access to the bit stream is super important for audio visualization and rhythm games. That way the developer can do a fast Fourier transform and make reactive visuals.

#5

Definitely need the ability to get the current position in the audio track. Getting the position in seconds would be great, but the index of PCM samples in the buffer would be perfect!

I’m thinking this would only be needed for background music.

NOTE: Not sure how hardware buffers work, but it has to be as accurate as possible because it will be used for rhythm games for syncing the audio with visual.

#6

As someone who has seen frequent usage of FMOD in games, I’m wondering if:

  • we can gleam anything from it’s API design or
  • we can create an adapter pattern for what kind of audio engines we want to use.

Honestly I think we should go the route of an audio adapter pattern. if this framework ever gets on platforms like PS Vita or if WP8 ever makes it back into the main branch, this makes it pretty damn painless for people. Bonus points if all you have to do is as for for an Audio Manager and it just gives you the best available.

#7

Excellent set of features. I think some of the basic features like the following if added initially will be of more help.

  1. Playing multiple BGM (not sure if this is possible)
  2. Being able to control each BGM or effect’s volume.
  3. Looping with a repeat count instead of a boolean.
  4. getTimeElapsed
  5. seekToTime

Apart from these basic features, the advanced features mentioned by Ricardo would be awesome!

#8

Hi, Ricardo.

Just couple of my minds.

* Integration with Node position
Not required, if programmist will need this - he’s do it by self in way, what is best in his game.

* MIDI
Not required - really, very old and unuseful.

* Play background music: loop, queue, volume
I think not required too. This is just sound effect - and if you provide it as is, programmer can use them as he wishes. In any serious game exist own SoundManager for this.
This is may be conflict with Android/iOS system fade… But I’ve not recommend done background music.
For example - we can fade out old sound and fade in new music in one moment (menu change / scene change / change game situation, etc.)

* Hardware accelerated play whenever available (eg: iOS)
May be. But please leave way behind if user want improve it.

* Formats: mp3, wav, ogg?
I think mp3 is not needed. I prefer ogg (like quality less) and wav (like as is quality). This is remove unused size.
If it’s possible - please do code, which can read blocks from pack (file stream, zip-stream, etc.) - so user can improve low-level by his reasons.

* Async loading
Need some manager, which can be customized easily. I think better is leave this in examples - openAL context must be in one thread, as I remember.

* Transitions: cross-fade between queue items
I think would be fine, if you put EXAMPLE in basic examples. If programmer will need it - he will copy and change it by self due his reasons.

What else?
Please provide BASIC things with easy customization - more complex things can be done by examples.

#9

We would definitely appreciate the ability to seek/skip within an audio track as this has been a big limitation for us recently.

#10

I would love this, being able to seek within a track would be extremely beneficial for my game.

In addition, I think being able to fade out/fade in sounds (volume, not crossfading) would be helpful.

#11

Sam Borenstein wrote:

I would love this, being able to seek within a track would be extremely beneficial for my game.
>
In addition, I think being able to fade out/fade in sounds (volume, not crossfading) would be helpful.

We made a small patch for seeking within audio tracks. Currently it only works on iOS but we could extend to Android. However, I’m not sure the best way to propose this as an extension to the public cocos2dx 2.x API.

#12

I have one feature request:

  • Possibility to load and play sounds from byte buffers as well as from files.

Right now we have hacked in support for this in CocosDenshion and SimpleAudioEngine ourselves but it would be great if that would just work out of the box with cocos2dx!

#13

The only thing that bugs me with the current audio engine is that .mp3 doesn’t seem to be supported on BlackBerry 10. It works fine with .ogg, but it’s inconvenient to have to maintain a separate set of files just for that one platform. I don’t know if this is a hardware limitation, but it’s my only real complaint with the current audio system.

Ben

#14

Have you considered using OpenAL Soft as a cross platform framework for the new audio engine?
The guys from YoyoGames (Gamemaker) use it for Windows Phone 8 and Android. Also it may help to evade bugs in several vendor audio implementations (e.g. SGS II).
http://kcat.strangesoft.net/openal.html

#15

Great list of features — I think you are on the right track.

I disagree with almost everything “Igor Nikolaev” posted.

Recent bugs in Android indicate that evading bugs in OS vendor SDK may be very important.

Most Important to Me:

  • Play background music: loop, queue, volume,
  • Hardware accelerated play whenever available
  • Formats: mp3, wav, ogg, caf
  • Async loading
  • Transitions: cross-fade between queue items
#16

I think it would be great if you will add such a thing like isEffectPlaying(). It makes a lot of troubles without it.

#17

It would be nice to modulate individual sound effects with pitch, pan, gain after the sound is already playing, not just when starting it.

#18

My list:

  • Background music: loop, queue, volume,
  • Hardware accelerated play everywhere
  • mp3, wav, ogg, caf
  • Async loading please!
  • Cross-fade between queue items
#19

I’d love to have MOD/XM support :slight_smile:

But on the other side: There are many very good and fast audio engines out there (and it’s not a trivial task to write one that is good) so for me it’s fine if cocos only supports some basic audio stuff and if someon needs something more complex he can choose a audio lib like FMOD or BASS.

#20

I’m no audio expert but why are “background music” and “sound effects” played in different audio systems? I’ve had to work around the length-of-playback bug in Android by playing some sound effects as bgm, so it’d be nice if that issue could be resolved. Maybe “background music” is just a pointer to a specific sound effect?

Features I want:

  • recording (I know that Android and iOS can encode to mp4), and check if device can record
  • playback formats: mp3, mp4
  • duration of audio file before playback, or at completion of recording
  • seek to time
  • current time
  • playback rate (forward, backward, half time, double time)