Multi-timbral AVAudioUnitMIDIInstrument
Multi-timbral AVAudioUnitMIDIInstrument in Swift
Table of Contents
Introduction
There is one sublcass of AVAudioUnitMIDIInstrument provided by Apple – the AVAudioUnitSampler. The only problem is that it is mono-timbral; it cannot play more than one timbre at a time.
To create a new AVAudioUnit, we need to use a bit of Core Audio.
So, I’ll give you two examples, one using Core Audio and an AUGraph and then one using AVFoundation using AVAudioEngine.
Core Audio Unit
We need to create an AUGraph and attach nodes to it.
Here’s the first step. Create your class, define instance variables, and create the graph using Core Audio’s C API.
|
1 2 3 4 5 6 7 8 9 10 11 |
class AudioUnitMIDISynth : NSObject { var processingGraph = AUGraph() var midisynthNode = AUNode() var midisynthUnit = AudioUnit() func augraphSetup() { var status = OSStatus(noErr) status = NewAUGraph(&processingGraph) AudioUtils.CheckError(status) ... |
Here is the item we’re interested in. Create a node that’s an Audio Unit Music Device with a subtype MIDISynth and add it to the graph.
|
1 2 3 4 5 6 7 8 9 |
func createSynthNode() { var cd = AudioComponentDescription( componentType: OSType(kAudioUnitType_MusicDevice), componentSubType: OSType(kAudioUnitSubType_MIDISynth), componentManufacturer: OSType(kAudioUnitManufacturer_Apple), componentFlags: 0,componentFlagsMask: 0) let status = AUGraphAddNode(self.processingGraph, &cd, &midisynthNode) AudioUtils.CheckError(status) } |
And also create the usual io node, kAudioUnitSubType_RemoteIO on iOS, in the same way. I’m not going to bother with a mixer in this example..
Get the audio units from the nodes using AUGraphNodeInfo in order to get/set properties on them later. Then connect them using AUGraphConnectNodeInput.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
// now do the wiring. The graph needs to be open before you call AUGraphNodeInfo status = AUGraphOpen(self.processingGraph) AudioUtils.CheckError(status) status = AUGraphNodeInfo(self.processingGraph, self.midisynthNode, nil, &midisynthUnit) AudioUtils.CheckError(status) status = AUGraphNodeInfo(self.processingGraph, self.ioNode, nil, &ioUnit) AudioUtils.CheckError(status) let synthOutputElement:AudioUnitElement = 0 let ioUnitInputElement:AudioUnitElement = 0 status = AUGraphConnectNodeInput(self.processingGraph, self.midisynthNode, synthOutputElement, // srcnode, SourceOutputNumber self.ioNode, ioUnitInputElement) // destnode, DestInputNumber AudioUtils.CheckError(status) |
To load the Sound Font, set the kMusicDeviceProperty_SoundBankURL property on your unit. I’m using a SoundFont from MuseScore here.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
func loadMIDISynthSoundFont() { if var bankURL = NSBundle.mainBundle().URLForResource("FluidR3 GM2-2", withExtension: "SF2") { let status = AudioUnitSetProperty( self.midisynthUnit, AudioUnitPropertyID(kMusicDeviceProperty_SoundBankURL), AudioUnitScope(kAudioUnitScope_Global), 0, &bankURL, UInt32(sizeof(bankURL.dynamicType))) AudioUtils.CheckError(status) } else { print("Could not load sound font") } print("loaded sound font") } |
The typical Sound Font contains dozens of patches. You don’t really want to load every single one of them. You should pre-load the patches you will actually use. The way to do that is a bit strange. You set the property kAUMIDISynthProperty_EnablePreload to true (1), send MIDI program change messages via MusicDeviceMIDIEvent for the patches you want to load, and then turn off kAUMIDISynthProperty_EnablePreload by setting it to 0. You need to have the AUGraph initialized via AUGraphInitialize before calling this.
Where is this documented? Damned if I know. Do you know? Tell me.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 |
func loadPatches() { if !isGraphInitialized() { fatalError("initialize graph first") } let channel = UInt32(0) var enabled = UInt32(1) var status = AudioUnitSetProperty( self.midisynthUnit, AudioUnitPropertyID(kAUMIDISynthProperty_EnablePreload), AudioUnitScope(kAudioUnitScope_Global), 0, &enabled, UInt32(sizeof(UInt32))) AudioUtils.CheckError(status) let pcCommand = UInt32(0xC0 | channel) status = MusicDeviceMIDIEvent(self.midisynthUnit, pcCommand, patch1, 0, 0) AudioUtils.CheckError(status) status = MusicDeviceMIDIEvent(self.midisynthUnit, pcCommand, patch2, 0, 0) AudioUtils.CheckError(status) enabled = UInt32(0) status = AudioUnitSetProperty( self.midisynthUnit, AudioUnitPropertyID(kAUMIDISynthProperty_EnablePreload), AudioUnitScope(kAudioUnitScope_Global), 0, &enabled, UInt32(sizeof(UInt32))) AudioUtils.CheckError(status) } |
Now when you want to play a note, you send a MIDI program change to tell the synth unit which patch to use.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
func playPatch1On() { let channel = UInt32(0) let noteCommand = UInt32(0x90 | channel) let pcCommand = UInt32(0xC0 | channel) var status = OSStatus(noErr) generateRandomPitch() // pitch is an instance variable since we need to send a note off later. status = MusicDeviceMIDIEvent(self.midisynthUnit, pcCommand, patch1, 0, 0) AudioUtils.CheckError(status) status = MusicDeviceMIDIEvent(self.midisynthUnit, noteCommand, pitch, 64, 0) AudioUtils.CheckError(status) } |
If you want to play a sequence, the traditional way to do that with an AUGraph is with the Audio Toolbox entities. MusicPlayer will play a MusicSequence. When you create your MusicSequence, you attach it to the AUGraph.
There are examples in my Github project for sending note on/note off messages as well as playing a MusicSequence through the AUGraph.
AVFoundation Unit
So we know the how to do this in Core Audio. How do you do it in AVFoundation?
The class hierarchy for AVAudioUnitSampler is:
AVAudioNode -> AVAudioUnit -> AVAudioUnitMIDIInstrument -> AVAudioUnitSampler
So, our AVAudioUnit will be:
AVAudioNode -> AVAudioUnit -> AVAudioUnitMIDIInstrument -> AVAudioUnitMIDISynth
That part was obvious. What you need to do though is not especially clear. As usual, Apple doesn’t give you a clue. So, this is how I got it to work. I don’t know if this is the “official” method. If you know, tell me.
I’ve noticed that the provided AVAudionUnits work with no-arg inits. So, I decided to create the AudioUnit’s AudioComponentDescription here and pass it up through the hierarchy to have one of those classes (probably AVAudioUnit) initialize it.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
class AVAudioUnitMIDISynth: AVAudioUnitMIDIInstrument { override init() { var description = AudioComponentDescription() description.componentType = kAudioUnitType_MusicDevice description.componentSubType = kAudioUnitSubType_MIDISynth description.componentManufacturer = kAudioUnitManufacturer_Apple description.componentFlags = 0 description.componentFlagsMask = 0 super.init(audioComponentDescription: description) } |
AVAudioUnit defines the audioUnit property. We can use that to set the kMusicDeviceProperty_SoundBankURL property for a Sound Font.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
func loadMIDISynthSoundFont() { if var bankURL = NSBundle.mainBundle().URLForResource("FluidR3 GM2-2", withExtension: "SF2") { let status = AudioUnitSetProperty( self.audioUnit, AudioUnitPropertyID(kMusicDeviceProperty_SoundBankURL), AudioUnitScope(kAudioUnitScope_Global), 0, &bankURL, UInt32(sizeof(bankURL.dynamicType))) if status != OSStatus(noErr) { print("error \(status)") } } else { print("Could not load sound font") } print("loaded sound font") } |
Remember that kAUMIDISynthProperty_EnablePreload chacha we did to pre-load patches? We can do that here too.
That’s it.
To use it, attach it to your audio engine.
|
1 2 3 4 5 |
engine = AVAudioEngine() midiSynth = AVAudioUnitMIDISynth() // this is our guy midiSynth.loadMIDISynthSoundFont() // overload to pass in a URL engine.attachNode(midiSynth) engine.connect(midiSynth, to: engine.mainMixerNode, format: nil) |
You can play a sequence via the AVAudioSequencer which is attached to your engine. If you don’t preload your patches, the sequencer will do that for you.
This is how to load a standard MIDI file into the sequencer.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
func setupSequencerFile() { self.sequencer = AVAudioSequencer(audioEngine: self.engine) let options = AVMusicSequenceLoadOptions.SMF_PreserveTracks if let fileURL = NSBundle.mainBundle().URLForResource("chromatic2", withExtension: "mid") { do { try sequencer.loadFromURL(fileURL, options: options) } catch { print("something screwed up \(error)") return } } sequencer.prepareToPlay() print(sequencer) } |
The sequencer can also be created with NSData. This is quite convenient – everyone loves creating an NSMutableData instance and then shove bytes into it. Right?
Have a MusicSequence? Your only option it to turn it into NSData.
This works. If you have a better way, let me know.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
func sequenceData(musicSequence:MusicSequence) -> NSData? { var status = OSStatus(noErr) var data:Unmanaged<CFData>? status = MusicSequenceFileCreateData(musicSequence, MusicSequenceFileTypeID.MIDIType, MusicSequenceFileFlags.EraseFile, 480, &data) if status != noErr { print("error turning MusicSequence into NSData") return nil } let ns:NSData = data!.takeUnretainedValue() data?.release() return ns } |
Summary
All this to create an AVAudioUnit subclass.
You should preload the patches you are going to use. If you’re going to use an AVAudioSequencer, you don’t have to; it will do it for you.
Create an AVAudioUnit subclass and pass a Core Audio AudioComponentDescription to a superclass in your init function.
You can access the audioUnit in your AVAudioUnit subclass and set properties on it using Core Audio.
I am looking for some way to tell if AVAudioSequencer has finished playing, so I can update the play/stop button. Some sort of notification or completion handler I imagine, have any of you guys had experience with
this situation.
Thanks
Z
No, Apple wasn’t gracious enough to do that.
I’d love to be wrong on that, so if anyone else knows a way, let me know.
Take a look at my example for how to accomplish “rewinding”.
Specifically look at the play() function in this controller.
Similar question to Zelda’s, is there a way to setup callbacks for any events on AVAudioSequencer? I’d love a callback when the sequencer is looping, or to be able to get callbacks on certain beats. Can this be added or extended? Thanks for the blog Gene, very helpful!
Hello,
Your blog is gold. 🙂
It looks like you are sending midi events only on channel 0. Isn’t it possible to preload patches in MIDISynth on various channels, and later to send midi notes on those channels? Sending a program change for every note seems to be overkill.
As with everything I write, I show an answer that works but not necessarily the answer 🙂
Did you try it to see if it works? If I weren’t swamped with work projects, I’d do it now.
Gene, Thanks alot for all your post! They are extremely valuable to the community… .
I am wondering whether you’ve experienced Memory Leak problems when disposing AudioGraph with SoundFonts and if you’ve managed to overcome it?
We have a simple program that uses the same procedure as your loadMIDISynthSoundFont() and loadSynthPatches() and everytime AudioGraph is disposed, we experience a 13Mb overhead! (We don’t use AV stuff..). We use the good old “FuildR3 GM2-2.SF2” for this test… .
Any hints?
If you’d like to send me an excerpt of your code, maybe just having an extra set of eyeballs on the problem would help.
Thank you Gene! I’ve been reading your blog posts and one you wrote about a year ago or so stated that the AVAudioSequencer technology is unstable so I was wondering what was the status of that? Is it more stable now? Does it have the ability to add or remove events from the Music Sequence?
What is the difference between the AVAudioEngine.musicSequence property and the one that we attach via AVAudioSequencer?
Is it possible to use the old Core Audio/Core MIDI APIs (MusicTrack, MusicSequence, MusicPlayer) for play back with AVAudioUnitNodes? We already have an existing architecture in AVAudioEngine, but need add/remove functionality with a sequencer. Would MusicSequence work for that?
As of Swift 3 beta 6, nothing has been added to
AVAudioSequencerandAVMusicTrack. They are for playback only right now. Doug says file a request on Radar – if there are “enough” then they might think about it. I’m sure they are tired of hearing from me about it.In the past I tested setting the musicSequence property on AVAudioEngine and it crashed. I suppose they want you to not use it directly. The documentation on that property is awful as usual.
Yes you can use the Audiotoolbox classes with AVAudioEngine. I have a 6.022 x 10^gazillion github repos with examples.
Here’s a blog post
Here’s a a repo
Hello.
If anyone can hint me to a useful replacement for the awful *AVAudioSequencer*,
written in Swift 3/4, I would be really very happy!
Everything seems to be possible with AVFoundation yet, loading/creating/editing/saving MIDI sequences, creating multiple sampler instances and sophisticated effect processing chains and so on…
except an *useful sequencer*, which is really missing.
*AVAudioSequencer* is still completely useless for creative coding of that kind. Its totally limited in possibilities. It’s even totally superfluous, because it does nothing that cannot be done with other methods.
Unfortunately, currently I have no clue how to build an own sequencer class, that is for instance able to drive multiple instances of *AVAudioUnitSampler* based on multi-channel *MusicSequence*s. Using simple multi-timbre DLS and Soundfont MidiSynths is just not that exciting, as one could rather just use a simple AVMidiPlayer for playback then.
Any advices would be highly appreciated. 😉
There’s the C Audiotoolbox MusicPlayer and MusicSequence which is usable from Swift but rather clunky.
The AVFoundation classes are half-baked right now. They are geared towards simply playing back a pre-existing MIDI file. The AVTrack has nothing in it to build it from scratch. Apple says “file a Radar”. I did. It told me that it was a “duplicate request”, but in Apple’s wisdom, Radars are private and not searchable.
Swift 4 and iOS 11 betas do not improve this even a little. Interesting, because they’re going to mark AUGraph as deprecated. I wonder if the Audiotoolbox will be deprecated and replaced with AVFoundation classes. I’m not holding my breath.
Update:
I am busy now with trying a ‘fake’ sequencer via CoreMIDI by using a virtual MIDI port and a *MusicPlayer* instance -> in connection with calling *MIDIDestinationCreateWithBlock()* to instantiate a callback for the MIDI events inside a *MusicSequence* in memory. So I am able to redirect these events to multiple (multi timbre) *AVAudioUnitSampler*s with success – finally achieving a multi channel sampler with independent effect chains. This way, I will have the desired control over midi/audio channels, which is not possible with a standard dls/sf2 MIDISynth. It basically works so far, but have not tested in depth yet. Requirement is that it will play Standard MIDI files.
Generally I still consider this to be a somehow “fast & dirty” solution, especially because this virtual MIDI Port may be visible to other apps, which is not always, what I want. I do not know, whether there is a possibility to hide this virtual port to others.
My initial idea was to code a sequencer from the scratch, based on a high resolution timer and a sorted event list from the MusicSequence in memory or such. I guess this would be much more work and time for testing .. and I am quite new to the world of that nifty swifty…
Update 2:
Hello again.
I am working on this for days now, and the BUGS in this entire framework (AudioToolbox) are so annoying!
At first, the SWIFT frameworks (especially the interesting parts) are badly (if even at all) documented.
By trial and error I discovered loads of bugs, i.e. with all kind of MIDI RAW data, i.e MIDIMetaEvent, MusicEventUserData, MIDIRawData with *MusicSequence*. So either I am totally stupid, or they did the SWIFT porting wrong…
However, excessively tweaking my code, I found a possibility to work around this. I would rather say, it is a kind of “Hack” to get this working. The latest “pointer access” stuff of SWIFT also play a big part to this. Probably SWIFT is the complete WRONG way to do this all with satisfaction …
At first, I wanted to build a complete MIDI Monitor with the possibility to edit, save and reload multichannel recorded MIDI data this way. But without the support of Meta and SysEx, which all require correct handling of MIDI ‘raw data’ of arbitrary variable length this would be completely senseless, as there is no chance to re-open a saved MusicTrack as a MIDI file and read back these data correctly. ^^
It seems, that ist is not even possible to simply save and load song and track names with the provided SWIFT AudioToolbox methods in a correct manner.
*If someone actually knows, how this correctly works, so I would be very happy to see some code, because mine is a HACK.*
The sequencer Part of my project based on the MusicPlayer with MIDIEndpoint basically works, but there are major design flaws of the entire AudioToolbox framework in my opinion. That is: There is no possibility to tag music events uniquely inside a MusicSequence, so that you cannot access events in any useful manner. Instead, you always and endless have to iterate all those events and you have to “guess”, whether the fond event is the one you wanted or possibly not. ^^ So that is a pain. And it is actually a totally unprofessional and quite useless implementation. Sorry, Apple, But that’s a fact!
I guess, if Apples Garageband would rely oh these frameworks alone with would be written completely in SWIFT, so the program actually won’t work at all. ^^
Hi Gene, Thanks for your post, It’s great and helped me a lot.
I use Core Audio Unit to implement multiple midi channel play with different instrument. I work it out by learn from this blog. But standard midi file always have some channel haven’t instrument audio output.
I use FluidR3 GM2-2 as my soundfont bank. It must have 128 kind of instrument tone.
I checked the midi command Cx (x is midi channel index , this command is set the instrument tone of current channel) send to apple core audio unit by the method of MusicDeviceMIDIEvent
Of course ,I checked the volume of every midi channel ,It’s not zero.
It’s strange, Do you met this situation? Any advices would be highly appreciated.
Do you use the default multi timbre/channel MidiSynth for that or an AuSampler (mono timbre, only channel omni – receives on all channels) with different audio busses ?
Try to check your sound bank and whether the patches (program changes) use MIDI Bank Select 0. Possibly there are no other sound banks (1 to 126) defined inside the sound font, which may cause this behaviour, if it is switched inside the MIDI file.
There may be other reasons too: I.e. usage of explression and sustain controllers, invalid samples inside the font or a corrupted mapping inside the sound font… Also the MIDI files may have some internals, that cause such behaviour.
I discovered similar things inside the iOS simulator. Did not find any exact reason, except, that it may have to do with internal memory allocation of the AudioToolbox framework. Very large sounfonts for instance do produce complete dropouts in sound at certain positions and situations…
Thank you Jens, But it’s not work for me. And I changed different sf2 file I will got different effect, That;s strange, I can assure these files all have 127 patches.
I think, the problem are these sf2 files. if the result is different.
Therefore again: please check the MIDI Bank Select messages in your MIDI files. ^^
We do not know something about the internal parsing and mapping of sf2 files by the Apple code…. So it could be, that Apple dismissed patches, that are not understood by them or they do not even support propper MIDI Bank Select messages to select patches on different program banks, defined in the files…
Unfortunately you won’t even get error messages about that! ^^
I got one more tip from my code. If I don’t send the midi command Cx (x is midi channel index , this command is set the instrument tone of current channel) send to apple core audio unit by the method of MusicDeviceMIDIEvent , The midi all channel will play by piano, It’s complete , don’t lose anything. So I also trust you said that Apple dismissed patches. If so that is the worst thing.
Send me a link to your github repo or email me your code and I’ll see.
Sorry,I don’t know what’s your email.
Hi Gene,
Thanks for your post , And I worked it done . But I meet a situation, When I want send midi to music midi sequencer by MusicDeviceMIDIEvent, I also send it to external midi device port by MIDIPacketListAdd and MIDISend method. If my iOS device connected external midi device. the audio midi sequencer worked so strangely. If there haven’t external midi device , It’s worked ideally.
Any suggestion, Thanks very much.
The big MIDI Raw Data problem in SWIFT traces down to a certain problem.
The data structs are obviously imported/converted wrong in the SWIFT headers. A Meta Message, for instance, is defined there as:
public struct MIDIMetaEvent {
public var metaEventType: UInt8
public var unused1: UInt8
public var unused2: UInt8
public var unused3: UInt8
public var dataLength: UInt32
public var data: (UInt8) <<<<<<<<<
public init()
public init(metaEventType: UInt8, unused1: UInt8, unused2: UInt8, unused3: UInt8, dataLength: UInt32, data: (UInt8))
}
That, (please note that ominous (UInt8) tuple), is the reason, why it won't work! SWIFT obviously internally copies this data when allocating events – we do not get direct access to the data for some reason. And this(re) allocates the space for exactly 1 element regarding this declaration… So only the first byte of the message is accessible for us – which is exactly what many of us have experienced – all other data is not there. Although, the data still validly exists anywhere in memory, as CAShow() shows…
So the trick for a solution is, to provide an own version of the MIDIMetaEvent struct definition, with a corrected data field and pass that to the functions which require it. But note, that MIDI raw data can have massively varying size…
I have seen many source code samples, where the users just simply access the pointers of the default data struct with exactly 1 byte allocation and actually write junk into it per UnsaveMutablePointer actions! This is a quite dangerous way, because we do not know anything about how AudioToolbox internally works…
Even if it worked this way for many tests, there is no chance to get the correct count of bytes inside the MIDI callbacks and MusicSequence iterations, if one uses the default (wrong) SWIFT struct definitions. My approach therefore is, to cast a user defined struct definition of MIDIMetaEvent & co. with a fixed buffer of properly aligned UInt8 data taking a sufficient size.
Tested it. It worked. Now I am able to receive Meta, Sysex and User Events with the correct count of bytes and data.
But it took an entire day to feature this out…
This comes up so often that I created a blog post with the answer I usually give.
But I’d be curious to see your “corrected struct”. Can you email me a bit of the code?
Hello Gene!
Yes, I will send you the stuff for the new *MIDIMetaEventEx* structure, I created. It’s really a simple solution… But I have to do excessive testing with that at first and finally pump a working “prove’ into the App Store…
By the way: It is a HACK of course. But what?! It actually works. ^^
Sorry to obviously have that posted into the wrong place here ! ^^
Thank you for everything!
Can you send me an email, I can’t find yours.
Your code:
withUnsafeMutablePointer(to: &metaEvent.data, {
ptr in
for i in 0 ..< data.count {
ptr[i] = data[i]
}
})
// I"m guessing that this copies the data
let status = MusicTrackNewMetaEvent(track, 0, &metaEvent)
… is HIGHLY INSECURE, because this way you actually pump junk into the devices (unknown) memory !! ^^
Yes it is. My paragraph following this example discourages doing this.
Hello Gene,
I just posted the code to the right thread.
You can just copy and past it into your example projects. It’s quasi the same mechanism as you did, but with a ‘corrected’ data structure definition (MIDIMetaEventEx). Use the new struct in all reading and writing operations and it’s save and it works.
Similarly define new structs for MIDIRawData(Ex) and MusicEventUserData(Ex) to make these work too…
… and yeah …
it is actually a pity, that APPLE ignores our concerns in that miserable way!!!
so we have to find working solutions ourselves ^^
btw: With these new struct definitions one ‘theoretical’ can even pass large user defined data to the a MusicSequence.
For whatever this could be any useful …
But i did not test that yet.
Much better, if there was a possibility to tag events inside a MusicSequence with a unique identifier for direct access… but that isn’t possible. sadly. we only have that awful iterator to the events of a music sequence.
So my personal aproach would be to copy all events of a loaded music sequence into my own sequencer memory data for modification and direct access. and then reload the music sequence with that data in realtime frequently. Do not know, whether this woud break the realtime playing flow of a sequence, if he/she is moving a note event with her/his fingers while a sequence is still playing ^^…
Has someone experiences with such behaviour? Real time editing of note events is a major requirement for a MIDI sequencer, I guess…
Hi Gene,
Thanks for this blog its very helpful!
Just wondering if its possible to also create a mult-timbral synth using a aupreset as the sample source rather than a soundfont file? I’ve tried using the your code but replacing the soundbank url with an aupreset url but that doesn’t work.. I also can’t find anything similar to kMusicDeviceProperty_SoundBankURL that might apply to aupresets..
many thanks!