Swift 2: AVFoundation to play audio or MIDI
Swift AVFoundation
There are many ways to play sound in iOS. Core Audio has been around for a while and it is very powerful. It is a C API, so using it from Objective-C and Swift is possible, but awkward. Apple has been moving towards a higher level API with AVFoundation. Here I will summarize how to use AVFoundation for several common audio tasks.
N.B. Some of these examples use new capabilities of iOS 8.
Playing an Audio file
Audio Session
Playing a MIDI file
Audio Engine
Playing MIDI Notes
Summary
Resources
Playing an Audio file
Let’s start by loading an audio file with an AVAudioPlayer instance. There are several audio formats that the player will grok. I had trouble with a few MP3 files that played in iTunes or VLC, but caused a cryptic exception in the player. So, check your source audio files first.
If you want other formats, your Mac has a converter named afconvert. See the man page.
|
1 |
afconvert -f caff -d LEI16 foo.mp3 foo.caf |
Let’s go step by step.
Get the file URL.
|
1 2 3 4 |
guard let fileURL = NSBundle.mainBundle().URLForResource("modem-dialing-02", withExtension: "mp3") else { print("could not read sound file") return } |
Create the player. You will need to make the player an instance variable. If you just use a local variable, it will be popped off the stack before you hear anything!
|
1 2 3 4 5 6 |
do { try self.avPlayer = AVAudioPlayer(contentsOfURL: fileURL) } catch { print("could not create AVAudioPlayer \(error)") return } |
You can provide the player a hint for how to parse the audio data. There are several constants for file type UTIs you can use. For our MP3 file, we’ll use AVFileTypeMPEGLayer3.
|
1 |
self.avPlayer = AVAudioPlayer(contentsOfURL: fileURL, fileTypeHint: AVFileTypeMPEGLayer3) |
Now configure the player. prepareToPlay() “pre-rolls” the audio file to reduce start up delays when you finally call play().
You can set the player’s delegate to track status.
|
1 2 3 |
avPlayer.delegate = self avPlayer.prepareToPlay() avPlayer.volume = 1.0 |
To set the delegate you have to make a class implement the player delegate protocol. My class has the clever name “Sound”. The delegate protocol requires the NSObjectProtocol, so Sound is a subclass of NSObject.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
// MARK: AVAudioPlayerDelegate extension Sound : AVAudioPlayerDelegate { func audioPlayerDidFinishPlaying(player: AVAudioPlayer, successfully flag: Bool) { print("finished playing \(flag)") } func audioPlayerDecodeErrorDidOccur(player: AVAudioPlayer, error: NSError?) { if let e = error { print("\(e.localizedDescription)") } } } |
Finally, the transport controls that can be called from an action.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
func stopAVPLayer() { if avPlayer.playing { avPlayer.stop() } } func toggleAVPlayer() { if avPlayer.playing { avPlayer.pause() } else { avPlayer.play() } } |
Audio Session
The Audio Session singleton is an intermediary between your app and the media daemon. Your app and all other apps (should) make requests to the shared session. Since we are playing an audio file, we should tell the session that is our intention by requesting that its category be AVAudioSessionCategoryPlayback, and then make the session active. You should do this in the code above right before you call play() on the player.
Setting a session for playback.
|
1 2 3 4 5 6 7 8 9 10 |
func setSessionPlayback() { let audioSession = AVAudioSession.sharedInstance() do { try audioSession.setCategory(AVAudioSessionCategoryPlayAndRecord, withOptions: AVAudioSessionCategoryOptions.MixWithOthers) try audioSession.setActive(true) } catch { print("couldn't set category \(error)") } } |
Playing a MIDI file
You use AVMIDIPlayer to play standard MIDI files. Loading the player is similar to loading the AVAudioPlayer. You need to load a soundbank from a Soundfont or DLS file. The player also has a pre-roll prepareToPlay() function.
I’m not interested in copyright infringement, so I have not included either a DLS or SF2 file. So do a web search for a GM SoundFont2 file. They are loaded in the same manner. I’ve tried the MuseCore SoundFont and it sounds ok. There is probably a General MIDI DLS on your OSX system already: /System/Library/Components/CoreAudio.component/Contents/Resources/gs_instruments.dls. Copy this to the project bundle if you want to try it.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
func createAVMIDIPlayerFromMIDIFIleDLS() { guard let midiFileURL = NSBundle.mainBundle().URLForResource("sibeliusGMajor", withExtension: "mid") else { fatalError("\"sibeliusGMajor.mid\" file not found.") } guard let bankURL = NSBundle.mainBundle().URLForResource("gs_instruments", withExtension: "dls") else { fatalError("\"gs_instruments.dls\" file not found.") } do { try self.midiPlayer = AVMIDIPlayer(contentsOfURL: midiFileURL, soundBankURL: bankURL) print("created midi player with sound bank url \(bankURL)") } catch let error as NSError { print("Error \(error.localizedDescription)") } self.midiPlayer.prepareToPlay() } |
Audio Engine
iOS 8 introduces a new audio engine which seems to be the successor to Core Audio’s AUGraph and friends. See my article on using these classes in Swift.
The new AVAudioEngine class is the analog to AUGraph. You create AudioNode instances and attach them to the engine. Then you start the engine to initiate data flow.
Here is an engine that has a player node attached to it. The player node is attached to the engine’s mixer. These are instance variables.
|
1 2 3 4 5 |
engine = AVAudioEngine() playerNode = AVAudioPlayerNode() engine.attachNode(playerNode) mixer = engine.mainMixerNode engine.connect(playerNode, to: mixer, format: mixer.outputFormatForBus(0)) |
Then you need to start the engine.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
func startEngine() { if engine.running { print("audio engine already started") return } do { try engine.start() print("audio engine started") } catch { print("oops \(error)") print("could not start audio engine") } } |
Cool. Silence.
Let’s give it something to play. It can be an audio file, or as we’ll see, a MIDI file or a computed buffer.
In this example we create an AVAudioFile instance from an MP3 file, and tell the playerNode to play it.
First, load an audio file. Or load an audio file into a buffer.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
var buffer:AVAudioPCMBuffer! func readFileIntoBuffer() { guard let fileURL = NSBundle.mainBundle().URLForResource("modem-dialing-02", withExtension: "mp3") else { print("could not read sound file") return } do { let file = try AVAudioFile(forReading: fileURL) buffer = AVAudioPCMBuffer(PCMFormat: file.processingFormat, frameCapacity: AVAudioFrameCount(file.length)) try file.readIntoBuffer(buffer) } catch { print("could not create AVAudioPCMBuffer \(error)") return } } |
Now we hand the buffer to the player node by “scheduling” it, then playing it.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
func finishedPlaying() { print("finished playing") playerNode.stop() } func togglePlayer() { if playerNode.playing { playerNode.stop() } else { setSessionPlayback() startEngine() playerNode.scheduleBuffer(buffer, completionHandler: finishedPlaying) playerNode.play() } } |
There are quite a few variations on scheduleBuffer. Have fun trying them out.
Playing MIDI Notes
How about triggering MIDI notes/events based on UI events? You need an instance of AVAudioUnitMIDIInstrument among your nodes. There is one concrete subclass named AVAudioUnitSampler. Create a sampler and attach it to the engine.
|
1 2 3 4 |
engine = AVAudioEngine() sampler = AVAudioUnitSampler() engine.attachNode(sampler) engine.connect(sampler, to: engine.mainMixerNode, format: nil) |
In your UI’s action function, load the appropriate instrument into the sampler. The program parameter is a General MIDI instrument number. You might want to set up constants. Soundbanks have banks of sound. You need to specify which bank to use with the bankMSB and bankLSB. I use Core Audio constants here to choose the “melodic” bank and not the “percussion” bank.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
// instance variables let melodicBank = UInt8(kAUSampler_DefaultMelodicBankMSB) let defaultBankLSB = UInt8(kAUSampler_DefaultBankLSB) let gmMarimba = UInt8(12) let gmHarpsichord = UInt8(6) func loadPatch(gmpatch:UInt8, channel:UInt8 = 0) { guard let soundbank = NSBundle.mainBundle().URLForResource("GeneralUser GS MuseScore v1.442", withExtension: "sf2") else { print("could not read sound font") return } do { try sampler.loadSoundBankInstrumentAtURL(soundbank, program:gmpatch, bankMSB: melodicBank, bankLSB: defaultBankLSB) } catch let error as NSError { print("\(error.localizedDescription)") return } self.sampler.sendProgramChange(gmpatch, bankMSB: melodicBank, bankLSB: defaultBankLSB, onChannel: channel) } |
Then send a MIDI program change by calling our load function. After that, you can send startNote and stopNote messages to the sampler. You need to match the parameters for each start and stop message.
|
1 2 3 4 5 6 |
loadPatch(gmHarpsichord) // play middle C, mezzo forte on MIDI channel 0 self.sampler.startNote(60, withVelocity: 64, onChannel: 0) ... // in another action self.sampler.stopNote(60, onChannel: 0) |
Summary
This is a good start I hope. There are other things I’ll cover soon, such as generating and processing the audio buffer data.
Hi
Thank you for your explanations,
I greatly assisted by your excellent blog!
I wanted to ask which way I’m connects the avmidiplayer to the sampler ֿ
i.e
If I want to apply midi commands (commands that exist in the sampler)
As noteOn, change channel … on the file that plays now in avmidiplayer
in real time.
How do we do this?
Thanks again
Rebecca
AVMidiPlayer is a simple class that simply plays a MIDI file through a soundbank.
I you want to send MIDI messages, it is the wrong class to use.
AVAudioUnitSampler has functions that it inherits from AVAudioUnitMIDIInstrument to send MIDI messages.
Thanks your post saved me a lot of time. I was looking for information on opening a MIDI file in Swift.
Hi Gene De Lisa,
I’m trying to convert a midi file to m4a using swift2 in iOS, it does seem doable, via AVAudioEngine.
My problem is how do I load a musicSequence into a AVAudioPCMBuffer?
Thanks
Z
You can’t load a MusicSequence into an AVAudioPCMBuffer. You load samples into it either by computing a waveform or loading a sound file.
Ok, good to know. I’ll try again. The musicSequence starts as a MIDIfile and is turned into audio somewhere along the way, presumably it is held in a AVAudioPcmBuffer?
Is it possible to capture or Tap the audio and save it as an M4a file.
I have watched the WWDC 2014-2015 coreAudio video’s and still finding it tricky getting my head around it.
Thank you again
Z
The engine renders the MIDI commands in the MusicSequence through the sampler. You probably have the sampler connected to the engine’s mixer. You can tap the mixer’s output and save that.
So, I can play a musicSequence through a musicPlayer, no problem there.
And according to the Apple doc’s in Playing Audio section, there is a music sequence variable available and also says …The music sequence that was previously attached to the audio engine…..How and where would I attach the musicSequence or musicPlayer to the audioEngine?
Thank you again.
Never mind, I found the answer in the 29th minute of the WWDC 2015 coreAudio video.
Thanks
Gene,
I’m trying to use your code above to play guitar notes and guitar chords. I am successfully using it for guitar notes, but chords are not working very well. I have tried playing multiple simultaneous notes like this….
func gstart(noteNumber: UInt8, channel: UInt8) {
loadPatch(gmNylonGuitar, channel: channel)
self.sampler.startNote(40, withVelocity: 100, onChannel: 1)
self.sampler.startNote(48, withVelocity: 100, onChannel: 2)
self.sampler.startNote(52, withVelocity: 100, onChannel: 3)
self.sampler.startNote(55, withVelocity: 100, onChannel: 1)
self.sampler.startNote(60, withVelocity: 100, onChannel: 2)
self.sampler.startNote(64, withVelocity: 100, onChannel: 3)
}
One or two notes works ok, if I add more notes I start getting SIGABRT in loadPatch. I have also experimented with trying to use multiple channels but that doesn’t help. As you can probably tell, I don’t know much about MIDI.
I am using Xcode 8.0, Swift 3, and running on iOS 10. I am also using gs_instruments.dls for the sound bank.
Do you have any suggestions on how I can play chords on iOS, preferably using midi? These chords will have up to 6 notes playing simultaneously. If I have to I can always record chords and play those sound files but I would prefer to try to use MIDI.
One particular use for sending a startNote message to a sampler is in response to a UI event like touch down – followed by an endNote on touch up.
For what you want, I suggest that you use an AudioToolbox MusicSequence and add notes to a MusicTrack with the same beat.
var mess = MIDINoteMessage(channel: 0,
note: 60,
velocity: 64,
releaseVelocity: 0,
duration: 1.0 )
var status = MusicTrackNewMIDINoteEvent(track, beat, &mess)
//create and add more notes at the same beat
Hi Gene De Lisa,
Is it possible to display text, karaoke style while playing a MIDI file in iOS with Swift?
Could you point me in the right direction?
Thanks again
Z