Swift AVFoundation to play audio or MIDI
Swift AVFoundation
There are many ways to play sound in iOS. Core Audio has been around for a while and it is very powerful. It is a C API, so using it from Objective-C and Swift is possible, but awkward. Apple has been moving towards a higher level API with AVFoundation. Here I will summarize how to use AVFoundation for several common audio tasks.
N.B. Some of these examples use new capabilities of iOS 8.
Playing an Audio file
Audio Session
Playing a MIDI file
Audio Engine
Playing MIDI Notes
Summary
Resources
Playing an Audio file
Let’s start by loading an audio file with an AVAudioPlayer instance. There are several audio formats that the player will grok. I had trouble with a few MP3 files that played in iTunes or VLC, but caused a cryptic exception in the player. So, check your source audio files first.
If you want other formats, your Mac has a converter named afconvert. See the man page.
|
1 |
afconvert -f caff -d LEI16 foo.mp3 foo.caf |
Let’s go step by step.
Get the file URL.
|
1 |
let fileURL:NSURL = NSBundle.mainBundle().URLForResource("modem-dialing-02", withExtension: "mp3") |
Create the player. You will need to make the player an instance variable, because if you just use a local variable, it will be popped off the stack before you hear anything.
|
1 2 3 4 5 6 7 |
var error: NSError? self.avPlayer = AVAudioPlayer(contentsOfURL: fileURL, error: &error) if avPlayer == nil { if let e = error { println(e.localizedDescription) } } |
You can provide the player a hint for how to parse the audio data. There are several constants you can use.
|
1 |
self.avPlayer = AVAudioPlayer(contentsOfURL: fileURL, fileTypeHint: AVFileTypeMPEGLayer3, error: &error) |
Now configure the player. prepareToPlay() “pre-rolls” the audio file to reduce start up delays when you finally call play().
You can set the player’s delegate to track status.
|
1 2 3 |
avPlayer.delegate = self avPlayer.prepareToPlay() avPlayer.volume = 1.0 |
To set the delegate you have to make a class implement the player delegate protocol. My class has the clever name “Sound”.
|
1 2 3 4 5 6 7 8 9 |
// MARK: AVAudioPlayerDelegate extension Sound : AVAudioPlayerDelegate { func audioPlayerDidFinishPlaying(player: AVAudioPlayer!, successfully flag: Bool) { println("finished playing \(flag)") } func audioPlayerDecodeErrorDidOccur(player: AVAudioPlayer!, error: NSError!) { println("\(error.localizedDescription)") } } |
Finally, the transport controls that can be called from an action.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
func stopAVPLayer() { if avPlayer.playing { avPlayer.stop() } } func toggleAVPlayer() { if avPlayer.playing { avPlayer.pause() } else { avPlayer.play() } } |
The complete gist for the AVAudioPlayer:
Audio Session
The Audio Session singleton is an intermediary between your app and the media daemon. Your app and all other apps (should) make requests to the shared session. Since we are playing an audio file, we should tell the session that is our intention by requesting that its category be AVAudioSessionCategoryPlayback, and then make the session active. You should do this in the code above right before you call play() on the player.
Setting a session for playback.
Go to Table of Contents
Playing a MIDI file
You use AVMIDIPlayer to play standard MIDI files. Loading the player is similar to loading the AVAudioPlayer. You need to load a soundbank from a Soundfont or DLS file. The player also has a pre-roll prepareToPlay() function.
I’m not interested in copyright infringement, so I have not included either a DLS or SF2 file. So do a web search for a GM SoundFont2 file. They are loaded in the same manner. I’ve tried the MuseCore SoundFont and it sounds ok. There is probably a General MIDI DLS on your OSX system already: /System/Library/Components/CoreAudio.component/Contents/Resources/gs_instruments.dls. Copy this to the project bundle if you want to try it.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
self.soundbank = NSBundle.mainBundle().URLForResource("GeneralUser GS MuseScore v1.442", withExtension: "sf2") // a standard MIDI file. var contents:NSURL = NSBundle.mainBundle().URLForResource("ntbldmtn", withExtension: "mid") var error:NSError? self.mp = AVMIDIPlayer(contentsOfURL: contents, soundBankURL: soundbank, error: &error) if self.mp == nil { println("nil midi player") } if let e = error { println("Error \(e.localizedDescription)") } self.mp.prepareToPlay() |
You can also load the MIDI player with an NSData instance like this:
|
1 2 3 |
var data = NSData(contentsOfURL: contents) var error:NSError? self.mp = AVMIDIPlayer(data: data, soundBankURL: soundbank, error: &error) |
Cool, so besides getting the data from a file, how about creating a sequence on the fly? There are the Core Audio MusicSequence and MusicTrack classes to do that. But damned if I can find a way to turn the sequence into NSData. Do you? FWIW, the AVAudioEngine q.v. has a barely documented musicSequence variable. Maybe we can use that in the future.
In your action, call the play() function on the player. There is only one play function, and that requires a completion handler.
|
1 2 3 |
self.mp.play({ println("midi done") }) |
Complete AVMIDIPlayer example gist.
Audio Engine
iOS 8 introduces a new audio engine which seems to be the successor to Core Audio’s AUGraph and friends. See my article on using these classes in Swift.
The new AVAudioEngine class is the analog to AUGraph. You create AudioNode instances and attach them to the engine. Then you start the engine to initiate data flow.
Here is an engine that has a player node attached to it. The player node is attached to the engine’s mixer. These are instance variables.
|
1 2 3 4 5 |
engine = AVAudioEngine() playerNode = AVAudioPlayerNode() engine.attachNode(playerNode) mixer = engine.mainMixerNode engine.connect(playerNode, to: mixer, format: mixer.outputFormatForBus(0)) |
Then you need to start the engine.
|
1 2 3 4 5 6 7 |
var error:NSError? if !engine.startAndReturnError(&error) { println("error couldn't start engine") if let e = error { println("error \(e.localizedDescription)") } } |
Cool. Silence.
Let’s give it something to play. It can be an audio file, or as we’ll see, a MIDI file or a computed buffer.
In this example we create an AVAudioFile instance from an MP3 file, and tell the playerNode to play it.
First, load an audio file. If you know the format of the file you can provide hints.
|
1 2 3 4 5 6 7 8 |
let fileURL = NSBundle.mainBundle().URLForResource("modem-dialing-02", withExtension: "mp3") var error: NSError? let audioFile = AVAudioFile(forReading: fileURL, error: &error) // OR //let audioFile = AVAudioFile(forReading: fileURL, commonFormat: .PCMFormatFloat32, interleaved: false, error: &error) if let e = error { println(e.localizedDescription) } |
Now hand the audio file to the player node by “scheduling” it, then playing it.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
engine.connect(playerNode, to: engine.mainMixerNode, format: audioFile.processingFormat) playerNode.scheduleFile(audioFile, atTime:nil, completionHandler:nil) if engine.running { playerNode.play() } else { if !engine.startAndReturnError(&error) { println("error couldn't start engine") if let e = error { println("error \(e.localizedDescription)") } } else { playerNode.play() } } |
Playing MIDI Notes
How about triggering MIDI notes/events based on UI events? You need an instance of AVAudioUnitMIDIInstrument among your nodes. There is one concrete subclass named AVAudioUnitSampler. Create a sampler and attach it to the engine.
|
1 2 3 |
sampler = AVAudioUnitSampler() engine.attachNode(sampler) engine.connect(sampler, to: engine.outputNode, format: nil) |
At init time, create a URL to your SoundFont or DLS file as we did previously.
|
1 |
soundbank = NSBundle.mainBundle().URLForResource("GeneralUser GS MuseScore v1.442", withExtension: "sf2") |
Then in your UI’s action function, load the appropriate instrument into the sampler. The program parameter is a General MIDI instrument number. You might want to set up constants. Soundbanks have banks of sound. You need to specify which bank to use with the bankMSB and bankLSB. I use a Core Audio constant here to choose the “melodic” bank and not the “percussion” bank.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
// probably instance variables let melodicBank:UInt8 = UInt8(kAUSampler_DefaultMelodicBankMSB) let gmMarimba:UInt8 = 12 let gmHarpsichord:UInt8 = 6 // then in the action var error:NSError? if !sampler.loadSoundBankInstrumentAtURL(soundbank, program: gmHarpsichord, bankMSB: melodicBank, bankLSB: 0, error: &error) { println("could not load soundbank") } if let e = error { println("error \(e.localizedDescription)") } |
Then send a MIDI program change to the sampler. After that, you can send startNote and stopNote messages to the sampler. You need to match the parameters for each start and stop message.
|
1 2 3 4 5 6 |
self.sampler.sendProgramChange(gmHarpsichord, bankMSB: melodicBank, bankLSB: 0, onChannel: 0) // play middle C, mezzo forte on MIDI channel 0 self.sampler.startNote(60, withVelocity: 64, onChannel: 0) ... // in another action self.sampler.stopNote(60, onChannel: 0) |
Summary
This is a good start I hope. There are other things I’ll cover soon, such as generating and processing the audio buffer data.
Dear Gene,
Thank you for your precious tutorial, it was exactly what I was looking for!
About the first example (playing a sound file): I had to make the Sound class inherit from NSObject, otherwise Xcode complains.
Everything seems OK, the console tells me that the file is playing, the volume is up… but I still get no sound.
I am running on the emulator, not on an iPhone. Could this be the issue?
Thanks in advance,
Best,
Elvio
PS: of course I tried different file formats (mp3 and aiff) and different files for each format.
Ah, the Gist doesn’t match the Github complete project. Thanks.
Yes, the Swift Sound class needs to inherit from NSObject for the key value things.
Do you get a sound if you try your sound file with the github project?
I did have a problem with one badly encoded MP3 file.
Thank you for the instantaneous answer!
It’s embarrassing, but after browsing
https://github.com/genedelisa
I still can’t find the github project… could you please give me the link in extenso?
Thanks!
The Github link is up there ^^^ under Resources 🙂
https://github.com/genedelisa/AvFoundationFrobs
IT WORKS! 😀
I will spend the next days studying your code, and trying to understand what I did wrong in mine.
Thanks again,
Best,
Elvio
I simply put
var mySound = Sound()
inside the viewDidLoad() method. When I got it out, everything worked.
Starting MIDI, now! 🙂
Dear Gene,
I finally had time to dig into the MIDI part of your code.
It looks like the sound bank is actually initialized twice, once in Sound and the other in the View Controller.
To avoid redundancy, I would suggest to delete the latter, and to refer to it with self.sound.soundbank
But apart from this detail, I have a weird problem: the harpsichord sound (or whatever midi sound) is audible only when I press the NoteOn/NoteOff button. When I try to play a midi file, everything is played with sinuses.
My impression is that something is broken in the chain, and that the engine connects to a default synth instead of connecting to the sampler with the sf2 bank.
Do you have any hint? Thanks in advance!
Best,
Elvio
They are 2 separate examples.
The ViewController has an example of using “low level” AVAudioEngine code.
Sound.swift is a higher level example that uses the AVMIDIPlayer and AVAudioPlayer.
Oh, ok! Thanks!
Dear Gene,
You mention the possibility to “feed” an audioNode with a sound file, with a MIDI file or with a buffer.
I can’t find any way to relate a MIDI file to an AVAudioNode (surprisingly, even if AVMIDIPlayer produces audio, it inherits just from NSObject). Could you give me a hint?
Thanks,
Elvio
Hi I’m wondering how to receive MIDI inputs directly into a playground. What I’m wanting to do is receive the note number whenever I hit a key on my keyboard what would be the most practical way of doing this?
Why is it when I do a basic search for midi players I get a whole bunch of smart phone crap instead of windows?
PBCAC
Hello,
Thanks for yours tutorials on this subject.
I have one question about the “sendProgramChange” function of the AVAudioUnitSampler object. It shouldn’t allow to change the preset of a soundfont ?
I want to play some musical notes of a soundfont file with multiple presets, but I’m unable to change the presets without reloading the soundfont files (which is too long to load).
My other solution is to split my soundfont file in multiple files with only one preset, and to create multiple AVAudioUnitSampler objects, but this solution consume more memory.
What works for me is to use a separate sampler for each sound font preset that I want to use. You hook up each sampler to the mixer. This is what I did in Core Audio graphs also.
Let me know if that helps.
Yes, it works well.
I allocates the samplers when it’s necessary and I release them after playing the sounds.
Thanks for your help.
Hi Gene, thanks for taking the time to share all of this work. I’m on the “learn swift” journey. I’d appreciate any thoughts you might be willing to share on the appropriate technique to de-activate the session. I’m assuming you inspect the event generated from the controller when the sound file is finished playing and then de-activate it but I’m not sure what best practices is here. Appreciate anything you can share. Regards, Doug
An easy way to activate/deactivate the session is in your AppDelegate’s applicationDidBecomeActive and applicationDidEnterBackground if you are not playing “background audio”. This is what Apples says in this Audio Session Programming Guide
Hi!
Wondering if anyone has any advice on getting a timestamp of a song when the audio starts to output. Right now there is about a second delay between the an avplayer action and audio being outputted. I would like to track the time when audio is being outputted on the iOS device rather than just the command action.
Thanks!
Hello Gene De Lisa!
Very thank you for examples – it’s awesome!
Please tell, how to implement real time audio processing to audio from music player, for example – add reverb to audio played from Apple Music player. Another words – insert audio processing chain right before audio output.
Best regards, Ivan.
One way is via v3 Audio Unit Extensions. Here is a WWDC video.
For common uses like delay, there are a few AVAudioUnitEffects already available. Here is AVAudioUnitDelay.
Hello Gene De Lisa!
Thanks for the explanation and example
It helped me a lot!
I have a questions :
How can I change channel volume For an object AVMIDIPlayer in real time?
Thank you …
Best regards, sapir.
None of Apple’s MIDI APIs allow you to do real time modification of the event stream.
But you can change the volume of the mixerNode in real time if you’re using the AVAudioUnitSampler.
PS
There is a Swift 2 version of this article.
Thank you for the answer
I try it ,
I use this : var sampler:AVAudioUnitSampler
my code:
self.sampler.sendController(7, withValue: 0, onChannel: 1)
self.sampler.sendController(7, withValue: 0, onChannel: 2)
self.sampler.sendController(7, withValue: 0, onChannel: 3)
self.sampler.sendController(7, withValue: 120, onChannel: 10)
I didnt hear eny change in the midi playing file in AVMIDIPLAYER
Thank you …
Best regards, sapir.
Hi,
When you load the sound font and set those values:
let gmMarimba:UInt8 = 12
let gmHarpsichord:UInt8 = 6
Where do you get them from? Is there a software that can load a dls file and show all the available instruments in there?
Thanks!
General MIDI patch numbers