The other day on my lunch I went for a stroll into the Apple Store in Glasgow and I was watching what kind of apps users were trying out. Just about every user was enjoying drawing stuff and interacting with complex visual apps on the new iPads. I totally understand this as some of the visuals are just stunning on those apps and it kind of reminded me of some of my early app ideas when I was getting into iOS development. The thing is, once you start to cut your teeth with the iOS repertoire, you find yourself working with Table Views, Collection Views and some custom UI components if you’re working on a new project. So I quickly realised that for this idea I would need to take a step down into Metal.

I had a brief play around with shaders last year when making a Unity game where I wanted a chicken character to become almost invisible. That was fun. However, in iOS I have found myself wanting to explore that space more and realised rather quickly that there is a bit of a learning curve involved. I like to say ‘If something has a bit of obsession value involved, categorically sign me up’ so naturally I jumped right in.

This post is a work in progress for me, I’m basically working my way through ‘Metal by Tutorials’ by Ray Wenderlich and conveying my findings. I always say:

If you need to learn fast, you can’t afford not to blog.

Starting Metal

Metal gives us near-direct access to the GPU meaning absolutely killer performance with an approachable API. This is how you’ll get astoundingly good 3d performance on an iPad that you just couldn’t get using a standard engine, plus we get the control. To be honest though, I’ve barely gotten started with Metal but wanted to kind of help people looking to get into it and write about my journey. Let’s begin.

The first thing you’ll need is access to a device. I always create iOS Playgrounds when learning something new, but in this case Metal can’t run on that as it needs hardware to do so. You’ll need to create a MacOS Playground, which will work exactly the same in this tutorial.

Device

Metal needs access to system default device which we can get through it’s handy API:

1
2
3
4
5
6
7
import PlaygroundSupport
import MetalKit

// Create the Metal Device to get started
guard let device = MTLCreateSystemDefaultDevice() else {
fatalError("No default preferred metal system device found")
}

Here we’ve called the static method that will be part of the initial setup to get the Metal ball rolling and will completely bail with a fatal error if it’s not possible. We’re also going to use PlaygroundSupport as this will give us access the live view as well which is really handy to actually see something for our efforts directly in the Playground: yay!

1
2
3
4
// Get the view ready for live viewing
let frame = CGRect(x: 0, y: 0, width: 400, height: 400)
// Rather than using a standard view, we use a Metal Kit View - MTKView
let view = MTKView(frame: frame, device: device)

One of the interesting things that’s different with Metal is that instead of UI Views where the drawing is taken care of for us, we use MTKViews where we get much more control as to how things are presented. We use that here and it takes a device (which we created earlier) as well as an actual frame to work with size-wise.

Next we’ll set the background colour of our view:

1
2
// View color in rgba value
view.clearColor = MTLClearColor(red: 0.5, green: 1, blue: 0.5, alpha: 1)

We use the ‘clearColor’ property and pass in rgba values to set the colour. Think of this as a standard backgroundColor property on a UIView.

Buffer Time

Interestingly, when you’re working with Metal, things are done and stored in buffers. I had seen some of this code before and always wondered what it was but it seems to be an abstraction from dealing directly with memory and how things are done handling commands to and from the GPU. Let’s create a buffer just now passing in the device we had earlier:

1
2
3
// Memory management in Metal apparantly is done with temporary buffers that we assign for work
// Loads in vertex information directory to the GPU for calculations
let allocator = MTKMeshBufferAllocator(device: device)

Now let’s create a primitive shape to work with rather than loading in something from Blender (which I don’t know how to do yet!).

1
let mdlMesh = MDLMesh(sphereWithExtent: [0.5, 0.5, 0.5], segments: [50, 50], inwardNormals: false, geometryType: .triangles, allocator: allocator)

Here we’ve created a sphere shape and given it draw dimensions. Everything is kind of drawn as triangles as GPUs are configured to always handle these. Apparently if you design an object in Blender or Maya you work with Quads as they’re better for creating 3d models, but at the end of the day it’ll always be about triangles: I guess Pythagoras was right…

Under the hood, something called Model I\O loaded our object but it needs to be created into a mesh now for displaying:

1
2
// We need to convert it to a standard mesh in order to actually display it with Metal
let mesh = try MTKMesh(mesh: mdlMesh, device: device)

Cool, so we’ve created our device, set the background colour of the Metal view and created our sphere as a mesh. Next let’s add the ingredients to start allowing the GPU to work with actual rendering of our amazing 3d sphere!

Shaders

Shaders, man they sounds scary. What exactly are they and what do they do? First off its interesting to know that most shaders are tiny little programs that run on a subset of C++ or C++ itself. This can freak people out as it is, but it’s all syntax. At the end of the day as long as you understand what is going on you’ll be fine.

The GPU will receive vertices, points in 3d space, and fragments (what colour those should be. Imagine a 3d object made of triangles, let’s say it is a giant frog. The frog is made of lots of triangles, so we’ll have lots of points to join (vertices), we’ll also make the whole thing one colour so it’ll have to run the fragment function for each of these and it’ll make each one green. You could go wild with shaders but they’re job is to take 3d objects and return a pixel value on the screen.

Let’s create our first shader. You should create this in a shader file, but we’ll just create a String for now using multiline syntax 3 x “:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
let myFirstShader = """
#include
using namespace metal;

struct VertexIn {
float4 position [[ attribute(0) ]];
};

vertex float4 vertex_main(const VertexIn vertex_in [[ stage_in ]]) {
return vertex_in.position;
}

fragment float4 fragment_main() {
return float4(0.5, 0.8, 0.5, 1);
}
"
""

We have a standard C++ include, like an import from Swift, which imports Metal’s functionality for this mini program. Next we use a namespace. Namespaces are simply ways of accessing methods from a class without having to remind the compiler that we’re using that name. If we didn’t use the namespace command here we’d need to do: ‘metal.someFunction()’, ‘metal.someProperty’ every time rather than ‘someFunction(), someProperty’. (Note this is not the correct syntax, just the idea – you would use ‘::’ to access properties and methods from metal in C++.

You might be somewhat familiar with ‘main’ functions – they are the entry point into applications. Not long ago we were used to seeing this method in the App Delegate before it was changed to ‘@UIApplicationMain’. Here you can see it in the vertex_main and the fragment_main functions. Metal will call these at its own disposal to render our 3d object as needed.

The vertex main function will just return whatever was passed in without altering them in anyway. BUT, you could have messed with this to create a blurry effect if you were working with water for example. The fragment just returns the colour that the vertices should be, creating a solid 3d triangle. Feel free to alter these.

Getting over the distinct absence of autocomplete when working with shaders is a bit difficult at first, but if you’ve entered things as you see above it will work just fine until you want to learn more (I know I do!).

metal ios basics for beginners

Imagine the points on the mountain above are the vertices and the fragments are the white colours in between.

Putting It Together

Now that we’ve created our basic shader, we need to notify ‘something’ of where they are and what they’re called. To do that we use a thing called a Library.

1
2
3
4
5
let library = try device.makeLibrary(source: shader, options: nil)
// Get the two functions ready
let vertexFunction = library.makeFunction(name: "vertex_main")
let fragmentFunction = library.makeFunction(name: "fragment_main")
// Compiler checks these exist and that they were successfully created

We create a library from our device and pass it the shader. We then carefully enter the name of the vertex and fragment functions we created earlier. Go you!

Once we’re good to go we need to create a rendering pipeline. Think of this as taking all of our commands to render frames. The thing is, it needs to be super-fast so we create some constant state so that we can do a one-time setup and make everything more efficient. The weird part is we don’t directly ‘make one’, we have to create a description of one and it will in turn give us access to the new state:

// Pipeline State – nothing is going to change, like caching for efficiency – holds positions, fragment and vertex shaders and rendering info: colour space, depth etc

1
2
3
4
5
6
// We create this via a description
let descriptor = MTLRenderPipelineDescriptor()
descriptor.colorAttachments[0].pixelFormat = .bgra8Unorm
descriptor.vertexFunction = vertexFunction
descriptor.fragmentFunction = fragmentFunction
descriptor.vertexDescriptor = MTKMetalVertexDescriptorFromModelIO(mesh.vertexDescriptor)

The descriptor gets access to the pixel format and the functions we need to render and can cache things like the colour space. Next, we’ll pass in the the descriptor to the device and ask it nicely to return us a pipeline state:

1
let pipelineState = try device.makeRenderPipelineState(descriptor: descriptor)

Now that we’ve created our pipeline and it is ready to go, let’s look at how to render our sphere.

1
2
3
4
5
6
// Every frame from now on - preferably 60fps +
// Rendering

guard let commandBuffer = commandQueue.makeCommandBuffer(),
let descriptor = view.currentRenderPassDescriptor,
let renderEncoder = commandBuffer.makeRenderCommandEncoder(descriptor: descriptor) else { fatalError() }

This is quite a bit of code. We need a command buffer as it is apparently how Metal handles memory. Remember the good old days of ‘autorelease’, well that thankfully abstracted away using buffers here. We access things we can render with ‘drawables’ in Metal. The render pass descriptor of our view is essentially everything Metal needs to know to actually draw the view such as textures and depth. Finally we make an encoder – think of this as an actual render task done in a frame. If we get all of these then we proceed.

Now that we have our encoder task ready, we’ll pass it the pipeline state we made earlier and the vertices it has to draw:

1
2
3
renderEncoder.setRenderPipelineState(pipelineState)

renderEncoder.setVertexBuffer(mesh.vertexBuffers[0].buffer, offset: 0, index: 0)

The sphere has submeshes and we want the 1st one so we access that using the buffer. Just imagine a complex model of a house: it would contain a lot of submeshes. One for the windows, one for the grass etc.

1
2
3
guard let subMesh = mesh.submeshes.first else {
fatalError()
}

Let’s Render

 

Finally let’s get some output!

1
2
3
4
5
6
7
8
9
10
11
// Draw the primitive sphere
renderEncoder.drawIndexedPrimitives(type: .triangle, indexCount: subMesh.indexCount, indexType: subMesh.indexType, indexBuffer: subMesh.indexBuffer.buffer, indexBufferOffset: 0)
// End rendering, the task is done
renderEncoder.endEncoding()
// Get our drawable from the view
guard let drawable = view.currentDrawable else { fatalError() }
// Show it!
commandBuffer.present(drawable)
commandBuffer.commit()
// Output it in the Playground
PlaygroundPage.current.liveView = view

Now you should see your shiny sphere showing in the live view.

Metal basic introduction ios

Whenever you start working with a new technology, you have to think of it like taking over a football team you’ve never met before. You understand the game, but you need to get to know your players. The big players in Metal are the Pipelines, the descriptors, the encoders and buffers. It’s still all new to me as well but the more we spend time working with a technology, the easier it becomes.