Installing Gaffer on AWS with EC2 Image Builder

This is a guide to setting up Amazon Web Services EC2 Image Builder with Gaffer. The EC2 Image Builder is a service from AWS that allows you to create an AMI (Amazon Machine Image) for use with EC2 with all the software and configuration needed to run jobs on AWS. An EC2 instance is a preset configuration of processor, memory and network resources made available by AWS - the virtual equivalent of hardware. An EC2 instance needs to be launched with an AMI that defines the virtual machine that will be running - the operating system and installed software.

This guide is intended for users with some AWS knowledge but not expert level. After completing this guide, you will have a Linux-based EC2 AMI available to your AWS users that can be run on any EC2 instance type for running Gaffer tasks, including Arnold rendering. These instances can be launched manually through the AWS management console, via Python scripts or the AWS terminal, or from applications like Thinkbox's Deadline.

Overview

The EC2 Image Builder is split into a number of different parts. We need to work with three of those to create our Gaffer image. Image Pipelines are run to create the AMI. With Image Pipelines, you determine the schedule with which it will be run (likely Manual for our purposes) and the Image Recipe that will be used to create the AMI.

An Image Recipe consists the the base image AMI that will be the starting point for the new AMI, optional additional storage to attach to the instance and a set of components that define the software to be installed.

A Component is a YAML script describing the set of steps to be performed to install and configure some logical unit of software.

Image Recipes and Components are both versioned. When you save, that version becomes immutable and you will create a new version when making changes or fixes.

Getting Started

To start, log into the AWS Console. Pay attention to the region you have set - this setting is in the top-right of the console. The EC2 Image Builder components as well as the generated AMIs (and therefore the EC2 instances you launch with them) are unique per-region.

Choose EC2 Image Builder from the Services menu in the top-left. You can navigate between Image Pipelines, Image Recipes and Components from the left-side menu.

Image Pipeline

Choose Image Pipelines to see the list of Pipelines, which is empty if this is your first time using it.

1. Choose "Create Image Pipeline" to get started.
2. Enter a name for the pipeline and optionally a description.
3. Change the schedule option to "Manual" so it will only run when you trigger it.
4. Proceed to the next step, to configure the Recipe.
5. Choose "Create new recipe". You will be given a variety of new options.
6. Set the Image type to "Amazon Machine Image (AMI)".
7. Enter a name for the Recipe.
8. Enter a version for the Recipe, such as 1.0.0.