Installing Gaffer on AWS with EC2 Image Builder

This is a guide to setting up Amazon Web Services EC2 Image Builder with Gaffer. The EC2 Image Builder is a service from AWS that allows you to create an AMI (Amazon Machine Image) for use with EC2 with all the software and configuration needed to run jobs on AWS. An EC2 instance is a preset configuration of processor, memory and network resources made available by AWS - the virtual equivalent of hardware. An EC2 instance needs to be launched with an AMI that defines the virtual machine that will be running - the operating system and installed software.

This guide is intended for users with some AWS knowledge but not expert level. After completing this guide, you will have a Linux-based EC2 AMI available to your AWS users that can be run on any EC2 instance type for running Gaffer tasks, including Arnold rendering. These instances can be launched manually through the AWS management console, via Python scripts or the AWS terminal, or from applications like Thinkbox's Deadline.

Overview

The EC2 Image Builder is split into a number of different parts. We need to work with three of those to create our Gaffer image. Image Pipelines are run to create the AMI. With Image Pipelines, you determine the schedule with which it will be run (likely Manual for our purposes) and the Image Recipe that will be used to create the AMI.

An Image Recipe consists the the base image AMI that will be the starting point for the new AMI, optional additional storage to attach to the instance and a set of components that define the software to be installed.

A Component is a YAML script describing the set of steps to be performed to install and configure some logical unit of software.

Image Recipes and Components are both versioned. When you save, that version becomes immutable and you will create a new version when making changes or fixes.

Getting Started

To start, log into the AWS Console. Pay attention to the region you have set - this setting is in the top-right of the console. The EC2 Image Builder components as well as the generated AMIs (and therefore the EC2 instances you launch with them) are unique per-region.

Choose EC2 Image Builder from the Services menu in the top-left. You can navigate between Image Pipelines, Image Recipes and Components from the left-side menu.

Gaffer Component

From the navigation menu, choose Components to see a list of your customized components, which is empty if this is your first time using it.

1. Choose "Create component".
2. Keep "Component type" set to Build
3. Set the Component Details
   1. Choose Linux as the "Image operating system". This indicates this recipe is compatible with Linux-based Image Pipelines. You determine the operating system for your AMI in the Image Pipeline configuration.
   2. From "Compatible OS Versions", choose the Linux versions you intend for this recipe to be compatible with.
   3. Enter a Component name.
   4. Enter a Component version, such as 1.0.0
4. Set the Content. This is the YAML script that will be run to install Gaffer. The YAML script below will setup Gaffer.
5. Finally, choose "Save Component" at the bottom of the screen to save it.

name: Worker Image
description: Installs software needed for rendering on AWS. This should be loaded into an EC2 Image Builder Configuration for this project.
schemaVersion: 1.0

parameters:
  - gafferVersion:
      type: string
      default: '1.3.7.0'
      description: 'The Gaffer version to install.'

phases:
  - name: build
    steps:
      - name: DownloadGaffer
        action: WebDownload
        inputs:
          - source: https://github.com/GafferHQ/gaffer/releases/download/{{gafferVersion}}/gaffer-{{gafferVersion}}-linux.tar.gz
            destination: /opt/gaffer-{{gafferVersion}}/gaffer-{{gafferVersion}}.tar.gz

      - name: ExtractGaffer
        action: ExecuteBash
        inputs:
          commands:
            - tar xvzf {{build.DownloadGaffer.inputs[0].destination}} --strip-components=1 -C /opt/gaffer-{{gafferVersion}}
            - rm {{build.DownloadGaffer.inputs[0].destination}}
            - echo 'export GAFFER_ROOT=/opt/gaffer-{{gafferVersion}}' >> /etc/profile

  - name: validate
    steps:
      - name: ValidateGaffer
        action: ExecuteBash
        inputs:
          commands:
            - $GAFFER_ROOT/bin/gaffer env python -c "print('Gaffer validated')"

The name, description, schemaVersion and the two items under phases are required. The parameters list is a helpful way of specifying variables. They can be referenced in multiple places, so they make your Components less error-prone. Their value can also be set from within an Image Recipe which makes for much fewer versions of your Component. Without parameter lists, you would need to version up your Component with each Gaffer version.

Build Phase

The first phase of a component build installs Gaffer. It first downloads Gaffer from the Github release URL. Then it extracts the archive to the /opt directory. Finally, an environment variable is set for other components to locate Gaffer's root directory.

The extraction step makes use of another helpful feature in Component scripts - referencing variables from a previous step. The {{build.DownloadGaffer.inputs[0].destination}} text indicates that the file path to extract is taken from the DownloadGaffer step of the build phase. This is another way to reduce errors from typos in your script.

Validate Phase

The second phase is for verifying that your installation succeeded. For our purposes, running a short Python script from within Gaffer will prove that the installation succeeded and Gaffer is operable. Implementing this phase helps identify problems before the AMI is created. If any of the validation steps fail, the Image Pipeline will be stopped and an error raised.

Arnold Component

Creating the Arnold Component is much like the Gaffer Component, with one significant difference. Arnold's installer is not available publicly so you will need to upload it to some location that EC2 Image Builder can access. It should be kept private to your organization, which makes AWS S3 a good solution. S3 is a storage repository you can manually or programmatically upload to. EC2 Image Builder Components have an action for downloading from S3, which makes it easy to incorporate into Components.

1. Upload the Arnold Linux archive to a bucket in your S3 storage and note it's URL. It will start with s3://.
2. Choose "Create Component" to start a new Component.
3. Set the Component details as you did with the Gaffer Component.
4. An example Component Content script is below.

name: Arnold Component
description: Installs Arnold and sets the ARNOLD_ROOT environment variable.
schemaVersion: 1.0

parameters:
  - arnoldVersion:
      type: string
      default: '7.2.5.3'
      description: 'The Arnold version to install.'

phases:
  - name: build
    steps:
      - name: DownloadArnold
        action: S3Download
        inputs:
          - source: s3://hypotheticalinc-image-sources/Arnold-{{arnoldVersion}}-linux.tgz
            destination: /opt/arnoldRoot/arnold-{{arnoldVersion}}.tgz

      - name: ExtractArnold
        action: ExecuteBash
        inputs:
          commands:
            - mkdir /opt/arnoldRoot/{{arnoldVersion}}
            - tar xvzf {{build.DownloadArnold.inputs[0].destination}} -C /opt/arnoldRoot/{{arnoldVersion}}
            - rm {{build.DownloadArnold.inputs[0].destination}}
            - echo 'export ARNOLD_ROOT=/opt/arnoldRoot/{{arnoldVersion}}' >> /etc/profile

  - name: validate
    steps:
      - name: ValidateGafferArnold
        action: ExecuteBash
        inputs:
          commands:
            - $GAFFER_ROOT/bin/gaffer env python -c "import GafferArnold;print(help(GafferArnold))"

The Arnold component is similar to the Gaffer Component. Instead of downloading from a publicly available Github release, we get the Arnold archive from our S3 upload. This method can be used for other private archives you may need to install, such as non-public Gaffer extensions.

The validation phase once again runs a Python script in Gaffer. Here we simply import GafferArnold and attempt to create an ArnoldLight to verify it is working.

Gaffer Prerequisites Component

name: Gaffer Prerequisites
description: Required components to run Gaffer.
schemaVersion: 1.0

phases:
  - name: build
    steps:
      - name: installPackages
        action: ExecuteBash
        inputs:
          commands:
            - yum install -y mesa-libGL mesa-libGLU

Image Pipeline

Choose Image Pipelines to see the list of Pipelines, which is empty if this is your first time using it.

1. Choose "Create Image Pipeline" to get started.
2. Enter a name for the pipeline and optionally a description.
3. Change the schedule option to "Manual" so it will only run when you trigger it.
4. Proceed to the next step, to configure the Recipe.