AWS Batch

What Problem Does AWS Batch Solve?

Batch computing is used for workloads that can be run automatically without end-user interaction, often for processing large volumes of data. Common examples include large-scale scientific simulations, financial risk analysis, or media transcoding.

Managing the infrastructure for these workloads is challenging:

• You need to provision servers, install software, and manage their lifecycle.
• Workloads are often spiky, leading to either underutilized (costly) or under-provisioned (slow) infrastructure.
• You need a mechanism to queue, schedule, and execute jobs in a fault-tolerant manner.

AWS Batch automates this entire process, allowing you to focus on your application logic instead of infrastructure management.

The Core Components of AWS Batch

AWS Batch is built on four fundamental components that you configure to create a complete batch computing environment.

1. Compute Environment

A Compute Environment is the set of compute resources that AWS Batch creates and manages to run your jobs.

• Managed Compute Environment (Most Common): You specify the desired configuration, and AWS Batch handles the rest. It will automatically launch, manage, and terminate compute resources based on the demands of your job queue. You can configure it to use:
  • EC2 On-Demand Instances.
  • EC2 Spot Instances for significant cost savings on fault-tolerant workloads.
  • AWS Fargate for a serverless compute option where you don't manage any EC2 instances at all.
  • Fargate Spot for cost-optimized serverless batch jobs.
• Unmanaged Compute Environment: You are responsible for creating and managing the EC2 instances in your AWS account. AWS Batch will use these pre-existing instances to run jobs but will not manage their lifecycle. This is for advanced use cases with very specific infrastructure requirements.

2. Job Queue

A Job Queue is where your submitted jobs reside until the scheduler can dispatch them to a Compute Environment.

• Priority: You can create multiple job queues with different priority levels. For example, a high-priority queue for urgent processing and a low-priority queue for background tasks.
• Mapping: Each job queue is mapped to one or more compute environments. The scheduler will attempt to place jobs from a queue into its associated compute environments based on priority.

3. Job Definition

A Job Definition is a template or blueprint for your jobs. It specifies how a job is to be run. It is similar to a Task Definition in Amazon ECS.

Key parameters in a Job Definition include:

• The Docker container image for the job.
• Resource requirements, such as vCPU and memory.
• An IAM role to grant the job permissions to access other AWS services (e.g., S3 buckets).
• Retry strategies and job timeouts.
• Environment variables and command-line parameters to pass to the container.

4. Job

A Job is the actual unit of work submitted to AWS Batch. When you submit a job, you reference a Job Definition and submit it to a specific Job Queue. You can also override parameters from the Job Definition at runtime. Jobs can be simple, single-container tasks or complex multi-node parallel jobs.

The AWS Batch Workflow

The components work together in a logical sequence:

1. Create a Compute Environment: You define the pool of resources you want Batch to use (e.g., a managed environment using Fargate Spot).
2. Create a Job Queue: You create a queue and associate it with the compute environment you just created.
3. Create a Job Definition: You create a template for the work you want to perform, specifying your Docker container and its requirements.
4. Submit a Job: You submit a job, pointing to your Job Definition and your Job Queue.
5. AWS Batch Takes Over:
   • The job is placed in the Job Queue.
   • The AWS Batch scheduler sees the job in the queue.
   • It evaluates the resource requirements defined in the Job Definition.
   • It provisions the necessary resources in the associated Compute Environment (e.g., launches a Fargate task). If the resources already exist, it uses them.
   • Once the compute resource is ready, the job is run (your container is executed).
   • After the job completes, Batch will terminate the compute resources if they are no longer needed, saving you costs.

Use Cases and Pricing

Common Use Cases

• Financial Services: Post-trade analysis, fraud detection, and risk modeling.
• Life Sciences: Drug screening, DNA sequencing, and bioinformatics research.
• Media & Entertainment: High-resolution rendering, transcoding, and media supply chain processing.
• Engineering: Deep learning, simulations, and complex scientific analysis.

Pricing Model

There is no additional charge for AWS Batch itself. You only pay for the underlying AWS resources that you consume to store and run your batch jobs. This includes:

• The cost of the EC2 instances or Fargate vCPU/memory used in your compute environments.
• Any associated costs for data storage (S3) or logging (CloudWatch).
  Sources