Reading time: 8 minutes

Amazon EMR Serverless provides a managed, serverless platform for big data processing. In this article, we’ll look at the benefits of this service over regular EMR clusters and Glue jobs, why you would need to run an EMR Serverless job locally, and finally, how to do it.

You will find a working demo of this article in this github repository.

When you want to run Spark jobs in AWS using an at least somewhat managed service, you don’t have much choice.

Historically, we start with EMR, the grandfather. But you must manage the underlying infrastructure yourself. Environment management is difficult to industrialize (bootstrap, Python versions, ad hoc dependencies for jobs, etc.), and log management is an overly complicated mess (but it’s still one of my questions). favorite technical interviews to ask on this subject).

If you don’t want to keep EC2s for your use case and want to go serverless, the Go-To solution is Glue work. Log management is much simpler than EMR, which is a big advantage. But you quickly realize that as your data context evolves, Glue work can be costly (because you can’t scale memory and vCPUs separately), and the environment management is better, but still clunky.

For example, you can’t install the C compiled library on your system and package them in a zip to include them in your task, like you can for Lambda functions. You must specify them using the –additional-python-modules parameter (which is nice (sort of), but makes it hard to keep your dependencies consistent across your work).

Serverless EMR has the advantage of being completely serverless (who could have guessed that from the name?) while being more granular than Glue jobs for scaling resources. It also has a huge advantage: you can dockize your work. This is a huge game-changer because you can now consistently create your task execution environment while still being confident that you have precise control over what is installed and what version. I could also mention the interface of the EMR Serverless Studiowhich is an absolute pleasure to use, whether in the construction phase or in the execution phase.

When you create Spark jobs to deploy in a Cloud environment, especially serverless, you often realize that if it’s not done well, it can be a painful process (this phrase also holds true for non-Spark applications, however ).

When you create your code, you have two choices for testing it.

You can test it locally in a Jupyter Notebook (even using Spark, as pointed out in a previous article of ours). However, you have no guarantee that your local compute environment will match that of AWS, so you run the risk of starting over due to a dependency mismatch, for example.

You can test it remotely by deploying the code in AWS and run the job with the code to be tested. This has the advantage of resolving the risk of dependency mismatches since you test your code directly in an environment identical to the production one. But then the problem of cold start arises (which is, for EMR Serverless, non-negligible, 3 or 4 minutes at least to start the application and launch a job) and, above all, the problem of cost. Indeed, leaving an EMR Serverless application running continuously while you iterate on your code can bring unpleasant surprises at the end of the month to your FinOps team (and if you don’t have one, a wicked surprise to your financial director at the end of the year).

Running locally with Docker to the rescue:

• Consistent environment: you have strong guarantees that your local development environment matches your AWS compute environment.
• Iterate faster: Test configurations, code, and dependencies without deploying to AWS and without having to wait for EMR Serverless to cold start.
• Reduce costs: Avoid the costs of deploying incomplete jobs to the cloud and iterating using often expensive AWS resources.

It is important to note that the benefit of running your Pyspark code locally is not to test your processing work on a large scalebecause you will likely lack sufficient resources to do this (and if not, you may not need to use Spark for this task). The goal is to test your code on a subset of data, and once you’ve made sure it’s functional, you can deploy and test it at scale in your AWS environment.

Prerequisites

1. Docker installed: Download and install Docker from docker.com (tested with 27.4.0).
2. DME Serverless Application: A configured EMR Serverless application, ready to deploy with the appropriate IAM role attached (tested with EMR 7.1.0).

Step 1: Define the application code

For this example, let’s assume you’re using a PySpark job to process a dataset. Save the following Python script as main.py:

from pyspark.sql import SparkSession
from pyspark.sql.types import *
# test custom dependency with c compiled library import
import pandas


def main():
	spark_session = (
		SparkSession.builder.enableHiveSupport()
        	.config("spark.sql.catalogImplementation", "hive") # setup hive catalog implementation to work with AWS Glue data catalog
        	.getOrCreate()
	)
	# create dummy Spark dataframe
	data = [("Alice", 34), ("Bob", 45), ("Cathy", 29)]
	columns = ["Name", "Age"]
	df = spark_session.createDataFrame(data, columns)
	database_name = "test_database"
	spark_session.sql(f"CREATE DATABASE IF NOT EXISTS {database_name}")
	s3_bucket_name = "CHANGE_BUCKET_NAME"
	# write dataframe in AWS
	df.write.mode("overwrite").option("path", f"s3a://{s3_bucket_name}/{database_name}/my_table/").saveAsTable(f"{database_name}.my_table")
	# try to request dataframe
	spark_session.sql(f"SELECT * FROM {database_name}.my_table").show()


# setup script entrypoint
if __name__ == "__main__":
	main()

Step 2: Create a Custom Dockerfile

As described in the AWS documentation, we use the AWS EMR serverless Docker image as the basis for our custom image. This Dockerfile should not be considered optimized; it is only used as a showcase example.

# syntax=docker/dockerfile:1
FROM --platform=linux/amd64 public.ecr.aws/emr-serverless/spark/emr-7.1.0:20240823

# 

USER root
# MODIFICATIONS GO HERE

# install python 3.10 as the default python version in the base image is a bit outdated
RUN yum install -y gcc openssl-devel bzip2-devel sqlite-devel libffi-devel tar gzip wget make zlib-static && \
	yum clean all && \
	wget  && \
	tar xzf Python-3.10.15.tgz && cd Python-3.10.15 && \
	./configure --enable-optimizations --enable-loadable-sqlite-extensions && \
	make altinstall && \
	ln -sf /usr/local/bin/python3.10 /usr/bin/python3 && \
	ln -sf /usr/bin/pip3 /usr/bin/pip

# setting custom work path in the system
# If you don't do that you will have a Java AccessDeniedException when trying to copy the iceberg jar file to the /usr/app/src directory in the spark executors instances
ENV HOME="/usr/app/src"
RUN mkdir -p $HOME && chown -R hadoop:hadoop $HOME
WORKDIR $HOME

ENV PYTHONPATH="${PYTHONPATH}:$HOME"

# setup jupyter notebook for local execution
RUN pip install jupyter notebook
# the following line is to setup files required by the source docker image which are mounted directly by EMR Serverless service when executing in AWS. We create empty file because we want to be able to run this image locally, and the scripts do not really need to do anything in this context
RUN mkdir -p /var/loggingConfiguration/spark/ && touch /var/loggingConfiguration/spark/run-fluentd-spark.sh && touch /var/loggingConfiguration/spark/run-adot-collector.sh

# install custom dependencies with c compiled library
RUN pip install pandas

# end of modifications
# EMRS will run the image as hadoop

USER hadoop:hadoop

Step 3: Create and run the Docker image

1. Build the image: Run the following command to create the Docker image:

docker buildx build . -t emr_serverless_local_image:1 --provenance=false

2. Run the container: Run the container to test your PySpark task

mkdir logs # create directory which will contain Spark logs
# AWS Authentication using profiles configured in your credentials do not work in the EMR Serverless image, you can only use environment variables to connect to AWS
export CREDENTIALS=$(aws configure export-credentials)
docker run \
    -e AWS_ACCESS_KEY_ID=$(echo $CREDENTIALS | jq -r '.Credentials.AccessKeyId') \
    -e AWS_SECRET_ACCESS_KEY=$(echo $CREDENTIALS | jq -r '.Credentials.SecretAccessKey') \
    -e AWS_SESSION_TOKEN=$(echo $CREDENTIALS | jq -r '.Credentials.SessionToken // ""') \
    -e AWS_REGION="eu-west-1" \
    -e AWS_DEFAULT_REGION="eu-west-1" \
    --mount type=bind,source=$(pwd)/logs,target=/var/log/spark/user/ \
    -v $(pwd)/main.py:/usr/app/src/main.py:rw \
    -p 8888:8888 \
    -e PYSPARK_DRIVER_PYTHON=jupyter \
    -e PYSPARK_DRIVER_PYTHON_OPTS='notebook --ip="0.0.0.0" --no-browser' \
    emr_serverless_local_image:1 \
    pyspark --master local \
    --conf spark.hadoop.fs.s3a.endpoint=s3.eu-west-1.amazonaws.com \
    --conf spark.hadoop.hive.metastore.client.factory.class=com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory

You should see the debug printout in your terminal and the process should hang (if it doesn’t and terminates, you’ll find the error, or at least more context, in the $HOME/logs file /stderr).

Now look at the $HOME/logs/stderr file and you will find the URL and token allowing you to connect to the Jupyter notebook instance running in your Docker container (you are looking for something like this “http://127.0. 0.1: 8888/tree?token=[TOKEN]”). Copy and paste this link into your browser and create a new notebook.

3. Test your code: Test your main.py directly from the Jupyter notebook instance

Note that in the docker run command, I added a main.py file to the $HOME directory of your Docker container. It allows you to modify your main.py code inside or outside of your Docker container while keeping both versions in sync, allowing you to iterate without having to rebuild and rerun your Docker container with each change of code.

You can now create a cell and run your main function from the main.py file:

from main import main
main()

Now you can edit your code inside or outside your Docker container and iterate locally, which will be faster and cheaper than testing in AWS every time.

Step 4: Deploy your Docker image to ECR and link it to EMR Serverless

Once you have validated your code, you can push the Docker image to your ECR.

# login to the ECR
aws ecr get-login-password --region ${aws_region_name}| docker login -u AWS ${aws_account_id}.dkr.ecr.${aws_region_name}.amazonaws.com --password-stdin
# tag the Docker image correctly
docker image tag emr_serverless_local_image:1 ${aws_account_id}.dkr.ecr.${aws_region_name}.amazonaws.com/${ecr_name}:${image_tag}
# push the Docker image to ECR
docker push ${aws_account_id}.dkr.ecr.${aws_region_name}.amazonaws.com/${ecr_name}:${image_tag}

You can then configure your EMR Serverless application to use your Docker image stored in your ECR (see this documentation for more detailed instructions on this part). Don’t forget to check that the EMR version of your EMR Serverless application matches that of the base Docker image used in your Dockerfile (in my example 7.1.0) and that the architecture too (here x84_64). Also set a resource policy on your ECR to allow EMR Serverless to pull the image.

Next, you need to upload your main.py file to an S3 bucket:

aws s3 cp main.py s3://${aws_s3_bucket_name}/main.py

Finally, submit a new job to your EMR Serverless application (see this documentation for more precise instructions on this part), don’t forget to specify the s3 key of your main.py file.

Testing EMR Serverless jobs locally with Docker provides a reliable and cost-effective way to develop and debug your applications. By replicating the environment locally, you can ensure smoother deployments and faster iterations, saving time and resources. With the steps outlined, you are ready to seamlessly bridge local development and execution in the cloud.

If you want to dig deeper into the code, you’ll find a working demo in this github repository.