Using Amazon S3 to Optimize Amazon EC2 Storage Workload

Amazon Web Services (AWS) is the leading Cloud provider in the world right now. From a research conducted in 2020, it was found out that AWS controls 32% of the cloud market. This is so because AWS offers a vast range of Cloud services such as Compute, Storage, Database, Networking, Hosting, AI and Analytics, Management, Internet of Things(IoT) etc. and also the level of trust AWS has been able to gain from customers worldwide. In 2020, AWS had a total of 175 services under these different specializations in the Tech field.

Amongst these numerous services, two popular ways by which customers use the cloud are for Hosting applications and also for Storage purposes. Hosting on AWS is widely used by a lot of customers because of its flexibility using the pay-as-you go model of payment where you are charged for only the time your service runs and also the different ways of deploying instances (i.e. Servers) where they can make use of spot instances and reserved instances at very great discounts. These features of AWS make it preferred over the traditional ways of Hosting.

Now, assuming after getting to know some of the benefits and advantages that AWS cloud could give to your business or enterprise such as Speed to market, Global reach, High Availability, High Durability and many more, you’ve decided to give it a go and host your application on the cloud. For your application be it an Ecommerce web-app, enterprise application or even a mobile application; to function well or give the best results, it is going to run on data and also going to produce data which needs to be stored. These data could be customer data, inventory data, files etc. And as usual, AWS has got you covered with some storage services like Elastic Block Store(EBS), Elastic File System(EFS) and Simple Storage Service(S3).

Virtual Servers provisioned in AWS are called EC2 Instances. EC2 stands for Elastic Compute Cloud meaning the instances can be scaled up or down depending on the demand. For every instance that you provision in the cloud, by default, you are given the option to provision to an EBS Storage Volume for that instance and this has been the common practice for many users of AWS cloud. But using this type of storage for your applications for the long term might cause some problems to spring up because of some limitations of EBS. EBS volumes have a maximum storage capacity which means as your application grows and more files are produced and need to be stored, you might find yourself in a sticky wicket. The good news is that, there is a way out and that way out is Amazon Simple Storage Service

(S3). S3 provides unlimited object storage in the cloud, you can store virtually any kind of data in any format and it is relatively cheaper than EBS volumes. Let’s take a deeper look at some of the advantages of using S3 as the storage for your application instead of EBS volumes.

• With Storage size and limitations, S3 has no limit on the number of objects but EBS volumes have a maximum size of 16 TB.
• Easy to manage security features: S3 is a fully managed service so users can store data and protect it from unauthorized access using its bucket policies and encryption tools already provided. Also S3 could be used in conjunction with Amazon Macie to reject any irregular access requests to your sensitive data. So as the name suggests, S3 is simple to use and many of the admin configuration tasks are handled for you. Lastly, S3 provides 11 nines of durability, meaning it is 99.999999999% durable. So your data is never lost or destroyed.
• Cost-efficient storage classes: S3 has different storage classes that has reduced costs. Some of these classes are S3 Standard, S3 standard-infrequent Access (S3 standard IA) and S3 Glacier with S3 Glacier giving you’re the lowest storage class in AWS. Using S3 lifecycle policies, you can move your data across the various S3 classes to reduce costs of infrequently accessed data. Also, S3 intelligent-tiering helps the storage of data that has changing or unknown access patterns by storing data across 2 access tiers, which reduces your cost of storage.
• Reliable disaster recovery and accessibility: S3 Cross-Region Replication make it easy to protect critical application data. It offers agile recovery from disasters or outages whether they are caused by system failures, natural disasters, or human error. Also, S3 object data can be made to be accessible from anywhere in the world; data can be accessed over internet from anywhere using console, CLI, REST or SOAP APIs. depending on the permissions preferred by the user but with EBS, it can be accessed only by EC2 instance.

S3 is an isolated service that stands alone unlike the EBS volumes that comes as default storage must be accessed through an EC2 instance. So then how do you then store your EC2 files on S3 if they are not dependent on each other in anyway? One way of doing that is through the AWS CLI.

Below is a demo of how you can make an EC2 instance a web server and then connect it to S3 to store files from your application hosted on the web server.

First of all, there are two ways of provisioning instances in AWS.

• Using the AWS Management Console: This is considered the manual way of provisioning resources. Using the management console, you would need an aws account. Once you have that, you can log in and provision any AWS service of your choice. It is easy to use for the layman since there is no programming or special expertise needed to navigate your way around it. It offers one interface that you can use for adding and removing services, for seeing a map of all IT infrastructure in the cloud, for reviewing the costs for all services in one place. To access the management console, click here.
• Using AWS Cloudformation: This is considered the programmatic way of provisioning resources. AWS Cloudformation presents a way to model and provision resources with their preferred specifications using code, either in JSON or YAML format. The JSON or YAML document created when using this method is called the Cloudformation Template and the resources specified in it is called the stack. One benefit of using this method is reusability. If you are to provision resources with the same specifications, you can use a single template many times to do that. So when you have a task that requires you to provision a fleet of resources, AWS Cloudformation is the advisable route to use. This way of provisioning resources is called Infrastructure as a Code. Cloudformation can be accessed from the AWS Management Console.

Now let’s get our hands dirty and set up our Instance (virtual Web server).

• Log into the management console, locate EC2 and create an EC2 instance.
• Or you could use the following Cloudformation template to create your instance: click here to download the Cloudformation template.

NB: Attach a role that gives your EC2 instance full access to s3 buckets in the creation process.

After the Instance has been created, locate S3 and create an S3 bucket.

For this task, we already have our application created and the code has been deposited on GitHub.

This is the architectural overview of what we are trying to do.

• To get the SSH link to your instance, click on your instance and Click on the Connect button, there you will find the link to your instance.

 

• Go to your Local computer’s Git bash command line and SSH into the EC2 instance using the link. NB: You will have to install Git to get the Git Bash Command Line.

 

Now to set up a web server environment for our application, we need to do the following:

• Install Apache2 web server with the command: Sudo apt-get install apache2
• Install Git. Since we are going to be getting the code of the application from GitHub, we need to download git unto the instance. Install with command: Sudo apt-get install git
• For my application, I need php installed in order for it to run and I’m going to do that with command: Sudo apt-get install php
• Lastly, I also need to install MySQL for my application: Sudo apt-get install mysql.
• Now change directory to folder “html” (this is where apache stores project files to be hosted). Use the following command: cd /var/www/html.
• Install your application from GitHub unto the instance using command: git clone https://github.com/ <path to app repository on GitHub>.

To be able to access your Bucket through your EC2 instance, you will need to install the AWS CLI.

The AWS CLI comes in a zip file so you will need to install the “unzip” tool to be able to unzip the

CLI when downloaded.

• To download the unzip tool, use command: sudo apt install unzip
• Now follow the commands below sequentially to download, unzip and install the AWS CLI:

1. curl “https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip” -o “awscliv2.zip“
2. unzip awscliv2.zip
3. sudo ./aws/install

Now to test if your EC2 instance can connect to your S3 bucket, upload some files into your S3 bucket.

• Type command: aws s3 ls s3://<NAME OF YOUR S3 BUCKET>.

This should list all the objects in your bucket. This means a connection has been established.

And that is a simple demo about how you can make S3 buckets accessible to your EC2 instance.

In conclusion, choosing the type of storage service solely depends on the type of Application you want to be hosting. If you want a storage service that can store and retrieve bulk data, archive data, cheap and fully managed by AWS, meaning you get to concentrate on optimizing your application rather than focusing on configuration and encryption tasks; Amazon Simple Storage Service (S3) should be your preferred choice.