Enhancing Data Integrity in Amazon S3 with Additional Checksums

In the security world, cryptography uses something called “hashing” to confirm that a file is unchanged. Usually, when a file is hashed, the hash result is published. Next, when a user downloads the file and applies the same hash method, the hash results, or checksums (a string of output that is a set size) are compared. This means if indeed the checksum of the downloaded file and the original file are the same, the two files are identical, confirming that there have been no unexpected changes — for example, file corruption, man-in-the-middle (MITM) attacks, etc. Since hashing is a one-way process, the hashed result cannot be reversed to expose the original data.

Verify the integrity of an object uploaded to Amazon S3

We can use Amazon S3 features to upload an object with the checksum flag “On” with the checksum algorithm that is used to validate the data during upload (or download) — in this example, as SHA-256. Optionally, you may also specify the checksum value of the object. When Amazon S3 receives an object, it calculates the checksum by leveraging the algorithm that you specified. Now, if the two checksum values do not match, Amazon S3 will generate an error.

Types of Additional Checksums

Various checksum algorithms can be used for verifying data integrity. Some common ones include:

MD5: A widely used algorithm, but less secure against collision attacks.

SHA-256: Provides a higher level of security and is more resistant to collisions.

CRC32: A cyclic redundancy check that is fast but not suitable for cryptographic purposes.

Implementing Additional Checksums

Sign in to the Amazon S3 console. From the AWS console services search bar, enter S3. Under the services search results section, select S3.

Choose Buckets from the Amazon S3 menu on the left and then choose the Create Bucket button.

Enter a descriptive globally unique name for your bucket. The default Block Public Access setting is appropriate, so leave this section as is.

You can leave the remaining options as defaults, navigate to the bottom of the page, and choose Create Bucket.

Our bucket has been successfully created.

Upload a file and specify the checksum algorithm

Navigate to the S3 console and select the Buckets menu option. From the list of available buckets, select the bucket name of the bucket you just created.

Next, select the Objects tab. Then, from within the Objects section, choose the Upload button.

Choose the Add Files button and then select the file you would like to upload from your file browser.

Navigate down the page to find the Properties section. Then, select Properties and expand the section.

Under Additional checksums select the on option and choose SHA-256.

If your object is less than 16 MB and you have already calculated the SHA-256 checksum (base64 encoded), you can provide it in the Precalculated value input box. To use this functionality for objects larger than 16 MB, you can use the CLI or SDK. When Amazon S3 receives the object, it calculates the checksum by using the algorithm specified. If the checksum values do not match, Amazon S3 generates an error and rejects the upload, but this is optional.

Navigate down the page and choose the Upload button.

After your upload completes, choose the Close button.

Checksum Verification

Select the uploaded file by selecting the filename. This will take you to the Properties page.

Locate the checksum value: Navigate down the properties page and you will find the Additional checksums section.

This section displays the base64 encoded checksum that Amazon S3 calculated and verified at the time of upload.

Compare

To compare the object in your local computer, open a terminal window and navigate to where your file is.

Use a utility like Shasum to calculate the file. The following command performs a sha256 calculation on the same file and converts the hex output to base64: shasum -a 256 image.jpg | cut -f1 -d\ | xxd -r -p | base64

When comparing this value, it should match the value in the Amazon S3 console.

Run this code by replacing it with your image.

Congratulations! You have learned how to upload a file to Amazon S3, calculate additional checksums, and compare the checksum on Amazon S3 and your local file to verify data integrity.

This brings us to the end of this blog, thanks for reading, and stay tuned for more.

If you have any questions concerning this article or have an AWS project that requires our assistance, please reach out to us by leaving a comment below or email us at sales@accendnetworks.com.

Thank you!

Understanding AWS Security Groups and Bootstrap Scripts

Understanding AWS Security Groups and Bootstrap Scripts: Enhancing Cloud Security and Automation.

In the realm of AWS, achieving a balance between robust security and streamlined automation is essential for efficient cloud management. AWS Security Groups and Bootstrap Scripts play pivotal roles in this Endeavor. In this article, we’ll delve into these two AWS components and provide a hands-on demo to illustrate how to leverage them effectively.

AWS Security Groups:

What are AWS Security Groups?

AWS Security Groups are a fundamental element of AWS’s network security model. They act as virtual firewalls for your instances to control inbound and outbound traffic. Security Groups are associated with AWS resources like EC2 instances and RDS databases and allow you to define inbound and outbound rules that control traffic to and from these resources.

Key Features of AWS Security Groups:

Stateful: Security Groups are stateful, meaning if you allow inbound traffic from a specific IP address, the corresponding outbound traffic is automatically allowed. This simplifies rule management.

Default Deny: By default, Security Groups deny all inbound traffic. You must explicitly define rules to allow traffic.

Dynamic Updates: You can modify Security Group rules anytime to adapt to changing security requirements.

Use Cases:

Web Servers: Security Groups are used to permit HTTP (port 80) and HTTPS (port 443) traffic for web servers while denying other unwanted traffic.

Database Servers: For database servers, Security Groups can be configured to only allow connections from known application servers while blocking access from the public internet.

Bastion Hosts: In a secure architecture, a bastion host’s Security Group can be set up to allow SSH (port 22) access only for specific administrators.

Demo: Creation of security group to open port 22 for SSH, port 80 for HTTP, and port 443 for HHTPS.

To create security group, click on this link and follow our previous demo on the security group.

https://accendnetworks.com/comprehensive-guide-to-creating-and-managing-security-groups-for-your-amazon-ec2-instances/

Bootstrap Scripts

What are Bootstrap Scripts?

A bootstrap script, often referred to as user data or initialization script, is a piece of code or script that is executed when an EC2 instance is launched for the first time. This script automates the setup and configuration of the instance, making it ready for use. Bootstrap scripts are highly customizable and allow you to install software, configure settings, and perform various tasks during instance initialization.

Key Features of Bootstrap Scripts:

Automation: Bootstrap scripts automate the instance setup and configuration process, reducing manual intervention and potential errors.

Flexibility: You have full control over the contents and execution of the script, making it adaptable to your specific use case.

Idempotent: Bootstrap scripts can be designed to be idempotent, meaning they can be run multiple times without causing adverse effects.

Use Cases:

Software Installation: You can use bootstrap scripts to install and configure specific software packages on an instance.

Configuration: Configure instance settings, such as setting environment variables or customizing application parameters.

Automated Tasks: Run scripts for backups, log rotation, and other routine maintenance tasks.

Combining Security Groups and Bootstrap Scripts

The synergy between Security Groups and Bootstrap Scripts offers a robust approach to enhancing both security and automation in your AWS environment.

Security Controls: Security Groups ensure that only authorized traffic is allowed to and from your EC2 instances. Bootstrap scripts can automate the process of ensuring that your instances are configured securely from the moment they launch.

Dynamic Updates: In response to changing security needs, Bootstrap Scripts can automatically update instance configurations.

Demo: Bootstrapping AWS EC2 Instance to update packages, install and start Apache HTTP server. (HTTP is on port 80)

In the EC2 dashboard, select instances then click launch

In the launch instance dashboard, under name and tags, give your instance a name. call it bootstrap-demo-server.

Under application and OS images, select the QuickStart tab then select Amazon Linux. Under Amazon Machine Image (AMI), select the drop-down button and select Amazon Linux 2 AMI. Scroll down.

Under instance type make sure it is t2. Micro because it is the free-tier one. Under keypair login select the dropdown and select your key-pair. Scroll down.

We will leave all the other options as default move all the way to the advanced details section, then click the drop-down button to expand it.

Move all the way down to the instance user data section. Copy and paste this code inside there.

#!/bin/bash
yum update -y
yum install httpd -y
systemctl start httpd
systemctl enable httpd

Tip: Once the instance is launched and you may have to go back and modify this User Data section, stop the instance, click Actions-Instance settings-Edit user data.

Under instance, summary, review, and click launch instance.

Click on the instance ID then wait for the status check to initialize. Copy the public IPv4 DNS and paste it into your browser.

We are getting this error because HTTP port 80 is not open, we will go back to our instance modify security group, and open port 80.

Select your instance and click the instance state dropdown move to security then click modify security group.

Under the associated security group, select the search box and look for the web traffic security group which opens port 80 and 443 select it, then click save.

Now come back to your instance copy its public DNS and paste it into your browser.

Congratulations, we can now access our HTTP web traffic on port 80. This clearly shows how security groups can allow or deny internet traffic.

Again, remember this instance was bootstrapped and installed Apache on launch.

Pull everything down and stay tuned for more.

If you have any questions concerning this article or have an AWS project that requires our assistance, please reach out to us by leaving a comment below or email us at sales@accendnetworks.com

Thank you!