Amazon Web Service (AWS) S3 - Simple Storage Service - Object Storage

Giriş

Object Storage

Açıklaması şöyle

S3 Is Object Storage. It’s Not a Filesystem.

A lot of systems today were originally built on local disks or EBS volumes. Then someone comes along and says, “Let’s just move it to S3.” Sounds simple. In reality? You’re in for a rewrite.

S3 isn’t a drop-in replacement for disk. It doesn’t support seek, append, or rename. It’s immutable, eventually consistent, and has high latency on every read and write. If your system relies on random access, sync writes, or file-level atomicity — S3 will break it.

Bucket

Dosya ve dizinlere bucket deniliyor. 

Çok büyük binary veri saklayabilir. Veriye key ile erişiriz. Şeklen şöyle

MinIO

MinIO S3 API'si ile uyumlu ancak veriyi Amazon'da değil kendi sunucumuzda saklayabileceğimiz bir alternatif. 

Java istemci uygulaması burada. MinIO API yazısına bakabilirsiniz.

S3 Ne Zaman Kullanılır

Açıklaması şöyle

When to use S3

- When you need to store large binary objects/files (up to 5TB each)

- When the amount of data you need to store is large (>10TB), continues to grow daily, and may need to be retrieved (can’t be deleted)

S3 Advantages

- Supports very high throughput
- Infinite scalability — No limit on amount of storage

S3 Disadvantages

- No Query support, only key-based retrieval
- Latency is 100–200 ms for small objects. Caching can alleviate this

S3 nesne değiştikçe notification/bildirim gönderir. Açıklaması şöyle

At a high level, S3 event notification is enabled. As new objects are uploaded to the bucket, S3 posts JSON messages to SNS topics. SNS acts as the event router. AWS Lambda is subscribed to these topics. It consumes the incoming messages, custom processes the artifacts and uploads them to other regions.

Bucket Policy 

1. AWS S3 console'a gidilir. 

2. Bucket seçilir

3. Permissions sekmesine gidilir ve bucket_policy.json dosyası düzenlenir

Örnek - Public Read Access

Şöyle yaparız

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::your-unique-bucket-name/*"
        }
    ]
}

Örnek

Şöyle yaparız

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PublicReadGetObject",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::testbucket/*"
    }
  ]
}

Cross-Region Replication (CRR)

1. Create an IAM role that grants the necessary permissions for replication

Açıklaması şöyle

Execute the following command to create an IAM role named CrossRegionReplicationRole with a trust policy specified in the crr-trust-policy.json file:

Şöyle yaparız

aws iam create-role 
  --role-name CrossRegionReplicationRole 
  --assume-role-policy-document file://crr-trust-policy.json

crr-trust-policy.json dosyası şöyle

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

2. Granting the required permissions for cross-region replication

crr-policy.json dosyası şöyle

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetReplicationConfiguration",
                "s3:ListBucket",
                "s3:GetObjectVersion",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::your-source-bucket",
                "arn:aws:s3:::your-source-bucket/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ReplicateObject",
                "s3:ReplicateDelete"
            ],
            "Resource": "arn:aws:s3:::your-destination-bucket/*"
        }
    ]
}

3. Attach the IAM role to your source bucket

Şöyle yaparız

aws s3api put-bucket-replication 
  --bucket your-source-bucket 
  --replication-configuration file://crr-configuration.json

crr-configuration.json dosyası şöyle

{
  "Role": "arn:aws:iam::123456789012:role/CrossRegionReplicationRole",
  "Rules": [
    {
      "ID": "Rule1",
      "Prefix": "",
      "Status": "Enabled",
      "SourceSelectionCriteria": {
        "SseKmsEncryptedObjects": {
          "Status": "Enabled"
        }
      },
      "Destination": {
        "Bucket": "arn:aws:s3:::your-destination-bucket",
        "StorageClass": "STANDARD_IA",
        "BucketAccountId": "123456789012"
      }
    }
  ]
}

Maven

Şu satırı dahil ederiz

<dependency>
  <groupId>com.amazonaws</groupId>
  <artifactId>aws-java-sdk</artifactId>
  <version>1.12.83</version>
</dependency>

Localstack

Localstack DockerCompose yazısına taşıdım

AWSCredentials Sınıfı

Örnek

Şöyle yaparız

import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.BasicAWSCredentials;

public AWSCredentials credentials() {
  AWSCredentials credentials = new BasicAWSCredentials(
    "accesskey",
    "secretkey"
  );
  return credentials;
}