Architecture of object-based storage and S3 standard specifications

Object storage has been growing in popularity among data storage architectures. Compared to file systems and block storage, object storage faces no limitations when handling petabytes of data. By design, the limitless nature of object storage makes it fit for Big Data and Cloud contexts.

Moreover, object storage is simple and efficient. It offers easy data replication, scalability and is fit for “Write Once Read Many” contexts such as data analytics. Those characteristics combined with its ease of implementation and programmability all account to its widely spread usage.

What exactly is an object? How does object storage work, and what enables it to scale? We aim to clarify this.

Object storage is not exclusive to Cloud services such as AWS Simple Storage Service (S3), and multiple local object storage solutions exist as alternatives. Because AWS S3 sets a standard for object storage’s API interface, storage solutions and applications consuming from them are federated under “S3 compatibility”. Any S3-compatible app works with a large number of S3-compliant object storage solutions and vice-versa, which enhances both of their growth.

This article is the first of a series of three:

Object storage: how it works, why it scales

As the name suggests, object storage contains data in the form of objects. The core paradigm of object storage is to optimize common data and metadata operations while coupling the two together. What is an object made of?

It is the combination of a key (granting access), a value (actual data) and

Read more