Object Storage
StorageScalable blob storage for unstructured data like images, videos, documents, and backups with lifecycle policies and CDN integration.
Overview
Object Storage is a storage architecture that manages data as objects — each object contains the data itself, metadata, and a unique identifier. Unlike file systems (hierarchical) or block storage (fixed-size blocks), object storage provides a flat namespace with virtually unlimited scalability, making it ideal for storing unstructured data: images, videos, documents, backups, logs, and data lake content. In Vetora's simulator, the Object Storage component models upload/download latency, throughput, and the interaction between object storage and CDN edge caches.
The object storage access pattern is fundamentally different from databases. Objects are accessed by key (a unique identifier like a path) and are typically written once, read many times, and rarely updated in place. Instead of updating an object, you write a new version and optionally delete the old one. This immutable write pattern enables powerful optimizations: objects can be replicated across availability zones without conflict resolution, cached aggressively at CDN edges, and tiered to cheaper storage classes as they age.
Multipart upload is essential for large objects. Instead of uploading a 5GB video in a single HTTP request (which would fail on any network interruption), multipart upload splits the object into parts (5MB–5GB each), uploads them in parallel or sequentially, and assembles them server-side. If a part fails, only that part needs to be retried. This approach provides reliable uploads over unreliable networks and improves throughput through parallelism. Vetora models multipart upload behavior and shows how part size and parallelism affect total upload time.
Lifecycle policies automate data management across storage tiers. A typical policy might keep objects in standard storage for 30 days (fast, expensive), transition to infrequent access storage at 30 days (cheaper, retrieval fee), archive to glacier/cold storage at 90 days (cheapest, slow retrieval), and delete at 365 days. These policies reduce storage costs by 50–80% compared to keeping all data in standard storage. Vetora models storage costs across tiers to help you optimize your data lifecycle strategy.
Versioning provides a safety net against accidental deletion and data corruption. When versioning is enabled, every overwrite or delete creates a new version rather than destroying the original. You can restore any previous version at any time. Combined with Object Lock (write-once-read-many), versioning provides compliance-grade data protection for financial records, medical imaging, and regulatory archives. The cost trade-off is that versioned objects consume more storage, as all versions are retained until explicitly deleted.
When to Use
Recommended
- +Storing user-uploaded content — images, videos, documents, profile pictures — at any scale
- +Static website hosting and SPA (Single Page Application) asset hosting, often fronted by a CDN
- +Data lake storage for analytics — raw event data, log archives, ML training datasets
- +Database backups and disaster recovery — automated backup pipelines store snapshots in object storage
- +Content distribution — object storage as CDN origin for globally distributed static content
Not Recommended
- -Transactional data requiring frequent updates and queries — use a database for structured, queryable data
- -Low-latency random access — object storage has 50–200ms first-byte latency, unsuitable for real-time lookups
- -Small, frequently accessed metadata — in-memory caches or databases are more appropriate
- -File system semantics (directory listing, rename, append) — object storage has a flat namespace without true directories
Key Parameters in Vetora
Real-World Examples
Amazon S3
The pioneer of cloud object storage with 99.999999999% (11 nines) durability. Stores over 100 trillion objects. Features include S3 Select (query in-place), event notifications, and 8 storage classes for cost optimization.
Azure Blob Storage
Microsoft's object storage with hot, cool, and archive tiers. Integrates with Azure CDN, Azure Functions triggers, and Data Lake Storage Gen2 for analytics workloads.
Google Cloud Storage
Unified object storage with Standard, Nearline, Coldline, and Archive classes. Features include Autoclass (automatic tier optimization), strong consistency, and native integration with BigQuery for analytics.
Frequently Asked Questions
What is object storage and how is it different from file storage?
Object storage manages data as discrete objects with metadata and unique identifiers in a flat namespace. File storage organizes data in a hierarchical directory structure with file paths. Object storage scales virtually infinitely (trillions of objects), provides 11 nines of durability, and supports HTTP-based access (REST APIs). File storage supports POSIX semantics (rename, append, directory listing) but has scalability limits. Use object storage for unstructured data at scale (images, videos, backups); use file storage when applications require file system semantics.
How does multipart upload work in object storage?
Multipart upload splits a large object into smaller parts (typically 5MB–100MB each), uploads them independently (in parallel or sequentially), and assembles them server-side into a single object. If a part upload fails, only that part needs to be retried — not the entire object. This provides reliable uploads over unreliable networks, improves throughput through parallel uploads, and is required for objects larger than 5GB in most providers. S3 supports up to 10,000 parts per upload.
What are storage lifecycle policies and how do they reduce costs?
Lifecycle policies automatically transition objects between storage classes based on age or access patterns. A typical policy: keep in Standard (fast, $0.023/GB) for 30 days, transition to Infrequent Access ($0.0125/GB) at 30 days, Archive ($0.004/GB) at 90 days, and delete at 365 days. This reduces storage costs by 50–80% compared to keeping all data in Standard. Some providers offer Intelligent Tiering that automatically moves objects between tiers based on actual access patterns without manual policy configuration.
How do you integrate object storage with a CDN?
Object storage serves as the CDN's origin — the source of truth for content. When a CDN edge node receives a request and does not have the content cached, it fetches from the object storage origin, caches it, and serves subsequent requests from the edge. Configuration involves creating a CDN distribution pointing to the storage bucket, setting cache headers (Cache-Control, Expires) on objects, and optionally using origin access identity to restrict direct bucket access. This pattern reduces storage egress costs and provides global low-latency content delivery.
What is object versioning and when should you enable it?
Object versioning retains all versions of an object when it is overwritten or deleted. Instead of replacing the original, each write creates a new version while preserving previous versions. Enable versioning when you need protection against accidental deletion, audit trails for compliance (financial records, medical data), or the ability to restore previous states. Combine with lifecycle policies to automatically delete old versions after a retention period. The cost trade-off is that all versions consume storage — a frequently updated object can accumulate significant version history.
Related Components
Content Delivery Network that caches and serves content from edge locations close to users, reducing...
Persistent data store supporting SQL or NoSQL models with ACID transactions, replication, sharding, ...
Application server or microservice that processes requests, runs business logic, and communicates wi...
Durable message streaming platform for pub/sub, event sourcing, and asynchronous communication betwe...
Try Object Storage in the Simulator
Build architectures with Object Storage and 13 other component types. Run discrete event simulations and get AI-powered feedback.
Open Playground