Photo from Pexels
Originally Posted On: https://medium.com/codex/strange-things-i-found-about-amazon-s3-9232891e89cd
The surprising truths lurking in the cloud.
When I first started working with Amazon S3, I thought it would be straightforward — just cloud storage, right? Upload files, download files, and organize them in folders. How hard could it be?
Well, S3 had other plans. Here are four strange things I discovered that completely changed how I think about object storage.
1. It’s Not Actually a File System
This was the first thing that threw me off. S3 looks like a file system. The console shows folders. You can navigate through what appears to be a directory structure. However, here’s the truth bomb: there are no folders in S3.
S3 is an object store, not a file system. What you see documents/reports/2024/annual.pdf is actually just a single object with a key named documents/reports/2024/annual.pdf. The slashes? They’re just characters in the key name. The “folders” you see in the console are basically an illusion—a helpful UI convention that groups objects by common prefixes.
This distinction may seem academic until you begin operating at scale. Listing all objects under a “folder”? That’s a prefix search. Moving a “folder”? You’re actually copying and deleting every single object individually. There’s no metadata saying “this is a folder containing these files.”
Once you understand this, a lot of S3’s behavior starts making sense.
2. You Can’t Rename Folders
Speaking of folders not being real: you can’t rename them.
I spent an embarrassing amount of time looking for a “rename folder” option in the S3 console.
Spoiler alert: it’s not there. And now we know why — you can’t rename something that doesn’t exist!
Want to move everything from old-name/ new-name/? You have to:
- Copy every object to the new prefix
- Delete every object from the old prefix
For a folder with thousands of objects, this isn’t just tedious — it’s time-consuming and costs money (you pay for both the PUT and DELETE operations). There are no shortcuts, no atomic rename operations. Each object is its own independent entity that needs to be handled individually.
Tools like AWS CLI or SDK can automate this process, but the underlying reality remains: S3 doesn’t do folder operations because it doesn’t have folders.
3. The Download Multiple Files Mystery Error
Here’s a fun one that had me scratching my head. I tried downloading multiple small files from the S3 console — maybe 5–10 files, each under a few MB. Should be easy, right?
Wrong. I encountered a generic error:
Object exceeds 5 TB limit. Use AWS CLI or SDK to download objects larger than 5 TB.
But it doesn’t make sense because my files were tiny.
Press enter or click to view image in full size
The S3 console has a hard limit on multi-file downloads that seems arbitrary and isn’t clearly documented. Even when your files are well under any reasonable size threshold, individually or combined, the console still refuses. The error message is vague and unhelpful, leaving you wondering whether it’s a browser limitation, a network issue, or simply S3 being unstable.
The workaround? Use the AWS CLI or SDK. Suddenly, those duplicate files download without any issues. Which brings us to…
4. The Console is Just the Tip of the Iceberg
The S3 console is suitable for basic operations, such as uploading a few files, checking if an object exists, and adjusting bucket policies. But it’s essentially a simplified interface sitting on top of the real power: the AWS CLI and SDKs.
Want to sync an entire directory? Use aws s3 sync.
Need to do batch operations efficiently? SDK with concurrent requests.
Can you actually rename that folder? CLI with a script.
Need to download multiple files? CLI doesn’t even break a sweat.
The console abstracts away S3’s true nature as an API-driven object store. Once you start using the CLI or SDKs, you realize the console was training wheels you didn’t know you were wearing. You can do things like:
- Set up lifecycle policies programmatically
- Handle massive parallel uploads/downloads
- Implement retry logic for failed operations
- Automate complex workflows
The CLI and SDKs expose S3 for what it really is: a robust, scalable API for object storage that happens to have a web interface.
Once you’ve embraced the CLI and SDK approach for S3 operations, document processing workflows become much cleaner. If you’re storing PDFs or scanned documents in S3 and need to extract text for indexing or search, IronOCR integrates well with this API-first mindset.
using Amazon.S3;
using IronOcr;
var s3Client = new AmazonS3Client();
var response = await s3Client.GetObjectAsync("my-bucket", "documents/scanned-invoice.pdf");
var ocr = new IronTesseract();
using var input = new OcrInput();
input.LoadPdf(response.ResponseStream);
var result = ocr.Read(input);
// Now you have searchable text from your S3-stored document
Console.WriteLine(result.Text);
The library accepts streams directly, so you skip the intermediate step of saving files to disk. For bulk processing across thousands of S3 objects, you can combine this with the SDK’s pagination and parallel processing patterns you’re already using to handle those “folder” operations.
The Takeaway
S3 is brilliant at what it does — storing objects at a massive scale with high durability. But it’s not a traditional file system, and it doesn’t pretend to be (even if the console does).
Understanding these quirks isn’t just about avoiding frustration. It’s about using S3 the way it was designed to be used: as an object store with an API-first approach, where keys are flat (even if we pretend they’re hierarchical), and automation beats manual clicking every time.
Now, if you’ll excuse me, I have some “folders” to not rename using a Python script.
