Warm tip: This article is reproduced from serverfault.com, please click

How to limit the amount of data stored by user via S3 buckets on AWS?

发布于 2020-12-06 05:29:03

I'm creating a platform whereby users upload data to us. We check the data to make sure it's safe and correctly formatted, and then store the data in buckets, tagging by user.

The size of the data upload is normally around 100MB. This is large enough to be concerning.

I'm worried about cases where certain users may try to store an unreasonable about of data on the platform, i.e. they make 1000s of transactions within a short period of time.

How do cloud service providers allow site admins to monitor the amount of data stored per user?

How is this actively monitored?

Any direction/insight appreciated. Thank you.

Questioner
EB2127
Viewed
11
John Rotenstein 2020-12-06 13:37:17

Amazon S3 does not have a mechanism for limiting data storage per "user".

Instead, your application will be responsible for managing storage and for defining the concept of an "application user".

This is best done by tracking data files in a database, including filename, owner, access permissions, metadata and lifecycle rules (eg "delete after 90 days"). Many applications allow files to be shared between users, such as a photo sharing website where you can grant view-access of your photos to another user. This is outside the scope of Amazon S3 as a storage service, and should be handled within your own application.

If such data is maintained in a database, it would be trivial to run a query to identify "data stored per user". I would recommend against storing such information as metadata against the individual objects in Amazon S3 because there is no easy way to query metadata across all objects (eg list all objects associated with a particular user).

Bottom line: Amazon S3 will store your data and keep it secure. However, it is the responsibility of your application to manage activities related to users.