A cloudwatch log insights query to retrieve bucket access history
I recently had the need to secure some S3 buckets so that I could restrict access to only the required principals. I was pretty confident that I knew which principals I need to add to the bucket policy but I wanted to make sure that I didn't omit any roles and break our CI/CD pipeline.
As I had both CloudTrail and Data Events enabled I knew that the information I needed would be in the logs.
After a little bit of experimentation I came up with with this this Cloudwatch Logs Insights query.
fields @message | filter eventCategory == "Data" | filter eventSource == 's3.amazonaws.com' | filter requestParameters.bucketName == 'lucas-test-bucket' | stats count(*) as count by eventSource, eventName, requestParameters.bucketName, userIdentity.sessionContext.sessionIssuer.userName
The query explained
The key parts of the query are as follows
- filter eventCategory == "Data" This filters out all non data events such as management events
- filter eventSource == 's3.amazonaws.com' I'm only concerned with S3 events
- filter requestParameters.bucketName == 'lucas-test-bucket' This narrows the results down to a single bucket and probably makes the previous filter clause redundant
- stats count(*) as count by This works similarly to DISTINCT in an SQL query and groups the results into unique rows that makes the results much more concise
- userIdentity.sessionContext.sessionIssuer.userName This is the name of the role that accessed the bucket and is what I need to add to the bucket policy
As you can see, the following results provide all of the information I need to identify not only which principals accessed the bucket but which operations they performed, allowing me to follow the principle of least privilege when creating the bucket policy.