TLDR:
Demo of the S3 pagination UI: https://coruscating-manatee-4afd46.netlify.app
Source code: https://github.com/headquarters/s3-viewer

Building a UI to traverse files and folders in S3 buckets can sometimes be confusing because S3 stores objects, not files and folders. It uses prefixes to fake a directory structure. This fake directory structure is what shows up in its own console UI.

Another potentially confusing matter in building a UI to traverse S3 objects is the use of the word “pagination” in the SDK client docs. If you’re used to “pagination” meaning a REST API returns results and a client app shows results with previous/next buttons, then this is not the pagination you’re looking for. The boto3 docs discuss paginators, for example, but in this case the pagination is meant to go through “pages” of your bucket’s objects all in one process. Say you have 10K objects and set your limit to 1000 per page, you can loop over the paginator 10 times to eventually fetch all the results. Now your server would be holding on to 10K objects, ready to send them…to the client? Given very few UIs would ever display 10K results in a single UI page, this is unrealistic for pagination with a standard client-server architecture. For server to server, it might make sense to fetch in pages like this, but not so much for a user interface.

A better approach is to work with the prefix and a “continuation token” to pick back up your querying where you leave off in between requests. This is the approach I took to build something akin to the console UI for paginating through an S3 bucket. I used SvelteKit to build the UI and a single API endpoint for fetching object lists with the AWS S3 client library. Going “into” a folder alters the prefix that is used, while moving back and forth between a list “inside” a folder utilizes the continuation token to pick up where you leave off in between requests. No “paginator” required or necessary.