Recently, we helped a customer migrate a massive dataset from another S3 provider to Summit's S3 object storage solution. We migrated just under a petabyte of data and over 3 billion objects. That's an average object size of only 250KB, with many even smaller. If you know anything about object storage, you know that migrating billions of tiny files isn’t easy, especially when a big cloud provider gives you just 60 days to move before egress fees kick back in. (Pro tip: If you notify your provider that you’re moving, you can often have those egress fees waived for 60 days.) 

Only one petabyte of data?! With a tad bit of hubris, we said, "No problem. We're a service provider. We have unlimited bandwidth and many servers to throw at the problem!"  

Spoiler alert: It wasn't easy.

Le'll break down the challenges we faced, how to properly quantify your data, and the tools that can help facilitate your migration. Future parts will cover migration strategies, progress monitoring, estimating timeframes and costs, and how to pull it all off under pressure. 

Real-World S3 Migration Challenges at the Petabyte Scale 

We learned much through this humbling migration experience and are now sharing that with you. Our specific migration scenario also felt unique in that we had to do an S3-to-S3 data transfer without local data staging. Many S3 tools for copying data are designed to upload from a local file system or download it to your local disk, but S3-to-S3 copies are a whole different matter. We did not have a petabyte local file system lying around, so staging the data locally was a non-starter for us. We needed to move data from S3 to S3 with no intermediate staging. Also, staging likely implied doubling the time required to perform the copy. 

The secret to properly completing your migration in 60 days? Monitor and tune your process and copies, be patient, and, most importantly, be persistent.  

The high-level process looks something like this: 

  • Understand your data 
  • Benchmark strategies and tools 
  • Pick a strategy 
  • Migrate 
  • Monitor and tune 
  • Verify 
  • Clean up 

So, let's define a few terms and delve into the "understanding your data" phase of the migration. 

Analyze and Profile Your S3 Bucket Before Migration

When we talk about object storage, we often refer to size, but the word size is ambiguous without context. Is a bucket large because it has petabytes of data or many objects? Can both be true? Does it matter for copying? In short, it matters, and copying one petabyte of data from a bucket with 1 billion objects is a very different experience than copying one petabyte of data from a bucket with 1 million objects. For clarity and completeness, we're defining three different kinds of sizes to help facilitate this conversation: 

  • Asize - Average file size in the bucket (i.e. 100k or 100Mb) 
  • Fcount - Number of files in the bucket (i.e. 100 objects) 
  • Bsize - Total size of the bucket (i.e. 1Pb) 

The relationship between these is quite simple: Bsize / Fcount = ASize.

Copying a bucket with Bsize = 1Pb and Asize = 1Gb differs from the same 1Pb bucket with an Asize=250Kb.  

So, why do Asize and Fcount matter more than Bsize when copying large buckets? Doesn't it take the same amount of time to transfer one petabyte regardless of ASize?  

High File Count Slows Down S3 Data Transfers 

S3 operates over HTTP, meaning every object retrieval requires an HTTP request-response cycle, including connection setup, authentication, and metadata exchange. When copying a large number of objects from S3, Asize matters because of the overhead involved in making individual HTTP requests for each object. When dealing with many small objects, this overhead adds up and becomes significant, as the time spent starting and handling these requests can outweigh the actual data transfer time. 

In contrast, objects with large ASize are more efficient to copy because they amortize the overhead across more data. Instead of constantly setting up new connections for many small files, a single large file is transferred in fewer requests, reducing latency and improving throughput.  

Additionally, S3 supports multipart transfers for large files, allowing parallelism at the data level rather than the request level, which further improves efficiency. As a result, copying a bucket with fewer large files is generally much faster than copying one with many small files.  

Unfortunately, ASize was pretty small for this migration, so we knew we'd lose a lot of time on HTTP overhead. But that wasn't the only challenge.  

S3 Bucket Structure Impacts Migration Speed  

Ok, now that we understand the various measures of bucket size, it's also helpful to know how the objects in the bucket are laid out. A thoughtful layout and knowledge of it can make a big difference in our copy's efficiency.  

The three most common scenarios are: 

  • Totally flat: all objects in the root of the bucket 
  • Single layer nesting: objects distributed one level down in sub folders. i.e. s3://bucket/folder1/objects, s3://bucket/folder2/objects, etc.
  • Multi-Layer nesting: i.e. objects are distributed in multiple layers and subfolders. i.e. s3://bucket/folder1/sub-folder1/objects, s3://bucket/folder1/sub-folder2/objects, etc. 

Accelerating S3 Migrations with Folder-Based Parallelism 

If your application team knows the layout information beforehand, it could save you lots of time. If you don't have it, you can try getting a bucket listing and then parse through it or attempt copying it with one instance. The downsides to a single copy instance are that it will be hard to monitor progress, keep track of resume locations if a failure occurs and speed will be limited. On the other hand, if you know the layout, you can set up separate jobs to copy objects in each folder independently. This will allow you to run parallel copies and resume at the folder level and likely save you from having huge object listing files to parse. 

Best Tools for Cloud-to-Cloud S3 Migrations 

After gaining some understanding about the nature of the data we need to migrate, we need to select some tools to perform the copy. There are a lot of options, and we've tested most of them! In our experience, s3Sync and rclone are the best tools for making a non-staged copy directly from one remote bucket to another. That rules out several options like s5cmd, s3cmd, and AWS CLI.  

The other option is to roll your own solution in your favorite programming language. (Anyone want to partner up and write a highly parallelized bucket-to-bucket copy tool in Zig?)

We did some quick and dirty experiments and found that we couldn't easily beat the performance of s3sync with a homegrown tool in Python.

Sure, we could've written something from scratch in Zig or Go to bypass the usual libraries everyone uses. But we wanted to run a migration, not develop new tools. 

Using s3Sync for Massive Object Storage Migrations 

Our go-to tool for buckets with lots of small objects is s3sync. s3Sync is open sourced under a GNU license, it's written in Go, is designed for highly parallel and fast copies. Most importantly, it supports S3-to-S3 copies. It's straightforward to use with only a handful of well-documented command line options. Some things to consider: 

  • Workers: This is the number of threads or copies s3sync will run simultaneously. You'll need to test and tune to figure out the optimal value here. Think more workers for high FCount and low Asize copies and fewer workers for low FCount but high Asize and Bsize. 
  • Memory: Memory requirements are a function of Asize and number of workers. So, Asize * Workers = Needed RAM. We never exceeded 64GB.  
  • Bandwidth: Bandwidth matters but not as much as you'd think. Don't assume more available bandwidth will mean a faster copy. You will need to monitor bandwidth and tune the number of workers and the number of instances of s3sync to utilize all your bandwidth. We had two bonded 10G connects and could never fully saturate them. (Mostly due to limitations on our source bucket platform). We'll talk more about this in the tunning and monitoring section below. A 1Gbps connection will probably limit you, but 10Gb per instance of s3sync will probably be just fine.
  • Latency: The less latency between the machine running s3sync and your source and target buckets, the better. Especially when we talk about resuming copies creates lots of metadata requests. Try to avoid copies across regions if possible (IAD to SJC, for example).  
  • Resuming: There are limited options to resume copying with s3sync. You can use the '--filter-not-exist' option to ensure s3sync only copies files that don't exist in the target bucket, however it's not a perfect solution. Let's assume you have a bucket with 100 million files in the bucket root and your copy process is interrupted half way through. When you restart the copy, s3sync needs to figure out if the source object exists in the target, and this results in 50 million metadata requests. While faster than recopying the objects, it still takes time and resuming can take hours to days depending on how fast your bucket providers are. Nevertheless, you should still use this option. 
  • Prefixes: Prefixes are a way to break up your copy and allow you have to run multiple instances of s3sync and further parallelize your copy. Let's assume you've maxed out memory or bandwidth on your copy host. By breaking out copies by prefix, you can run more instances of s3sync on other hosts at the same time. In our case, we had a bucket with single layer nesting and folders names 0 through 1000 and each folder containing anywhere from a few million to a couple hundred million objects. We'll talk more about this in the strategies section below, but tldr, we took advantage of this so we could run 10 copies of s3sync and have more granular "resume resolution." 
  • Filters: There are quite a few options for filtering your copy that can be handy depending on what your migration/cut-over strategy is. Just keep in mind these usually come at the expense of more metadata requests, which impacts throughput.

Other important flags/notes: 

  • --on-fail skip: Don't abort the copy if an object fails to copy. Just note it and keep going. We can investigate and clean up later. 
  • --sync-log: It's handy to monitor progress in real-time. 
  • Use a script to execute s3sync.  
  • Redirect all output to log files that you can parse for errors later and monitor logs. 

Using rclone for Large File S3 Migrations

For buckets with low FCount and high ASize, rclone is a great option too. It's free, open-source, and works across many platforms, including Linux and Windows. Like s3cmd, it supports S3-to-S3 copies with no need for local staging and has a more robust set of command options and switches. Its most significant downside is memory usage when copying buckets with a high FCount (i.e., more than 50 million objects in any given prefix). 

The difference in memory usage between rclone and s3sync
  • Resuming Jobs and File Integrity - The probability that any long-running copy job will be interrupted is high. Being able to resume where you left off is key; otherwise, you risk losing hours or days of work. We leveraged rclone's --ignore-existing and --update flags to resume transfers in copy mode vs. sync mode. This allowed us to resume while avoiding accidental deletion of files uploaded from other sources. The --dry-run flag helped us understand how much is left to copy without committing to the actual copy.
  • Roll Your Own Concurrency - Earlier, we talked about data structure and prefixes. With an understanding of how the data was laid out in the bucket, we could tune multiple copies of rclone using the --transfers flag to achieve optimal throughput.
  • Logging and Monitoring - Like with s3sync, we recommend using the --log-file, --log-level=DEBUG and -vv flags. This preserves any errors and also helps with post-copy analysis and recovery.

Additional Tips:

  • If your destination bucket requires versioning, consider not enabling it until the migration has been successfully completed to avoid unnecessary overhead and storage growth.
  • Since we primarily used rclone for buckets with low FCount and high ASize, we configured a lifecycle policy to automatically expire aborted multipart uploads, as there were instances where copy jobs had to be aborted and adjusted.

Avoid Migration Interruptions with tmux and mosh 

For migrations at this scale, session reliability, monitoring, and reporting are key to a successful migration. Having a copy job interrupted because of connectivity or a crash is awful because we'll lose hours to days of actual copy time in almost all scenarios at this scale. Resuming is expensive. Let's walk through those three at a high level, explain why they matter, and discuss some practical approaches.  

We're assuming you have several Linux hosts dedicated to running copies. That means you are likely logged in via SSH, and your connection is susceptible to interruption. Your computer goes to sleep, your VPN drops, or your ISP has issues. Since these copies are long-running and interruptions often mean days of lost copy time, you don't want to risk a lost session interrupting your copy.  

We recommend two invaluable tools. First is tmux, a popular terminal multiplexer. Tmux allows you to start a session and, within that session, start a copy. You can log back into your copy host and "reattach" to your disconnected session if you get disconnected. The value here is that your terminal never disconnects from the perspective of the processes (rclone, s3sync, etc.) spawned from it. In other words, your copy continues to run even if your ssh session disconnects. Moreover, tmux lets you run multiple windows and panes. So you have one session running various instances of rclone or s3cmd, which you'll probably need to do if you have a high FCount and low Asize.  

Also, consider mosh; while not needed, it's a massive quality of life improvement if you're a road warrior and work remotely. Mosh allows you to open an SSH session and have that session survive network disruptions. So, if your computer goes to sleep or your VPN disconnects, your session automagically resumes with no re-login needed when you wake your computer or reconnect your VPN. Mosh combined with tmux will ensure your copies are never interrupted due to connectivity interruptions. 

Lessons Learned from Repatriating S3 Buckets 

Repatriating billions of small objects from S3 isn’t just about moving data, it’s a test of orchestration, tooling, and a deep understanding of your data’s structure.  

The biggest lesson here is that object count and size matter just as much as the total bucket size. High file counts with tiny objects can overwhelm even robust tools. Utilities like s3sync and rclone can perform well, but only when tuned to your environment and use case. Metadata overhead, prefix planning resumption strategies, and copy job concurrency all need to be considered. Success comes from preparation, understanding your data layout, and choosing the right tools.  

Eric Dynowski

Eric Dynowski has been developing software, designing global infrastructures, and managing large technology installs for over 25 years. His background in complex infrastructure design and integration has reduced customer budgets by millions.

Eric Dynowski