Seven Matryoshki. Source: Ye Olde Curiosity Shop, Seattle, WA

Migrations are an art, not a science. A migration is a juggling act between breaking glass, keeping the lights on, and opening new doors.

Migrations are the evolution of a system through incremental changes.

Migrations are not limited to infrastructure; customer/product changes can be migrations too.

The difference between a single commit and a migration is that a migration is a long series of commits. Also, migrations generally affect a lot more than one person.

Migrations require updating docs, people’s brains, and muscle memory.

Done well, migrations reduce complexity and accelerate product development.

Done poorly, migrations increase complexity and slow product development.

Why did I write this?

Circa November 2024, some seasoned engineers, a few newer folks, and I, had a passionate discussion:

We’ve increasingly moved our product towards being easy – wizards, fast onboarding – at the cost of simplicity – removing non-essential complexity. Does that resonate with anyone else? The easy things become quite hard to manage in the medium-to-long term.

From what I heard, yes. It resonated, resoundingly.

How do we think about the tradeoff between simple and easy in a longer view? If we build something easy today, how do we think about incremental architectural changes to reduce our complexity? Likely easier said than done, but how have we thought about it in the past, vs. now?

What are the biggest sources of pain and complexity right now in the platform, and are there ways to reduce that complexity over time, balancing it with product roadmap, new features, etc.?

These are great questions!

This conversation happened in November 2024, when I had just completed a 3-year migration project of my own. So I was already in a headspace of reflecting on the past 5 years and what wisdom can be gleaned from it.

So what’s the goal?

At Amperity, the word “migration” carries a lot of historical baggage. Let’s sit with the baggage, process it, and accept it.

Let’s:

• Share examples of good outcomes of migrations.
• Share examples of troubled migrations and how they turned successful.
• Share common themes from the above.

How do big migrations start?

Usually, but not always, a design-and-architecture or staff engineering group gets together and studies a big problem. A design doc, or series of design docs get written. It may be very long, 20+ pages. Usually, an engineer is chosen as the Directly Responsible Individual (DRI) to start and navigate the project.

Examples of successful big migrations

We did it! Yay, us! This is in no particular order, and I’m obviously forgetting some as I write this:

TODO:
I’ve worked on backend/infra most of my time at Amperity, so I’m probably missing some of the more direct customer-facing migrations. Feel free to email me if I missed anything! I will update this post, if so.

What’s in a successful big migration?

Usually:

• A Directly-Responsible Individual (DRI) who is also a direct contributor to the code (or holds enough sway with engineers who are direct contributors)
• A central Design Doc, kept up to date. A Google Doc works well.
• Maintain buy-in from engineering and product
  • You will be asked, “What?” and “Why?”. Do Repeat Yourself, and link to the Design Doc.
• Small focused team: Team-to-team handoffs and trying to coordinate across teams is the path of pain.
  • This may include embedding: bringing together SMEs from different teams (e.g. a product domain SME with a cloud infrastructure SME) into a “tiger team” or “v(irtual) team”
  • Observe the Churn Rule: take the initiative on updating callers, in backward-compatible fashion. Avoid imposing work on teams who are likely busy and far from the migration details
    • Friction in the migration strategy directly feeds back into accelerating the migration (e.g. they may start with a runbook, then feel the repetitive parts, and then automate them)
• Ratcheting: when the new pattern is ready and in use, prevent new usages of the old pattern from being added - usages of the old pattern monotonically decrease.
  • See also: Strangler Fig Pattern
  • See also: Shitlists (CI tests that fail when old pattern is re-introduced)
  • Examples at Amperity:
    • mulch is a homegrown Shitlist for Clojure code. It blocks new usages of deprecated functions in our Clojure and ClojureScript code.
    • After we migrated a few services from Finagle/Thrift to Prodigal, we produced a runbook to migrate services and made Prodigal the officially blessed path for new services.
    • After we migrated a few services from Aurora to K8s, we produced a repeatable migration runbook. We updated the “add a new service” tools/docs/guidance to target K8s. Soon after, 5 new services needed to be added, and they were stood up directly on K8s.
• They are finished and recognized as finished. How?
  • Written and shared definition of done
  • Announcement that it’s done
• Deliver an early win and/or other incremental wins
  • Caveat: Avoid starting with the easiest parts
    • Up front research to find the riskiest, hardest parts and derisk them
    • Sequence work so that the hard parts get easier
      • Example: we don’t know how to use K8s, and have no K8s tooling, which is a big operational risk we need to solve before using it widely in production → Let’s start by using K8s in a lesser-used part of the product to build expertise and foundational tooling
    • Or, start with the riskiest part to prove the migration strategy
      • Example: in both the Prodigal and K8s migration, one of the first services we tackled was Tenant Service, which is a high-traffic, core service that could bring the whole app down
    • Sometimes migrations stall because they dive head-first into the easy parts, and then the harder parts lead to re-considering earlier design decisions.
  • Also, Scenic Routes: take a little detour to get obvious wins for customers, which are clearly beneficial to the migration
    • WARNING: when Scenic Routes keep getting pulled into the critical path, and they grow in scope, this can lead to a Matryoshka Migration
• Learn from Incidents
  • We run AARs and incorporate the learnings into the Design Doc
• Delete the dead code!
  • If the only dead stuff is code, it’s often easiest to delete it all at once, when we’re confident it’s unused. Big red line PRs are tasty!
• Shut down the dead infra!
  • In infra migrations targeting cost efficiency (e.g. shutdown of Accumulo/HDFS), slowly scaling down the old system helps deliver incremental wins.

How do big migrations struggle?

Pattern: Matryoshka Migration

Commonly, migrations turn into Matryoshka Migrations. Matryoshka is the Russian nesting doll. 🪆Migrations nested in migrations.

A Matryoshka Migration is one that has diverged a bit too far from its original goals.

This migration has gone on so long that we started other migrations that affect the same parts of the system.

The critical path of the migration has gotten longer and more complicated, rather than shorter and simpler.

You can smell it:

• A migration started
• It has been almost done for a really long time: months, or years.
• Though it’s done with good intentions (e.g. Scenic Routes motivated by product/customers), the migration keeps going… and going… and going… or pausing inexplicably.

Pattern: Drained DRI

For example, a migration was started 1 year ago. Then, re-orgs and staffing changes (e.g. hiring, attrition, reduction in force) happened.
All the teams, code ownership, and managers have shuffled around.
Company strategy can change, too.

However, the friction that motivated the migration, and the friction of unfinished migrations, usually rear their ugly head again.
It’s hard work for the DRI to re-establish buy-in and kickstart the migration. Sometimes they have to do this multiple times.

What can help:

• Design Doc
• Switch DRI
• Embedding, re-establishing a focus team
• Let the DRI take a break

Turn this Matryoshka around!

Some of the successful migrations became Matryoshka Migrations and then turned around. How?

• Recognize that it’s a Matryoshka Migration.
  • Compare the original goals/timeline with the current timeline. A smell is that the original timeline hasn’t been re-evaluated.
• Re-evaluate the cost of complexity inherent in having multiple, large in-progress migrations.
• Re-focus on the original goals. Telling the customer value, and buy-in of those goals.
• Re-evaluate what’s really in the critical path, and focus on it.
• Re-establish a small focused team with a “mandate”, free up their time to focus on it.

Own up to blocked and stalled big migrations

Let’s be leaders: communicate, and have empathy.

Yes, it is okay, and sometimes better for the company to wait to start, pause, or stop a migration.

There’s a negative outcome we’ve seen sometimes at Amperity. Example:

1. Engineers or customers feel pain from a problem that exists today
2. Engineers have designed a solution, socialized it with engineers, and feel that they have clear buy-in.
3. Engineers feel blocked from starting or continuing work that they believe would solve the problem.
4. This blockage lacks a clear explanation: nameless leaders withhold their buy-in
5. Outcome: a tenured, high-value engineer is demoralized.

If we decide to block, pause, or cancel a migration, let’s communicate clearly and honestly why.

Reflect on finished big migrations

When a migration finishes, we can run retrospectives to gather and share what we learned. Consider this post a “Meta-Retrospective” of 5 years of migrations at Amperity.

One example retrospective is the Spark 2-to-3 Retrospective.

So… what should we do?

It’s a fact. We have a complex product and complex infrastructure.
What are the biggest sources of complexity right now?
Are there ways to reduce that complexity over time, balancing it with product roadmap, new features, etc.?

Yes!

We’ll do all the migrations, all at the same time, all as fast as possible. No - sorry, that’s a joke. Today, we can’t do all of them at full speed.

• As engineering leaders (e.g. staff-eng community), let’s catalog and prioritize all our major migrations that are executing and proposed.
• If some migrations are entering Matryoshka status, and still valuable, let’s focus on finishing them.
• We should decide which ones to pause, and which ones to hold off on starting, with empathy.
  • Note that pausing may mean some progress may still happen - it just will be in the background with gradual progress, likely not on any team’s Jira board. Perhaps interspersed among new feature commits, written on quiet Wednesday and Friday afternoons. (See below: Tidy as we go)

We’ve done a lot of the hard thinking on ways to simplify that complexity, but we haven’t started or implemented them fully.

Examples:

• Datasets: groups of named tables. Tables are tables!
• Amperity’s internal library, comp-graph for forming Computation Graphs of complex layered transformations. (Credit to Kevin Litwack)
• Amperity Sandboxes: a massive enabler for testing, sharing changes. I think this is done from a product standpoint – what is left from the engineering perspective?
• Coordinated Changes
• Explicit vs. implicit configuration / semantic action-at-a-distance
• Workbench for exploring Datasets (state)
• Drafts

Tidy as we go

We can pragmatically “Tidy as we go”. Tidy First? is a nice 2-hour read on this topic.

Quick summary:

• “Tidying” is at the level of individual commits, a conversation among engineers and the code; it’s not called out as a “migration project”.
• Tidying means effectively carrying out a “low-urgency migration” bit-by-bit, interspersed with new-feature commits. In other words, gradual refactoring towards a higher quality, lower-friction codebase.
• Tidying requires trust and good judgment.

Example at Amperity:

• UI Code Modernization: Reframe Spaghetti → React/Helix

Sources and inspiration

This post is a synthesis of my own experience and several other people’s great ideas.

(iterate inc @amperity-engineering)

I’ve chatted about this topic with these people at Amperity (please note, I took 5 seconds on this list, it is not exhaustive, and in no particular order):

• … many others!
• John Rush
• Jeff Stokes
• Greg Look
• Stephen Meyles
• Hemanth Srinivas
• Aria Haghighi
• Bryce Covert
• Brandon Vincent
• Ace Levenberg
• Drew Inglis
• Graeme Roche
• Kevin Litwack
• Cary Lee
• Joe Christianson
• … many others!

In general, thanks to Amperity’s #staff-engineering discussions.

… and in the public domain:

• Kent Beck – “Tidy First?”
• Dan Na – “Pushing Through Friction”
  • What is friction? It is resistance; when things feel harder than they ought to be.
  • Friction in engineering demotivates otherwise smart and highly-motivated engineers.
  • Friction in a product turns customers away.
  • “WTF Factor” and “Normalization of Deviance”: The harms when tech debt, stalled migrations, or struggling migrations “swept under the rug” become normalized in an organization.
  • “Pushing Through Friction Is The Job [of a tech lead or staff+ engineer].”
• Tanya Reilly – “The Staff Engineer’s Path”
  • Chapter: Leading big projects
  • Section: Why have we stopped?
• Will Larson’s astute articles
  • Migrations: the sole scalable fix to tech debt.
  • Your migration probably isn’t failing due to insufficient staffing.
• Matt Schellhas – “You will always have more problems than engineers”
• While we were pretty proud of “coining” the term Matryoshka Migration, it turns out we aren’t really that original. Except when it comes to naming things. Check out this travelling blog: Migrating Matryoshka

The first one

A server needed to be created and configured. This is a special snowflake server. It’s owned by team B.

My team provisions servers all the time. I understood the scope of changes: I could’ve done it myself, or elected an engineer on my team.

Instead, I reached out to the team’s manager, and asked something like:

Hey, this is nominally in your team’s area of ownership. You all have a full plate, and don’t do server config often. It could be a learning opportunity for your team, but we could also just do it. What do you think?

She said yes, there’s interest on the team in a learning opportunity and a cross-team win, so let’s have an engineer on her team take it on. Nice!

Later, I reached out to the engineer and asked:

Hey, have you provisioned or configured a server before?

No, she hadn’t. All good, just checking, I said, and asked if it’d be helpful to have a call where we go through the tooling and typical workflows. We had that call, and it went great – she was over-prepared by reading some of our docs ahead of time. I asked lots of socratic questions and she engaged with them and asked her own questions too.

It got done! She showed her work to the engineering department.

The second one

With a different team and a different engineer, I “gave” them a task in a similar way: reached out to their manager, etc. They reached out to me with maybe one or two questions total. I reached out to them with a question, basically “hey, how can we fix this error?” and they replied with a “yep I can fix it nod”.

Basically, they did not engage with the socratic question, so I did not ask more.

It got done-done-done! They showed their work to the engineering department.

I had reached out to their manager – like in the first one – and the task getting done happened about 2 months later.

The learning

Patience pays off for everyone.

Both of these stories happened in the context of a multi-month project which I was leading.

As a result of choosing this path instead of the “do it all ourselves” path, we have strengthened the organization. The skills to provision and configure servers, and provision cloud resources, are not frequently needed by other teams, but when they are needed it is often a late discovery in a project that already has a lot of momentum.

In the grand scheme of the software-shipping machine, it is better for these skills and familiarity to be distributed in pockets around the organization.

That’s an example of force multiplication.

noun

(colloquial) A lot, a large amount

Thanks a heap!

– Wiktionary

Right, if we do heaps of work, we’ll get loads of money!

Serve pizza on plastic throwaway plates. It’s better for business!

Okay, wut?

To stay motivated and engaged, I’ve started thinking about what kind of work I should be doing in my current role and team.

A helpful motto I’ve learned is: any time a task comes onto your radar, you can:

1. Do it now
2. Drop it now
3. Delegate it
4. Do it later

Some people talk about stacks and queues of work. That’s pretty helpful. It makes a lot of sense, when every task is of equal priority. But time is finite. Not everything can be of equal priority!

As time and priority become more important factors in my personal algorithm, heaps – also known as PriorityQueue in Java, possibly a a deque in Python, and possibly a PersistentQueue in Clojure – is the analogy I’ve landed on. I don’t even know if it’s original, but it’s useful.

But also, big projects should not be rapidly moving on to and off of the heap. They have external commitments (stakeholders around the business, and maybe deadlines) that need to be kept in alignment. And context switching between big projects is expensive.

So, what’s on that heap?

Snacks

At any moment, there’s likely a real, important project that I could be working on, but so many ideas and potential things to work on come up every day.

An idea that stuck with me is the clever label for work that is enticing and easy, but not a great use of one’s time: snacking.

All the credit to The Staff Engineer’s Path and Staff Engineer for the clever analogy of snacking.

• Big projects are on a queue-ish and stack-ish system.
• Snacks stay on the heap.

What’s worth my time?

Besides the real, important projects, what’s worth my time?

A few ideas that seem obviously worth my time are:

• Org-wide sources of friction
• Org-wide drags on velocity
• Preventing the next incident we haven’t thought of yet

When I’m on-call, maybe a little different:

• In incident response, fill the gaps
• Fixing recurring sources of incidents in the system and on-call lives
• Fixing recurring sources of on-call friction and cognitive load

What should I do with snacks?

It’s okay to snack from time to time. It breaks up the monotony of the week. When I’m in the middle of large projects, it’s good to have a small quick thing that I can just ship and get validation on.

Instead of snacking all the time, I can:

• Delegate them
• Keep a list of snacks
• Block out an hour or two a week to have a snack

Might look like a snack, but actually important work

There’s a class of work that is actually important for future success but feels like a snack in the moment: researching big ideas that could become org-wide improvements.

To nurture these ideas, I can:

• Keep a list of pie-in-the-sky ideas
• Use an hour or two every week to continue the research and prototyping
• Sneakily build consensus. Get people sold on the idea, before it’s even working

Funny thing: the ops team literally sat in a corner of the office, in their own room. That’s where ops is, in that little room. It sounds like a meme.

The ops team had a nice tool to spin up a VM inside the company’s infrastructure. I appreciated that – my whole team used it all the time. I needed to train recurrent neural networks using GPUs and 20+ gigabytes of RAM. No way that was going to run on my laptop, so this workflow was invaluable to my work.

Here’s the big catch: production deployments happened once every two weeks. Full stop.

If something went wrong, the deployment had to wait another two weeks. Unless you were lucky: if the current ops on the weekly rotation was particularly nice, and not dealing with evening plans, and if you were online to respond to their questions, you could push through and fix that random error that only happens in production.

From time to time, I would wander into the ops corner and chat with people about strange issues my team saw in the production database, in our latest attempt to deploy to production, and so on.

The production deployment challenge

My team was fundamentally a data science team. We were training ML models, building and running data pipelines to collect training data and train models on the latest data. All Python code. That’s all fine.

There was a big problem: the models in production were misbehaving, and customers were noticing:

Your API returned this classifier result. That makes no sense. Why?

It would get sent to our team of analysts, and eventually my team to figure out. After a long back-and-forth, we’d conclude there’s probably a deficiency in the training data, the model, or business logic wrapping the model.

Okay, how do we fix this? We need to fix the model, or update the business logic wrapping the model. And then what? We can test it and deploy it to production. Okay, but we can’t deploy to production because only engineers and ops can do that. What? How do they … deploy to production? Go talk to them.

This became my problem to solve, perhaps heroically.

To give you a picture of where we were at, reviewing PRs was… simply not a thing. We worked out of GitHub repositories that were largely just snapshots of the current code. That’s great - it’s a backup of our code. In the ideal case, we’d push to master, ssh to an internal VM, pull the code, and run it. In the normal case, we’d ssh to the VM and edit things there, rerun the models, copy the code from the VM into the GitHub repository if we remember.

How do we build a new model and deploy it to production? Who knows. Maybe other people’s problem, but that wasn’t a term I knew back then.

The production deployment solution

This post isn’t about my heroism, so to keep it brief, I leaned hard into what I understood as “DevOps”:

• Went and talked to the engineering teams and the ops team. Figured out how everything fits together.
• Learned what the heck Chef is.
• Wrote and deployed an internal PyPi repository that used git tags as versions, and resolved dependencies using our internal GitHub repositories. With the support of a partner in the ops team. Not a ton of code - maybe 100 lines of Python - but deeply embedded and tough to test and deploy on my own.
• Established a pattern of not just pushing to master, but also tagging versions to release, and sometimes reviewing code in PRs before merging to master
• Created a Chef recipe template for Python apps
• Created a Chef recipe for our Python app
• Deployed the thing to production, and fixed the customer’s concerns!

Since then

Since then, I’ve spent most of my time the other side of the table, so to speak. Not to mention in a completely different company with different values, and a more flexible organization.

I’ve reflected a bit on what makes 2018 different from 2026.

2018: Production Operations Team

We are the production operations team. Our mission is to protect production.

If a developer wants a change in production, it should be possible, but we need to cover our butts. There’s going to be a handoff from the developers to us. Make a full paper trail with tickets.

If production is too hard to change for a regular developer - well, that sucks, but it kind of works in our favor. Any deviation from what’s in that ticket is - by default - a liability for us. Nobody has time for debugging a random issue that happens “only in production”. We have lives outside of work!

From time to time, a hero might push through the friction and make a self-service path. But again, that’s a hero, and heroes are few and far between. They also might quickly move on to another company where change is easier.

2026: Platform Engineering Team

Our mission is to accelerate development and make production resilient.

Developer experience has to be smooth. CI/CD has to be quick. If a developer is ever waiting for CI or CD, treat that as a mini-incident.

When a problem is discovered in production, it should be obvious what the problem was, and the internal signals for developers should make it easy to diagnose and fix.

It still feels early to be prescriptive on the how and what, so it’s worth another post in the future.

I stumbled upon that today and thought dang, that looks like something I can try in about 15 minutes… let me go make a cup of tea. And here it is!

Thank you, Chad.

More coming soon.