Seniority Is Knowing What Not to Build

The system forwarded customers’ event data to tools such as analytics services. Segment had given each destination a service and a queue, so a slow destination wouldn’t delay deliveries to the others. But changing shared code meant testing and deploying across a growing collection of services. Consolidating more than 140 destination services into one removed the need to deploy them all whenever a shared library changed.

Getting there required new machinery. Segment built Centrifuge to replace the individual queues and send events to the combined service. The team also changed its tests to replay recorded requests and responses rather than depend on live destination endpoints. Combining the code required making it practical to test together.

The new arrangement spread cached data across many more processes, each with its own in-memory store. A value already fetched by one process might be missing from the next process handling that destination’s work. A shared cache could make those values reusable across processes, but would add another system to operate and scale.

Alexandra Noonan’s account doesn’t quantify the cache tradeoff in customer latency or dollars. It records the choice and its rationale. The team also acknowledged other costs of consolidation, including the risk that a crash in one destination could take down the combined service.

Our reading is that the team accepted a local regression to protect a larger gain. That puts a burden on whoever argues against the next build: explain which loss the business can afford.

Which requirement would the build satisfy?

A worse metric gives you something to investigate. Whether it justifies a new system depends on what the business needs from the existing one.

Suppose you’re reviewing a web service that caches database query results. A lower hit rate sends a larger share of requests to the database instead of reusing a stored answer. That matters if the extra queries push response times past the agreed limit, overload the database, or cost more than the business can accept. If the service and database can absorb that extra work within those constraints, less effective caching may be tolerable.

Redis could be the right answer if sharing cached results removes the repeated queries causing the problem. So could reducing unnecessary requests or changing an expensive query. Compare those options under the same workload. A smaller code change that leaves the database overloaded hasn’t met the requirement.

The engineer arguing against the build has work to do. Show that the existing system, or the smaller intervention, can meet the requirement. Include the workload or failure condition that would change the decision. A successful test under light traffic leaves the question open if the service must handle a much heavier load.

A planned rollout can make today’s acceptable performance the wrong evidence. If it would exceed the load you’ve validated, test that workload before deferring capacity work. Set the condition for reopening the decision while there’s still room to implement and validate the next option. Waiting for failed requests leaves users carrying the cost of the delay.

The cost of declining belongs in the decision too. Someone may have to keep doing a manual task, tolerate slower responses within the agreed limit, or accept less headroom for growth. Those costs deserve the same scrutiny as the new system’s maintenance burden. Otherwise the engineer gets a simpler architecture by handing the work to someone else.

Standing should get the question heard

Ask a senior engineer to review only a cache’s implementation and you’ve already excluded the question of whether the cache belongs. They can improve the code while the more consequential choice goes unexamined. That’s a costly way to use someone who understands why the surrounding systems exist.

A junior engineer can supply the evidence too. If their objection is being dismissed on rank, a senior colleague can use their standing to get it considered. Using rank to veto a build without explaining the tradeoff merely reverses who’s being asked to defer.

The person who requested the work still needs their problem handled. They should leave the discussion knowing how the requirement will be met and which compromise they’re being asked to accept. Whoever owns the affected service or workflow has to agree that the compromise is acceptable. Closing an infrastructure ticket alone doesn’t establish that agreement.

Keep the evidence for declining the build in the design record, along with the condition that would reopen it. The next engineer should be able to challenge the decision without needing the original senior engineer in the room.