CSCD01 Engineering Large Software Systems
Mitigating Architecture Risk
Cho Yin Yong / Aleksander Bodurri

The hardest part of the job

Code is easy write, and technical problems can be constrained, analyzed and solutioned for.

But, we do not build in isolation. You can be an expert in all the things we’ve discussed in lecture up until this point and still be ineffective in the real world.

The ability to communicate architecture to others, both technical and non-technical, is the most challenging aspect of the software architect role.

Architecture is about decisions

The role of an architect is to make architectural decisions, these are decisions that are considered to be architecturally significant.

Question: Are technology choices architectural decisions?

Answer: It depends… (Let’s find out in this lecture)

What does it mean to be a good decision maker?

Afraid to make the wrong decision? Some architects avoid this by not making any decision. Talk is not making a decision.

Make decisions from a position of knowledge. Acquire knowledge by making communication a priority for your team.

An architect should be a bridge between the the technical and non-technical members of the team.

Architecturally Significant

Why do we care about the distinction between decisions and architecture decisions?

We said earlier that an architecture decision is a decision that is architecturally significant. What do we mean by that?

An analogy on significance: A restaurant

Cho and Alex co-own a restaurant called Deo One. The restaurant has servers, runners, hosts, chefs, managers, etc.

Alex has a series of changes to improve the restaurant that he wants to implement.

Let’s help Alex decide which changes are significant.

An analogy on significance: A restaurant

Alex suggests

  • Changing the layout of the restaurant from one with communal tables to individual tables.
  • Changing the colour of server uniforms from black to navy blue.
  • Changing the hours of operation from 3:00 PM - 10:00 PM to 5:00 PM - 12:00 AM.
  • Changing our food supplier from a wholesale provider to a local farm.
  • Removing physical menus and making the menu available through QR codes that are attached to the tables.

Is this a significant decision?

Architecturally Significant

StructureArchitectural styles
Distributed vs Monolithic
At the restaurantCommunal tables to individual tables
Non-functional requirementsThe NFRs that we’ve discussed in first part of this course.
At the restaurantNew hours of operation
CouplingConcerned with the points of dependency between components and services in a system.
At the restaurantWholesale supplier to a local farm
ContractsThe interfaces in a system. API gateways, integration hubs, proxies, etc.
At the restaurantPhysical menus to QR codes
The uniform colour change falls under none of these areas.

July 8, 2022

On a Friday morning, Rogers wireless and home internet service stops across Canada.

Interac runs on the Rogers network, so debit payments stop working in stores. For a large share of Rogers customers, a call to 911 does not connect.

By the time service returns more than a day later, the outage has cut off over 12 million customers.

Rogers Network Upgrade

For several weeks, Rogers engineers have been upgrading the core of the network, which carries all wireless and home internet traffic.

The upgrade is a series of configuration changes to the routers, made in seven phases. Each change receives a risk rating, which decides how much review and lab testing it must pass.

With five phases finished without a problem, the team begins phase six on the morning of July 8.

Architecture Risk

An architecture risk arises when the structure of a system allows a failure. Architects rate each risk by the impact of that failure and the likelihood that it occurs.

Risk Matrix

The risk matrix rates impact and likelihood from 1 to 3 and multiplies them. A product of 1 or 2 counts as low risk, 3 or 4 as medium and 6 to 9 as high.

Rogers Network in July 2022

Simplified from the Xona Partners report for the CRTC. The road comparisons are added for this lecture.

Phase Six Plan

A route maps a block of addresses to the next router on the path. A copy job on the distribution routers passes routes from the internet table to the core, limited by an allow list.

Risk Rating History

Rogers rated the first change in the series high risk and the next ones medium.

Because its algorithm weighs earlier successes, the rating for phase six came out as low.

A low rating required no extra review, no higher approval and no lab testing.

Phase Six Outcome

Observability

Observability is the ability to work out what a running system is doing from the signals it emits. OpenTelemetry is the open specification that defines those signals.

Diagnosing a Slow Checkout

November 2015

Every message that Discord users have sent lives in one MongoDB replica set.

The count passes 100 million messages this month.

Around the same time, response times become unpredictable.

Risk Mitigation

Mitigation changes the architecture to lower a risk rating. Because each change costs money and effort, the business decides whether the lower rating justifies it.

2017

To get past the limits of one database, Discord moves its messages to Cassandra.

By 2017 the database runs on 12 servers, with every message stored on 3 of them.

The team picks Cassandra knowing that it trades strong consistency for availability.

MongoDB and Cassandra at Discord

A cluster is a group of database servers acting as one database, where each server is called a node.

Replication

Risk Losing a node loses the data stored on it.

Replication stores each record on several nodes, so the data survives the loss of one node. Each stored copy is a replica. A replication factor of 3 gives Discord three replicas of every message.

Cassandra has no primary node, so any replica accepts a write.

Quorum

Risk A read reaches a replica that missed the latest write.

A read or write reaches quorum once a set number of replicas confirm it. With N replicas, W confirmations per write and R per read, every read includes the latest write when R + W > N.

Partitioning

Risk The data outgrows the memory and disk of one node.

Partitioning divides a dataset across nodes, so no single node has to store all of it.
Node 1
general
art
Node 2
gaming
memes
Node 3
music
world cup

Each coloured group is a partition, which holds the messages that share one partition key. The key here is the channel.

Key Range Partitioning

Risk With no rule for where keys live, a lookup asks every node.

Key range partitioning gives each node a contiguous range of partition keys. When some ranges receive more data or traffic than others, the partitions are skewed.
Node
Names starting with
Channels
Share of traffic
Node 1
A to H
artgaminggeneral
Node 2
I to P
memesmusic
Node 3
Q to Z
world cup

Hash Partitioning

Risk Popular key ranges overload one node.

Hash partitioning passes each partition key through a hash function and stores it on node number hash(key) mod N, where N is the node count.

Node Changes under Hash Partitioning

Although hash partitioning spreads keys evenly, a change in the node count N reassigns most of them.

Consistent Hashing

Risk A change in node count moves most keys.

Consistent hashing places nodes and partition keys on a ring, where each key belongs to the next node clockwise. A new node therefore takes keys from one neighbour only.

Virtual Nodes

Risk Uneven arcs on the ring give one node more load.

Virtual nodes give each node several points on the ring, which evens out the share of keys per node. Amazon's Dynamo introduced them in 2007.

Six Months on Cassandra

A public channel on the Puzzles & Dragons subreddit server takes 20 seconds to load.

Anyone who waits sees a channel with one message in it.

Someone had deleted millions of messages. Cassandra marks each delete so that it reaches every replica, and each read has to scan those markers.

Early 2022

Five years later, the Cassandra cluster has grown from 12 nodes to 177 and holds trillions of messages.

The engineers on call are paged for the database so often that the team calls it a high toil system.

Whenever one channel gets busy, latency climbs for unrelated channels too.

Hot Partition

A hot partition receives far more traffic than the other partitions. It follows from two mitigations of 2017, partitioning and quorum.

Request Coalescing

Risk One busy channel overloads its nodes.

Request coalescing answers concurrent identical requests with one database query. Discord added data services between its API and the database to do this.

Remaining Cassandra Problems

Compaction backlogCassandra merges its data files in the background. When that work falls behind, reads slow down.
Seen at DiscordNodes left rotation to catch up
Garbage collection pausesCassandra runs on the Java virtual machine, whose memory cleanup can freeze a node for seconds.
Seen at DiscordOperators rebooted nodes by hand

May 2022

Discord finishes moving every message to ScyllaDB, which is compatible with Cassandra and written in C++.

Because C++ needs no garbage collector, the database stops pausing to reclaim memory.

177 nodes shrank to 72.

Hot partitions can still occur in ScyllaDB, so the data services stay in place.

Summary

Rate impact and likelihood separately.At Rogers, five successful changes lowered the rating to low while the impact of a core failure stayed at its maximum.
Each mitigation adds a new risk.Cassandra removed Discord's memory limit and brought hot partitions.
Ratings go out of date as a system grows.The design that suited Discord at 12 nodes paged its engineers constantly at 177.