8 Fallacies of Distributed Computing

1. The network is reliable

Effects: Software applications are written with little error handling for network errors. During a network outage, such applications may stall or infinitely wait for an answer.

Solution: Implement fault-tolerant design patterns within your applications, API gateway, and service mesh e.g. timeouts, retries, bulkheads, and circuit breaker.

2. Latency is zero

Effects: Ignorance of network latency, and of the packet loss it can cause, means that inaccurate assumptions can be coded into applications.

Solution: Learn “Latency Numbers Every Programmer Should Know”. Implement retries and rate limiting, as appropriate, in your API gateway and service-to-service communications.

3. Bandwidth is infinite

Effects: Ignorance of bandwidth limits can result in bottlenecks and dropped packets.

Solution: Ensure developers work with the platform team, operations, and SRE to understand the network capabilities.

4. The network is secure

Effects: Complacency regarding network security results in being blindsided by malicious users and programs that continually adapt to security measures.

Solution: Conduct threat modeling. Implement authn/authz and end-to-end TLS via your API gateway and service mesh.

5. Topology doesn't change

Effects: Changes in network topology can have effects on both bandwidth and latency issues, and therefore can present similar problems.

Solution: Regularly announce and audit network changes. Recognize that with cloud networking everything changes all the time. See #3 for more information.

6. There is one administrator

Effects: Multiple administrators may institute conflicting policies of which senders of network traffic must be aware in order to complete their desired paths.

Solution: Work with the platform team, operations, and SRE to understand the network capabilities and policies.

7. Transport cost is zero

Effects: The “hidden” costs of building and maintaining a network are non-negligible. Developers also discount serialization costs, learning to high CPU and memory usage.

Solution: Budget time and money to build and maintain networks, API gateways, and service meshes. Be aware of application serialization costs: consider binary protocols like gRPC, and transport costs: consider implementing HTTP/3.

8. The network is homogeneous

Effects: If a developer assumes a homogeneous network, then it can lead to the same problems that result from the first three fallacies.

Solution: See solutions #1-3