8 Fallacies of Distributed Computing
1. The network is reliable
Effects: Software applications are written with little error handling for network errors. During a network outage, such applications may stall or infinitely wait for an answer.
Solution: Implement fault-tolerant design patterns within your applications, API gateway, and service mesh e.g. timeouts, retries, bulkheads, and circuit breaker.
2. Latency is zero
Effects: Ignorance of network latency, and of the packet loss it can cause, means that inaccurate assumptions can be coded into applications.
Solution: Learn “Latency Numbers Every Programmer Should Know”. Implement retries and rate limiting, as appropriate, in your API gateway and service-to-service communications.
3. Bandwidth is infinite
Effects: Ignorance of bandwidth limits can result in bottlenecks and dropped packets.
Solution: Ensure developers work with the platform team, operations, and SRE to understand the network capabilities.
4. The network is secure
Effects: Complacency regarding network security results in being blindsided by malicious users and programs that continually adapt to security measures.
Solution: Conduct threat modeling. Implement authn/authz and end-to-end TLS via your API gateway and service mesh.
5. Topology doesn't change
Effects: Changes in network topology can have effects on both bandwidth and latency issues, and therefore can present similar problems.
Solution: Regularly announce and audit network changes. Recognize that with cloud networking everything changes all the time. See #3 for more information.
6. There is one administrator
Effects: Multiple administrators may institute conflicting policies of which senders of network traffic must be aware in order to complete their desired paths.
Solution: Work with the platform team, operations, and SRE to understand the network capabilities and policies.
7. Transport cost is zero
Effects: The “hidden” costs of building and maintaining a network are non-negligible. Developers also discount serialization costs, learning to high CPU and memory usage.
Solution: Budget time and money to build and maintain networks, API gateways, and service meshes. Be aware of application serialization costs: consider binary protocols like gRPC, and transport costs: consider implementing HTTP/3.
8. The network is homogeneous
Effects: If a developer assumes a homogeneous network, then it can lead to the same problems that result from the first three fallacies.
Solution: See solutions #1-3