Questions

What is the scope? AZ, Regional, Global

When or why will you choose both CloudFront AND Load balancer?

What is an AZ?

• AZ or availability zone is an independent data center within a specific geographic region.
• An AWS Region is a collection of multiple availability zones.
• Regarded as a fault boundary.
• Data transfer across AZ is not free.
• Cross AZ traffic should be controlled for cost optimization and performance efficiency.

Why use multiple AZ?

• Improves reliability since failures in one AZ do not cause full outage of the workload.
• Improves performance by reducing latency.
• Helpful in disaster recovery?

Is Route53 AZ aware?

In the example with app servers trying to reach the AD domain controller, how does the name resolution help keep the traffic in the same AZ?

Create AZ specific DNS entries using network interfaces in the same AZ. For ex - NLB provides a static IP for network interface in each AZ. Use this to create endpoints such as euw-az1.myservice.com, instead of letting clients use myservice.com.

Challenges

Random AZ name to AZ id mapping

• AWS maps AZ names to different AZ ids in each account, to distribute load more evenly across all it’s data centers.
• AZ name format is eu-west-1a, where as AZ id format is euw1-az1.
• AZ name to AZ id mapping can differ between accounts. AZ id is the same everywhere. Example from an AWS Account (eu-west-1 region) -> EC2 Dashboard

No visibility to others’ AZ choices

Multi-account setups usually have multiple VPC peering and/or transit gateway attachments, often with differing AZ choices.

One team may not be aware of the other teams’ AZ choices. This can cause traffic to cross the AZ boundary when workloads deployed in different accounts/VPCs need to interact.

Guidelines

Due to the above mentioned challenges of multi-AZ architectures, it is important to follow some guidelines when designing.

Manage AZ assignments centrally

Just like managing VPC CIDR ranges centrally with IPAM, manage the preferred AZ centrally as well.

Example, when using VPC with CIDR range of /16, it can be divided further into /20 blocks. The first two /20 blocks can then be used in the first two AZs in all accounts. This greatly simplifies the routing table of the Transit Gateway in the shared network account.

AZ aware shared networking

AWS Transit Gateway is AZ aware (except in appliance mode) and keeps the traffic in the same AZ as source, until it reaches the destination or is routed to a different AZ. Refer examples.

When attaching a TGW to a VPC, use the same AZ that subnets in the VPC use.

Disable cross-zone load balancing

Load Balancers route traffic across AZ, by default (cross-zone load balancing = enabled). Consider disabling cross-zone load balancing when using cross-account load balancer. Be aware of the following trade-offs

• Capacity planning in each AZ to ensure adequate redundancy
• No support for stickiness

AWS Network Load Balancer does not route traffic across AZ, by default (cross-zone load balancing = disabled).

Use AZ local resources

Use local resources within the AZ, if possible.

Example, if there are AD controllers an app needs to reach, consider configuring the app instances in each AZ with the address of the AD controller of that specific AZ. This would ensure there is no risk of traffic crossing the AZ boundary.

Monitor cost and performance metrics

Monitoring relevant metrics is essential to continually improve the workload design and performance. Some things to keep an eye on -

• Analyse network traffic patterns using vpc flow logs to identify cross-AZ traffic.
• Use network manager to monitor inter/intra-AZ latency specific information.
• Analyse data transfer charges using cost and usage reports to identify cross-AZ traffic, and optimise workloads.

Examples

Using transit gateway to inspect all ingress/egress traffic

A workload in VPC A (application account) connects to the AD controller deployed in VPC B (shared services account) through the transit gateway (shared networking account)

[East-West-VPC-to-VPC-traffic-inspection-scenario.png]

Centralized egress with ingress/egress inspection

All egress traffic is routed through the transit gateway, and AZ specific NAT gateway. Separate NAT gateways should be provisioned in each AZ to avoid cross traffic.

Note the route table in the central egress TGW subnet. While sending traffic to internet, care should be taken to prevent [[Work/Network#Private IPs|private IPs]] from hitting the public internet. So, be sure to add routes like 10.0.0.0/8, 100.64.0.0/18, 192.168.0.0, 172.168.0.0

[Routing-traffic-through-NAT-Gateway.png]

Centralized ingress with ingress/egress inspection

Incoming internet is accepted by the internet gateway, and passed to the ALB, which in turn routes it to the destination app workload via the transit gateway, and inspection appliance.

The key steps to note here are 2, and 10. Load balancer (cross-zone disabled) picks a network interface in the same AZ (AZ B) as the destination. If cross-zone is enabled, it can pick any AZ, so a scenario like 10a becomes possible/probable.

[AZ-aware-load-balancing-scenario.png]

Storage service spanning multiple AZ

Figure shows a storage service which has 3 different components. A request router, worker nodes (lead, tail, and middle), and a load balancer (cross-zone load balancing = enabled).

This architecture results in data transfer across AZ about 8 times. This can add performance overhead as cross AZ latency is in single digit ms compared to communication within an AZ, where latency is sub ms (< 1 ms).

Improvements

• Switch from ALB to NLB (cross-zone disabled, static ip in each AZ)
• Create AZ specific DNS endpoints using the NLB network interface in AZ, like euw1-az1.foo.com
• Clients use the AZ specific DNS name above.
• This ties the clients to a specific AZ and reduce the benefit of using multiple AZ.
• Implement a client library to include logic for retries with exponential backoff, circuit breakers etc for cases when an AZ fails. The clients should be able to switch to a healthy AZ.
• Implement service discovery to reduce the need of LB, clients can directly connect to their AZ specific request routers (see figure above).

Full post

References

• https://community.aws/posts/improving-availability-and-performance-with-multi-az-architecture

• https://aws.amazon.com/blogs/networking-and-content-delivery/centralized-inspection-architecture-with-aws-gateway-load-balancer-and-aws-transit-gateway/

• https://aws.amazon.com/builders-library/static-stability-using-availability-zones/