Server-Side Discovery: Load Balancer-Based Service Routing

In a microservices architecture, something has to figure out where requests should go. Every service instance has its own IP address, and those addresses change as containers spin up and crash. Figuring out the current location of healthy service instances is the service discovery problem.

If you are coming from the client-side approach, check out Client-Side Discovery for the alternative pattern. Many systems also rely on a Service Registry to track where instances live.

Server-side discovery moves the routing logic out of your clients and into a centralized component. Instead of clients querying a registry and deciding which address to call, they send requests to a known endpoint. A load balancer or reverse proxy sits in the middle, knows about available instances, and handles the routing.

This approach makes clients simpler. They skip the discovery libraries, skip the caching logic, skip the reconnection handling when instances change. The infrastructure takes care of all of that.

Introduction

The flow goes like this: a client sends a request to a well-known address, the load balancer receives it, looks up which service instances are currently available, picks one based on the configured algorithm, and forwards the request there.

graph TD
    Client[Client Application] --> LB[Load Balancer]
    LB --> Registry[Service Registry]
    LB --> S1[Service Instance 1]
    LB --> S2[Service Instance 2]
    LB --> S3[Service Instance 3]
    Registry -.->|health checks| S1
    Registry -.->|health checks| S2
    Registry -.->|health checks| S3

The load balancer maintains a connection to the service registry, either pulling data periodically or receiving updates when instances register and deregister. It uses this information to keep its routing table current.

Health checks are critical here. The registry or the load balancer itself periodically pings instances to verify they are still responding correctly. Unhealthy instances get removed from the routing pool automatically.

Load Balancer as the Discovery Point

Traditional load balancers like HAProxy, NGINX, or cloud-provided options (AWS ALB, GCP Cloud Load Balancer) fit the server-side discovery role naturally. You configure them with a virtual IP for a service, and they route requests to healthy backends. For more on load balancing fundamentals, see Load Balancing.

The service registry keeps track of available instances. When an instance starts, it registers itself with the registry. When it shuts down cleanly, it deregisters. If it crashes, health checks detect the failure and the registry marks it as unavailable.

Your client code does not care about any of that. It connects to the load balancer address and sends requests. The load balancer decides which backend gets the traffic.

graph LR
    A[Client] -->|1 request| B[Load Balancer]
    B -->|2 lookup| C[Service Registry]
    C -->|3 healthy instances| B
    B -->|4 forward| D[Selected Instance]

This decoupling has a practical benefit: clients become thinner. They do not need to know the location of every service, just the load balancer addresses for the services they use. Service addresses can change without touching client configuration.

For systems already using load balancers for traffic distribution, adding service discovery on top requires minimal new infrastructure.

Ingress Controllers in Kubernetes

Kubernetes ingress controllers implement server-side discovery for HTTP traffic. An ingress controller watches for Ingress resources in the cluster. When you create an Ingress that routes traffic to a service, the controller configures itself to forward requests to the appropriate pod endpoints. For a full walkthrough of Kubernetes concepts, see Kubernetes.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-ingress
spec:
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /users
            pathType: Prefix
            backend:
              service:
                name: user-service
                port:
                  number: 80
          - path: /orders
            pathType: Prefix
            backend:
              service:
                name: order-service
                port:
                  number: 80

The ingress controller handles load balancing across all pods backing a service. It uses endpoints to track which pods exist and whether they are ready to receive traffic. When you scale a deployment, the Ingress automatically routes to the new pods once they become healthy.

Most ingress controllers also handle TLS termination, request rewriting, and canary deployments. They are the main entry point for external traffic into a Kubernetes cluster, solving the discovery problem for north-south traffic.

For east-west traffic within the cluster, Kubernetes services provide similar functionality through cluster IPs. The kube-proxy component handles routing, but it works at the network level rather than the HTTP level.

Python Implementation: NGINX Configuration Generator

In server-side discovery, the load balancer needs to dynamically update its configuration as service instances change:

import requests
import time
import json
from typing import Dict, List

class NginxConfigGenerator:
    """Generates NGINX upstream configuration from service registry."""

    def __init__(self, registry_url: str, nginx_config_path: str):
        self.registry_url = registry_url
        self.nginx_config_path = nginx_config_path

    def fetch_instances(self, service_name: str) -> List[Dict]:
        """Fetch healthy instances from service registry."""
        try:
            response = requests.get(
                f"{self.registry_url}/services/{service_name}/instances",
                timeout=5
            )
            response.raise_for_status()
            return [
                inst for inst in response.json()
                if inst.get("healthy", True)
            ]
        except requests.RequestException:
            return []

    def generate_upstream_block(self, service_name: str, instances: List[Dict]) -> str:
        """Generate NGINX upstream block for a service."""
        lines = [f"upstream {service_name} {{"]
        lines.append("    least_conn;")
        for inst in instances:
            host = inst.get("host", inst.get("ip", "127.0.0.1"))
            port = inst.get("port", 8080)
            lines.append(f"    server {host}:{port};")
        lines.append("}")
        return "\n".join(lines)

    def generate_location_block(self, service_name: str, public_path: str) -> str:
        """Generate NGINX location block routing to upstream."""
        return f"""
location {public_path} {{
    proxy_pass http://{service_name};
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}}
"""

    def update_nginx_config(self, services: Dict[str, str]):
        """Update NGINX configuration for all services.

        Args:
            services: Dict mapping service_name to public URL path
        """
        config_parts = ["# Auto-generated upstream blocks"]

        for service_name, public_path in services.items():
            instances = self.fetch_instances(service_name)
            if instances:
                config_parts.append(self.generate_upstream_block(service_name, instances))
                config_parts.append(self.generate_location_block(service_name, public_path))

        config_content = "\n\n".join(config_parts)

        with open(self.nginx_config_path, "w") as f:
            f.write(config_content)

        print(f"Updated NGINX config with {len(services)} services")


class LoadBalancerHealthMonitor:
    """Monitors backend health and removes unhealthy instances."""

    def __init__(self, upstream_config_path: str):
        self.upstream_config_path = upstream_config_path

    def check_backend_health(self, host: str, port: int, path: str = "/health") -> bool:
        """Check if a backend is healthy."""
        try:
            response = requests.get(
                f"http://{host}:{port}{path}",
                timeout=3
            )
            return response.status_code == 200
        except requests.RequestException:
            return False

    def get_unhealthy_backends(self, config_path: str) -> List[Dict]:
        """Parse NGINX config and check health of all backends."""
        # Simplified parsing - in production use nginx -t and parse output
        unhealthy = []
        # Implementation would parse config and check each backend
        return unhealthy

AWS ALB and NLB Integration Patterns

Amazon Web Services offers two main load balancing options for service discovery: the Application Load Balancer (ALB) and the Network Load Balancer (NLB). Both can serve as discovery points in a server-side architecture.

The ALB works at layer 7 and understands HTTP semantics. You can route based on URL paths, host headers, or query parameters. Target groups let you group instances or IP addresses behind the ALB. As you register and deregister targets, the ALB updates its routing automatically.

graph TD
    User[Internet User] --> ALB[AWS ALB]
    ALB --> TG1[Target Group: us-east-1]
    ALB --> TG2[Target Group: eu-west-1]
    TG1 --> I1[Instance]
    TG1 --> I2[Instance]
    TG2 --> I3[Instance]
    TG2 --> I4[Instance]

The NLB works at layer 4 and handles TCP/UDP traffic. It has lower latency and higher throughput than the ALB. For non-HTTP protocols or when raw performance matters most, the NLB is the better choice.

AWS also provides Amazon ECS Service Discovery and AWS Cloud Map for registering service instances. These integrate with ALB and NLB through target group registration. Your services register their IP addresses and ports with Cloud Map, and the load balancers automatically route to registered instances.

This tight integration between discovery and load balancing is a real advantage of cloud-managed solutions. You get health checking, routing, and registration without running your own infrastructure.

Service Mesh as a Discovery Layer

Service meshes like Istio and Linkerd implement server-side discovery through sidecar proxies. Every service instance gets a proxy sidecar that intercepts all incoming and outgoing traffic. The control plane manages routing rules, and the mesh handles service discovery without your application noticing. Read more in Service Mesh.

graph TD
    subgraph Cluster
        Client[Client Pod] --> Sidecar1[Envoy Sidecar]
        Sidecar1 --> Mesh[Service Mesh Control Plane]
        Sidecar1 --> S1[Service A]
        Sidecar1 --> S2[Service B]
        Sidecar2[Envoy Sidecar] --> S2
    end
    Mesh -.->|configures| Sidecar1
    Mesh -.->|configures| Sidecar2

The client sends requests to a logical service name. The sidecar intercepts the request, queries the control plane for available endpoints, and forwards to one of them. From the developer’s perspective, calling another service works exactly like making a regular network call.

Service meshes give you more than basic load balancers. They support circuit breaking, retries with budgets, traffic shifting for canary deployments, and mTLS between services. The cost is added complexity and resource overhead from running sidecar proxies on every node.

If you are already running Kubernetes, a service mesh fits naturally into the discovery picture. If you are on VMs or want simpler infrastructure, a traditional load balancer setup might be more appropriate.

Advantages of Server-Side Discovery

Client simplicity is the big win. Clients connect to a fixed address and do not worry about where services live. They skip discovery libraries, skip caching registry data, skip retry logic for registry unavailability. This makes client code easier to write and maintain.

Centralized routing logic means you can change how traffic flows without updating dozens of client applications. Want to shift 10% of traffic to a new version? Update the load balancer configuration. With client-side discovery, you would need to deploy updated clients to every service that calls that endpoint.

Consistency across clients follows from this. Every client uses the same routing algorithm, the same health check configuration, the same failover behavior. You get uniform behavior without relying on each team getting discovery right.

Observability improves when routing happens in a central place. You see all traffic through the load balancer, making it easier to spot anomalies, measure latency, and debug issues. Distributed tracing still helps, but you have a single point for metrics.

Disadvantages of Server-Side Discovery

Server-side discovery has real drawbacks.

An additional network hop adds latency. Every request goes through the load balancer, which then forwards to a backend. For low-latency systems, this overhead matters. Direct client-to-service communication cuts out the middleman.

The load balancer can become a bottleneck or a single point of failure. If the load balancer itself goes down, no requests get through. High availability configurations mitigate this, but they add complexity. You need redundant load balancers, floating IP addresses, and health checking for the load balancers themselves.

Operational complexity increases. You are now operating infrastructure that clients depend on. That infrastructure needs monitoring, capacity planning, and incident response. For small teams, this overhead can outweigh the benefits.

Less flexibility for clients matters in some scenarios. If you need very specific routing behavior that does not fit the load balancer model, you end up fighting the tooling. Client-side discovery lets each client implement exactly the logic it needs.

Comparing with Client-Side Discovery

Client-side discovery flips the model. Instead of sending requests to a load balancer, clients query a service registry directly, cache the results, and pick an instance themselves. See Client-Side Discovery for a detailed comparison.

graph TD
    Client[Client Application] --> Registry[Service Registry]
    Registry --> Client
    Client --> S1[Service Instance 1]
    Client --> S2[Service Instance 2]
    Client --> S3[Service Instance 3]

Netflix’s Eureka works this way. Services register with Eureka, and clients poll Eureka periodically to get the current list of instances. The client uses a round-robin or random algorithm to pick one.

Client-side discovery removes the load balancer hop, which can reduce latency. It also means there is no single component that blocks all traffic if it fails.

But client-side discovery puts more burden on each client. Libraries like Eureka client need to be integrated into every service. Caching logic needs to handle staleness. Reconnection logic needs to handle registry outages gracefully. Different clients might implement the same logic differently, leading to inconsistent behavior.

Server-side discovery centralizes that complexity. You write the routing logic once in the load balancer and let all clients benefit. The cost is accepting the load balancer as a required dependency.

For simple systems with few clients, client-side discovery might add unnecessary components. For large systems with many teams and hundreds of services, server-side discovery provides consistency and reduces per-client complexity.

When to Use / When Not to Use

When to Use Server-Side Discovery

Server-side discovery works well in these scenarios:

• Kubernetes environments where ingress controllers and service meshes provide built-in server-side discovery
• Multi-language service ecosystems where you want consistent routing without maintaining client libraries in each language
• Centralized policy enforcement where you need uniform handling of canary deployments, blue-green releases, and traffic shaping
• Operational teams handling routing where dedicated infrastructure teams can manage load balancers and proxies
• Standard HTTP services where layer 7 load balancing with URL-based routing adds value

When Not to Use Server-Side Discovery

Server-side discovery has trade-offs. Consider alternatives when:

• Latency is critical - every request goes through an additional hop (load balancer to service instead of client to service)
• You have single-language microservices - client-side discovery may reduce infrastructure complexity
• Small teams with simple needs - the operational overhead of managing load balancers may not be justified
• High-throughput low-latency systems - the load balancer can become a bottleneck under extreme load
• You need per-request routing logic - client-side discovery can make more nuanced routing decisions based on local state

Decision Flow

graph TD
    A[Service Discovery Approach] --> B{Running Kubernetes?}
    B -->|Yes| C[Ingress Controller / Service Mesh]
    B -->|No| D{AWS Environment?}
    D -->|Yes| E[ALB/NLB + Cloud Map]
    D -->|No| F{Multi-Language Services?}
    F -->|Yes| C
    F -->|No| G{Latency Critical?}
    G -->|Yes| H[Client-Side Discovery]
    G -->|No| I{Team Manages LB?}
    I -->|Yes| C
    I -->|No| H

Quick Recap Checklist

• Server-side discovery uses load balancers and reverse proxies to handle routing, keeping clients simple
• The load balancer queries the service registry and routes requests to healthy instances
• In Kubernetes, ingress controllers provide HTTP-level routing; service meshes add east-west traffic management
• On AWS, ALB/NLB combined with Cloud Map or ECS Service Discovery provides managed server-side discovery
• Advantages: simpler clients, centralized policy enforcement, consistent routing behavior across all services
• Disadvantages: additional network hop adds latency, load balancer can become a bottleneck or SPOF
• Combine with health checks and high availability configurations for resilient routing

Real-world Failure Scenarios

Scenario	What Happens	Root Cause	Mitigation
Load balancer SPOF	All traffic stops	Single LB instance fails	Run HA configuration with floating IP
Registry out of sync	LB routes to terminated instances	Registry update not propagated	Increase health check frequency
Config reload race	New connections fail during reload	NGINX reloads drop connections	Use connection draining
Target group limit	Cannot register new instances	AWS limit on targets per group	Pre-configure larger limits
Cross-zone routing	Unnecessary latency and cost	AZ failure forces cross-zone	Configure AZ affinity

---

Trade-off Comparison: Server-Side Discovery Approaches

Aspect	NGINX/HAProxy	AWS ALB/NLB	Envoy Proxy	Kubernetes Ingress
Protocol Support	HTTP/TCP	HTTP/TCP/UDP	HTTP/TCP/gRPC	HTTP/HTTPS
Service Discovery Integration	Via template	Native Cloud Map	EDS API	Native K8s API
Load Balancing Algorithms	Basic (RR, leastconn)	3 algorithms	Many	Basic
Health Checking	Passive + active	Active	Active	Active
mTLS Support	Via config	Native	Native	Via annotations
Circuit Breaking	No	No	Yes	No
Operational Complexity	Medium	Low	High	Low
Cost	Software cost (or free)	AWS usage fees	Open source	Included in K8s
Best For	Traditional deployments	AWS-native deployments	Service mesh	Kubernetes deployments

For continued learning, explore the Microservices Architecture Roadmap and System Design fundamentals.

---

Interview Questions

Server-side discovery typically uses reverse proxies to handle traffic routing. Understanding the options helps you choose the right approach.

NGINX as a Discovery Point

NGINX can dynamically update its upstream configuration based on registry data:

# Static configuration with Consul template
upstream backend {
    least_conn;
    server 10.0.1.10:8080;
    server 10.0.1.11:8080;
    server 10.0.1.12:8080;
}

# Consul template regenerates this block when instances change

The typical flow: Consul template watches the service registry and rewrites the NGINX config when instances change, then reloads NGINX.

Envoy as a Service Proxy

Envoy provides sophisticated server-side load balancing with built-in service discovery:

static_resources:
  listeners:
    - address:
        socket_address:
          address: 0.0.0.0
          port_value: 8080
      filter_chains:
        - filters:
            - name: envoy.filters.network.http_connection_manager
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                route_config:
                  name: service_route
                clusters:
                  - name: my_service
                    type: EDS # External Discovery Service
                    lb_policy: LEAST_REQUEST
                    health_checks:
                      - timeout: 5s
                        interval: 10s
                        unhealthy_threshold: 3
                        healthy_threshold: 2

Envoy’s EDS (Endpoint Discovery Service) integrates with service registries to dynamically update cluster membership.

AWS ALB Target Groups

ALB integrates with AWS service discovery for automatic target registration:

# Register targets with ALB via target group
aws elbv2 register-targets \
    --target-group-arn arn:aws:elasticloadbalancing:us-east-1:123456789:targetgroup/my-service/abc123 \
    --targets "Id=10.0.1.10,Port=8080" "Id=10.0.1.11,Port=8080"

Cloud Map provides the registry, and ALB automatically routes to registered targets.

Service Router Patterns

A service router sits in front of multiple services and routes based on request attributes:

class ServiceRouter:
    def __init__(self, registry):
        self.registry = registry

    def route(self, request):
        path = request.path
        # Look up which service handles this path
        service_name = self.routes.get(path)
        if not service_name:
            return 404

        instances = self.registry.get_healthy(service_name)
        if not instances:
            return 503

        # Server-side load balancing
        selected = self.load_balancer.select(instances)
        return self.forward(request, selected)

Health Check Integration

Server-side discovery requires the proxy to perform health checks:

Check Type	What It Verifies	Configuration
TCP connect	Port accepting connections	host:port, interval
HTTP GET	Health endpoint returns 200	URL, expected status
HTTPS/TLS	Certificate valid, service responding	TLS handshake
Custom script	Application-specific health	script path

The proxy marks an instance unhealthy after consecutive failures and stops routing traffic until it passes health checks again.

---

Further Reading

• Client-Side Discovery — Alternative discovery pattern where clients query the registry directly
• Service Registry — Core component that tracks service instance locations
• Load Balancing — Fundamentals of distributing traffic across instances
• Service Mesh — Sidecar proxy-based discovery for east-west traffic
• Kubernetes — Container orchestration with built-in service discovery concepts
• API Gateway — Combines server-side discovery with API management features
• Microservices Architecture Roadmap — Comprehensive learning path for microservices
• AWS Documentation: Application Load Balancer and Network Load Balancer
• Envoy Proxy Documentation — Service proxy with advanced load balancing
• Istio Documentation — Service mesh implementation with built-in discovery

Conclusion

Server-side discovery centralizes routing logic in load balancers, reverse proxies, and service meshes, simplifying client implementations at the cost of an additional network hop. The load balancer queries the service registry on behalf of clients and handles retry logic, health routing, and policy enforcement in one place.

Ingress controllers in Kubernetes handle north-south traffic discovery natively, while service meshes manage east-west traffic between services. AWS ALB/NLB integrated with Cloud Map provides managed server-side discovery without building your own infrastructure.

The tradeoff: server-side discovery adds latency through the intermediary hop and creates a potential single point of failure. Mitigate both with high availability configurations, health check integration, and appropriately aggressive timeouts.

For most microservice architectures, server-side discovery through an API gateway or service mesh strikes the right balance. Client-side discovery suits ultra-low-latency paths where every millisecond matters and your team can manage the added complexity.