Server-Side Discovery: Load Balancer-Based Service Routing

In a microservices architecture, something has to figure out where requests should go. Every service instance has its own IP address, and those addresses change as containers spin up and crash. Figuring out the current location of healthy service instances is the service discovery problem.

If you are coming from the client-side approach, check out Client-Side Discovery for the alternative pattern. Many systems also rely on a Service Registry to track where instances live.

Server-side discovery moves the routing logic out of your clients and into a centralized component. Instead of clients querying a registry and deciding which address to call, they send requests to a known endpoint. A load balancer or reverse proxy sits in the middle, knows about available instances, and handles the routing.

This approach makes clients simpler. They skip the discovery libraries, skip the caching logic, skip the reconnection handling when instances change. The infrastructure takes care of all of that.

How Server-Side Discovery Works

The flow goes like this: a client sends a request to a well-known address, the load balancer receives it, looks up which service instances are currently available, picks one based on the configured algorithm, and forwards the request there.

graph TD
    Client[Client Application] --> LB[Load Balancer]
    LB --> Registry[Service Registry]
    LB --> S1[Service Instance 1]
    LB --> S2[Service Instance 2]
    LB --> S3[Service Instance 3]
    Registry -.->|health checks| S1
    Registry -.->|health checks| S2
    Registry -.->|health checks| S3

The load balancer maintains a connection to the service registry, either pulling data periodically or receiving updates when instances register and deregister. It uses this information to keep its routing table current.

Health checks are critical here. The registry or the load balancer itself periodically pings instances to verify they are still responding correctly. Unhealthy instances get removed from the routing pool automatically.

Load Balancer as the Discovery Point

Traditional load balancers like HAProxy, NGINX, or cloud-provided options (AWS ALB, GCP Cloud Load Balancer) fit the server-side discovery role naturally. You configure them with a virtual IP for a service, and they route requests to healthy backends. For more on load balancing fundamentals, see Load Balancing.

The service registry keeps track of available instances. When an instance starts, it registers itself with the registry. When it shuts down cleanly, it deregisters. If it crashes, health checks detect the failure and the registry marks it as unavailable.

Your client code does not care about any of that. It connects to the load balancer address and sends requests. The load balancer decides which backend gets the traffic.

graph LR
    A[Client] -->|1 request| B[Load Balancer]
    B -->|2 lookup| C[Service Registry]
    C -->|3 healthy instances| B
    B -->|4 forward| D[Selected Instance]

This decoupling has a practical benefit: clients become thinner. They do not need to know the location of every service, just the load balancer addresses for the services they use. Service addresses can change without touching client configuration.

For systems already using load balancers for traffic distribution, adding service discovery on top requires minimal new infrastructure.

Ingress Controllers in Kubernetes

Kubernetes ingress controllers implement server-side discovery for HTTP traffic. An ingress controller watches for Ingress resources in the cluster. When you create an Ingress that routes traffic to a service, the controller configures itself to forward requests to the appropriate pod endpoints. For a full walkthrough of Kubernetes concepts, see Kubernetes.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-ingress
spec:
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /users
            pathType: Prefix
            backend:
              service:
                name: user-service
                port:
                  number: 80
          - path: /orders
            pathType: Prefix
            backend:
              service:
                name: order-service
                port:
                  number: 80

The ingress controller handles load balancing across all pods backing a service. It uses endpoints to track which pods exist and whether they are ready to receive traffic. When you scale a deployment, the Ingress automatically routes to the new pods once they become healthy.

Most ingress controllers also handle TLS termination, request rewriting, and canary deployments. They are the main entry point for external traffic into a Kubernetes cluster, solving the discovery problem for north-south traffic.

For east-west traffic within the cluster, Kubernetes services provide similar functionality through cluster IPs. The kube-proxy component handles routing, but it works at the network level rather than the HTTP level.

Python Implementation: NGINX Configuration Generator

In server-side discovery, the load balancer needs to dynamically update its configuration as service instances change:

import requests
import time
import json
from typing import Dict, List

class NginxConfigGenerator:
    """Generates NGINX upstream configuration from service registry."""

    def __init__(self, registry_url: str, nginx_config_path: str):
        self.registry_url = registry_url
        self.nginx_config_path = nginx_config_path

    def fetch_instances(self, service_name: str) -> List[Dict]:
        """Fetch healthy instances from service registry."""
        try:
            response = requests.get(
                f"{self.registry_url}/services/{service_name}/instances",
                timeout=5
            )
            response.raise_for_status()
            return [
                inst for inst in response.json()
                if inst.get("healthy", True)
            ]
        except requests.RequestException:
            return []

    def generate_upstream_block(self, service_name: str, instances: List[Dict]) -> str:
        """Generate NGINX upstream block for a service."""
        lines = [f"upstream {service_name} {{"]
        lines.append("    least_conn;")
        for inst in instances:
            host = inst.get("host", inst.get("ip", "127.0.0.1"))
            port = inst.get("port", 8080)
            lines.append(f"    server {host}:{port};")
        lines.append("}")
        return "\n".join(lines)

    def generate_location_block(self, service_name: str, public_path: str) -> str:
        """Generate NGINX location block routing to upstream."""
        return f"""
location {public_path} {{
    proxy_pass http://{service_name};
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}}
"""

    def update_nginx_config(self, services: Dict[str, str]):
        """Update NGINX configuration for all services.

        Args:
            services: Dict mapping service_name to public URL path
        """
        config_parts = ["# Auto-generated upstream blocks"]

        for service_name, public_path in services.items():
            instances = self.fetch_instances(service_name)
            if instances:
                config_parts.append(self.generate_upstream_block(service_name, instances))
                config_parts.append(self.generate_location_block(service_name, public_path))

        config_content = "\n\n".join(config_parts)

        with open(self.nginx_config_path, "w") as f:
            f.write(config_content)

        print(f"Updated NGINX config with {len(services)} services")


class LoadBalancerHealthMonitor:
    """Monitors backend health and removes unhealthy instances."""

    def __init__(self, upstream_config_path: str):
        self.upstream_config_path = upstream_config_path

    def check_backend_health(self, host: str, port: int, path: str = "/health") -> bool:
        """Check if a backend is healthy."""
        try:
            response = requests.get(
                f"http://{host}:{port}{path}",
                timeout=3
            )
            return response.status_code == 200
        except requests.RequestException:
            return False

    def get_unhealthy_backends(self, config_path: str) -> List[Dict]:
        """Parse NGINX config and check health of all backends."""
        # Simplified parsing - in production use nginx -t and parse output
        unhealthy = []
        # Implementation would parse config and check each backend
        return unhealthy

AWS ALB and NLB Integration Patterns

Amazon Web Services offers two main load balancing options for service discovery: the Application Load Balancer (ALB) and the Network Load Balancer (NLB). Both can serve as discovery points in a server-side architecture.

The ALB works at layer 7 and understands HTTP semantics. You can route based on URL paths, host headers, or query parameters. Target groups let you group instances or IP addresses behind the ALB. As you register and deregister targets, the ALB updates its routing automatically.

graph TD
    User[Internet User] --> ALB[AWS ALB]
    ALB --> TG1[Target Group: us-east-1]
    ALB --> TG2[Target Group: eu-west-1]
    TG1 --> I1[Instance]
    TG1 --> I2[Instance]
    TG2 --> I3[Instance]
    TG2 --> I4[Instance]

The NLB works at layer 4 and handles TCP/UDP traffic. It has lower latency and higher throughput than the ALB. For non-HTTP protocols or when raw performance matters most, the NLB is the better choice.

AWS also provides Amazon ECS Service Discovery and AWS Cloud Map for registering service instances. These integrate with ALB and NLB through target group registration. Your services register their IP addresses and ports with Cloud Map, and the load balancers automatically route to registered instances.

This tight integration between discovery and load balancing is a real advantage of cloud-managed solutions. You get health checking, routing, and registration without running your own infrastructure.

Service Mesh as a Discovery Layer

Service meshes like Istio and Linkerd implement server-side discovery through sidecar proxies. Every service instance gets a proxy sidecar that intercepts all incoming and outgoing traffic. The control plane manages routing rules, and the mesh handles service discovery without your application noticing. Read more in Service Mesh.

graph TD
    subgraph Cluster
        Client[Client Pod] --> Sidecar1[Envoy Sidecar]
        Sidecar1 --> Mesh[Service Mesh Control Plane]
        Sidecar1 --> S1[Service A]
        Sidecar1 --> S2[Service B]
        Sidecar2[Envoy Sidecar] --> S2
    end
    Mesh -.->|configures| Sidecar1
    Mesh -.->|configures| Sidecar2

The client sends requests to a logical service name. The sidecar intercepts the request, queries the control plane for available endpoints, and forwards to one of them. From the developer’s perspective, calling another service works exactly like making a regular network call.

Service meshes give you more than basic load balancers. They support circuit breaking, retries with budgets, traffic shifting for canary deployments, and mTLS between services. The cost is added complexity and resource overhead from running sidecar proxies on every node.

If you are already running Kubernetes, a service mesh fits naturally into the discovery picture. If you are on VMs or want simpler infrastructure, a traditional load balancer setup might be more appropriate.

Advantages of Server-Side Discovery

Client simplicity is the big win. Clients connect to a fixed address and do not worry about where services live. They skip discovery libraries, skip caching registry data, skip retry logic for registry unavailability. This makes client code easier to write and maintain.

Centralized routing logic means you can change how traffic flows without updating dozens of client applications. Want to shift 10% of traffic to a new version? Update the load balancer configuration. With client-side discovery, you would need to deploy updated clients to every service that calls that endpoint.

Consistency across clients follows from this. Every client uses the same routing algorithm, the same health check configuration, the same failover behavior. You get uniform behavior without relying on each team getting discovery right.

Observability improves when routing happens in a central place. You see all traffic through the load balancer, making it easier to spot anomalies, measure latency, and debug issues. Distributed tracing still helps, but you have a single point for metrics.

Disadvantages of Server-Side Discovery

Server-side discovery has real drawbacks.

An additional network hop adds latency. Every request goes through the load balancer, which then forwards to a backend. For low-latency systems, this overhead matters. Direct client-to-service communication cuts out the middleman.

The load balancer can become a bottleneck or a single point of failure. If the load balancer itself goes down, no requests get through. High availability configurations mitigate this, but they add complexity. You need redundant load balancers, floating IP addresses, and health checking for the load balancers themselves.

Operational complexity increases. You are now operating infrastructure that clients depend on. That infrastructure needs monitoring, capacity planning, and incident response. For small teams, this overhead can outweigh the benefits.

Less flexibility for clients matters in some scenarios. If you need very specific routing behavior that does not fit the load balancer model, you end up fighting the tooling. Client-side discovery lets each client implement exactly the logic it needs.

Comparing with Client-Side Discovery

Client-side discovery flips the model. Instead of sending requests to a load balancer, clients query a service registry directly, cache the results, and pick an instance themselves. See Client-Side Discovery for a detailed comparison.

graph TD
    Client[Client Application] --> Registry[Service Registry]
    Registry --> Client
    Client --> S1[Service Instance 1]
    Client --> S2[Service Instance 2]
    Client --> S3[Service Instance 3]

Netflix’s Eureka works this way. Services register with Eureka, and clients poll Eureka periodically to get the current list of instances. The client uses a round-robin or random algorithm to pick one.

Client-side discovery removes the load balancer hop, which can reduce latency. It also means there is no single component that blocks all traffic if it fails.

But client-side discovery puts more burden on each client. Libraries like Eureka client need to be integrated into every service. Caching logic needs to handle staleness. Reconnection logic needs to handle registry outages gracefully. Different clients might implement the same logic differently, leading to inconsistent behavior.

Server-side discovery centralizes that complexity. You write the routing logic once in the load balancer and let all clients benefit. The cost is accepting the load balancer as a required dependency.

For simple systems with few clients, client-side discovery might add unnecessary components. For large systems with many teams and hundreds of services, server-side discovery provides consistency and reduces per-client complexity.

When to Use / When Not to Use

When to Use Server-Side Discovery

Server-side discovery works well in these scenarios:

• Kubernetes environments where ingress controllers and service meshes provide built-in server-side discovery
• Multi-language service ecosystems where you want consistent routing without maintaining client libraries in each language
• Centralized policy enforcement where you need uniform handling of canary deployments, blue-green releases, and traffic shaping
• Operational teams handling routing where dedicated infrastructure teams can manage load balancers and proxies
• Standard HTTP services where layer 7 load balancing with URL-based routing adds value

When Not to Use Server-Side Discovery

Server-side discovery has trade-offs. Consider alternatives when:

• Latency is critical - every request goes through an additional hop (load balancer to service instead of client to service)
• You have single-language microservices - client-side discovery may reduce infrastructure complexity
• Small teams with simple needs - the operational overhead of managing load balancers may not be justified
• High-throughput low-latency systems - the load balancer can become a bottleneck under extreme load
• You need per-request routing logic - client-side discovery can make more nuanced routing decisions based on local state

Decision Flow

graph TD
    A[Service Discovery Approach] --> B{Running Kubernetes?}
    B -->|Yes| C[Ingress Controller / Service Mesh]
    B -->|No| D{AWS Environment?}
    D -->|Yes| E[ALB/NLB + Cloud Map]
    D -->|No| F{Multi-Language Services?}
    F -->|Yes| C
    F -->|No| G{Latency Critical?}
    G -->|Yes| H[Client-Side Discovery]
    G -->|No| I{Team Manages LB?}
    I -->|Yes| C
    I -->|No| H

Quick Recap

• Server-side discovery uses load balancers and reverse proxies to handle routing, keeping clients simple
• The load balancer queries the service registry and routes requests to healthy instances
• In Kubernetes, ingress controllers provide HTTP-level routing; service meshes add east-west traffic management
• On AWS, ALB/NLB combined with Cloud Map or ECS Service Discovery provides managed server-side discovery
• Advantages: simpler clients, centralized policy enforcement, consistent routing behavior across all services
• Disadvantages: additional network hop adds latency, load balancer can become a bottleneck or SPOF
• Combine with health checks and high availability configurations for resilient routing

For continued learning, explore the Microservices Architecture Roadmap and System Design fundamentals.