Skip to content

architecture overview

Jump to bottom
Thomas Mangin edited this page Nov 10, 2025 · 1 revision

ExaBGP Architecture Overview

Purpose: Technical reference for understanding ExaBGP's internal architecture Audience: Documentation writers, contributors, advanced users Last Updated: 2025-11-09

High-Level Architecture

┌─────────────────────────────────────────────────────────────┐
│                        External Process                       │
│              (Python/Shell/Any Language Script)              │
│                                                               │
│         Reads STDIN ←──→ Writes STDOUT (Commands)           │
└────────────────────────────┬────────────────────────────────┘
                             │ STDIN/STDOUT Pipes
┌────────────────────────────┴────────────────────────────────┐
│                        ExaBGP Core                           │
│                                                               │
│  ┌──────────────┐  ┌────────────┐  ┌────────────────────┐  │
│  │   Process    │  │   Parser   │  │   BGP Protocol     │  │
│  │   Manager    │  │  (Commands)│  │   State Machine    │  │
│  └──────────────┘  └────────────┘  └────────────────────┘  │
│                                                               │
│  ┌──────────────┐  ┌────────────┐  ┌────────────────────┐  │
│  │   Encoder    │  │  Message   │  │   Neighbor         │  │
│  │  (JSON/Text) │  │  Builder   │  │   Management       │  │
│  └──────────────┘  └────────────┘  └────────────────────┘  │
└────────────────────────────┬────────────────────────────────┘
                             │ TCP/BGP
┌────────────────────────────┴────────────────────────────────┐
│                     BGP Neighbor (Router)                     │
│                                                               │
│              Receives/Sends BGP Protocol Messages            │
└───────────────────────────────────────────────────────────────┘

Core Principles

1. No RIB/FIB Manipulation

ExaBGP is a pure BGP protocol implementation:

  • Does NOT maintain routing tables (RIB)
  • Does NOT modify kernel routing tables (FIB)
  • Does NOT forward packets
  • ONLY speaks the BGP protocol

Rationale: Separation of concerns - ExaBGP handles protocol, external processes handle routing decisions.

2. API-First Design

Everything is controlled via API:

  • Configuration defines neighbors and processes
  • External processes send commands via STDOUT
  • ExaBGP sends BGP messages via STDIN
  • Simple, language-agnostic integration

3. External Process Model

Business logic lives outside ExaBGP:

  • Health checks → Python/Shell scripts
  • Route decisions → Application logic
  • Integration → Any programming language
  • ExaBGP → BGP protocol only

Component Architecture

Process Manager

Responsibilities:

  • Launch configured external processes
  • Monitor process health
  • Manage STDIN/STDOUT pipes
  • Handle process restarts on failure

Configuration:

process healthcheck {
    run /usr/local/bin/healthcheck.py;
    encoder json;
}

Process Lifecycle:

  1. ExaBGP reads configuration
  2. Forks external process
  3. Establishes STDIN/STDOUT pipes
  4. Monitors process (restarts on crash)
  5. Cleans up on ExaBGP shutdown

Command Parser

Responsibilities:

  • Read STDOUT from external processes
  • Parse text/JSON commands
  • Validate command syntax
  • Queue commands for execution

Supported Command Formats:

# Text format
announce route 203.0.113.0/24 next-hop self

# JSON format (future)
{"exabgp": "4.0", "type": "announce", "route": {...}}

Command Types:

  • Route announcements (announce route)
  • Route withdrawals (withdraw route)
  • FlowSpec rules (announce flow)
  • Control commands (shutdown, reload, restart)
  • Information queries (show adj-rib out)

BGP Protocol State Machine

Responsibilities:

  • Implement BGP FSM (RFC 4271)
  • Handle BGP session establishment
  • Process BGP messages (OPEN, UPDATE, KEEPALIVE, NOTIFICATION)
  • Manage timers (hold time, keepalive)
  • Validate BGP protocol correctness

BGP States:

Idle → Connect → OpenSent → OpenConfirm → Established

Message Types:

  1. OPEN: Session establishment, capability negotiation
  2. UPDATE: Route announcements/withdrawals
  3. NOTIFICATION: Error reporting, session termination
  4. KEEPALIVE: Session maintenance
  5. ROUTE-REFRESH: Request route re-advertisement

Message Builder

Responsibilities:

  • Construct BGP UPDATE messages from commands
  • Encode BGP attributes correctly
  • Support IPv4, IPv6, FlowSpec, VPN, etc.
  • Validate message format before sending

BGP UPDATE Structure:

+-----------------------------------------------+
| Withdrawn Routes Length (2 octets)            |
+-----------------------------------------------+
| Withdrawn Routes (variable)                   |
+-----------------------------------------------+
| Total Path Attribute Length (2 octets)        |
+-----------------------------------------------+
| Path Attributes (variable)                    |
+-----------------------------------------------+
| Network Layer Reachability Info (variable)    |
+-----------------------------------------------+

Supported Attributes:

  • ORIGIN (Type 1)
  • AS_PATH (Type 2)
  • NEXT_HOP (Type 3)
  • MULTI_EXIT_DISC (Type 4)
  • LOCAL_PREF (Type 5)
  • COMMUNITIES (Type 8)
  • MP_REACH_NLRI (Type 14) - IPv6, FlowSpec, VPN
  • MP_UNREACH_NLRI (Type 15)
  • EXTENDED_COMMUNITIES (Type 16)
  • LARGE_COMMUNITIES (Type 32)

Encoder (JSON/Text)

Responsibilities:

  • Format received BGP messages for external processes
  • Support text and JSON output formats
  • Send formatted messages to process STDIN

Text Format Example:

neighbor 192.0.2.1 192.0.2.2 received update route 203.0.113.0/24 \
  next-hop 192.0.2.1 origin igp as-path [65001 65002] med 100

JSON Format Example:

{
  "exabgp": "4.0.0",
  "time": 1234567890,
  "type": "update",
  "neighbor": {
    "address": {"local": "192.0.2.2", "peer": "192.0.2.1"},
    "asn": {"local": 65000, "peer": 65001}
  },
  "message": {
    "update": {
      "announce": {
        "ipv4 unicast": {
          "192.0.2.1": [{
            "nlri": "203.0.113.0/24",
            "attribute": {
              "origin": "igp",
              "as-path": [65001, 65002],
              "med": 100
            }
          }]
        }
      }
    }
  }
}

Neighbor Management

Responsibilities:

  • Maintain neighbor configurations
  • Manage TCP connections to peers
  • Track neighbor state
  • Apply per-neighbor policies
  • Handle BGP capabilities negotiation

Neighbor Configuration:

neighbor 192.0.2.1 {
    router-id 10.0.0.1;
    local-address 192.0.2.2;
    local-as 65000;
    peer-as 65001;

    family {
        ipv4 unicast;
        ipv6 unicast;
        ipv4 flow;
    }

    capability {
        graceful-restart;
        add-path receive;
        extended-message receive;
    }

    api {
        processes [healthcheck];
        receive {
            parsed;
            update;
            notification;
        }
    }
}

Data Flow

Outbound (Announce Route)

1. External Process
   └─> STDOUT: "announce route 203.0.113.0/24 next-hop self"

2. Process Manager
   └─> Reads STDOUT, forwards to Parser

3. Command Parser
   └─> Parses command, extracts route details

4. Message Builder
   └─> Constructs BGP UPDATE message
   └─> Adds attributes (ORIGIN, AS_PATH, NEXT_HOP, etc.)

5. BGP Protocol
   └─> Sends UPDATE message to neighbor via TCP

6. Router (Neighbor)
   └─> Receives UPDATE, installs route

Inbound (Receive Route)

1. Router (Neighbor)
   └─> Sends BGP UPDATE message via TCP

2. BGP Protocol
   └─> Receives UPDATE, validates message

3. Encoder
   └─> Formats message as JSON/Text

4. Process Manager
   └─> Writes to process STDIN

5. External Process
   └─> Reads STDIN, processes received route
   └─> Makes routing decisions, sends commands via STDOUT

Configuration Architecture

Configuration File Structure

# Process definitions
process <name> {
    run <command>;
    encoder <json|text>;
}

# Global template (optional)
template {
    neighbor <ip> {
        # Shared neighbor configuration
    }
}

# Neighbor definitions
neighbor <ip> {
    router-id <id>;
    local-address <ip>;
    local-as <asn>;
    peer-as <asn>;

    family {
        ipv4 unicast;
        ipv6 unicast;
        ipv4 flow;
        l2vpn vpls;
    }

    capability {
        graceful-restart;
        add-path receive;
    }

    api {
        processes [<name>];
        receive {
            parsed;
            update;
        }
    }

    # Static routes (optional)
    static {
        route 203.0.113.0/24 next-hop self;
    }
}

Configuration Loading

1. ExaBGP starts
   └─> Loads configuration file

2. Validates syntax
   └─> Checks neighbor parameters
   └─> Validates process definitions

3. Applies templates
   └─> Merges template with neighbor configs

4. Initializes components
   └─> Creates neighbor objects
   └─> Forks external processes

5. Establishes BGP sessions
   └─> Connects to neighbors
   └─> Negotiates capabilities

Address Family Support

IPv4 Unicast (AFI=1, SAFI=1)

  • Traditional IPv4 routing
  • Most common use case

IPv6 Unicast (AFI=2, SAFI=1)

  • IPv6 routing
  • Uses MP_REACH_NLRI attribute

IPv4/IPv6 FlowSpec (AFI=1/2, SAFI=133)

  • Traffic filtering rules
  • DDoS mitigation
  • ExaBGP is only open-source implementation

IPv4 MPLS VPN (AFI=1, SAFI=128)

  • L3VPN over MPLS
  • Route distinguisher support

L2VPN VPLS (AFI=25, SAFI=65)

  • Virtual Private LAN Service
  • Layer 2 VPN support

BGP-LS (AFI=16388, SAFI=71)

  • Link-State distribution
  • SDN integration

RT Constraint (AFI=1, SAFI=132)

  • Route Target filtering
  • VPN route optimization

Protocol Features

RFC Implementation (55+ RFCs)

Core Protocol:

  • RFC 4271: BGP-4
  • RFC 4760: Multiprotocol Extensions for BGP-4
  • RFC 5492: Capabilities Advertisement

Route Refresh:

  • RFC 2918: Route Refresh Capability

Communities:

  • RFC 1997: BGP Communities Attribute
  • RFC 4360: BGP Extended Communities
  • RFC 8092: BGP Large Communities

FlowSpec:

  • RFC 5575: Dissemination of Flow Specification Rules
  • RFC 8955: Dissemination of Flow Specification Rules (updated)

Graceful Restart:

  • RFC 4724: Graceful Restart Mechanism for BGP

Add-Path:

  • RFC 7911: Advertisement of Multiple Paths in BGP

See RFC-SUPPORT.md and RFC-Information.md for complete list

Capability Negotiation

During OPEN message exchange, ExaBGP negotiates:

  • Multiprotocol Extensions (IPv6, FlowSpec, VPN)
  • Route Refresh
  • Graceful Restart
  • 4-byte AS numbers
  • Add-Path
  • Extended Message (for large UPDATEs)

Threading Model

Single-Threaded Event Loop

ExaBGP uses single-threaded architecture:

  • Main event loop handles all I/O
  • Non-blocking sockets for BGP sessions
  • Select/poll for process STDOUT monitoring
  • Timer management for BGP keepalives

Advantages:

  • Simple, no race conditions
  • Low overhead
  • Predictable behavior

Considerations:

  • External processes must not block
  • Long-running health checks should use separate processes
  • Python GIL not a concern (no threading)

Security Architecture

BGP Session Security

TCP MD5 Authentication (RFC 2385):

neighbor 192.0.2.1 {
    md5-password "secret";
}

Process Isolation

  • External processes run as separate OS processes
  • Limited privilege escalation risk
  • Process crashes don't affect ExaBGP core
  • STDOUT/STDIN are isolated pipes

Input Validation

  • All BGP messages validated against RFCs
  • Command parser validates syntax
  • Attribute validation before message construction
  • Protection against malformed BGP messages

Performance Characteristics

Resource Usage

  • Memory: 20-50MB typical, scales with routes
  • CPU: Minimal, event-driven I/O
  • Network: TCP keepalives every 60s (configurable)

Scalability

  • Routes: 100K+ routes per instance
  • Neighbors: 100+ simultaneous neighbors
  • Processes: Multiple external processes supported
  • Speakers: 450+ speakers simulated from single host

Limits

  • Python GIL for CPU-intensive tasks
  • Network I/O bounded by TCP
  • External process performance dependent on implementation

Deployment Patterns

Pattern 1: Single Node

[ExaBGP + Health Check] ----BGP----> [Router]

Simple, single service announcement.

Pattern 2: Multiple Services

[ExaBGP + Multiple Processes] ----BGP----> [Router]

Multiple health checks, multiple service IPs.

Pattern 3: Route Server

[ExaBGP Nodes] ----BGP----> [Route Server] ----BGP----> [Edge Routers]

Scalable, with route server for path selection.

Pattern 4: Redundant ExaBGP

[ExaBGP Primary]   ----BGP----> [Router]
[ExaBGP Secondary] ----BGP----> [Router]

High availability for ExaBGP itself.

Pattern 5: Distributed

[ExaBGP Node 1] ----BGP----> [Router 1]
[ExaBGP Node 2] ----BGP----> [Router 2]
[ExaBGP Node 3] ----BGP----> [Router 3]

Geographic distribution, anycast.

Logging and Debugging

Log Levels

  • CRITICAL: Fatal errors
  • ERROR: Errors (session failures, etc.)
  • WARNING: Warnings (retries, etc.)
  • INFO: Informational (route changes)
  • DEBUG: Detailed debugging

Log Destinations

  • stdout: Console output
  • stderr: Error output
  • File: Specified log file path
  • Syslog: System logger

Debug Configuration

[exabgp.log]
all = true
destination = /var/log/exabgp.log
level = DEBUG

[exabgp.api]
ack = true
encoder = json

Comparison with Traditional BGP Implementations

Feature ExaBGP Quagga/FRR BIRD
RIB No Yes Yes
FIB No Yes Yes
API Full Limited Limited
Language Python C C
FlowSpec Full Partial Partial
Programmability High Low Medium
Resource Usage Low Medium Low
Primary Use Case API Control Router Daemon Router Daemon

Code Organization (Reference)

exabgp/
├── application/         # CLI and main application
├── bgp/                # BGP protocol implementation
│   ├── message/        # BGP message types
│   ├── neighbor/       # Neighbor management
│   └── peer.py         # BGP peer state machine
├── configuration/      # Configuration parsing
├── network/           # Network I/O
├── reactor/           # Event loop
├── rib/               # Adj-RIB-Out (not RIB!)
└── version.py         # Version info

Note: Despite rib/ directory name, ExaBGP only maintains Adj-RIB-Out (routes sent to neighbors), not a full RIB.

Extension Points

1. Custom Encoders

Add new output formats for external processes.

2. Custom Attributes

Support new BGP attributes via message builder.

3. Custom AFI/SAFI

Add new address family support.

4. Custom Health Checks

Unlimited flexibility via external processes.

5. Integration Libraries

Language-specific libraries for ExaBGP interaction.

Summary

ExaBGP's architecture is designed for:

  • Simplicity: External process model, STDIN/STDOUT API
  • Correctness: Pure BGP protocol implementation (55+ RFCs)
  • Flexibility: Language-agnostic integration
  • Separation: Protocol (ExaBGP) vs. Logic (external processes)
  • Scalability: Lightweight, event-driven, single-threaded

Key Insight: By separating BGP protocol from routing logic, ExaBGP enables application-layer network control without requiring BGP expertise or router access.

For Documentation Writers: Use this architecture overview to:

  1. Explain how ExaBGP works internally
  2. Justify design decisions in documentation
  3. Help users understand the process model
  4. Clarify the "no RIB/FIB" principle
  5. Reference when explaining advanced features

🏠 Home

🚀 Getting Started

🔧 API

🛡️ Use Cases

🌐 Address Families

⚙️ Configuration

🔍 Operations

📚 Reference

🔄 Migration

🌍 Community

🔗 External

👻 Ghost written by Claude (Anthropic AI)

Clone this wiki locally