-
Notifications
You must be signed in to change notification settings - Fork 459
Traffic Engineering
Programmatic BGP traffic control for network optimization
π£οΈ Application-driven traffic engineering - control routing paths dynamically with BGP
- Overview
- Traffic Engineering Concepts
- BGP Attributes for Traffic Control
- AS-PATH Prepending
- MED Manipulation
- Communities for Policy
- Local Preference
- Inbound Traffic Engineering
- Outbound Traffic Engineering
- Multi-Homing Scenarios
- Implementation Examples
- Best Practices
- Monitoring and Validation
- Troubleshooting
Traffic Engineering (TE) with ExaBGP enables programmatic control of routing paths using BGP attributes.
Traffic Engineering is the process of controlling how traffic flows through a network to:
- Optimize link utilization
- Avoid congestion
- Reduce latency
- Lower costs (prefer cheaper transit)
- Implement policy routing
- Enable graceful maintenance
Traditional approach:
Manual router configuration
β SSH to routers
β Configure static routes, route-maps, policies
β Error-prone, slow, not dynamic
ExaBGP approach:
Application-driven automation
β ExaBGP API announces routes with specific attributes
β Dynamic, automated, application-aware
β Can react to network conditions in real-time
BGP selects best path using this order:
- Highest Local Preference (outbound TE)
- Shortest AS-PATH (inbound TE via prepending)
- Lowest Origin Type (IGP < EGP < Incomplete)
- Lowest MED (inbound TE)
- Prefer eBGP over iBGP
- Lowest IGP metric to next-hop
- Lowest router ID
ExaBGP controls: Local-pref, AS-PATH, MED, Communities
Inbound Traffic: Traffic entering your network
- Controlled by attributes visible to upstream/peers
- Use: AS-PATH prepending, MED, communities
Outbound Traffic: Traffic leaving your network
- Controlled by attributes within your network
- Use: Local-preference, communities (internal)
| Attribute | Scope | Direction | Use Case |
|---|---|---|---|
| Local-Pref | iBGP | Outbound | Prefer specific exit point |
| AS-PATH | eBGP | Inbound | Make path less attractive |
| MED | eBGP | Inbound | Suggest preferred entry point |
| Communities | Both | Both | Policy tagging and signaling |
AS-PATH prepending adds extra AS numbers to make a path artificially longer, making it less preferred.
Example:
Normal announcement: AS-PATH = 65001
Prepended 2x: AS-PATH = 65001 65001 65001
Prepended 5x: AS-PATH = 65001 65001 65001 65001 65001 65001
Result: Upstream routers prefer shorter AS-PATH (normal announcement)
1. Traffic Balancing Between Multiple Upstreams
Your network (AS 65001) has two transit providers:
- ISP A (fast, expensive)
- ISP B (slower, cheaper)
Goal: Send most traffic via ISP A, only overflow to ISP B
Solution: Prepend AS-PATH on announcements to ISP B
2. Maintenance Draining
Need to perform maintenance on link to ISP A
β Prepend AS-PATH heavily on ISP A announcements
β Traffic shifts to ISP B
β Perform maintenance on ISP A with minimal traffic
β Remove prepending when maintenance complete
3. Geographic Traffic Steering
Datacenter A (US East): Announce normally
Datacenter B (US West): Prepend 3x for European peers
Result: European traffic prefers Datacenter A (shorter AS-PATH)
Basic AS-PATH prepending:
#!/usr/bin/env python3
"""
AS-PATH prepending for traffic engineering
"""
import sys
import time
PREFIX = "203.0.113.0/24"
LOCAL_AS = 65001
# Prepend configuration
# 0 = no prepending (normal announcement)
# 1 = prepend once (AS-PATH = 65001 65001)
# 3 = prepend 3 times (AS-PATH = 65001 65001 65001 65001)
PREPEND_COUNT = 0 # Set per peer/prefix
def build_as_path(prepend_count):
"""Build AS-PATH with prepending"""
if prepend_count == 0:
return str(LOCAL_AS)
else:
# Prepend LOCAL_AS multiple times
as_path = ' '.join([str(LOCAL_AS)] * (prepend_count + 1))
return as_path
time.sleep(2)
while True:
as_path = build_as_path(PREPEND_COUNT)
sys.stdout.write(
f"announce route {PREFIX} "
f"next-hop self "
f"as-path [ {as_path} ]\n"
)
sys.stdout.flush()
time.sleep(30)Automatically prepend when link is congested:
#!/usr/bin/env python3
"""
Dynamic AS-PATH prepending based on link utilization
Prepend more when link is congested
"""
import sys
import time
import subprocess
import re
PREFIX = "203.0.113.0/24"
LOCAL_AS = 65001
INTERFACE = "eth0" # Interface to monitor
def get_interface_utilization(interface):
"""
Get interface utilization percentage
Returns: 0-100 (percentage of capacity)
"""
try:
# Read interface stats
with open(f'/sys/class/net/{interface}/statistics/tx_bytes', 'r') as f:
tx_bytes_1 = int(f.read())
time.sleep(1)
with open(f'/sys/class/net/{interface}/statistics/tx_bytes', 'r') as f:
tx_bytes_2 = int(f.read())
# Calculate bytes per second
bps = (tx_bytes_2 - tx_bytes_1) * 8
# Assume 10 Gbps interface
capacity_bps = 10 * 1000 * 1000 * 1000
utilization = (bps / capacity_bps) * 100
return utilization
except:
return 0
def calculate_prepend_count(utilization):
"""
Calculate prepend count based on utilization
0-40%: No prepending (normal traffic)
40-60%: Prepend 1x (reduce incoming traffic slightly)
60-80%: Prepend 3x (reduce incoming traffic moderately)
80-100%: Prepend 5x (reduce incoming traffic heavily)
"""
if utilization < 40:
return 0
elif utilization < 60:
return 1
elif utilization < 80:
return 3
else:
return 5
time.sleep(2)
sys.stderr.write("[TE] AS-PATH prepending based on link utilization\n")
while True:
utilization = get_interface_utilization(INTERFACE)
prepend_count = calculate_prepend_count(utilization)
# Build AS-PATH
if prepend_count == 0:
as_path = str(LOCAL_AS)
else:
as_path = ' '.join([str(LOCAL_AS)] * (prepend_count + 1))
sys.stdout.write(
f"announce route {PREFIX} "
f"next-hop self "
f"as-path [ {as_path} ]\n"
)
sys.stdout.flush()
sys.stderr.write(
f"[TE] Utilization: {utilization:.1f}%, "
f"Prepend: {prepend_count}x, "
f"AS-PATH: {as_path}\n"
)
time.sleep(30)MED (Multi-Exit Discriminator) suggests to neighbors which entry point to use.
- Lower MED = Preferred entry point
- Higher MED = Less preferred
- MED only compared among routes from same neighbor AS
1. Multi-Homed with Same Provider
Your network connected to ISP in two locations:
- Connection A (New York)
- Connection B (San Francisco)
For prefix 203.0.113.0/24:
- Announce MED 100 via Connection A (preferred)
- Announce MED 200 via Connection B (backup)
Result: ISP sends traffic via Connection A
2. Load Distribution
Multiple links to same ISP:
- Link 1: MED 100 (10 Gbps)
- Link 2: MED 100 (10 Gbps) β Equal cost, traffic split
- Link 3: MED 150 (1 Gbps backup)
Static MED announcement:
#!/usr/bin/env python3
"""
MED-based traffic engineering
"""
import sys
import time
PREFIX = "203.0.113.0/24"
MED = 100 # Lower = preferred
time.sleep(2)
while True:
sys.stdout.write(
f"announce route {PREFIX} "
f"next-hop self "
f"med {MED}\n"
)
sys.stdout.flush()
time.sleep(30)Dynamic MED based on link cost:
#!/usr/bin/env python3
"""
Dynamic MED based on link cost/utilization
Announce lower MED on cheaper/less utilized links
"""
import sys
import time
PREFIX = "203.0.113.0/24"
# Link configuration
LINK_COST = 100 # Cost per Mbps (e.g., expensive=200, cheap=50)
def get_link_utilization():
"""Get current link utilization (0-100%)"""
# TODO: Implement actual monitoring
return 50
def calculate_med():
"""Calculate MED based on link cost and utilization"""
utilization = get_link_utilization()
# MED = base_cost + utilization_penalty
# Higher utilization β higher MED β less preferred
base_med = LINK_COST
utilization_penalty = int(utilization)
med = base_med + utilization_penalty
return med
time.sleep(2)
while True:
med = calculate_med()
sys.stdout.write(
f"announce route {PREFIX} "
f"next-hop self "
f"med {med}\n"
)
sys.stdout.flush()
sys.stderr.write(f"[TE] Announced with MED={med}\n")
time.sleep(30)Communities are tags attached to routes for policy enforcement.
Format: AS:VALUE (e.g., 65000:100)
Common uses:
- Signal routing preferences to providers
- Tag routes for filtering
- Implement complex policies
Major ISPs publish community-based policies:
Hurricane Electric (AS 6939):
6939:100 = Set local-pref 100 (default)
6939:150 = Set local-pref 150 (prefer this route)
6939:50 = Set local-pref 50 (depref this route)
6939:666 = Blackhole this prefix
Cogent (AS 174):
174:10 = Prepend 1x to all peers
174:20 = Prepend 2x to all peers
174:30 = Prepend 3x to all peers
174:70 = Do not announce to peers
Announce with communities for traffic control:
#!/usr/bin/env python3
"""
BGP communities for traffic engineering
"""
import sys
import time
PREFIX = "203.0.113.0/24"
# Communities configuration
# Example: Hurricane Electric communities
COMMUNITIES = [
"6939:150", # Prefer this route (higher local-pref at HE)
]
time.sleep(2)
while True:
# Build community string
community_str = ' '.join(COMMUNITIES)
sys.stdout.write(
f"announce route {PREFIX} "
f"next-hop self "
f"community [{community_str}]\n"
)
sys.stdout.flush()
time.sleep(30)Dynamic communities based on traffic patterns:
#!/usr/bin/env python3
"""
Dynamic community tagging based on DDoS detection
Automatically signal blackhole community when under attack
"""
import sys
import time
PREFIX = "203.0.113.0/24"
BLACKHOLE_COMMUNITY = "6939:666" # HE blackhole community
def is_under_ddos_attack():
"""
Check if prefix is under DDoS attack
TODO: Integrate with DDoS detection system
"""
# Placeholder - check traffic rate, packet patterns, etc.
return False
time.sleep(2)
sys.stderr.write("[TE] DDoS-aware community announcements\n")
while True:
if is_under_ddos_attack():
# Under attack - signal blackhole
sys.stdout.write(
f"announce route {PREFIX} "
f"next-hop self "
f"community [{BLACKHOLE_COMMUNITY}]\n"
)
sys.stderr.write(f"[TE] Under attack, blackholing {PREFIX}\n")
else:
# Normal operation - announce without blackhole
sys.stdout.write(
f"announce route {PREFIX} "
f"next-hop self\n"
)
sys.stdout.flush()
time.sleep(30)Local Preference controls outbound traffic (exit point selection).
- Higher local-pref = Preferred exit
- Default = 100
- iBGP only (not sent to eBGP peers)
Your network (AS 65001) has two ISPs:
- ISP A (AS 1000) - expensive, low latency
- ISP B (AS 2000) - cheap, higher latency
Goal: Prefer ISP A for all traffic
Configuration:
#!/usr/bin/env python3
"""
Local-pref for outbound traffic engineering
Announce routes learned from ISP A with higher local-pref
"""
import sys
import time
import json
def handle_bgp_message(message):
"""Process BGP messages from ExaBGP"""
try:
msg = json.loads(message)
if 'neighbor' in msg and 'update' in msg:
neighbor_ip = msg['neighbor']['address']['peer']
update = msg['update']
# Check if this is a route from ISP A
if neighbor_ip == '192.168.1.1': # ISP A
# Announce to route reflector with higher local-pref
if 'announce' in update and 'ipv4 unicast' in update['announce']:
for prefix_data in update['announce']['ipv4 unicast'].values():
for prefix in prefix_data:
# Re-announce with local-pref 150
sys.stdout.write(
f"announce route {prefix} "
f"next-hop 192.168.1.1 "
f"local-preference 150\n"
)
elif neighbor_ip == '192.168.2.1': # ISP B
# Announce with default local-pref 100
if 'announce' in update and 'ipv4 unicast' in update['announce']:
for prefix_data in update['announce']['ipv4 unicast'].values():
for prefix in prefix_data:
sys.stdout.write(
f"announce route {prefix} "
f"next-hop 192.168.2.1 "
f"local-preference 100\n"
)
sys.stdout.flush()
except Exception as e:
sys.stderr.write(f"[ERROR] {e}\n")
time.sleep(2)
# Process messages from ExaBGP
while True:
line = sys.stdin.readline()
if line:
handle_bgp_message(line)
time.sleep(0.1)Tools:
- AS-PATH prepending (make path less attractive)
- MED (suggest entry point)
- Communities (provider-specific policies)
Scenario:
Your AS 65001 connected to ISP (AS 1000) via:
- Link A (10 Gbps in New York)
- Link B (10 Gbps in Los Angeles)
Goal: Balance traffic 50/50
Solution: Announce same prefixes with equal MED
#!/usr/bin/env python3
"""
Inbound traffic balancing between two links
"""
import sys
import time
# Prefix to announce
PREFIXES = [
"203.0.113.0/24",
"203.0.113.128/25",
]
# Same MED on both links for equal distribution
MED = 100
time.sleep(2)
while True:
for prefix in PREFIXES:
sys.stdout.write(
f"announce route {prefix} "
f"next-hop self "
f"med {MED}\n"
)
sys.stdout.flush()
time.sleep(30)Scenario:
Link A = Primary (prefer this)
Link B = Backup (only use if A fails)
Solution: Different MED values
# Link A configuration
MED = 50 # Lower = preferred
# Link B configuration
MED = 150 # Higher = backupTools:
- Local-preference (within your AS)
- Static routes with higher preference
- Communities (for policy enforcement)
Scenario:
ISP A (AS 1000): Expensive but fast
ISP B (AS 2000): Cheap but slower
Goal: Use ISP B for most traffic, ISP A only for important services
Implementation:
#!/usr/bin/env python3
"""
Outbound traffic engineering with local-preference
Prefer cheap ISP by default, expensive ISP for critical traffic
"""
import sys
import time
import json
CRITICAL_PREFIXES = [
"8.8.8.0/24", # Google DNS
"1.1.1.0/24", # Cloudflare DNS
# Add critical destinations
]
def is_critical_prefix(prefix):
"""Check if prefix is critical"""
return prefix in CRITICAL_PREFIXES
def handle_bgp_update(message):
"""Process BGP updates and set local-pref"""
try:
msg = json.loads(message)
if 'neighbor' in msg and 'update' in msg:
neighbor_ip = msg['neighbor']['address']['peer']
update = msg['update']
if 'announce' in update and 'ipv4 unicast' in update['announce']:
for prefix_data in update['announce']['ipv4 unicast'].values():
for prefix in prefix_data:
if neighbor_ip == '192.168.1.1': # ISP A (expensive)
if is_critical_prefix(prefix):
# Critical traffic via expensive ISP
local_pref = 200
else:
# Non-critical via expensive ISP (backup)
local_pref = 50
else: # ISP B (cheap)
# Most traffic via cheap ISP
local_pref = 100
sys.stdout.write(
f"announce route {prefix} "
f"next-hop {neighbor_ip} "
f"local-preference {local_pref}\n"
)
sys.stdout.flush()
except Exception as e:
sys.stderr.write(f"[ERROR] {e}\n")
time.sleep(2)
while True:
line = sys.stdin.readline()
if line:
handle_bgp_update(line)Setup:
Your AS 65001 β Two connections to ISP (AS 1000)
- Link A (primary)
- Link B (backup)
Traffic Engineering:
Inbound (from ISP):
# Link A (primary)
announce with MED 50
# Link B (backup)
announce with MED 150Outbound (to ISP):
# Prefer Link A
Set local-pref 150 for routes from Link A
Set local-pref 100 for routes from Link BSetup:
Your AS 65001 β Connections to:
- ISP A (AS 1000) - Tier 1, expensive
- ISP B (AS 2000) - Regional, cheap
Traffic Engineering:
Inbound:
# Announce to ISP A with normal AS-PATH
as_path = [65001]
# Announce to ISP B with prepending (make less attractive globally)
as_path = [65001, 65001, 65001]Outbound:
# Default: Prefer cheaper ISP B (local-pref 150)
# Fallback: ISP A (local-pref 100)Setup:
Datacenter A (US East) - AS 65001
Datacenter B (US West) - AS 65002
Goal: US traffic to nearest DC, EU traffic to US East
Implementation:
#!/usr/bin/env python3
"""
Geographic traffic engineering
Different AS-PATH prepending based on peer location
"""
import sys
import time
PREFIX = "203.0.113.0/24"
LOCAL_AS = 65001
# Peer configuration
PEERS = {
'192.168.1.1': { # US peer
'location': 'US',
'prepend': 0, # No prepending for US peers
},
'192.168.2.1': { # EU peer
'location': 'EU',
'prepend': 0 if DATACENTER == 'US-EAST' else 3, # Prefer US-EAST for EU
},
}
DATACENTER = 'US-EAST' # Set per datacenter
time.sleep(2)
# Note: This is simplified; in production, use neighbor-specific configurations
while True:
for peer_ip, config in PEERS.items():
prepend_count = config['prepend']
if prepend_count == 0:
as_path = str(LOCAL_AS)
else:
as_path = ' '.join([str(LOCAL_AS)] * (prepend_count + 1))
sys.stdout.write(
f"announce route {PREFIX} "
f"next-hop self "
f"as-path [ {as_path} ]\n"
)
sys.stdout.flush()
time.sleep(30)ExaBGP Configuration:
# /etc/exabgp/traffic-engineering.conf
neighbor 192.168.1.1 {
router-id 192.168.1.10;
local-address 192.168.1.10;
local-as 65001;
peer-as 1000;
family {
ipv4 unicast;
}
api {
processes [ te-controller ];
}
}
neighbor 192.168.2.1 {
router-id 192.168.1.10;
local-address 192.168.1.10;
local-as 65001;
peer-as 2000;
family {
ipv4 unicast;
}
api {
processes [ te-controller ];
}
}
process te-controller {
run /etc/exabgp/te-controller.py;
encoder text;
}Traffic Engineering Controller:
#!/usr/bin/env python3
"""
Comprehensive traffic engineering controller
Handles AS-PATH prepending, MED, communities based on policy
"""
import sys
import time
# Prefix announcements
PREFIXES = {
"203.0.113.0/24": {
# ISP A (AS 1000) - Primary
"192.168.1.1": {
"med": 50,
"prepend": 0,
"communities": ["65001:100"],
},
# ISP B (AS 2000) - Backup
"192.168.2.1": {
"med": 150,
"prepend": 2,
"communities": ["65001:200"],
},
},
}
LOCAL_AS = 65001
def announce_prefix(prefix, neighbor_ip, config):
"""Announce prefix with traffic engineering attributes"""
med = config.get('med', 100)
prepend = config.get('prepend', 0)
communities = config.get('communities', [])
# Build AS-PATH
if prepend == 0:
as_path = str(LOCAL_AS)
else:
as_path = ' '.join([str(LOCAL_AS)] * (prepend + 1))
# Build community string
community_str = ' '.join(communities) if communities else ''
# Build announcement
announcement = f"announce route {prefix} next-hop self"
announcement += f" med {med}"
announcement += f" as-path [ {as_path} ]"
if community_str:
announcement += f" community [{community_str}]"
sys.stdout.write(announcement + "\n")
time.sleep(2)
sys.stderr.write("[TE] Traffic engineering controller started\n")
while True:
for prefix, neighbors in PREFIXES.items():
for neighbor_ip, config in neighbors.items():
announce_prefix(prefix, neighbor_ip, config)
sys.stdout.flush()
time.sleep(30)Don't over-prepend:
Good: Prepend 1-3 times
Bad: Prepend 10+ times (unnecessary, can cause issues)
# Bad - no documentation
MED = 50
# Good - clear documentation
MED = 50 # Primary link via NYC POP, prefer this entry point# Announce test prefix first
TEST_PREFIX = "203.0.113.254/32"
# Verify at upstream
# show ip bgp 203.0.113.254
# Check AS-PATH, MED, communities
# Then deploy to production prefixesimport subprocess
def check_traffic_flow():
"""Monitor which link is receiving traffic"""
# Check interface statistics
result = subprocess.run(
['ifstat', '-i', 'eth0,eth1', '1', '1'],
capture_output=True
)
# Log and alert on unexpected shifts# Don't immediately shift all traffic
# Gradually change MED/prepending
# Step 1: Small change
MED = 120 # Was 100
# Wait and observe
time.sleep(300)
# Step 2: Larger change
MED = 150
# Continue graduallyCheck announcements received by upstream:
# Request looking glass from ISP
# Or use public route servers
telnet route-server.example.com
> show ip bgp 203.0.113.0Expected output:
Network Next Hop Metric LocPrf Weight Path
*> 203.0.113.0/24 192.168.1.10 50 0 65001 i
* 203.0.113.0/24 192.168.2.10 150 0 65001 65001 65001 i
^^^^^^^^^^^^^^^
Prepended AS-PATH
#!/usr/bin/env python3
"""
Monitor traffic flow distribution across links
"""
import time
import subprocess
INTERFACES = ['eth0', 'eth1'] # Links to ISPs
def get_interface_traffic(interface):
"""Get traffic in Mbps"""
with open(f'/sys/class/net/{interface}/statistics/tx_bytes', 'r') as f:
bytes_1 = int(f.read())
time.sleep(1)
with open(f'/sys/class/net/{interface}/statistics/tx_bytes', 'r') as f:
bytes_2 = int(f.read())
mbps = ((bytes_2 - bytes_1) * 8) / (1000 * 1000)
return mbps
while True:
print(f"\nTraffic Distribution:")
for interface in INTERFACES:
mbps = get_interface_traffic(interface)
print(f" {interface}: {mbps:.2f} Mbps")
time.sleep(10)Symptoms: Traffic still coming via prepended path
Diagnosis:
# Check if upstream sees prepending
telnet route-server.isp.net
> show ip bgp 203.0.113.0Possible causes:
- Upstream ignores AS-PATH length (uses other attributes)
- No alternative path available
- Local-pref or MED overrides AS-PATH
Solutions:
- Verify upstream BGP policy
- Use MED in addition to prepending
- Request ISP's policy documentation
Symptoms: Traffic not following MED preference
Diagnosis:
# Check if upstream receives MED
show ip bgp neighbors 192.168.1.10 advertised-routesPossible causes:
- ISP doesn't honor MED (policy decision)
- MED only compared within same AS (multiple ISPs)
- Always-compare-med not configured
Solutions:
- Use AS-PATH prepending instead
- Use provider communities
- Engage ISP support
Symptoms: Traffic flowing via wrong link
Check:
# Verify announcements
exabgpcli show adj-rib out
# Check BGP decision process on upstream
show ip bgp 203.0.113.0 bestpathCommon causes:
- Local-pref overriding your attributes
- Routing policy filters
- BGP decision order
- Load Balancing - Load distribution patterns
- Service High Availability - HA architectures
- SDN Integration - SDN controller integration
- Monitoring - Monitoring setup
- Debugging - Troubleshooting guide
- Configuration Syntax - Config reference
- API Overview - API patterns
Ready to implement traffic engineering? See Quick Start β
π» Ghost written by Claude (Anthropic AI)
π Home
π Getting Started
π§ API
π‘οΈ Use Cases
π Address Families
βοΈ Configuration
π Operations
π Reference
- Architecture
- BGP State Machine
- Communities (RFC)
- Extended Communities
- BGP Ecosystem
- Capabilities (AFI/SAFI)
- RFC Support
π Migration
π Community
π External
- GitHub Repo β
- Slack β
- Issues β
π» Ghost written by Claude (Anthropic AI)