Twitter
Midwest Internet Exchange

Making the most of your IX experience

Obviously, we have a vested interest in you making the most of your IX experience. Better traffic flow at a lower cost is better for everybody. We’ve linked to pages on local pref and MED in the past, but we thought we’d come up with our own post, even if the information is largely recycled.

Note that the current router sending the packet decides where it goes. You can ask it nicely, but it could completely ignore your request. It also ignores whatever any router that had that packet before thought was best for the packet.

Different platforms have different terms for how you apply this setting. We’re not going to go into platform-specific differences. Google likely has the answer for how to configure it for your platform. Route maps and route filters are the first places to start looking.

Cisco has a great page on BGP Best Path selection. One thing they leave out is that prefix length is king. Routes are calculated for each prefix, with the smallest prefix winning. Unless you’re on a Cisco and have a local (on-router) weight configured, local pref is the highest importance.

Local Pref

Local Pref is a setting that you configure on your router. You will assign routes learned from a particular peer a given local pref. Like most sports, the higher score wins. Also, what numbers you use are less important than their position relative to each other. Your customers get the highest local pref and your transit gets the lowest local pref. To steal a table from Noction:

 

 

 

 

 

 

 

  • Customers: 150
  • Private peering: 130
  • Preferred internet exchange peering: 120
  • Other internet exchange peering: 110
  • Transit ISPs: 100

I would set bilateral sessions on an IX to 120 and route servers on 110. If you have a bilateral session, the other network was important enough to you to establish said session, so you’d want the most responsive, unmodified announcements you can get.

Now, you may be on multiple IXes. You’ll want to arrange your local prefs to accordingly prefer one IX or the other. One may be further away or more expensive, so you will use it if you must, but you prefer the other one if possible. If they’re in the same market and similarly priced, you may have the bilateral sessions on one IX high and the route server low, but buck the above convention on the other IX, having the route server high and the bilateral sessions low. This would create some sort of load balancing. Adjust as necessary for your network.

Why aren’t the different local prefs just 1, 2, 3, 4, and 5? This way gives you room to tweak individual peers up or down as desired without having to renumber everything. If you really want to tweak and you’ll have a lot of peers, maybe you’ll set 100, 200, 300, etc.

Another use case for local pref is for downstream customers with primary and backup connections. You would set a better local pref for the primary connection and a lower for the second. Maybe the primary is a 10G and the backup is a 1G. Maybe the primary is a wavelength and the backup is an Ethernet service.

We like to motivate content networks to set their local prefs to prefer the IX. It’s the cheapest destination and will most likely be the highest quality connection. Eyeballs won’t benefit as much because they don’t have a lot of upload traffic. Hurricane Electric is on many IXes (currently the most of anyone), so they’re likely on your IX too. They also have more AS adjacencies than anyone else. Some of these are peers and some are customers. As a content provider with HE transit, you can still gain a lot from HE on an IX. The full IPv4 routing table from HE is 735,013 routes as of the time I wrote that. It’s already up to 735,015. Anyway, I’m also getting 117,556 IPv4 routes from them on the IX. I’m not sure what that entails, but likely their customers. By setting a better local pref for a session with Hurricane over the IX, the traffic I *CAN* send over the IX does go over the IX, with the rest going transit. HE is certainly a low-cost provider, but why pay more than you have to?

MED

Another, complimentary method is MED, Multi Exit Discriminator. This is a property that you advertise to your neighbors to state your preference as to what connection to your neighbor you’d like to use for traffic they send to you. If MED was a sport, it would be like golf. The lowest number wins.

This is useful if you connect to a peer (say Hurricane Electric) in multiple markets. You’d prefer your Chicago prefixes came to you in Chicago and your Indianapolis prefixes came to you in Indianapolis so that you didn’t have to pay to haul it yourself to the right market, but in the event of some kind of failure, traffic will still go the other path. This is also useful to tell the other network to prefer your private peering (direct cross connect between networks) instead of the IX you’re on in the same market.

Communities

You can generally apply the above settings (Local Pref and MED) based on community strings instead of hard-setting them to specific prefixes. You would assign communities to routes as they enter your network. You can use communities to convey a ton of information. You can use communities to convey where the route was learned, geographically. You can also convey what the relationship is with the entity where you learned the route from.

This information is useful, so you can make decisions about whether to export that route to other (or specific) networks or not. You can use it to apply the local pref and MED settings, based on how you’re connected to a given network and how far away that other connection is. You may need to prepend an announcement a certain number of times, based on the community. You may need to pass on blackhole routes.

The community’s role is to pass along information so that you can act upon that information accordingly as it leaves your network. Many networks support passing information to you and will receive information from you via communities as well. Community support varies widely. One of Hurricane Electric’s major downsides is a rather sparse set of supported communities. There is a page out there that hosts many network’s BGP communities guides.

Summary

If you take away anything from this post, have it be that there are ways to influence how traffic enters and exits your network and they are worth your time to learn how they work. Also, disaggregating your prefixes to control your traffic is the very last thing you should consider doing. It makes the Internet worse for the rest of us.

 

References

https://www.cisco.com/c/en/us/support/docs/ip/border-gateway-protocol-bgp/13753-25.html

https://www.noction.com/blog/what-you-should-know-about-bgps-local_pref

https://www.noction.com/blog/bgp-med-attribute

https://www.noction.com/blog/using-med-to-optimize-internet-exchange-paths

https://www.noction.com/blog/using-communities-to-filter-bgp-announcements

https://onestep.net/communities/

Post a Comment

%d bloggers like this: