Check out our updated peer list for our St. Louis Fabric.
We’ve had networks on our IXes that have had a gigabit/s or more of traffic going to our cache boxes when they had their own on-net cache boxes that were working as designed.
Many Content Delivery Networks (CDNs) have programs where you can get a node of theirs in your network to maximize performance, minimize costs, etc. Netflix has their Open Connect program. Google has their Google Global Cache nodes. Akamai has their Akamai Accelerated Network Partner program. Cloudflare, Facebook and others have programs as well, but I can’t readily find links to them.
If you have one of those nodes, why do you need to connect to them over an Internet exchange? Simply put, not all content a given CDN has is stored on every node. They all have algorithms they use to determine what content users of that box are likely to access. Those boxes are then loaded with that content. If something your customers want to watch isn’t on your local box, the CDN must direct you somewhere else to get it. It could be a different show, it could be a show you have, but at a different bit rate. Either way, it doesn’t exist on your cache box. If you aren’t peered with other boxes on an IX, guess where that goes. Yup, it goes out your transit. You’re now paying more for your customer to have a worse experience.
Maybe you’re concerned about traffic preferring the boxes you’re peered with over your own. While they all have a secret sauce that they won’t reveal to the public, they have documentation available on how to influence your use of them. Netflix explains here about half-way down the page how they do it. Google has a page that is all about how to configure your BGP environment. They all support some methods such as BGP communities, Local Pref, MED, AS-Path, etc. to influence how that traffic gets to you. Needing some help on how to set those and what they mean? We did a post last month that should provide some assistance on that.
Most important of all, don’t let misconceptions, misunderstandings, doubt, etc. get in the way of maximizing your use of what’s available to you. Reach out to support and we can have a conversation with whatever CDNs you like to make sure you’re comfortable with how everything is being used.
Obviously, we have a vested interest in you making the most of your IX experience. Better traffic flow at a lower cost is better for everybody. We’ve linked to pages on local pref and MED in the past, but we thought we’d come up with our own post, even if the information is largely recycled.
Note that the current router sending the packet decides where it goes. You can ask it nicely, but it could completely ignore your request. It also ignores whatever any router that had that packet before thought was best for the packet.
Different platforms have different terms for how you apply this setting. We’re not going to go into platform-specific differences. Google likely has the answer for how to configure it for your platform. Route maps and route filters are the first places to start looking.
Cisco has a great page on BGP Best Path selection. One thing they leave out is that prefix length is king. Routes are calculated for each prefix, with the smallest prefix winning. Unless you’re on a Cisco and have a local (on-router) weight configured, local pref is the highest importance.
Local Pref is a setting that you configure on your router. You will assign routes learned from a particular peer a given local pref. Like most sports, the higher score wins. Also, what numbers you use are less important than their position relative to each other. Your customers get the highest local pref and your transit gets the lowest local pref. To steal a table from Noction:
- Customers: 150
- Private peering: 130
- Preferred internet exchange peering: 120
- Other internet exchange peering: 110
- Transit ISPs: 100
I would set bilateral sessions on an IX to 120 and route servers on 110. If you have a bilateral session, the other network was important enough to you to establish said session, so you’d want the most responsive, unmodified announcements you can get.
Now, you may be on multiple IXes. You’ll want to arrange your local prefs to accordingly prefer one IX or the other. One may be further away or more expensive, so you will use it if you must, but you prefer the other one if possible. If they’re in the same market and similarly priced, you may have the bilateral sessions on one IX high and the route server low, but buck the above convention on the other IX, having the route server high and the bilateral sessions low. This would create some sort of load balancing. Adjust as necessary for your network.
Why aren’t the different local prefs just 1, 2, 3, 4, and 5? This way gives you room to tweak individual peers up or down as desired without having to renumber everything. If you really want to tweak and you’ll have a lot of peers, maybe you’ll set 100, 200, 300, etc.
Another use case for local pref is for downstream customers with primary and backup connections. You would set a better local pref for the primary connection and a lower for the second. Maybe the primary is a 10G and the backup is a 1G. Maybe the primary is a wavelength and the backup is an Ethernet service.
We like to motivate content networks to set their local prefs to prefer the IX. It’s the cheapest destination and will most likely be the highest quality connection. Eyeballs won’t benefit as much because they don’t have a lot of upload traffic. Hurricane Electric is on many IXes (currently the most of anyone), so they’re likely on your IX too. They also have more AS adjacencies than anyone else. Some of these are peers and some are customers. As a content provider with HE transit, you can still gain a lot from HE on an IX. The full IPv4 routing table from HE is 735,013 routes as of the time I wrote that. It’s already up to 735,015. Anyway, I’m also getting 117,556 IPv4 routes from them on the IX. I’m not sure what that entails, but likely their customers. By setting a better local pref for a session with Hurricane over the IX, the traffic I *CAN* send over the IX does go over the IX, with the rest going transit. HE is certainly a low-cost provider, but why pay more than you have to?
Another, complimentary method is MED, Multi Exit Discriminator. This is a property that you advertise to your neighbors to state your preference as to what connection to your neighbor you’d like to use for traffic they send to you. If MED was a sport, it would be like golf. The lowest number wins.
This is useful if you connect to a peer (say Hurricane Electric) in multiple markets. You’d prefer your Chicago prefixes came to you in Chicago and your Indianapolis prefixes came to you in Indianapolis so that you didn’t have to pay to haul it yourself to the right market, but in the event of some kind of failure, traffic will still go the other path. This is also useful to tell the other network to prefer your private peering (direct cross connect between networks) instead of the IX you’re on in the same market.
You can generally apply the above settings (Local Pref and MED) based on community strings instead of hard-setting them to specific prefixes. You would assign communities to routes as they enter your network. You can use communities to convey a ton of information. You can use communities to convey where the route was learned, geographically. You can also convey what the relationship is with the entity where you learned the route from.
This information is useful, so you can make decisions about whether to export that route to other (or specific) networks or not. You can use it to apply the local pref and MED settings, based on how you’re connected to a given network and how far away that other connection is. You may need to prepend an announcement a certain number of times, based on the community. You may need to pass on blackhole routes.
The community’s role is to pass along information so that you can act upon that information accordingly as it leaves your network. Many networks support passing information to you and will receive information from you via communities as well. Community support varies widely. One of Hurricane Electric’s major downsides is a rather sparse set of supported communities. There is a page out there that hosts many network’s BGP communities guides.
If you take away anything from this post, have it be that there are ways to influence how traffic enters and exits your network and they are worth your time to learn how they work. Also, disaggregating your prefixes to control your traffic is the very last thing you should consider doing. It makes the Internet worse for the rest of us.
A coupler created by Macquarie University in Australia, combined with a fibre fabricated by Hokkaido University and equipment maker Fujikura, and a transmission system developed by the National Institute of Information and Communications Technology in Japan, has led to transmission speeds in excess of 1 petabit.
The new four-core, three-mode fibre was touted as being the same width as existing standard fibre, but was capable of 12 times the data speed. Macquarie University said the fibre was less prone to damage due to its narrower diameter, and could be used with existing equipment.
If you are an IX client and are looking for a faster way to connect to a local SpeedTest server here is a faster way.
MidWest-IX has completed several recent capacity upgrades on our Indianapolis fabric.
- We have upgraded the capacity between 733 and 401 north Shadeland to 40 gigs of capacity.
- We have upgraded switch interconnects at 733 to 40 gig.
- Switch uplinks at 401 Shadeland have been upgraded to 20 gig.
- A switch upgrade inside 365 Data Centers at 701 Henry added additional 10 gig ports.
Round two will consist of route server upgrades, increased capacity to Google and Netflix caches, and updates of IOS on switches. Look for more information soon.
Our Peering DB for Cleveland is now live.
Transparency Route Servers, unlike regular BGP routers, provide transparency features to facilitate optimal path selection in Internet Exchange infrastructure. FD-IX Route Servers provide the following:
1) AS path transparency by hiding its own ASN. This shortens AS path by one hop and avoids the need for direct peering with other customers on our fabric. There may be other advantages of a direct peer, but that is specific to the individual peer. For example, some content networks don’t advertise all of their routes to the route server, but will if you establish a direct peer.
2) Next Hop transparency. The route servers will provide original Next Hop address as it is received from the peer. Note that you should never use the route server addresses as Next Hop for any prefix. The servers will drop any traffic not destined to them.
3) MED transparency. This allows optimizing the path in case an AS has two (or more) peers with FD-IX. Note that you can use MED to direct other peer’s customer traffic towards some of your interfaces. MEDs are encouraged for optimal traffic flow.
Route servers have some unique configuration options due to the next-hop and AS transparency. For example, Cisco IOS needs the command “no bgp enforce-first-as” in the bgp configuration. IOS-XR does it on a per-neighbor basis. Others call it force next-hop-self. These are commands many folks are not familiar with due to they are not needed for direct peering.