The folks over at PeeringDB have a presentation on setting up an account and adding your network.
The folks over at PeeringDB have a presentation on setting up an account and adding your network.
Brad from Beck’s Hybrids talks on the InterVison podcast about the role technology has played in agriculture.
MidWest is in the LifeLine facilities. Rich Banta gives a talk on understanding FISMA and FedRamp
Some light reading.
This document outlines an approach to mitigate the negative impact on networks resulting from maintenance activities. It includes guidance for both IP networks and Internet Exchange Points (IXPs). The approach is to ensure BGP-4 sessions that will be affected by maintenance are forcefully torn down before the actual maintenance activities commence.
We have many exciting things going on here at MidWest-IX.
Switch and Router Upgrades
MidWest-IX has been hard at work upgrading our Indianapolis Infrastructure to all 40 GIG interconnects. Recently we completed a cache fill router upgrade. This will allow extra capacity to our Netflix, Google, and other caches. At the same time, this simplifies our network design on the cache network. Simpler means more reliable and better performance.
We will be transitioning more and more to the FD-IX brand as our new sites in Texas, Cleveland, and other locations come online. We have expanded outside the Midwest and feel it is time to embrace the brand we introduced about a year ago. This will be the same people, same network, just a different name.
Indy IXP Manager upgrade
Mike has been busy upgrading our IXP manager installation for Indianapolis. This will allow many new features as well as sFlow for traffic reporting to the software. This is a very complex upgrade, so it is taking some time. We will have an announcement for it soon.
Many of you may have already seen this, but we have been working on a 10 GIG transport ring to connect many of our markets together. If you are looking for transport between any of the cities, please reach out to us.
Just a reminder to keep your peering DB entries up to date. The future IXP manager upgrade will be more tightly integrated with PeeringDB. We have been running the new version on our St. Louis fabric for a while, and the PeeringDB integration relies on your information is up-to-date. More and more companies are integrating their peering automation with PeeringDB. We recommend a yearly review of your data or anytime you receive new IP space, co-lo at a new facility, or staff changes.
City Specific news
Look for some exciting news regarding St. Louis, Cleveland and a few others coming soon.
We’ve had networks on our IXes that have had a gigabit/s or more of traffic going to our cache boxes when they had their own on-net cache boxes that were working as designed.
Many Content Delivery Networks (CDNs) have programs where you can get a node of theirs in your network to maximize performance, minimize costs, etc. Netflix has their Open Connect program. Google has their Google Global Cache nodes. Akamai has their Akamai Accelerated Network Partner program. Cloudflare, Facebook and others have programs as well, but I can’t readily find links to them.
If you have one of those nodes, why do you need to connect to them over an Internet exchange? Simply put, not all content a given CDN has is stored on every node. They all have algorithms they use to determine what content users of that box are likely to access. Those boxes are then loaded with that content. If something your customers want to watch isn’t on your local box, the CDN must direct you somewhere else to get it. It could be a different show, it could be a show you have, but at a different bit rate. Either way, it doesn’t exist on your cache box. If you aren’t peered with other boxes on an IX, guess where that goes. Yup, it goes out your transit. You’re now paying more for your customer to have a worse experience.
Maybe you’re concerned about traffic preferring the boxes you’re peered with over your own. While they all have a secret sauce that they won’t reveal to the public, they have documentation available on how to influence your use of them. Netflix explains here about half-way down the page how they do it. Google has a page that is all about how to configure your BGP environment. They all support some methods such as BGP communities, Local Pref, MED, AS-Path, etc. to influence how that traffic gets to you. Needing some help on how to set those and what they mean? We did a post last month that should provide some assistance on that.
Most important of all, don’t let misconceptions, misunderstandings, doubt, etc. get in the way of maximizing your use of what’s available to you. Reach out to support and we can have a conversation with whatever CDNs you like to make sure you’re comfortable with how everything is being used.
Obviously, we have a vested interest in you making the most of your IX experience. Better traffic flow at a lower cost is better for everybody. We’ve linked to pages on local pref and MED in the past, but we thought we’d come up with our own post, even if the information is largely recycled.
Note that the current router sending the packet decides where it goes. You can ask it nicely, but it could completely ignore your request. It also ignores whatever any router that had that packet before thought was best for the packet.
Different platforms have different terms for how you apply this setting. We’re not going to go into platform-specific differences. Google likely has the answer for how to configure it for your platform. Route maps and route filters are the first places to start looking.
Cisco has a great page on BGP Best Path selection. One thing they leave out is that prefix length is king. Routes are calculated for each prefix, with the smallest prefix winning. Unless you’re on a Cisco and have a local (on-router) weight configured, local pref is the highest importance.
Local Pref is a setting that you configure on your router. You will assign routes learned from a particular peer a given local pref. Like most sports, the higher score wins. Also, what numbers you use are less important than their position relative to each other. Your customers get the highest local pref and your transit gets the lowest local pref. To steal a table from Noction:
- Customers: 150
- Private peering: 130
- Preferred internet exchange peering: 120
- Other internet exchange peering: 110
- Transit ISPs: 100
I would set bilateral sessions on an IX to 120 and route servers on 110. If you have a bilateral session, the other network was important enough to you to establish said session, so you’d want the most responsive, unmodified announcements you can get.
Now, you may be on multiple IXes. You’ll want to arrange your local prefs to accordingly prefer one IX or the other. One may be further away or more expensive, so you will use it if you must, but you prefer the other one if possible. If they’re in the same market and similarly priced, you may have the bilateral sessions on one IX high and the route server low, but buck the above convention on the other IX, having the route server high and the bilateral sessions low. This would create some sort of load balancing. Adjust as necessary for your network.
Why aren’t the different local prefs just 1, 2, 3, 4, and 5? This way gives you room to tweak individual peers up or down as desired without having to renumber everything. If you really want to tweak and you’ll have a lot of peers, maybe you’ll set 100, 200, 300, etc.
Another use case for local pref is for downstream customers with primary and backup connections. You would set a better local pref for the primary connection and a lower for the second. Maybe the primary is a 10G and the backup is a 1G. Maybe the primary is a wavelength and the backup is an Ethernet service.
We like to motivate content networks to set their local prefs to prefer the IX. It’s the cheapest destination and will most likely be the highest quality connection. Eyeballs won’t benefit as much because they don’t have a lot of upload traffic. Hurricane Electric is on many IXes (currently the most of anyone), so they’re likely on your IX too. They also have more AS adjacencies than anyone else. Some of these are peers and some are customers. As a content provider with HE transit, you can still gain a lot from HE on an IX. The full IPv4 routing table from HE is 735,013 routes as of the time I wrote that. It’s already up to 735,015. Anyway, I’m also getting 117,556 IPv4 routes from them on the IX. I’m not sure what that entails, but likely their customers. By setting a better local pref for a session with Hurricane over the IX, the traffic I *CAN* send over the IX does go over the IX, with the rest going transit. HE is certainly a low-cost provider, but why pay more than you have to?
Another, complimentary method is MED, Multi Exit Discriminator. This is a property that you advertise to your neighbors to state your preference as to what connection to your neighbor you’d like to use for traffic they send to you. If MED was a sport, it would be like golf. The lowest number wins.
This is useful if you connect to a peer (say Hurricane Electric) in multiple markets. You’d prefer your Chicago prefixes came to you in Chicago and your Indianapolis prefixes came to you in Indianapolis so that you didn’t have to pay to haul it yourself to the right market, but in the event of some kind of failure, traffic will still go the other path. This is also useful to tell the other network to prefer your private peering (direct cross connect between networks) instead of the IX you’re on in the same market.
You can generally apply the above settings (Local Pref and MED) based on community strings instead of hard-setting them to specific prefixes. You would assign communities to routes as they enter your network. You can use communities to convey a ton of information. You can use communities to convey where the route was learned, geographically. You can also convey what the relationship is with the entity where you learned the route from.
This information is useful, so you can make decisions about whether to export that route to other (or specific) networks or not. You can use it to apply the local pref and MED settings, based on how you’re connected to a given network and how far away that other connection is. You may need to prepend an announcement a certain number of times, based on the community. You may need to pass on blackhole routes.
The community’s role is to pass along information so that you can act upon that information accordingly as it leaves your network. Many networks support passing information to you and will receive information from you via communities as well. Community support varies widely. One of Hurricane Electric’s major downsides is a rather sparse set of supported communities. There is a page out there that hosts many network’s BGP communities guides.
If you take away anything from this post, have it be that there are ways to influence how traffic enters and exits your network and they are worth your time to learn how they work. Also, disaggregating your prefixes to control your traffic is the very last thing you should consider doing. It makes the Internet worse for the rest of us.