Twitter
Midwest Internet Exchange

Making the most of your IX experience

Obviously, we have a vested interest in you making the most of your IX experience. Better traffic flow at a lower cost is better for everybody. We’ve linked to pages on local pref and MED in the past, but we thought we’d come up with our own post, even if the information is largely recycled.

Note that the current router sending the packet decides where it goes. You can ask it nicely, but it could completely ignore your request. It also ignores whatever any router that had that packet before thought was best for the packet.

Different platforms have different terms for how you apply this setting. We’re not going to go into platform-specific differences. Google likely has the answer for how to configure it for your platform. Route maps and route filters are the first places to start looking.

Cisco has a great page on BGP Best Path selection. One thing they leave out is that prefix length is king. Routes are calculated for each prefix, with the smallest prefix winning. Unless you’re on a Cisco and have a local (on-router) weight configured, local pref is the highest importance.

Local Pref

Local Pref is a setting that you configure on your router. You will assign routes learned from a particular peer a given local pref. Like most sports, the higher score wins. Also, what numbers you use are less important than their position relative to each other. Your customers get the highest local pref and your transit gets the lowest local pref. To steal a table from Noction:

 

 

 

 

 

 

 

  • Customers: 150
  • Private peering: 130
  • Preferred internet exchange peering: 120
  • Other internet exchange peering: 110
  • Transit ISPs: 100

I would set bilateral sessions on an IX to 120 and route servers on 110. If you have a bilateral session, the other network was important enough to you to establish said session, so you’d want the most responsive, unmodified announcements you can get.

Now, you may be on multiple IXes. You’ll want to arrange your local prefs to accordingly prefer one IX or the other. One may be further away or more expensive, so you will use it if you must, but you prefer the other one if possible. If they’re in the same market and similarly priced, you may have the bilateral sessions on one IX high and the route server low, but buck the above convention on the other IX, having the route server high and the bilateral sessions low. This would create some sort of load balancing. Adjust as necessary for your network.

Why aren’t the different local prefs just 1, 2, 3, 4, and 5? This way gives you room to tweak individual peers up or down as desired without having to renumber everything. If you really want to tweak and you’ll have a lot of peers, maybe you’ll set 100, 200, 300, etc.

Another use case for local pref is for downstream customers with primary and backup connections. You would set a better local pref for the primary connection and a lower for the second. Maybe the primary is a 10G and the backup is a 1G. Maybe the primary is a wavelength and the backup is an Ethernet service.

We like to motivate content networks to set their local prefs to prefer the IX. It’s the cheapest destination and will most likely be the highest quality connection. Eyeballs won’t benefit as much because they don’t have a lot of upload traffic. Hurricane Electric is on many IXes (currently the most of anyone), so they’re likely on your IX too. They also have more AS adjacencies than anyone else. Some of these are peers and some are customers. As a content provider with HE transit, you can still gain a lot from HE on an IX. The full IPv4 routing table from HE is 735,013 routes as of the time I wrote that. It’s already up to 735,015. Anyway, I’m also getting 117,556 IPv4 routes from them on the IX. I’m not sure what that entails, but likely their customers. By setting a better local pref for a session with Hurricane over the IX, the traffic I *CAN* send over the IX does go over the IX, with the rest going transit. HE is certainly a low-cost provider, but why pay more than you have to?

MED

Another, complimentary method is MED, Multi Exit Discriminator. This is a property that you advertise to your neighbors to state your preference as to what connection to your neighbor you’d like to use for traffic they send to you. If MED was a sport, it would be like golf. The lowest number wins.

This is useful if you connect to a peer (say Hurricane Electric) in multiple markets. You’d prefer your Chicago prefixes came to you in Chicago and your Indianapolis prefixes came to you in Indianapolis so that you didn’t have to pay to haul it yourself to the right market, but in the event of some kind of failure, traffic will still go the other path. This is also useful to tell the other network to prefer your private peering (direct cross connect between networks) instead of the IX you’re on in the same market.

Communities

You can generally apply the above settings (Local Pref and MED) based on community strings instead of hard-setting them to specific prefixes. You would assign communities to routes as they enter your network. You can use communities to convey a ton of information. You can use communities to convey where the route was learned, geographically. You can also convey what the relationship is with the entity where you learned the route from.

This information is useful, so you can make decisions about whether to export that route to other (or specific) networks or not. You can use it to apply the local pref and MED settings, based on how you’re connected to a given network and how far away that other connection is. You may need to prepend an announcement a certain number of times, based on the community. You may need to pass on blackhole routes.

The community’s role is to pass along information so that you can act upon that information accordingly as it leaves your network. Many networks support passing information to you and will receive information from you via communities as well. Community support varies widely. One of Hurricane Electric’s major downsides is a rather sparse set of supported communities. There is a page out there that hosts many network’s BGP communities guides.

Summary

If you take away anything from this post, have it be that there are ways to influence how traffic enters and exits your network and they are worth your time to learn how they work. Also, disaggregating your prefixes to control your traffic is the very last thing you should consider doing. It makes the Internet worse for the rest of us.

 

References

https://www.cisco.com/c/en/us/support/docs/ip/border-gateway-protocol-bgp/13753-25.html

https://www.noction.com/blog/what-you-should-know-about-bgps-local_pref

https://www.noction.com/blog/bgp-med-attribute

https://www.noction.com/blog/using-med-to-optimize-internet-exchange-paths

https://www.noction.com/blog/using-communities-to-filter-bgp-announcements

https://onestep.net/communities/

1 Petabit per second optical fiber

A coupler created by Macquarie University in Australia, combined with a fibre fabricated by Hokkaido University and equipment maker Fujikura, and a transmission system developed by the National Institute of Information and Communications Technology in Japan, has led to transmission speeds in excess of 1 petabit.

The new four-core, three-mode fibre was touted as being the same width as existing standard fibre, but was capable of 12 times the data speed. Macquarie University said the fibre was less prone to damage due to its narrower diameter, and could be used with existing equipment.

https://www.zdnet.com/article/researchers-develop-optical-fibre-capable-of-over-1-petabit-per-second/

Direct link to our Indy Speedtest Server

If you are an IX client and are looking for a faster way to connect to a local SpeedTest server here is a faster way.

http://midwest-ix.speedtest.net

MidWest-IX completes Phase 1 Capacity Upgrade for Indianapolis

MidWest-IX has completed several recent capacity upgrades on our Indianapolis fabric.

  1. We have upgraded the capacity between 733 and 401 north Shadeland to 40 gigs of capacity.
  2. We have upgraded switch interconnects at 733 to 40 gig.
  3. Switch uplinks at 401 Shadeland have been upgraded to 20 gig.
  4. A switch upgrade inside 365 Data Centers at 701 Henry added additional 10 gig ports.

Round two will consist of route server upgrades, increased capacity to Google and Netflix caches, and updates of IOS on switches.  Look for more information soon.

St. Louis Event August 15th

Register and Information
http://go.netrality.com/peering-at-peering

PeeringDB entry for Cleveland

Our Peering DB for Cleveland is now live.

https://www.peeringdb.com/ix/2286

What do route servers do?

Transparency Route Servers, unlike regular BGP routers, provide transparency features to facilitate optimal path selection in Internet Exchange infrastructure. FD-IX Route Servers provide the following:

1) AS path transparency by hiding its own ASN. This shortens AS path by one hop and avoids the need for direct peering with other customers on our fabric. There may be other advantages of a direct peer, but that is specific to the individual peer.  For example, some content networks don’t advertise all of their routes to the route server, but will if you establish a direct peer.

2) Next Hop transparency. The route servers will provide original Next Hop address as it is received from the peer. Note that you should never use the route server addresses as Next Hop for any prefix. The servers will drop any traffic not destined to them.

3) MED transparency. This allows optimizing the path in case an AS has two (or more) peers with FD-IX. Note that you can use MED to direct other peer’s customer traffic towards some of your interfaces.  MEDs are encouraged for optimal traffic flow.

Route servers have some unique configuration options due to the next-hop and AS transparency. For example, Cisco IOS needs the command “no bgp enforce-first-as” in the bgp configuration.  IOS-XR does it on a per-neighbor basis. Others call it force next-hop-self. These are commands many folks are not familiar with due to they are not needed for direct peering.

FD-IX expanding to Cleveland

MidWest-IX/FD-IX is in the process of bringing on a new node in Cleveland, Ohio.  Look for more updates shortly. 

Member Spotlight: PDS Connect

PDS Connect is a locally owned full-service internet provider and IT consulting company located in Mooresville, Indiana. We care most about providing the best service and customer satisfaction, which sets us apart from other internet and phone companies. Our primary services are Fixed Wireless Broadband, Fiber-Based Internet, Server Co-Location, Data Center Services, and Wireless LAN/WAN Deployment and Consulting. PDS Connect is also a provider of phone service and corporate email solutions.

Our broadband Internet is available throughout Morgan County and surrounding areas at speeds of up to one gigabit! Fixed wireless and fiber optic broadband offer several options to cover residential, small businesses and large corporation’s needs.

Eric Rogers from PDS Connect says: We have found several advantages to being a Midwest Internet Exchange member. First and foremost is that we keep traffic local. This is something that creates the fastest Internet experience because the route that traffic takes is shorter, and therefore faster. The next is that it allows us to create partnerships and deploy our services faster than traditional methods of connectivity. Our broadband subscribers represent thousands of content consumers, and having access to that content without leaving the Midwest data center saves our company money on transit costs, as well as providing a lower latency and optimal path for the best customer experience. Because it is a faster build time and we save money on transit, we pass those savings onto our customers.

Justin Wilson, from MidWest-IX, says “The Internet traffic PDS Connect adds to the exchange is a diverse mix of eyeballs and hosting traffic. By having PDS we are adding my routes and diversity to our Indianapolis exchange.”

PDS Connect can be contacted at info@precisionds.com, 317-831-3000, or via their website at www.pdsconnect.me.

Make sure you are subscribed to our mailing list

MidWest-IX/FD-IX  has two e-mail mailing lists for each of our markets.

The first one is our announcements list. We post any relevant maintenance, updates, new members, and relevant news to this list.

The second is our tech mailing list.  We periodically post tech tidbits and other relevant tech topics to help our members get the most out of their connections to MidWest-IX/FD-IX.

In order to see what you are subscribed to please login to your IXP Manager account.  If you do not have one please contact us. Once logged in click on “My Account” at the top left, and select “My profile”.  At the lower left, you will see a heading that says “My Subscriptions”. If you are subscribed to the mailing lists there will be a checkmark next to them.  If not, select the left box and click “update my subscriptions” below.

%d bloggers like this: