When I start a new position with an organization, the very first thing I do is review the policy framework and its contents. I don’t dig into the network diagrams. I don’t pester securi…

The most disrespected person in America is the black woman.The most unprotected person in America is the black woman.The most neglected person in America is the black woman. —El Hajj Malik El Shabazz

The Internet of Things is now ubiquitous, but even twenty two years after its introduction, much of it, even the simplest parts, remain much less customizable than many of us would like. Luckily, mos

How to capture VLAN tags that are used by tcpdump? Server unable to ping the gateway which is on VLAN.

Packer is a free and open source tool for creating golden images for multiple platforms from a single source configuration.

This guide is intended to help you gain a true understanding of SQL query speeds. It includes research that demonstrates the speed of slow and fast query typ...

All models are wrong, but some are useful is a common saying in statistics. It does not merely apply to statistics, however. It is general observation. Box (1976) wrote an  influential article on the topic. He says that you make progress by successively making models followed by experiments, followed by more models, and then more … Continue reading All models are wrong

For decades, the solution to Black Americans’ distrust of cops has often been to not call them. Now white people are catching on too.

Every time I write about unequal traffic distribution across a link aggregation group (LAG, aka Etherchannel or Port Channel) or ECMP fabric, someone asks a simple question “is there no way to reshuffle the traffic to make it more balanced?” TL&DR summary: there are ways to do it, and some vendors already implemented them. The Problem The algorithm that spreads the traffic across a group of outbound links (LAG or set of ECMP next hops) has to satisfy a few requirements: It has to work reasonably well in typical environments; It should not reorder packets of the same flow (here’s why); It has to be simple enough to be implementable in reasonably cheap ASICs; The second and third requirement result in what the chipset manufacturers (and subsequently the hardware vendors) are offering today: hash-based distribution of packets. In case you need a step-by-step overview of this process, here’s how it works: Create an array of buckets and assign each outgoing link to one or more buckets. The bucket size is the number you see in marketing papers as “we support N-way ECMP” or “we have N-way LAG”. Take N fields from the outgoing packet header. The fields could be MAC addresses (source and/or destination), IP addresses (source and/or destination), IP port numbers, or even some other fixed-position fields in the packet header. Some vendors – for example Arista – allow you to configure which fields you want to use (assuming the platform chipset supports this functionality). Hash the fields from the packet header to get an integer between 0 and bucket size – 1. Example: for bucket sizes that are power of two take the low-order N bits of the hash. Enqueue the packet into the output queue of the interface that is associated with the bucket selected by the packet hash. Have you noticed that the algorithm never checks the size of the output queue? If the hashing algorithm decides to send the packet through Interface#1, the switch will send the packet through Interface#1 even though that interface might be dropping packets like crazy due to continuous congestion, and all the other interfaces sit idle. The reason the load-balancing algorithm never checks the load on the outbound interface is simple: the typical environment mentioned above is usually assumed to be a healthy mix of numerous independent mice flows. Throw a few elephants in the mix and the assumptions start breaking down. The only vendor that was always able to cope with the elephants in the mix is Brocade due to the fact that their traditional typical environment (storage networks) consists mainly of elephants. Can We Solve the Problem? Here’s an intriguingly simple question: Why can’t we change the mix of outgoing interfaces in the N-way ECMP table to reflect the actual interface load? Wouldn’t that allow us to push the mice flows away from elephants crowding some of the interfaces? In principle, the answer is “Sure, we could do that”, but we have to solve three challenges: Coarse-grained reshuffling could make matters worse. If your hardware supports 8-way ECMP and you have four uplinks, you might shift a large proportion of the traffic when you reassign the buckets to less-loaded interfaces, resulting in a nasty oscillation. Modern chipsets support at least 256-way ECMP, so that shouldn’t be a problem. The hardware you use has to support per-bucket counters. All hardware supports per-interface counters, but while they help you identify the congested interfaces, the won’t help you reshuffle the traffic – if the control-plane software cannot see how much traffic goes through each bucket, it makes no sense to randomly reshuffle the buckets hoping for the best. We shall not reorder the packets (at least within the data center), which means that we cannot reshuffle active buckets, but it’s relatively safe to change the outgoing interface of a currently inactive bucket. You could still reorder packets within a TCP session under an unlikely set of circumstances (figuring out what those circumstances are is left as an exercise for the reader), but we just might have to accept that slight risk of temporary performance degradation if we want to get better link utilization. Would the reshuffle inactive buckets idea work in practice? Are there inactive buckets in a typical high-volume data center environment? Welcome to the weird world of flowlets. What Are Flowlets? It seems the idea of flowlets first appeared in the Harnessing TCP’s Burstiness with Flowlet Switching paper (see also corresponding PPT) – due to the bursty nature of TCP, you might be able to do pretty reliable bucket reshuffling with 256 or more buckets, as some buckets always tend to be empty. Microsoft started using flowlets in Windows Server 2012 R2, and recently Cisco implemented flowlet-based dynamic load balancing in the ACI leaf-and-spine fabrics. Juniper is doing something similar (adaptive load balancing) on MX routers in Junos 14.1, and did Adaptive Flowlet Splicing within a Virtual Chassis Fabric (a nice rehash of the topic). Need more information? Data Center Fabrics webinar describes data center solutions from leading networking vendors; ipSpace.net webinar subscription also gives you access to further data center topics, including leaf-and-spine fabric architectures and design scenarios;

In part 1, we discussed how Fastly started down the slippery slope of network software. Our previous experience with routing suggested that avoiding traditional network devices would not only dramatically cut capital expenditure, but also quickly outpace existing solutions. Having already eschewed routers, we turned our attention towards obsoleting load balancers entirely. In this post, we’ll detail our in-house load balancing system which performs seamless failover with minimal processing overhead.

I'm fairly old (40+) for a programmer, and you'd think most skill jumps would have occurred by now, and I think that in terms of raw ability...

me

Pete Lumbis and Network Ninja mentioned an interesting Unequal-Cost Multipathing (UCMP) data center use case in their comments to my UCMP-related blog posts: anycast servers. Here’s a typical scenario they mentioned: a bunch of servers, randomly connected to multiple leaf switches, is offering a service on the same IP address (that’s where anycast comes from). Typical Data Center Anycast Deployment

Cities are finally being analyzed for the equity of their green space.

This is a post about building tools to validate data. I wanted to share a few reflections based on helping to design and build a few different public and private tools, as well as my experience as …

Posted on Thursday 27 May 2021. 1,095 words, 12 links. By Matt Webb.

This is very different from “someday.” Choose any date you like, as far in the future as you like. But a date, circled on the calendar. By that date, what will you have implemented? Wha…

When unexpected changes are requested during the development process, your final product may be a lot more complicated than what your spec originally called for. This phenomenon is called “scope creep.” Add a fully remote team with thin work-life boundaries on top of that, and you've got problems.

A few of the most essential and helpful building blocks that can help your workflow take off.

Understand how to use schema markup for e-commerce websites to improve SEO. Take advantage of Schema.org & structured data for better results on the SERP.

A new Developer Velocity Lab effort from Microsoft and GitHub digs into what makes a day "good" to help boost developer productivity and satisfaction.

What is Container Security and How to Secure Containers? What, Why and How of Container Security? So, Why Do We Need Container Security? But, What is Container Security? How to Secure Containers?

I was once asked why computers are called "computers" when they do so much more than compute numbers. A modern PC browses the internet, plays audio and video, generates beautiful graphics for video games and movies, simulates and predicts complex weather patterns and epidemiological risks, brings architectural and engineering blueprints to life, and much more. The reason computers can do all of this because all these problems can be expressed as numerical equations, and the computer's CPU—its central processing unit—is actually little more than a simple calculator.

This year, it’ll be 100 years since the extremely grotesque act of domestic terrorism known as the Tulsa Race Massacre, which occurred on May 31 and June 1, 1921.