IPSec functions at Layer 3, providing security by using end-to-end tunnels. These are encrypted only at the ends of each tunnel. A major drawback to IPSec is its complexity. Not only does it typically entail a dedicated encryption engine, but IPSec significantly enlarges the size of the Ethernet header. This compounds network inefficiencies and adds to overall solution cost.
In contrast, MACsec is a relatively simple protocol, which only minimally expands the header. Because MACsec is usually PHY port-based, it supports easy upgrades and high-speed connectivity up to 100G at low power and low cost. Unlike IPSec, it’s possible to implement MACsec as a simple line-card upgrade and without a dedicated security processor (see "Security Essentials for the Internet of Things").
Furthermore, MACsec can scale linearly with the number of links in hop-by-hop scenarios, and with the number of endpoints in end-to-end applications. An IPSec engine, on the other hand, can support only a certain amount of total capacity and a specific number of tunnels per port.
However, the two protocols are compatible and can be very complementary. A tag- and flow-based MACsec enhances IPSec on two levels. First, in network equipment that’s either too costly or overly power-hungry, it’s now feasible to convert it to something MACsec-based only. Second, looking at wireless network security to the level of small cells, the last mile-link between the small cell and central office no longer must be IPSec—it, too, could be purely MACsec-based.