👀 ix.watch: who's leaking onto the peering LAN?

· 7 min read

Back in March I wrote a small Python script that sniffs my own IXP ports for traffic that has no business being there. Router advertisements, spanning tree BPDUs, DHCP servers, that kind of thing. When it caught something, it figured out who owned the MAC address, saved a pcap and drafted a polite email to their NOC. I’d read the draft, click send, and wait.

That worked. It also meant every single leak on every single exchange went through me, one email at a time. So the script grew up.

At some point last month I published it as a page on bgp.rodeo and casually dropped it in an IRC-channel when someone brought up weird packets they say on an IXPs LAN. Since then the tool got linked and praised a couple of times.

So, when things get serious, they need their own domain: ix.watch. Worth every cent right? It’s such a cool name.

What it is

ix.watch is a public, live list of every network sending something onto an IXP peering LAN that shouldn’t be there.

A peering LAN is a big shared Layer 2 segment. Every member’s router sits on the same broadcast domain, which means that one router advertisement or one stray BPDU reaches every port on the fabric. The rules are simple: IP unicast, ARP and IPv6 neighbour discovery between routers, and nothing else. Most networks get this right. A surprising number don’t.

Right now ix.watch watches seven exchanges: AMS-IX, Speed-IX, Frys-IX, InterIX, Nine-IX, Clue-IX and ENS-IX. At the time of writing, it lists 142 sources from 112 different networks. (A “source” is one MAC address on one IX. It drops off the list 7 days after its last frame.)

The front page is the table: who’s sending, on which IX, what, and when it was last seen. It updates live over server-sent events, so you can literally watch the leaks roll in. Below that there’s a faceplate, one line card per IXP and one port per source, with an LED that flickers whenever a frame comes in. Is that necessary? No. Did I enjoy building it? Very much.

What it looks for

The sensor flags everything it can recognise that isn’t plain unicast, ARP or ND:

  • IPv6 router advertisements, by far the most common one (67 of the 142 sources right now)
  • ARP or ND from outside the LAN: someone asking for a peering LAN address from an IP that isn’t on the peering LAN. That’s 49 of them, mostly people with the wrong subnet or a secondary address on the IX interface.
  • DHCP and DHCPv6, with the port pair required to match, because false positives are embarrassing when you publish them
  • LLDP, CDP and spanning tree BPDUs
  • MikroTik’s whole family: MNDP, MAC-Telnet/Winbox, loop protect and RoMON
  • Syncthing local discovery, which means somebody’s workstation is bridged onto an exchange.
  • Wrong netmask: IPv4 broadcasts to a smaller subnet’s broadcast address, which means the sender’s prefix is too long
  • Ubiquiti discovery, reverse ARP, IPv6 to the Ethernet broadcast address, and a catch-all for other broadcast and multicast

The vendor breakdown is fun too. Arista and Cisco lead, which mostly says something about who’s on the big exchanges. But there are also 12 sources with a Proxmox MAC address. That’s someone’s VM, bridged straight onto an internet exchange. I’m not judging. (I’m judging a little.)

Pcaps or it didn’t happen

Every network gets its own page at ix.watch/as<ASN>. It shows what we caught, the decoded headers of the last few frames, hourly counts over the past week, and a pcap you can open in Wireshark. “Trust me bro” doesn’t work in networking, and it definitely doesn’t work when you’re publicly listing someone’s network.

The page also shows the fix. Almost all of these come down to one line of config on the IX-facing interface, and the page picks the snippet for the hardware we saw. Sending RAs from a Cisco?

interface TenGigabitEthernet0/0/0
 ipv6 nd ra suppress all

That’s it. That’s the whole fix. The point is that a network should be able to land on its page, understand the problem and fix it in five minutes, without anyone emailing anyone.

How it works

There are a few moving parts, each in its own repo:

  • The sensor is a single Python file. It opens a raw socket in promiscuous mode on the peering interface, and a kernel BPF filter drops plain unicast IP before it ever reaches Python, so it barely uses any CPU even on a busy LAN. It reports what it sees over MQTT.
  • The watcher is what runs on my own routers. It’s the sensor plus the old March bits: pcaps, MAC-to-network resolution (neighbour table and PeeringDB first, then bgp.tools) and NOC email drafts.
  • The API collects everything from MQTT, ties sources to networks and serves it publicly.
  • The site is a static Astro build that reads the API.
  • The alerts service sends mail. More on that below.

All of it runs on my own network, AS202585, on the Talos cluster in my rack.

Alerts

A public list is nice, but it only helps if the right person looks at it. So at alerts.ix.watch you can log in with PeeringDB and subscribe to your own network. You get a mail when one of your sources starts leaking, a daily reminder while it goes on, and a mail when it stops (a full day without frames). Every mail includes the pcap.

Logging in through PeeringDB OAuth means I don’t have to verify who belongs to which network. PeeringDB already did that. If you’re affiliated with the network there, you can subscribe to it.

It’s free for networks that bgp.tools tags as Personal. Hobby networks like mine are exactly the ones on the small free exchanges, and exactly the ones that occasionally forget to turn off RAs on a MikroTik. Commercial networks will be a paid tier eventually. That part isn’t built yet, so for now you can subscribe and you’ll sit on a waiting list.

If you run an IX, you get something extra. Log in with an account affiliated with the organisation that runs the exchange, and you get a page for your IX with everyone currently leaking onto your LAN, plus free alerts for the whole fabric.

The data is yours

Everything on ix.watch is public, and the site is built on the same API anyone can use. No key, no sign-up, CORS is open.

# who's sending right now
curl -s https://api.ix.watch/ix-violators.json | jq '.sources[] | select(.last > now - 300) | .name'

# follow the live stream
curl -N https://api.ix.watch/ix-violators/events

There’s a per-network JSON and pcap too. The data page has the details.

Add your IX

ix.watch only sees the LANs it’s connected to, which is currently the exchanges I happen to peer at. There are two ways to change that.

You’re a member somewhere else? Run the sensor. It’s one Python file, MIT licensed, needs Python 3.9+ and paho-mqtt, and the config is a handful of lines:

{
  "name": "as64500",
  "interfaces": { "eth1.100": "NL-IX" },
  "mqtt": { "url": "wss://mqtt.bgp.rodeo/",
            "username": "as64500", "password": "..." }
}

Your IX shows up on ix.watch, credited “via” your network. Mail noc at ix dot watch with your ASN and IX and I’ll send you an account.

You run an IX? Give me a port. The smallest one you offer is plenty, since all it does is listen to broadcast, multicast and what’s sent to it. Your members get a page telling them exactly what’s wrong and how to fix it, which is fewer tickets for your NOC to chase. Rather keep it in-house? Run the sensor on something that’s already on the LAN, like a route server or a looking glass.

Keeping the LEDs on

ix.watch is free to read and free to use. It does cost money though. The domain alone was €65. (Yes, really. For two letters and a dot.) Every new IX needs a port and usually some transport to get there, and then there’s power, colocation and my evenings. If it helped you find a leak, or you just like a clean peering LAN, there’s a donate page.

And if your network shows up on the list: no hard feelings. Check your page, paste the line, and you’ll drop off within a week. That’s the whole idea.

If that site says ‘clean’ for all IXs within a week; the mission is done. But I’ll keep watching! 👀