Nobody's Fault but Mine

Just before going to the upcoming 2024 IEC 61850 Interoperability Meeting (IOP) of the UCA International User Group in Birmingham, Alabama and also triggered by a similar case that came up during a proof of concept installation, I want to share an experience that we made already back at the IOP 2022 in Milan.

There was an OMICRON StationGuard intrusion detection system connected to observe the communication network. StationGuard had no official role in the testing schedule. We were just curious what we would get to see. And our curiosity was well awarded.

There was a mirror port configured on an Ethernet switch to sniff off the traffic from one of the PRP LANs and StationGuard was connected to this.

StationGuard is made for well defined, mostly static scenarios, where the devices are described in an SCD file. But at the IOP, which is effectively a big plugfest, only parts of the devices on the network (the ones which were part of the IEC 61850 application) were properly described in the SCD file. There were at least as many devices (networking devices, notebooks for device setting and test control, etc.) that were not in the SCD file. They popped up as unknown, requiring manual identification and role assignments.

But there was even more going on. Under certain conditions, new devices popped up, about two to four times per minute. And they came in pairs which seemed to communicate to each other only once. I quickly recognized that these are most likely not real devices.

That the devices appeared in pairs is easy to explain in hindsight. Whenever StationGuard encounters an Ethernet packet with either a yet unknown destination or source address, it creates a new unknown device entry. If there is traffic to a formerly unknown destination, it may draw a red line to there from an already existing source. If there is traffic from a formerly unknown source, it may draw a red line from there to an already existing destination. Whether the line is drawn depends if the traffic violates the permissions assigned to the roles of the involved devices. As the newly created devices have no role assigned by default, it is always a violation and the line is drawn.

In the Ethernet packets detected in this case, both the destination and the source MAC address were formerly unknown, so both devices were created and a line between them was drawn. But no further traffic between the two took place later on.

Before going further, one thing must be pointed out: In the course of the investigations, source MAC addresses of the form 00-01-00-61-xx-30 played a prominent role. And yes, we know that the OUI field (the organizationally unique identifier, the first three bytes designated to the "owner" of the address space) would refer to the company Equip'trans in France. But this hint is entirely misleading. There was no equipment of Equip'trans involved, the MAC addresses came not from a real Ethernet controller, but emerged from a malfunction of a component on the network.

As said, source MAC addresses of the from 00-01-00-61-xx-30 appeared more often than others. The "xx" was also not entirely random, but counting up in increments of two to four between occurrences of MAC addresses with this pattern. This caused always new sources to be created. This bit of determinism helped us to recognize a pattern and track down the issue. The destination addresses were more random, also causing new destinations to be created, forming the mentioned pairs.

Patrice Roussel from MOXA had captured a 250MB PCAP file covering the time span when I first noticed the malformed packets. The capture file contained traffic from both PRP LANs and the malformed packets appeared on both of them. So they must have been coming from a doubly attached node or through a RedBox.

Joining with Goran Pregrad from Pro Integris we analyzed the capture files which are routinely saved by StationGuard for such forensic purposes. Goran developed a particular eagerness to get to the bottom of the matter. He was incredibly adept at using filter conditions in Wireshark and finding patterns. And we found something. The malformed packets were fragments from valid MMS packets that were exchanged before. This is depicted in the packet dumps shown below.

In the case shown above, the malformed packet appeared about 270 milliseconds after the MMS packet, but the times between such pairs varied up to several seconds. The malformed packets were always 60 bytes in length. In the MMS packet dump, the portion that became the malformed packet is outlined. These are the 56 bytes from position $38 to $6f. The four bytes at the end of the malformed packet are not copied from position $70 to $73 of the MMS packet, instead they are always set to zero. This pattern appeared over and over again, always causing a new transaction between unknown devices popping up in StationGuard.

We were unable to conclude why these fragments were sent out to the network, but we were able to pinpoint where they were coming from. As we explained this to the owner of the suspected equipment, he guessed that his USB-to-Ethernet adapter might be the suspect. He gave it to me for further investigations, so I could possibly recreate the issue and confirm the suspicion. I did set up a similar client/server scenario with this USB-to-Ethernet adapter involved, but the malformed packets did not appear in my setup. This suggests that the cause might be the software running on the device issuing the malformed packets.

What we experienced with StationGuard at the IOP 2022 had no effect on the testing activities at all. The malformed packets were not caused by a cyber attack and did not cause any damage. But imagine they were: we got aware of something which had gone completely unnoticed if there was not StationGuard capturing it. This shows that we need the right tools to detect such issues and the right skills to trace them back to the source. The teamwork with the colleagues to track down the cause was a great experience and we got a refresher in how set up filters in Wireshark.

That the whole thing happened at all, was nobody's fault but mine, since I brought the StationGuard to the IOP. And I will do so again.