1. Network Troubleshooting for Industrial Automation
In recent weeks, I’ve been immersed in network basics training for colleagues. However, I’ve taken time to compile my recent PROFINET IRT network troubleshooting experience to share critical insights with you.
This time I share the problems I found when I was doing network diagnosis in a pharmaceutical equipment manufacturer. The company is planning to develop a new pharmaceutical machine. The automatic control system of this model uses S7-1500 CPU, ET 200 SP distributed I/O and SINAMICS S120. By debugging the communication method between S7-1500 CPU and SINAMICS S120 drive using PROFINET RT, the system can run normally and meet the requirements of existing processes. However, in order to compete with competitors in future product performance differentiation, users hope to test the application of PROFINET IRT isochronous synchronization between S7-1500 CPU and SINAMICS S120 drive on the existing basis.
The network topology of the site is shown in Figure 1. The red Switch 1 is a Weidmuller switch. Interface 1 of the S7-1500 CPU PN port is connected to the Weidmuller switch, and interface 2 of the S7-1500 CPU PN port is connected to the subsequent SINAMICS S120. The S7-1500 CPU and other distributed IOs such as ET200SP, BPS, and EX260 communicate via PROFINET RT through Switch 1. The S7-1500 CPU and SINAMICS S120 communicate via PROFINET IRT.
When the communication between the S7-1500 CPU and SINAMICS S120 is changed from RT mode to IRT mode, the CPU can work normally at first, but after a few minutes, an error message as shown in Figure 2 will appear and the communication will be interrupted.
Figure 2. Diagnostic information of CPU diagnostic cache.
The project configuration was checked on site and it was found that the IRT configuration was correct. There was no problem with the synchronization domain configuration error. Check whether the problem was caused by the Weidmuller switch. So the data packets were captured on the network cable between the Weidmuller switch and the S7-1500 CPU to see if there were any abnormal packets. The captured packets are shown in Figure 3 below.
Figure 3. Data frames captured by Wireshark.
From the packets in the figure above, it can be seen that there are both RT packets and wire delay measurement packets passing through the Weidmuller switch. Filter the clock synchronization packets and analyze the results shown in Figure 4 below.
Figure 4. PROFINET line delay measurement data frame
From the above figure, we can see that there are a lot of line delay measurement messages in the network (multiple messages appear in almost 1ms), and these line delay measurement messages come from different devices (source MAC addresses are not unique). Replace the Weidmuller switch with a Siemens switch and capture the message filtering clock synchronization message in the same place as shown in Figure 5 below.
Figure 5: PROFINET line delay measurement data frame after replacing with Siemens switch
From Figure 5, we can see that the line delay is sent every 30ms, and there is only one source MAC address. This is the difference between Siemens switch and Weidmuller switch. After replacing with Siemens switch, the system ran for nearly 2 hours without any fault.
The cause of the fault is that Weidmuller switch cannot filter the multicast message of clock synchronization, which causes the CPU to be unable to correctly calculate the line delay of IRT, and finally causes the synchronization of IRT synchronization domain to be abnormal, and finally leads to system interruption.
The conclusion drawn from the handling of this problem is that it is best to use PN switch in PROFINET network to avoid some unnecessary troubles.