Safety first
- Do not move switch ports, change VLANs, disable spanning tree, or cycle network switches on a live cell without operations approval.
- Do not mirror or capture traffic from networks that include regulated or confidential data without site approval.
- Changing speed/duplex, QoS, IGMP, VLANs, or IP addresses can interrupt I/O. Plan the change and rollback.
- If remote I/O or drives are involved, understand the machine state before intentionally breaking a connection for testing.
Common symptoms
- Controller I/O tree shows intermittent connection timeouts.
- HMI or SCADA tags are stale or slow.
- Drive or adapter reports embedded EtherNet/IP network loss.
- Ping works but implicit I/O or CIP messaging fails.
- Device works on a bench switch but not in the panel.
- Problems appear after switch replacement, VLAN change, duplicate device install, or cable work.
Quick checks
- Start with the exact failing path: source device, destination device, switch ports, VLANs, subnet, gateway, and protocol.
- Check link state, negotiated speed/duplex, CRC/FCS errors, late collisions, port flaps, and switch logs before touching PLC code.
- Verify IP address, subnet mask, gateway, and duplicate IP evidence on every device in the path.
- Compare controller I/O connection timeout text with managed switch timestamps.
- Confirm whether the issue is Layer 1/2 local switching, Layer 3 routing, multicast, firewall/ACL, or application-level CIP.
- Use the Networking Tools page to calculate subnets and confirm whether devices are actually in the same network.
Field procedure
Follow the sequence before changing parameters, replacing hardware, or cycling power.
1. Draw the actual path
- 1List device A, device B, all switches between them, VLANs, trunks, routed interfaces, firewall interfaces, and NAT devices.
- 2Record switch port numbers and cable labels from the physical panel, not only the drawing.
- 3Record IP, subnet mask, gateway, MAC address, device name, firmware, and controller slot/path.
- 4Identify whether the traffic is cyclic I/O, produced/consumed tags, MSG instructions, HMI polling, OPC, Modbus TCP, or web diagnostics.
2. Prove Layer 1 and Layer 2 first
- 1Check link up/down history, speed, duplex, and port errors on both sides of each link.
- 2Replace or test suspect patch cables and field terminations before changing switch configuration.
- 3Look for error counters that increment during the failure window.
- 4Confirm forced speed/duplex settings are matched on both device and managed switch port, or leave unmanaged links on auto-negotiation.
- 5Check unmanaged switches, media converters, and copper/fiber adapters that may not show in drawings.
3. Prove addressing and routing
- 1Calculate the network, broadcast, and usable host range for each endpoint.
- 2Check duplicate IP alarms, ARP table changes, and MAC address movement between switch ports.
- 3Confirm gateway settings only when traffic must leave the local subnet.
- 4If routing or firewalling is involved, prove bidirectional rules for the protocol and ports, not only ICMP ping.
- 5Check whether NAT changes the CIP path or device identity expected by the controller.
4. Prove EtherNet/IP behavior
- 1Check controller I/O tree module status, connection fault text, requested packet interval, and ownership.
- 2Compare CIP connection drops with switch port flap/error logs.
- 3For multicast I/O, verify IGMP snooping, querier presence, VLAN boundaries, and whether multicast flooding is occurring.
- 4For produced/consumed tags or MSG instructions, verify controller path, firmware compatibility, and connection limits.
- 5Capture packets only after you know the exact source, destination, and failure window.
Diagnostic groups
Use these buckets to separate evidence and avoid chasing unrelated symptoms.
Duplex or speed mismatch
Network is slow, intermittent, or works for light traffic but fails under I/O or HMI load.
Likely causes
- Device and switch port negotiated different duplex modes.
- One side is forced while the other side is auto.
- Media converter or unmanaged switch changes negotiation behavior.
- Firmware or hardware replacement changed link defaults.
Checks
- Read speed and duplex from the managed switch and device diagnostics.
- Check CRC, FCS, late collision, and alignment counters.
- Check whether errors climb when the machine runs or HMI screens poll heavily.
- Check both ends after any setting change and power cycle if required by the device.
Corrective actions
- For managed switch links, match speed and duplex on both ends when forcing settings.
- For unmanaged switch links, leave auto-negotiation enabled where the device manual requires it.
- Document the before/after port status and error counters.
- Do not assume ping success proves duplex is healthy.
Duplicate IP or wrong subnet
Device appears and disappears, ARP tables change, controller connects to the wrong device, or two devices work only when one is disconnected.
Likely causes
- Replacement device shipped with a reused static IP.
- BOOTP/DHCP assignment was not disabled or reserved correctly.
- Subnet mask lets devices appear local when they should route, or route when they should be local.
- NAT or VPN overlap creates ambiguous addressing.
Checks
- Compare ARP entries over time for the same IP address.
- Check switch MAC address table for movement between ports.
- Ping with one suspected device disconnected at a time only if production allows it.
- Use subnet calculations to verify local and remote network boundaries.
Corrective actions
- Assign unique static or reserved addresses and document them.
- Disable BOOTP/DHCP only after the intended static address is confirmed.
- Correct subnet mask and gateway settings consistently.
- Update PLC I/O tree, HMI shortcuts, drive parameters, and documentation after address changes.
Multicast flooding or I/O overload
Remote I/O drops, HMIs slow down, or non-participating ports see high traffic when multicast I/O is active.
Likely causes
- IGMP snooping disabled or no querier present on the VLAN.
- Cyclic I/O RPIs are too aggressive for the network or device.
- Multicast crosses VLANs or trunks where it should not.
- Too many devices share a low-quality unmanaged switch.
Checks
- Check switch multicast table, IGMP querier state, and high-utilization ports.
- Compare controller I/O RPIs and device connection counts.
- Mirror the port only after identifying the affected VLAN and failure time.
- Check whether unicast I/O options are available and appropriate for the device family.
Corrective actions
- Configure IGMP snooping and querier according to the site switch standard.
- Segment high-volume I/O from HMI/business traffic where practical.
- Adjust RPIs only after understanding process requirements.
- Replace unmanaged switches in critical I/O paths with managed industrial switches where evidence supports it.
Routing, firewall, or VLAN boundary issue
Local devices communicate, but cross-VLAN HMI, historian, gateway, or engineering traffic fails.
Likely causes
- Missing default gateway.
- Firewall/ACL allows ping but blocks CIP, OPC, web diagnostics, or vendor service ports.
- VLAN trunk or native VLAN mismatch.
- DNS name resolves to a different interface than expected.
Checks
- Compare local subnet traffic with routed traffic.
- Verify source and destination IPs as seen by the firewall or router.
- Check VLAN assignment on access ports and allowed VLANs on trunks.
- Test the exact application port, not only ICMP.
Corrective actions
- Correct gateway and routing only for devices that require routed traffic.
- Add narrow firewall rules for approved OT protocols and endpoints.
- Fix VLAN membership or trunk allowed lists.
- Update DNS or host records when server names are used by FactoryTalk, ThinManager, or RDS.
Evidence to capture
- Topology sketch with device names, IPs, VLANs, switch names, switch ports, and media converters.
- Managed switch port status: link, speed, duplex, errors, discards, flaps, VLAN, PoE if applicable, and MAC table.
- Controller I/O module status, fault text, RPI, and connection path.
- ARP tables or duplicate IP alarms.
- Packet capture time window, source, destination, protocol, and switch mirror point.
- Before/after configuration snippets for switch, router, firewall, and device network settings.
Escalate when
- Network change would interrupt controller I/O, drives, safety-adjacent diagnostics, or operator visibility.
- Duplicate IP evidence involves a production controller, gateway, drive, or remote I/O rack.
- Switch logs show port security, spanning tree, loop protection, or storm control events that need network-owner approval.
- Firewall or routing ownership is outside the controls team.
- Issue crosses IT/OT boundary, VPN, NAT, or iDMZ architecture.