Content
# SCADA Network VLAN consult
> **Thread Summary:** Kevin Seifert reached out to Chad Crawford for VLAN implementation expertise to address slow SCADA data transfer issues at South County Sanitation's CP4000 plant. After onsite network documentation and discussions, they identified potential fiber line hardware problems and planned to first troubleshoot the fiber connection before isolating IO traffic via VLANs on the PLC's NICs. Mason prepared a detailed network isolation plan, which Chad reviewed positively, and they agreed to proceed with hardware troubleshooting followed by network segregation if needed.
---
## 1. From: Kevin Seifert <kevin@autosysnet.com> — Thu, 26 Feb 2026 17:42:57 +0000
Hey Chad!
Do you have a good background in VLAN implementation? If so we could use your expertise at South County Sanitation.
Kevin
---
## 2. From: Chad Crawford <ccrawford@spiceintegration.com> — Thu, 26 Feb 2026 12:45:47 -0500
Hey Kevin! Yes, I do! I would love the opportunity to show them our IT
capabilities as we have been trying to land them as an IT client for a good
while.
What can I do to help?
*Chad Crawford*
*Chief Technical Officer*
SPICE Integration
*805.464.4111*
C-10 License: *1110179*
California Small Business Certification ID: *2036507*
On Thu, Feb 26, 2026 at 12:43 PM Kevin Seifert <kevin@autosysnet.com> wrote:
> Hey Chad!
>
> Do you have a good background in VLAN implementation? If so we could use
[... earlier replies trimmed ...]
---
## 3. From: Kevin Seifert <kevin@autosysnet.com> — Thu, 26 Feb 2026 18:01:37 +0000
Mason and I are out here now doing the leg work to document the current SCADA/PLC network here in the new side of the plant called CP4000.
Our issue is SCADA is extremely slow to get data to/from this new PLC. There is a fiber line coming out here, so pretty sure it's not due to a slow trunk line.
There are about 30 VFDs/Smart Overloads in this MCC, plus three other PLCs that the CP4000 PLC communicates with. All of this is on the same managed switch and the same port on the CompactLogix PLC, along with the connection to SCADA.
We are thinking that by isolating all the IO to one of the two NICs on the PLC, thereby getting it off of the main scada net, that will help. But we are unsure of the exact logistics of such.
I will forward the result of our work here today, and we can discuss further.
-Kevin Seifert, AutoSys LLC
________________________________
From: Chad Crawford <ccrawford@spiceintegration.com>
Sent: Thursday, February 26, 2026 9:45 AM
To: Kevin Seifert <kevin@autosysnet.com>
Cc: John Bowers <jbowers@spiceintegration.com>; Mason Radke <mason@autosysnet.com>
Subject: Re: SCADA Network VLAN consult
Hey Kevin! Yes, I do! I would love the opportunity to show them our IT capabilities as we have been trying to land them as an IT client for a good while.
What can I do to help?
Chad Crawford
Chief Technical Officer
SPICE Integration
805.464.4111
C-10 License: 1110179
California Small Business Certification ID: 2036507
[https://ci3.googleusercontent.com/mail-sig/AIorK4wQmJMpYT2The9LtRL2Xv-kJwd9DoLcdOXjleFUqX99PYaxvwrfFkcagMimKJ0X9cNcvP_05EUH06U9]
On Thu, Feb 26, 2026 at 12:43 PM Kevin Seifert <kevin@autosysnet.com<mailto:kevin@autosysnet.com>> wrote:
Hey Chad!
Do you have a good background in VLAN implementation? If so we could use your expertise at South County Sanitation.
Kevin
---
## 4. From: Chad Crawford <ccrawford@spiceintegration.com> — Thu, 26 Feb 2026 13:10:22 -0500
Perfect. Sounds like a plan! I'm fairly certain both NICs on the PLC share
the same internal microcontroller so regarding overall throughput I'm not
certain that would improve things. However this fiber line that connects
CP4000 to the rest of the network; does it use Media converters/SFP
Modules? It may be that the SFP modules are unmatched/faililng/not the
correct wavelength or fiber line is kinked somewhere. Just to rule it out
completely I would do a throughput test on each side of that fiber run,
possibly use an OTDR to see if there is any signal loss or leakage
happening in the specific fiber strand.
You can call me whenever you'd like to discuss.
*Chad Crawford*
*Chief Technical Officer*
SPICE Integration
*805.464.4111*
C-10 License: *1110179*
California Small Business Certification ID: *2036507*
On Thu, Feb 26, 2026 at 1:01 PM Kevin Seifert <kevin@autosysnet.com> wrote:
> Mason and I are out here now doing the leg work to document the current
> SCADA/PLC network here in the new side of the plant called CP4000.
>
[... earlier replies trimmed ...]
---
## 5. From: Kevin Seifert <kevin@autosysnet.com> — Fri, 27 Feb 2026 02:53:51 +0000
Chad,
Mason worked up a highly detailed report and plan of attack for the network isolation. Take a look at it and let us know if you see any faults or red flags here.
After my discussions with you today I think we will proceed to first do some troubleshooting of the network hardware as you suggested, and then move to segregation if all checks out.
Thanks for your time!
-Kevin Seifert, AutoSys LLC
________________________________
From: Chad Crawford <ccrawford@spiceintegration.com>
Sent: Thursday, February 26, 2026 10:10 AM
To: Kevin Seifert <kevin@autosysnet.com>
Cc: John Bowers <jbowers@spiceintegration.com>; Mason Radke <mason@autosysnet.com>
Subject: Re: SCADA Network VLAN consult
Perfect. Sounds like a plan! I'm fairly certain both NICs on the PLC share the same internal microcontroller so regarding overall throughput I'm not certain that would improve things. However this fiber line that connects CP4000 to the rest of the network; does it use Media converters/SFP Modules? It may be that the SFP modules are unmatched/faililng/not the correct wavelength or fiber line is kinked somewhere. Just to rule it out completely I would do a throughput test on each side of that fiber run, possibly use an OTDR to see if there is any signal loss or leakage happening in the specific fiber strand.
You can call me whenever you'd like to discuss.
Chad Crawford
Chief Technical Officer
SPICE Integration
805.464.4111
C-10 License: 1110179
California Small Business Certification ID: 2036507
[https://ci3.googleusercontent.com/mail-sig/AIorK4wQmJMpYT2The9LtRL2Xv-kJwd9DoLcdOXjleFUqX99PYaxvwrfFkcagMimKJ0X9cNcvP_05EUH06U9]
On Thu, Feb 26, 2026 at 1:01 PM Kevin Seifert <kevin@autosysnet.com<mailto:kevin@autosysnet.com>> wrote:
Mason and I are out here now doing the leg work to document the current SCADA/PLC network here in the new side of the plant called CP4000.
Our issue is SCADA is extremely slow to get data to/from this new PLC. There is a fiber line coming out here, so pretty sure it's not due to a slow trunk line.
There are about 30 VFDs/Smart Overloads in this MCC, plus three other PLCs that the CP4000 PLC communicates with. All of this is on the same managed switch and the same port on the CompactLogix PLC, along with the connection to SCADA.
We are thinking that by isolating all the IO to one of the two NICs on the PLC, thereby getting it off of the main scada net, that will help. But we are unsure of the exact logistics of such.
I will forward the result of our work here today, and we can discuss further.
-Kevin Seifert, AutoSys LLC
________________________________
From: Chad Crawford <ccrawford@spiceintegration.com<mailto:ccrawford@spiceintegration.com>>
Sent: Thursday, February 26, 2026 9:45 AM
To: Kevin Seifert <kevin@autosysnet.com<mailto:kevin@autosysnet.com>>
Cc: John Bowers <jbowers@spiceintegration.com<mailto:jbowers@spiceintegration.com>>; Mason Radke <mason@autosysnet.com<mailto:mason@autosysnet.com>>
Subject: Re: SCADA Network VLAN consult
Hey Kevin! Yes, I do! I would love the opportunity to show them our IT capabilities as we have been trying to land them as an IT client for a good while.
What can I do to help?
Chad Crawford
Chief Technical Officer
SPICE Integration
805.464.4111
C-10 License: 1110179
California Small Business Certification ID: 2036507
[https://ci3.googleusercontent.com/mail-sig/AIorK4wQmJMpYT2The9LtRL2Xv-kJwd9DoLcdOXjleFUqX99PYaxvwrfFkcagMimKJ0X9cNcvP_05EUH06U9]
On Thu, Feb 26, 2026 at 12:43 PM Kevin Seifert <kevin@autosysnet.com<mailto:kevin@autosysnet.com>> wrote:
Hey Chad!
Do you have a good background in VLAN implementation? If so we could use your expertise at South County Sanitation.
Kevin
## Attachments
- [[20260226_19c9d0463169_ICS_Network_Segmentation_Plan_Rev3.2_FINAL.pdf]]
---
## 6. From: Chad Crawford <ccrawford@spiceintegration.com> — Fri, 27 Feb 2026 12:40:53 -0500
Sounds good Kevin! This is a beautiful write-up, Mason. I'd agree that
those are important security considerations, and that the network
segmentation overall is worth doing.
I'd feel more confident your solution will solve the problem if we can
prove first that the latency experienced is being caused by aforementioned
ARP broadcast.
You mentioned the network may be experiencing a broadcast storm, this most
commonly happens when a switching loop exists somewhere in the circuit
(which Ethernet/IP I/O devices can inadvertently cause). Your document
confirms that RSTP (Rapid Spanning Tree Protocol) is disabled on all ports.
This protocol protects against Switching loops (which could very well be
the cause of this latency issue). RTSP creates a loop-free logical
topology, blocking redundant paths while allowing for rapid convergence.
Did you guys happen to do a wireshark scan while onsite?
This would give you the details on exactly what is going on (Packets
Dropped, ARP Consistency, Etc.)
Let me know your thoughts!
*Chad Crawford*
*Chief Technical Officer*
SPICE Integration
*805.464.4111*
C-10 License: *1110179*
California Small Business Certification ID: *2036507*
On Thu, Feb 26, 2026 at 9:53 PM Kevin Seifert <kevin@autosysnet.com> wrote:
> Chad,
>
> Mason worked up a highly detailed report and plan of attack for the
[... earlier replies trimmed ...]
---
## 7. From: Chad Crawford <ccrawford@spiceintegration.com> — Mon, 2 Mar 2026 13:45:06 -0500
Understood!
SPICE is happy to support AutoSys however we can, I just wanted to ensure
our transparency in case this was already seen as the definitive solution.
Either way we are here to help get to the bottom of it.
Segmentation is always an excellent choice for reliability *and *security.
Let me know what you guys need from me to get started. I'd be more than
happy to be your eye in the sky next time you are onsite so we can examine
the physical network layout and the symptoms together at your convenience.
Thanks Mason!
*Chad Crawford*
*Chief Technical Officer*
SPICE Integration
*805.464.4111*
C-10 License: *1110179*
California Small Business Certification ID: *2036507*
On Fri, Feb 27, 2026 at 10:18 PM Mason Radke <mason@autosysnet.com> wrote:
> I dont know how confident I am at all that this will solve the problem!
> Haha.. but as a test, we wanted to try to segregate this traffic.. so this
> plan is geared towards the segregation in order to see if it helps. I love
[... earlier replies trimmed ...]
---
## 8. From: Kevin Seifert <kevin@autosysnet.com> — Tue, 3 Mar 2026 01:15:45 +0000
Chad,
I will let you know when I receive the fiber tester and am ready to head back out there for some diagnostics. We can schedule it then.
Thanks!
-Kevin Seifert, AutoSys LLC
________________________________
From: Chad Crawford <ccrawford@spiceintegration.com>
Sent: Monday, March 2, 2026 10:45 AM
To: Mason Radke <mason@autosysnet.com>
Cc: Kevin Seifert <kevin@autosysnet.com>; John Bowers <jbowers@spiceintegration.com>
Subject: Re: SCADA Network VLAN consult
Understood!
SPICE is happy to support AutoSys however we can, I just wanted to ensure our transparency in case this was already seen as the definitive solution. Either way we are here to help get to the bottom of it.
Segmentation is always an excellent choice for reliability and security. Let me know what you guys need from me to get started. I'd be more than happy to be your eye in the sky next time you are onsite so we can examine the physical network layout and the symptoms together at your convenience.
Thanks Mason!
Chad Crawford
Chief Technical Officer
SPICE Integration
805.464.4111
C-10 License: 1110179
California Small Business Certification ID: 2036507
[https://ci3.googleusercontent.com/mail-sig/AIorK4wQmJMpYT2The9LtRL2Xv-kJwd9DoLcdOXjleFUqX99PYaxvwrfFkcagMimKJ0X9cNcvP_05EUH06U9]
On Fri, Feb 27, 2026 at 10:18 PM Mason Radke <mason@autosysnet.com<mailto:mason@autosysnet.com>> wrote:
I dont know how confident I am at all that this will solve the problem! Haha.. but as a test, we wanted to try to segregate this traffic.. so this plan is geared towards the segregation in order to see if it helps. I love the idea of running some tests to see if we can identify other issues or bottle necks.
—Mason Radke
—Autosys, LLC
From: Chad Crawford <ccrawford@spiceintegration.com<mailto:ccrawford@spiceintegration.com>>
Date: Friday, February 27, 2026 at 9:41 AM
To: Kevin Seifert <kevin@autosysnet.com<mailto:kevin@autosysnet.com>>
Cc: John Bowers <jbowers@spiceintegration.com<mailto:jbowers@spiceintegration.com>>, Mason Radke <mason@autosysnet.com<mailto:mason@autosysnet.com>>
Subject: Re: SCADA Network VLAN consult
Sounds good Kevin! This is a beautiful write-up, Mason. I'd agree that those are important security considerations, and that the network segmentation overall is worth doing.
I'd feel more confident your solution will solve the problem if we can prove first that the latency experienced is being caused by aforementioned ARP broadcast.
You mentioned the network may be experiencing a broadcast storm, this most commonly happens when a switching loop exists somewhere in the circuit (which Ethernet/IP I/O devices can inadvertently cause). Your document confirms that RSTP (Rapid Spanning Tree Protocol) is disabled on all ports. This protocol protects against Switching loops (which could very well be the cause of this latency issue). RTSP creates a loop-free logical topology, blocking redundant paths while allowing for rapid convergence.
Did you guys happen to do a wireshark scan while onsite?
This would give you the details on exactly what is going on (Packets Dropped, ARP Consistency, Etc.)
Let me know your thoughts!
Chad Crawford
Chief Technical Officer
SPICE Integration
805.464.4111
C-10 License: 1110179
California Small Business Certification ID: 2036507
[https://ci3.googleusercontent.com/mail-sig/AIorK4wQmJMpYT2The9LtRL2Xv-kJwd9DoLcdOXjleFUqX99PYaxvwrfFkcagMimKJ0X9cNcvP_05EUH06U9]
On Thu, Feb 26, 2026 at 9:53 PM Kevin Seifert <kevin@autosysnet.com<mailto:kevin@autosysnet.com>> wrote:
Chad,
Mason worked up a highly detailed report and plan of attack for the network isolation. Take a look at it and let us know if you see any faults or red flags here.
After my discussions with you today I think we will proceed to first do some troubleshooting of the network hardware as you suggested, and then move to segregation if all checks out.
Thanks for your time!
-Kevin Seifert, AutoSys LLC
________________________________
From: Chad Crawford <ccrawford@spiceintegration.com<mailto:ccrawford@spiceintegration.com>>
Sent: Thursday, February 26, 2026 10:10 AM
To: Kevin Seifert <kevin@autosysnet.com<mailto:kevin@autosysnet.com>>
Cc: John Bowers <jbowers@spiceintegration.com<mailto:jbowers@spiceintegration.com>>; Mason Radke <mason@autosysnet.com<mailto:mason@autosysnet.com>>
Subject: Re: SCADA Network VLAN consult
Perfect. Sounds like a plan! I'm fairly certain both NICs on the PLC share the same internal microcontroller so regarding overall throughput I'm not certain that would improve things. However this fiber line that connects CP4000 to the rest of the network; does it use Media converters/SFP Modules? It may be that the SFP modules are unmatched/faililng/not the correct wavelength or fiber line is kinked somewhere. Just to rule it out completely I would do a throughput test on each side of that fiber run, possibly use an OTDR to see if there is any signal loss or leakage happening in the specific fiber strand.
You can call me whenever you'd like to discuss.
Chad Crawford
Chief Technical Officer
SPICE Integration
805.464.4111
C-10 License: 1110179
California Small Business Certification ID: 2036507
[https://ci3.googleusercontent.com/mail-sig/AIorK4wQmJMpYT2The9LtRL2Xv-kJwd9DoLcdOXjleFUqX99PYaxvwrfFkcagMimKJ0X9cNcvP_05EUH06U9]
On Thu, Feb 26, 2026 at 1:01 PM Kevin Seifert <kevin@autosysnet.com<mailto:kevin@autosysnet.com>> wrote:
Mason and I are out here now doing the leg work to document the current SCADA/PLC network here in the new side of the plant called CP4000.
Our issue is SCADA is extremely slow to get data to/from this new PLC. There is a fiber line coming out here, so pretty sure it's not due to a slow trunk line.
There are about 30 VFDs/Smart Overloads in this MCC, plus three other PLCs that the CP4000 PLC communicates with. All of this is on the same managed switch and the same port on the CompactLogix PLC, along with the connection to SCADA.
We are thinking that by isolating all the IO to one of the two NICs on the PLC, thereby getting it off of the main scada net, that will help. But we are unsure of the exact logistics of such.
I will forward the result of our work here today, and we can discuss further.
-Kevin Seifert, AutoSys LLC
________________________________
From: Chad Crawford <ccrawford@spiceintegration.com<mailto:ccrawford@spiceintegration.com>>
Sent: Thursday, February 26, 2026 9:45 AM
To: Kevin Seifert <kevin@autosysnet.com<mailto:kevin@autosysnet.com>>
Cc: John Bowers <jbowers@spiceintegration.com<mailto:jbowers@spiceintegration.com>>; Mason Radke <mason@autosysnet.com<mailto:mason@autosysnet.com>>
Subject: Re: SCADA Network VLAN consult
Hey Kevin! Yes, I do! I would love the opportunity to show them our IT capabilities as we have been trying to land them as an IT client for a good while.
What can I do to help?
Chad Crawford
Chief Technical Officer
SPICE Integration
805.464.4111
C-10 License: 1110179
California Small Business Certification ID: 2036507
[https://ci3.googleusercontent.com/mail-sig/AIorK4wQmJMpYT2The9LtRL2Xv-kJwd9DoLcdOXjleFUqX99PYaxvwrfFkcagMimKJ0X9cNcvP_05EUH06U9]
On Thu, Feb 26, 2026 at 12:43 PM Kevin Seifert <kevin@autosysnet.com<mailto:kevin@autosysnet.com>> wrote:
Hey Chad!
Do you have a good background in VLAN implementation? If so we could use your expertise at South County Sanitation.
Kevin
---