Content
# Re: Laguna SCADA issue
> **Thread Summary:** The email thread discusses troubleshooting and resolving a Laguna SCADA system issue where one engine was stuck in a sync cycle between two servers. David Noellsch stopped the problematic engines, temporarily ran everything on server 2, and then performed a repair install on server 1. The system was successfully restored with both servers operational, engines running normally, and performance monitored to be nominal.
---
## 1. From: Kevin Seifert <kevin@autosysnet.com> — Tue, 21 Apr 2026 19:15:10 +0000
Adding @Mason Radke<mailto:mason@autosysnet.com> to the chain
-Kevin Seifert, AutoSys LLC
________________________________
From: David Noellsch <DavidN@CannonCorp.us>
Sent: Tuesday, April 21, 2026 12:13 PM
To: Kevin Seifert <kevin@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
Kevin,
Thanks for sending. This is some good background info. I agree a repair install is likely needed. Let me try to login. I assume VPN remote connection is still working.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://www.linkedin.com/company/cannon->
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 2:01 PM
To: David Noellsch <DavidN@CannonCorp.us>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Here is the hand-off information Mason gave me.
Let me know what you think.
Thanks in advance.
Kevin Seifert
![[20260421_19db177da855_Outlook-0m0loqwi.png]]
AutoSys LLC
kevin@autosysnet.com<mailto:kevin@autosysnet.com>
805-835-9595
---
## 2. From: David Noellsch <DavidN@cannoncorp.us> — Tue, 21 Apr 2026 19:49:53 +0000
I've stopped the platform and all engines on app-server-1. One of the engines was stuck in a sync cycle with its redundant partner on server 2. I stopped the partner engine on server 2 temporarily to allow a clean termination on server 1.
Everything is running on server 2 now. I'm monitoring for a few minutes. Then I will perform a repair install on server 1.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://www.linkedin.com/company/cannon->
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 2:15 PM
To: David Noellsch <DavidN@CannonCorp.us>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Adding @Mason Radke<mailto:mason@autosysnet.com> to the chain
-Kevin Seifert, AutoSys LLC
________________________________
From: David Noellsch <DavidN@CannonCorp.us>
Sent: Tuesday, April 21, 2026 12:13 PM
To: Kevin Seifert <kevin@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
Kevin,
Thanks for sending. This is some good background info. I agree a repair install is likely needed. Let me try to login. I assume VPN remote connection is still working.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://www.linkedin.com/company/cannon->
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 2:01 PM
To: David Noellsch <DavidN@CannonCorp.us>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Here is the hand-off information Mason gave me.
Let me know what you think.
Thanks in advance.
Kevin Seifert
![[20260421_19db197b27a4_Outlook-0m0loqwi.png]]
AutoSys LLC
kevin@autosysnet.com<mailto:kevin@autosysnet.com>
805-835-9595
---
## 3. From: Kevin Seifert <kevin@autosysnet.com> — Tue, 21 Apr 2026 20:08:32 +0000
Thank you David
-Kevin Seifert, AutoSys LLC
________________________________
From: David Noellsch <DavidN@CannonCorp.us>
Sent: Tuesday, April 21, 2026 12:49 PM
To: Kevin Seifert <kevin@autosysnet.com>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
I've stopped the platform and all engines on app-server-1. One of the engines was stuck in a sync cycle with its redundant partner on server 2. I stopped the partner engine on server 2 temporarily to allow a clean termination on server 1.
Everything is running on server 2 now. I'm monitoring for a few minutes. Then I will perform a repair install on server 1.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://www.linkedin.com/company/cannon->
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 2:15 PM
To: David Noellsch <DavidN@CannonCorp.us>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Adding @Mason Radke<mailto:mason@autosysnet.com> to the chain
-Kevin Seifert, AutoSys LLC
________________________________
From: David Noellsch <DavidN@CannonCorp.us>
Sent: Tuesday, April 21, 2026 12:13 PM
To: Kevin Seifert <kevin@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
Kevin,
Thanks for sending. This is some good background info. I agree a repair install is likely needed. Let me try to login. I assume VPN remote connection is still working.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://www.linkedin.com/company/cannon->
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 2:01 PM
To: David Noellsch <DavidN@CannonCorp.us>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Here is the hand-off information Mason gave me.
Let me know what you think.
Thanks in advance.
Kevin Seifert
![[20260421_19db1a8b97c5_Outlook-0m0loqwi.png]]
AutoSys LLC
kevin@autosysnet.com<mailto:kevin@autosysnet.com>
805-835-9595
---
## 4. From: David Noellsch <DavidN@cannoncorp.us> — Tue, 21 Apr 2026 20:50:32 +0000
System is restored. Both servers are operational, and engines are running on both servers. I've been monitoring and checking for the last ~20 minutes. Performance looks nominal, including Historian/trending.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://www.linkedin.com/company/cannon->
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 3:08 PM
To: David Noellsch <DavidN@CannonCorp.us>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Thank you David
-Kevin Seifert, AutoSys LLC
________________________________
From: David Noellsch <DavidN@CannonCorp.us>
Sent: Tuesday, April 21, 2026 12:49 PM
To: Kevin Seifert <kevin@autosysnet.com>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
I've stopped the platform and all engines on app-server-1. One of the engines was stuck in a sync cycle with its redundant partner on server 2. I stopped the partner engine on server 2 temporarily to allow a clean termination on server 1.
Everything is running on server 2 now. I'm monitoring for a few minutes. Then I will perform a repair install on server 1.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://www.linkedin.com/company/cannon->
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 2:15 PM
To: David Noellsch <DavidN@CannonCorp.us>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Adding @Mason Radke<mailto:mason@autosysnet.com> to the chain
-Kevin Seifert, AutoSys LLC
________________________________
From: David Noellsch <DavidN@CannonCorp.us>
Sent: Tuesday, April 21, 2026 12:13 PM
To: Kevin Seifert <kevin@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
Kevin,
Thanks for sending. This is some good background info. I agree a repair install is likely needed. Let me try to login. I assume VPN remote connection is still working.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://www.linkedin.com/company/cannon->
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 2:01 PM
To: David Noellsch <DavidN@CannonCorp.us>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Here is the hand-off information Mason gave me.
Let me know what you think.
Thanks in advance.
Kevin Seifert
![[20260421_19db1cf3e612_Outlook-0m0loqwi.png]]
AutoSys LLC
kevin@autosysnet.com<mailto:kevin@autosysnet.com>
805-835-9595
---
## 5. From: Kevin Seifert <kevin@autosysnet.com> — Tue, 21 Apr 2026 20:52:56 +0000
Fantastic.
Will you please verify that all data is showing on SCADA for the MBR process? Jesse had mentioned earlier that the analogs were reporting but not the digitals.
Also, please forward a service report.
Thank you David!
-Kevin Seifert, AutoSys LLC
________________________________
From: David Noellsch <DavidN@CannonCorp.us>
Sent: Tuesday, April 21, 2026 1:50 PM
To: Kevin Seifert <kevin@autosysnet.com>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
System is restored. Both servers are operational, and engines are running on both servers. I've been monitoring and checking for the last ~20 minutes. Performance looks nominal, including Historian/trending.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://www.linkedin.com/company/cannon->
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 3:08 PM
To: David Noellsch <DavidN@CannonCorp.us>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Thank you David
-Kevin Seifert, AutoSys LLC
________________________________
From: David Noellsch <DavidN@CannonCorp.us>
Sent: Tuesday, April 21, 2026 12:49 PM
To: Kevin Seifert <kevin@autosysnet.com>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
I've stopped the platform and all engines on app-server-1. One of the engines was stuck in a sync cycle with its redundant partner on server 2. I stopped the partner engine on server 2 temporarily to allow a clean termination on server 1.
Everything is running on server 2 now. I'm monitoring for a few minutes. Then I will perform a repair install on server 1.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://www.linkedin.com/company/cannon->
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 2:15 PM
To: David Noellsch <DavidN@CannonCorp.us>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Adding @Mason Radke<mailto:mason@autosysnet.com> to the chain
-Kevin Seifert, AutoSys LLC
________________________________
From: David Noellsch <DavidN@CannonCorp.us>
Sent: Tuesday, April 21, 2026 12:13 PM
To: Kevin Seifert <kevin@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
Kevin,
Thanks for sending. This is some good background info. I agree a repair install is likely needed. Let me try to login. I assume VPN remote connection is still working.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://www.linkedin.com/company/cannon->
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 2:01 PM
To: David Noellsch <DavidN@CannonCorp.us>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Here is the hand-off information Mason gave me.
Let me know what you think.
Thanks in advance.
Kevin Seifert
![[20260421_19db1d17087d_Outlook-0m0loqwi.png]]
AutoSys LLC
kevin@autosysnet.com<mailto:kevin@autosysnet.com>
805-835-9595
---
## 6. From: David Noellsch <DavidN@cannoncorp.us> — Tue, 21 Apr 2026 21:51:09 +0000
The MBR PLC appears to be off, stopped, or disconnected. Is it currently running? Neither server is able to read data from it. It's also not showing up on the Data Highway Plus network station list.
Service Report - SCADA Application Server Instability
Date: April 21, 2026
System: AVEVA System Platform (Galaxy: LCSD)
Servers: APP-SRVR1, APP-SRVR2
Issue Summary
Operators reported slow or unresponsive SCADA graphics and trend displays beginning April 20, 2026. Investigation identified that APP-SRVR1 was experiencing a repeated application engine restart condition. One AppEngine was continuously cycling between Syncing and Stopped, causing repeated reinitialization of PLC subscriptions and Historian client connections. This resulted in delayed graphic loading, intermittent blank trend data, and overall system sluggishness.
Findings
Log review indicated a corrupted AVEVA System Platform component on APP-SRVR1 triggering a Windows Installer self-repair loop. The repair process repeatedly failed, causing the affected AppEngine to continuously restart and preventing stable object hosting. This created excessive resubscription traffic to PLC data sources and repeated Historian client reconnections, degrading system performance.
APP-SRVR1 was isolated from the Galaxy, allowing APP-SRVR2 to temporarily operate the SCADA system independently. An AVEVA Change Installer repair was then performed on APP-SRVR1 to correct the corrupted component installation.
Resolution
The repair installation completed successfully. Application engines on APP-SRVR1 were brought back online sequentially and verified stable. Object hosting was transferred back to the primary server without recurrence of the synchronization issue. Historian trending and client performance returned to normal.
Current Status
SCADA system operating normally with full redundancy restored between APP-SRVR1 and APP-SRVR2. No further application engine instability observed.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://www.linkedin.com/company/cannon->
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 3:52 PM
To: David Noellsch <DavidN@CannonCorp.us>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Fantastic.
Will you please verify that all data is showing on SCADA for the MBR process? Jesse had mentioned earlier that the analogs were reporting but not the digitals.
Also, please forward a service report.
Thank you David!
-Kevin Seifert, AutoSys LLC
________________________________
From: David Noellsch <DavidN@CannonCorp.us>
Sent: Tuesday, April 21, 2026 1:50 PM
To: Kevin Seifert <kevin@autosysnet.com>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
System is restored. Both servers are operational, and engines are running on both servers. I've been monitoring and checking for the last ~20 minutes. Performance looks nominal, including Historian/trending.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://www.linkedin.com/company/cannon->
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 3:08 PM
To: David Noellsch <DavidN@CannonCorp.us>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Thank you David
-Kevin Seifert, AutoSys LLC
________________________________
From: David Noellsch <DavidN@CannonCorp.us>
Sent: Tuesday, April 21, 2026 12:49 PM
To: Kevin Seifert <kevin@autosysnet.com>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
I've stopped the platform and all engines on app-server-1. One of the engines was stuck in a sync cycle with its redundant partner on server 2. I stopped the partner engine on server 2 temporarily to allow a clean termination on server 1.
Everything is running on server 2 now. I'm monitoring for a few minutes. Then I will perform a repair install on server 1.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://www.linkedin.com/company/cannon->
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 2:15 PM
To: David Noellsch <DavidN@CannonCorp.us>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Adding @Mason Radke<mailto:mason@autosysnet.com> to the chain
-Kevin Seifert, AutoSys LLC
________________________________
From: David Noellsch <DavidN@CannonCorp.us>
Sent: Tuesday, April 21, 2026 12:13 PM
To: Kevin Seifert <kevin@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
Kevin,
Thanks for sending. This is some good background info. I agree a repair install is likely needed. Let me try to login. I assume VPN remote connection is still working.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://www.linkedin.com/company/cannon->
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 2:01 PM
To: David Noellsch <DavidN@CannonCorp.us>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Here is the hand-off information Mason gave me.
Let me know what you think.
Thanks in advance.
Kevin Seifert
![[20260421_19db206b6faf_Outlook-0m0loqwi.png]]
AutoSys LLC
kevin@autosysnet.com<mailto:kevin@autosysnet.com>
805-835-9595
---
## 7. From: Jerry Nichols <jenichl@countyofsb.org> — Tue, 21 Apr 2026 21:53:53 +0000
I will go look at the MBR PLC.
Jerry Nichols
Laguna County Sanitation District
Chief Plant Operator
Tel: 805-934-6281
Cell: 805-310-1163
________________________________
From: David Noellsch <DavidN@CannonCorp.us>
Sent: Tuesday, April 21, 2026 2:51 PM
To: Kevin Seifert <kevin@autosysnet.com>; Mason Radke <mason@autosysnet.com>
Cc: Jerry Nichols <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
Caution: This email originated from a source outside of the County of Santa Barbara. Do not click links or open attachments unless you verify the sender and know the content is safe.
The MBR PLC appears to be off, stopped, or disconnected. Is it currently running? Neither server is able to read data from it. It's also not showing up on the Data Highway Plus network station list.
Service Report – SCADA Application Server Instability
Date: April 21, 2026
System: AVEVA System Platform (Galaxy: LCSD)
Servers: APP-SRVR1, APP-SRVR2
Issue Summary
Operators reported slow or unresponsive SCADA graphics and trend displays beginning April 20, 2026. Investigation identified that APP-SRVR1 was experiencing a repeated application engine restart condition. One AppEngine was continuously cycling between Syncing and Stopped, causing repeated reinitialization of PLC subscriptions and Historian client connections. This resulted in delayed graphic loading, intermittent blank trend data, and overall system sluggishness.
Findings
Log review indicated a corrupted AVEVA System Platform component on APP-SRVR1 triggering a Windows Installer self-repair loop. The repair process repeatedly failed, causing the affected AppEngine to continuously restart and preventing stable object hosting. This created excessive resubscription traffic to PLC data sources and repeated Historian client reconnections, degrading system performance.
APP-SRVR1 was isolated from the Galaxy, allowing APP-SRVR2 to temporarily operate the SCADA system independently. An AVEVA Change Installer repair was then performed on APP-SRVR1 to correct the corrupted component installation.
Resolution
The repair installation completed successfully. Application engines on APP-SRVR1 were brought back online sequentially and verified stable. Object hosting was transferred back to the primary server without recurrence of the synchronization issue. Historian trending and client performance returned to normal.
Current Status
SCADA system operating normally with full redundancy restored between APP-SRVR1 and APP-SRVR2. No further application engine instability observed.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://urldefense.com/v3/__https://www.linkedin.com/company/cannon-__;!!Ifs0MJmijOm0!vS5XfNk1OJiX_IjFckdo62E_MgSt4wusesBqIZ7offs6BIRPu9jPMi0RyD-KBaXUzS3Wa2LZYVwNu5GwAJx84ts$>
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 3:52 PM
To: David Noellsch <DavidN@CannonCorp.us>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Fantastic.
Will you please verify that all data is showing on SCADA for the MBR process? Jesse had mentioned earlier that the analogs were reporting but not the digitals.
Also, please forward a service report.
Thank you David!
-Kevin Seifert, AutoSys LLC
________________________________
From: David Noellsch <DavidN@CannonCorp.us>
Sent: Tuesday, April 21, 2026 1:50 PM
To: Kevin Seifert <kevin@autosysnet.com>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
System is restored. Both servers are operational, and engines are running on both servers. I've been monitoring and checking for the last ~20 minutes. Performance looks nominal, including Historian/trending.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://urldefense.com/v3/__https://www.linkedin.com/company/cannon-__;!!Ifs0MJmijOm0!vS5XfNk1OJiX_IjFckdo62E_MgSt4wusesBqIZ7offs6BIRPu9jPMi0RyD-KBaXUzS3Wa2LZYVwNu5GwAJx84ts$>
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 3:08 PM
To: David Noellsch <DavidN@CannonCorp.us>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Thank you David
-Kevin Seifert, AutoSys LLC
________________________________
From: David Noellsch <DavidN@CannonCorp.us>
Sent: Tuesday, April 21, 2026 12:49 PM
To: Kevin Seifert <kevin@autosysnet.com>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
I've stopped the platform and all engines on app-server-1. One of the engines was stuck in a sync cycle with its redundant partner on server 2. I stopped the partner engine on server 2 temporarily to allow a clean termination on server 1.
Everything is running on server 2 now. I'm monitoring for a few minutes. Then I will perform a repair install on server 1.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://urldefense.com/v3/__https://www.linkedin.com/company/cannon-__;!!Ifs0MJmijOm0!vS5XfNk1OJiX_IjFckdo62E_MgSt4wusesBqIZ7offs6BIRPu9jPMi0RyD-KBaXUzS3Wa2LZYVwNu5GwAJx84ts$>
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 2:15 PM
To: David Noellsch <DavidN@CannonCorp.us>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Adding @Mason Radke<mailto:mason@autosysnet.com> to the chain
-Kevin Seifert, AutoSys LLC
________________________________
From: David Noellsch <DavidN@CannonCorp.us>
Sent: Tuesday, April 21, 2026 12:13 PM
To: Kevin Seifert <kevin@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
Kevin,
Thanks for sending. This is some good background info. I agree a repair install is likely needed. Let me try to login. I assume VPN remote connection is still working.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://urldefense.com/v3/__https://www.linkedin.com/company/cannon-__;!!Ifs0MJmijOm0!vS5XfNk1OJiX_IjFckdo62E_MgSt4wusesBqIZ7offs6BIRPu9jPMi0RyD-KBaXUzS3Wa2LZYVwNu5GwAJx84ts$>
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 2:01 PM
To: David Noellsch <DavidN@CannonCorp.us>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Here is the hand-off information Mason gave me.
Let me know what you think.
Thanks in advance.
Kevin Seifert
![[20260421_19db2093c73a_Outlook-0m0loqwi.png]]
AutoSys LLC
kevin@autosysnet.com<mailto:kevin@autosysnet.com>
805-835-9595
---
## 8. From: Kevin Seifert <kevin@autosysnet.com> — Tue, 21 Apr 2026 21:54:01 +0000
The MBR should be online and operating. I will verify with Jesse.
-Kevin Seifert, AutoSys LLC
________________________________
From: David Noellsch <DavidN@CannonCorp.us>
Sent: Tuesday, April 21, 2026 2:51 PM
To: Kevin Seifert <kevin@autosysnet.com>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
The MBR PLC appears to be off, stopped, or disconnected. Is it currently running? Neither server is able to read data from it. It's also not showing up on the Data Highway Plus network station list.
Service Report – SCADA Application Server Instability
Date: April 21, 2026
System: AVEVA System Platform (Galaxy: LCSD)
Servers: APP-SRVR1, APP-SRVR2
Issue Summary
Operators reported slow or unresponsive SCADA graphics and trend displays beginning April 20, 2026. Investigation identified that APP-SRVR1 was experiencing a repeated application engine restart condition. One AppEngine was continuously cycling between Syncing and Stopped, causing repeated reinitialization of PLC subscriptions and Historian client connections. This resulted in delayed graphic loading, intermittent blank trend data, and overall system sluggishness.
Findings
Log review indicated a corrupted AVEVA System Platform component on APP-SRVR1 triggering a Windows Installer self-repair loop. The repair process repeatedly failed, causing the affected AppEngine to continuously restart and preventing stable object hosting. This created excessive resubscription traffic to PLC data sources and repeated Historian client reconnections, degrading system performance.
APP-SRVR1 was isolated from the Galaxy, allowing APP-SRVR2 to temporarily operate the SCADA system independently. An AVEVA Change Installer repair was then performed on APP-SRVR1 to correct the corrupted component installation.
Resolution
The repair installation completed successfully. Application engines on APP-SRVR1 were brought back online sequentially and verified stable. Object hosting was transferred back to the primary server without recurrence of the synchronization issue. Historian trending and client performance returned to normal.
Current Status
SCADA system operating normally with full redundancy restored between APP-SRVR1 and APP-SRVR2. No further application engine instability observed.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://www.linkedin.com/company/cannon->
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 3:52 PM
To: David Noellsch <DavidN@CannonCorp.us>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Fantastic.
Will you please verify that all data is showing on SCADA for the MBR process? Jesse had mentioned earlier that the analogs were reporting but not the digitals.
Also, please forward a service report.
Thank you David!
-Kevin Seifert, AutoSys LLC
________________________________
From: David Noellsch <DavidN@CannonCorp.us>
Sent: Tuesday, April 21, 2026 1:50 PM
To: Kevin Seifert <kevin@autosysnet.com>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
System is restored. Both servers are operational, and engines are running on both servers. I've been monitoring and checking for the last ~20 minutes. Performance looks nominal, including Historian/trending.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://www.linkedin.com/company/cannon->
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 3:08 PM
To: David Noellsch <DavidN@CannonCorp.us>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Thank you David
-Kevin Seifert, AutoSys LLC
________________________________
From: David Noellsch <DavidN@CannonCorp.us>
Sent: Tuesday, April 21, 2026 12:49 PM
To: Kevin Seifert <kevin@autosysnet.com>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
I've stopped the platform and all engines on app-server-1. One of the engines was stuck in a sync cycle with its redundant partner on server 2. I stopped the partner engine on server 2 temporarily to allow a clean termination on server 1.
Everything is running on server 2 now. I'm monitoring for a few minutes. Then I will perform a repair install on server 1.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://www.linkedin.com/company/cannon->
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 2:15 PM
To: David Noellsch <DavidN@CannonCorp.us>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Adding @Mason Radke<mailto:mason@autosysnet.com> to the chain
-Kevin Seifert, AutoSys LLC
________________________________
From: David Noellsch <DavidN@CannonCorp.us>
Sent: Tuesday, April 21, 2026 12:13 PM
To: Kevin Seifert <kevin@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
Kevin,
Thanks for sending. This is some good background info. I agree a repair install is likely needed. Let me try to login. I assume VPN remote connection is still working.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://www.linkedin.com/company/cannon->
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 2:01 PM
To: David Noellsch <DavidN@CannonCorp.us>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Here is the hand-off information Mason gave me.
Let me know what you think.
Thanks in advance.
Kevin Seifert
![[20260421_19db2094fd2f_Outlook-0m0loqwi.png]]
AutoSys LLC
kevin@autosysnet.com<mailto:kevin@autosysnet.com>
805-835-9595
---
## 9. From: Kevin Seifert <kevin@autosysnet.com> — Tue, 21 Apr 2026 22:00:22 +0000
![[20260421_19db20f1aa72_image.png]]
-Kevin Seifert, AutoSys LLC
________________________________
From: Jerry Nichols <jenichl@countyofsb.org>
Sent: Tuesday, April 21, 2026 2:53 PM
To: David Noellsch <DavidN@CannonCorp.us>; Kevin Seifert <kevin@autosysnet.com>; Mason Radke <mason@autosysnet.com>
Subject: Re: Laguna SCADA issue
I will go look at the MBR PLC.
Jerry Nichols
Laguna County Sanitation District
Chief Plant Operator
Tel: 805-934-6281
Cell: 805-310-1163
________________________________
From: David Noellsch <DavidN@CannonCorp.us>
Sent: Tuesday, April 21, 2026 2:51 PM
To: Kevin Seifert <kevin@autosysnet.com>; Mason Radke <mason@autosysnet.com>
Cc: Jerry Nichols <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
Caution: This email originated from a source outside of the County of Santa Barbara. Do not click links or open attachments unless you verify the sender and know the content is safe.
The MBR PLC appears to be off, stopped, or disconnected. Is it currently running? Neither server is able to read data from it. It's also not showing up on the Data Highway Plus network station list.
Service Report – SCADA Application Server Instability
Date: April 21, 2026
System: AVEVA System Platform (Galaxy: LCSD)
Servers: APP-SRVR1, APP-SRVR2
Issue Summary
Operators reported slow or unresponsive SCADA graphics and trend displays beginning April 20, 2026. Investigation identified that APP-SRVR1 was experiencing a repeated application engine restart condition. One AppEngine was continuously cycling between Syncing and Stopped, causing repeated reinitialization of PLC subscriptions and Historian client connections. This resulted in delayed graphic loading, intermittent blank trend data, and overall system sluggishness.
Findings
Log review indicated a corrupted AVEVA System Platform component on APP-SRVR1 triggering a Windows Installer self-repair loop. The repair process repeatedly failed, causing the affected AppEngine to continuously restart and preventing stable object hosting. This created excessive resubscription traffic to PLC data sources and repeated Historian client reconnections, degrading system performance.
APP-SRVR1 was isolated from the Galaxy, allowing APP-SRVR2 to temporarily operate the SCADA system independently. An AVEVA Change Installer repair was then performed on APP-SRVR1 to correct the corrupted component installation.
Resolution
The repair installation completed successfully. Application engines on APP-SRVR1 were brought back online sequentially and verified stable. Object hosting was transferred back to the primary server without recurrence of the synchronization issue. Historian trending and client performance returned to normal.
Current Status
SCADA system operating normally with full redundancy restored between APP-SRVR1 and APP-SRVR2. No further application engine instability observed.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://urldefense.com/v3/__https://www.linkedin.com/company/cannon-__;!!Ifs0MJmijOm0!vS5XfNk1OJiX_IjFckdo62E_MgSt4wusesBqIZ7offs6BIRPu9jPMi0RyD-KBaXUzS3Wa2LZYVwNu5GwAJx84ts$>
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 3:52 PM
To: David Noellsch <DavidN@CannonCorp.us>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Fantastic.
Will you please verify that all data is showing on SCADA for the MBR process? Jesse had mentioned earlier that the analogs were reporting but not the digitals.
Also, please forward a service report.
Thank you David!
-Kevin Seifert, AutoSys LLC
________________________________
From: David Noellsch <DavidN@CannonCorp.us>
Sent: Tuesday, April 21, 2026 1:50 PM
To: Kevin Seifert <kevin@autosysnet.com>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
System is restored. Both servers are operational, and engines are running on both servers. I've been monitoring and checking for the last ~20 minutes. Performance looks nominal, including Historian/trending.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://urldefense.com/v3/__https://www.linkedin.com/company/cannon-__;!!Ifs0MJmijOm0!vS5XfNk1OJiX_IjFckdo62E_MgSt4wusesBqIZ7offs6BIRPu9jPMi0RyD-KBaXUzS3Wa2LZYVwNu5GwAJx84ts$>
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 3:08 PM
To: David Noellsch <DavidN@CannonCorp.us>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Thank you David
-Kevin Seifert, AutoSys LLC
________________________________
From: David Noellsch <DavidN@CannonCorp.us>
Sent: Tuesday, April 21, 2026 12:49 PM
To: Kevin Seifert <kevin@autosysnet.com>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
I've stopped the platform and all engines on app-server-1. One of the engines was stuck in a sync cycle with its redundant partner on server 2. I stopped the partner engine on server 2 temporarily to allow a clean termination on server 1.
Everything is running on server 2 now. I'm monitoring for a few minutes. Then I will perform a repair install on server 1.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://urldefense.com/v3/__https://www.linkedin.com/company/cannon-__;!!Ifs0MJmijOm0!vS5XfNk1OJiX_IjFckdo62E_MgSt4wusesBqIZ7offs6BIRPu9jPMi0RyD-KBaXUzS3Wa2LZYVwNu5GwAJx84ts$>
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 2:15 PM
To: David Noellsch <DavidN@CannonCorp.us>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Adding @Mason Radke<mailto:mason@autosysnet.com> to the chain
-Kevin Seifert, AutoSys LLC
________________________________
From: David Noellsch <DavidN@CannonCorp.us>
Sent: Tuesday, April 21, 2026 12:13 PM
To: Kevin Seifert <kevin@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
Kevin,
Thanks for sending. This is some good background info. I agree a repair install is likely needed. Let me try to login. I assume VPN remote connection is still working.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://urldefense.com/v3/__https://www.linkedin.com/company/cannon-__;!!Ifs0MJmijOm0!vS5XfNk1OJiX_IjFckdo62E_MgSt4wusesBqIZ7offs6BIRPu9jPMi0RyD-KBaXUzS3Wa2LZYVwNu5GwAJx84ts$>
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 2:01 PM
To: David Noellsch <DavidN@CannonCorp.us>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Here is the hand-off information Mason gave me.
Let me know what you think.
Thanks in advance.
Kevin Seifert
![[20260421_19db20f1aa72_Outlook-0m0loqwi.png]]
AutoSys LLC
kevin@autosysnet.com<mailto:kevin@autosysnet.com>
805-835-9595
---
## 10. From: Kevin Seifert <kevin@autosysnet.com> — Tue, 21 Apr 2026 22:04:42 +0000
Jesse indicates the MBR is running, PLC looks fine, and local HMI (on DH+) is working.
But we don't see it back at the server room.
-Kevin Seifert, AutoSys LLC
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 3:00 PM
To: Jerry Nichols <jenichl@countyofsb.org>; David Noellsch <DavidN@CannonCorp.us>; Mason Radke <mason@autosysnet.com>
Subject: Re: Laguna SCADA issue
![[20260421_19db21315391_image.png]]
-Kevin Seifert, AutoSys LLC
________________________________
From: Jerry Nichols <jenichl@countyofsb.org>
Sent: Tuesday, April 21, 2026 2:53 PM
To: David Noellsch <DavidN@CannonCorp.us>; Kevin Seifert <kevin@autosysnet.com>; Mason Radke <mason@autosysnet.com>
Subject: Re: Laguna SCADA issue
I will go look at the MBR PLC.
Jerry Nichols
Laguna County Sanitation District
Chief Plant Operator
Tel: 805-934-6281
Cell: 805-310-1163
________________________________
From: David Noellsch <DavidN@CannonCorp.us>
Sent: Tuesday, April 21, 2026 2:51 PM
To: Kevin Seifert <kevin@autosysnet.com>; Mason Radke <mason@autosysnet.com>
Cc: Jerry Nichols <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
Caution: This email originated from a source outside of the County of Santa Barbara. Do not click links or open attachments unless you verify the sender and know the content is safe.
The MBR PLC appears to be off, stopped, or disconnected. Is it currently running? Neither server is able to read data from it. It's also not showing up on the Data Highway Plus network station list.
Service Report – SCADA Application Server Instability
Date: April 21, 2026
System: AVEVA System Platform (Galaxy: LCSD)
Servers: APP-SRVR1, APP-SRVR2
Issue Summary
Operators reported slow or unresponsive SCADA graphics and trend displays beginning April 20, 2026. Investigation identified that APP-SRVR1 was experiencing a repeated application engine restart condition. One AppEngine was continuously cycling between Syncing and Stopped, causing repeated reinitialization of PLC subscriptions and Historian client connections. This resulted in delayed graphic loading, intermittent blank trend data, and overall system sluggishness.
Findings
Log review indicated a corrupted AVEVA System Platform component on APP-SRVR1 triggering a Windows Installer self-repair loop. The repair process repeatedly failed, causing the affected AppEngine to continuously restart and preventing stable object hosting. This created excessive resubscription traffic to PLC data sources and repeated Historian client reconnections, degrading system performance.
APP-SRVR1 was isolated from the Galaxy, allowing APP-SRVR2 to temporarily operate the SCADA system independently. An AVEVA Change Installer repair was then performed on APP-SRVR1 to correct the corrupted component installation.
Resolution
The repair installation completed successfully. Application engines on APP-SRVR1 were brought back online sequentially and verified stable. Object hosting was transferred back to the primary server without recurrence of the synchronization issue. Historian trending and client performance returned to normal.
Current Status
SCADA system operating normally with full redundancy restored between APP-SRVR1 and APP-SRVR2. No further application engine instability observed.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://urldefense.com/v3/__https://www.linkedin.com/company/cannon-__;!!Ifs0MJmijOm0!vS5XfNk1OJiX_IjFckdo62E_MgSt4wusesBqIZ7offs6BIRPu9jPMi0RyD-KBaXUzS3Wa2LZYVwNu5GwAJx84ts$>
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 3:52 PM
To: David Noellsch <DavidN@CannonCorp.us>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Fantastic.
Will you please verify that all data is showing on SCADA for the MBR process? Jesse had mentioned earlier that the analogs were reporting but not the digitals.
Also, please forward a service report.
Thank you David!
-Kevin Seifert, AutoSys LLC
________________________________
From: David Noellsch <DavidN@CannonCorp.us>
Sent: Tuesday, April 21, 2026 1:50 PM
To: Kevin Seifert <kevin@autosysnet.com>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
System is restored. Both servers are operational, and engines are running on both servers. I've been monitoring and checking for the last ~20 minutes. Performance looks nominal, including Historian/trending.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://urldefense.com/v3/__https://www.linkedin.com/company/cannon-__;!!Ifs0MJmijOm0!vS5XfNk1OJiX_IjFckdo62E_MgSt4wusesBqIZ7offs6BIRPu9jPMi0RyD-KBaXUzS3Wa2LZYVwNu5GwAJx84ts$>
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 3:08 PM
To: David Noellsch <DavidN@CannonCorp.us>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Thank you David
-Kevin Seifert, AutoSys LLC
________________________________
From: David Noellsch <DavidN@CannonCorp.us>
Sent: Tuesday, April 21, 2026 12:49 PM
To: Kevin Seifert <kevin@autosysnet.com>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
I've stopped the platform and all engines on app-server-1. One of the engines was stuck in a sync cycle with its redundant partner on server 2. I stopped the partner engine on server 2 temporarily to allow a clean termination on server 1.
Everything is running on server 2 now. I'm monitoring for a few minutes. Then I will perform a repair install on server 1.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://urldefense.com/v3/__https://www.linkedin.com/company/cannon-__;!!Ifs0MJmijOm0!vS5XfNk1OJiX_IjFckdo62E_MgSt4wusesBqIZ7offs6BIRPu9jPMi0RyD-KBaXUzS3Wa2LZYVwNu5GwAJx84ts$>
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 2:15 PM
To: David Noellsch <DavidN@CannonCorp.us>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Adding @Mason Radke<mailto:mason@autosysnet.com> to the chain
-Kevin Seifert, AutoSys LLC
________________________________
From: David Noellsch <DavidN@CannonCorp.us>
Sent: Tuesday, April 21, 2026 12:13 PM
To: Kevin Seifert <kevin@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
Kevin,
Thanks for sending. This is some good background info. I agree a repair install is likely needed. Let me try to login. I assume VPN remote connection is still working.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://urldefense.com/v3/__https://www.linkedin.com/company/cannon-__;!!Ifs0MJmijOm0!vS5XfNk1OJiX_IjFckdo62E_MgSt4wusesBqIZ7offs6BIRPu9jPMi0RyD-KBaXUzS3Wa2LZYVwNu5GwAJx84ts$>
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 2:01 PM
To: David Noellsch <DavidN@CannonCorp.us>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Here is the hand-off information Mason gave me.
Let me know what you think.
Thanks in advance.
Kevin Seifert
![[20260421_19db21315391_Outlook-0m0loqwi.png]]
AutoSys LLC
kevin@autosysnet.com<mailto:kevin@autosysnet.com>
805-835-9595
---
## 11. From: Kevin Seifert <kevin@autosysnet.com> — Tue, 21 Apr 2026 22:14:07 +0000
I will go out tomorrow to investigate.
Thanks again for your assist today David.
-Kevin Seifert, AutoSys LLC
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 3:04 PM
To: Jerry Nichols <jenichl@countyofsb.org>; David Noellsch <DavidN@CannonCorp.us>; Mason Radke <mason@autosysnet.com>
Subject: Re: Laguna SCADA issue
Jesse indicates the MBR is running, PLC looks fine, and local HMI (on DH+) is working.
But we don't see it back at the server room.
-Kevin Seifert, AutoSys LLC
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 3:00 PM
To: Jerry Nichols <jenichl@countyofsb.org>; David Noellsch <DavidN@CannonCorp.us>; Mason Radke <mason@autosysnet.com>
Subject: Re: Laguna SCADA issue
![[20260421_19db21bc813b_image.png]]
-Kevin Seifert, AutoSys LLC
________________________________
From: Jerry Nichols <jenichl@countyofsb.org>
Sent: Tuesday, April 21, 2026 2:53 PM
To: David Noellsch <DavidN@CannonCorp.us>; Kevin Seifert <kevin@autosysnet.com>; Mason Radke <mason@autosysnet.com>
Subject: Re: Laguna SCADA issue
I will go look at the MBR PLC.
Jerry Nichols
Laguna County Sanitation District
Chief Plant Operator
Tel: 805-934-6281
Cell: 805-310-1163
________________________________
From: David Noellsch <DavidN@CannonCorp.us>
Sent: Tuesday, April 21, 2026 2:51 PM
To: Kevin Seifert <kevin@autosysnet.com>; Mason Radke <mason@autosysnet.com>
Cc: Jerry Nichols <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
Caution: This email originated from a source outside of the County of Santa Barbara. Do not click links or open attachments unless you verify the sender and know the content is safe.
The MBR PLC appears to be off, stopped, or disconnected. Is it currently running? Neither server is able to read data from it. It's also not showing up on the Data Highway Plus network station list.
Service Report – SCADA Application Server Instability
Date: April 21, 2026
System: AVEVA System Platform (Galaxy: LCSD)
Servers: APP-SRVR1, APP-SRVR2
Issue Summary
Operators reported slow or unresponsive SCADA graphics and trend displays beginning April 20, 2026. Investigation identified that APP-SRVR1 was experiencing a repeated application engine restart condition. One AppEngine was continuously cycling between Syncing and Stopped, causing repeated reinitialization of PLC subscriptions and Historian client connections. This resulted in delayed graphic loading, intermittent blank trend data, and overall system sluggishness.
Findings
Log review indicated a corrupted AVEVA System Platform component on APP-SRVR1 triggering a Windows Installer self-repair loop. The repair process repeatedly failed, causing the affected AppEngine to continuously restart and preventing stable object hosting. This created excessive resubscription traffic to PLC data sources and repeated Historian client reconnections, degrading system performance.
APP-SRVR1 was isolated from the Galaxy, allowing APP-SRVR2 to temporarily operate the SCADA system independently. An AVEVA Change Installer repair was then performed on APP-SRVR1 to correct the corrupted component installation.
Resolution
The repair installation completed successfully. Application engines on APP-SRVR1 were brought back online sequentially and verified stable. Object hosting was transferred back to the primary server without recurrence of the synchronization issue. Historian trending and client performance returned to normal.
Current Status
SCADA system operating normally with full redundancy restored between APP-SRVR1 and APP-SRVR2. No further application engine instability observed.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://urldefense.com/v3/__https://www.linkedin.com/company/cannon-__;!!Ifs0MJmijOm0!vS5XfNk1OJiX_IjFckdo62E_MgSt4wusesBqIZ7offs6BIRPu9jPMi0RyD-KBaXUzS3Wa2LZYVwNu5GwAJx84ts$>
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 3:52 PM
To: David Noellsch <DavidN@CannonCorp.us>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Fantastic.
Will you please verify that all data is showing on SCADA for the MBR process? Jesse had mentioned earlier that the analogs were reporting but not the digitals.
Also, please forward a service report.
Thank you David!
-Kevin Seifert, AutoSys LLC
________________________________
From: David Noellsch <DavidN@CannonCorp.us>
Sent: Tuesday, April 21, 2026 1:50 PM
To: Kevin Seifert <kevin@autosysnet.com>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
System is restored. Both servers are operational, and engines are running on both servers. I've been monitoring and checking for the last ~20 minutes. Performance looks nominal, including Historian/trending.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://urldefense.com/v3/__https://www.linkedin.com/company/cannon-__;!!Ifs0MJmijOm0!vS5XfNk1OJiX_IjFckdo62E_MgSt4wusesBqIZ7offs6BIRPu9jPMi0RyD-KBaXUzS3Wa2LZYVwNu5GwAJx84ts$>
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 3:08 PM
To: David Noellsch <DavidN@CannonCorp.us>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Thank you David
-Kevin Seifert, AutoSys LLC
________________________________
From: David Noellsch <DavidN@CannonCorp.us>
Sent: Tuesday, April 21, 2026 12:49 PM
To: Kevin Seifert <kevin@autosysnet.com>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
I've stopped the platform and all engines on app-server-1. One of the engines was stuck in a sync cycle with its redundant partner on server 2. I stopped the partner engine on server 2 temporarily to allow a clean termination on server 1.
Everything is running on server 2 now. I'm monitoring for a few minutes. Then I will perform a repair install on server 1.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://urldefense.com/v3/__https://www.linkedin.com/company/cannon-__;!!Ifs0MJmijOm0!vS5XfNk1OJiX_IjFckdo62E_MgSt4wusesBqIZ7offs6BIRPu9jPMi0RyD-KBaXUzS3Wa2LZYVwNu5GwAJx84ts$>
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 2:15 PM
To: David Noellsch <DavidN@CannonCorp.us>; Mason Radke <mason@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Adding @Mason Radke<mailto:mason@autosysnet.com> to the chain
-Kevin Seifert, AutoSys LLC
________________________________
From: David Noellsch <DavidN@CannonCorp.us>
Sent: Tuesday, April 21, 2026 12:13 PM
To: Kevin Seifert <kevin@autosysnet.com>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Re: Laguna SCADA issue
Kevin,
Thanks for sending. This is some good background info. I agree a repair install is likely needed. Let me try to login. I assume VPN remote connection is still working.
David Noellsch, PE
Principal Control System Engineer
D: 661.616.6228 C: 213.215.9673
DavidN@CannonCorp.us<mailto:DavidN@CannonCorp.us> | LinkedIn<https://urldefense.com/v3/__https://www.linkedin.com/company/cannon-__;!!Ifs0MJmijOm0!vS5XfNk1OJiX_IjFckdo62E_MgSt4wusesBqIZ7offs6BIRPu9jPMi0RyD-KBaXUzS3Wa2LZYVwNu5GwAJx84ts$>
Cannon
Reliable Responsive Solutions
________________________________
From: Kevin Seifert <kevin@autosysnet.com>
Sent: Tuesday, April 21, 2026 2:01 PM
To: David Noellsch <DavidN@CannonCorp.us>
Cc: Nichols, Jerry <jenichl@countyofsb.org>
Subject: Laguna SCADA issue
** WARNING: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. **
Here is the hand-off information Mason gave me.
Let me know what you think.
Thanks in advance.
Kevin Seifert
![[20260421_19db21bc813b_Outlook-0m0loqwi.png]]
AutoSys LLC
kevin@autosysnet.com<mailto:kevin@autosysnet.com>
805-835-9595
---