ServiceControl v6.7.3

[ags-tombrennan]

Describe the bug

Description

Windows 2022, IIS, ServicePulse 2.30, with ServiceControl v6.7.3, connecting to RabbitMQ

Connection string is
host=rmq.domain.net;virtualhost=vhost;username=rmq_user;password=redacted;DisableRemoteCertificateValidation=true;ManagementApiUrl=https://rabbitmq01.domain.net:15672;ManagementApiUserName=rmq_mgmt_user;ManagementApiPassword=redacted

Can confirm from ServiceControl that the following instances are running

• particular.monitoring, Transport is RabbitMQ - Conventional routing topology (quorum queues)
  • instance/queue name is particular.monitoring
  • url is http://localhost:33633/
• particular.servicecontrol, Transport is RabbitMQ - Conventional routing topology (quorum queues)
  • instance/queue name is particular.servicecontrol
  • url is http://localhost:33333/api
• particular.servicecontrol.audit, , Transport is RabbitMQ - Conventional routing topology (quorum queues)
  • instance/queue name is particular.servicecontrol.audit

Expected behavior

Navigating to the endpoint https://rmq-psp.domain.net/#/configuration/connections and entering the following connection properties

• ServiceControl connection url https://rmq-psp.domain.net/api/ would expect to show "Connection successful" after clicking on button Test
• ServiceControl Monitoring url https://rmq-psp.domain.net/monitoring/ would expect to show "Connection successful" after clicking on button Test

Actual behavior

Navigating to the endpoint https://rmq-psp.domain.net/#/configuration/connections and entering the following connection properties

• ServiceControl connection url https://rmq-psp.domain.net/api/ shows "Connection Failed" after clicking on button Test
• ServiceControl Monitoring url https://rmq-psp.domain.net/monitoring/ shows "Connection successful" after clicking on button Test

Versions

Windows Server 2022 Datacenter OS Build 20348.4052

ServiceControl v6.7.3
ServicePulse v2.3.0

dotnet --info is showing

Host:
  Version:      9.0.9
  Architecture: x64
  Commit:       893c2ebbd4
  RID:          win-x64

.NET SDKs installed:
  No SDKs were found.

.NET runtimes installed:
  Microsoft.AspNetCore.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 8.0.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 7.0.20 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 8.0.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 9.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

Other architectures found:
  x86   [C:\Program Files (x86)\dotnet]
    registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

Please list the version of the relevant packages or applications in which the bug exis3

Steps to reproduce

Event Viewer is showing this in sequence below, the data is in XML format to make it easier

Relevant log output

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
  <Provider Name="Application Error" /> 
  <EventID Qualifiers="0">1000</EventID> 
  <Version>0</Version> 
  <Level>2</Level> 
  <Task>100</Task> 
  <Opcode>0</Opcode> 
  <Keywords>0x80000000000000</Keywords> 
  <TimeCreated SystemTime="2026-01-29T10:53:25.8362661Z" /> 
  <EventRecordID>224658</EventRecordID> 
  <Correlation /> 
  <Execution ProcessID="0" ThreadID="0" /> 
  <Channel>Application</Channel> 
  <Computer>psprmq.domain.net</Computer> 
  <Security /> 
  </System>
<EventData>
  <Data>ServiceControl.exe</Data> 
  <Data>6.7.3.0</Data> 
  <Data>68a40000</Data> 
  <Data>KERNELBASE.dll</Data> 
  <Data>10.0.20348.3932</Data> 
  <Data>4c7b412e</Data> 
  <Data>e0434352</Data> 
  <Data>000000000003f46c</Data> 
  <Data>ab0</Data> 
  <Data>01dc910d67cde2fe</Data> 
  <Data>D:\ServiceControl\Bin\ServiceControl.exe</Data> 
  <Data>C:\Windows\System32\KERNELBASE.dll</Data> 
  <Data>7876b6ac-09b6-4799-9f8b-d7571296ca91</Data> 
  <Data /> 
  <Data /> 
  </EventData>
  </Event>


<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
  <Provider Name=".NET Runtime" /> 
  <EventID Qualifiers="0">1026</EventID> 
  <Version>0</Version> 
  <Level>2</Level> 
  <Task>0</Task> 
  <Opcode>0</Opcode> 
  <Keywords>0x80000000000000</Keywords> 
  <TimeCreated SystemTime="2026-01-29T10:53:25.3210486Z" /> 
  <EventRecordID>224657</EventRecordID> 
  <Correlation /> 
  <Execution ProcessID="0" ThreadID="0" /> 
  <Channel>Application</Channel> 
  <Computer>psprmq.domain.net</Computer> 
  <Security /> 
  </System>
<EventData>
  <Data>Application: ServiceControl.exe CoreCLR Version: 8.0.2025.41914 .NET Version: 8.0.20 Description: The process was terminated due to an unhandled exception. Exception Info: System.Threading.Tasks.TaskCanceledException: A task was canceled. at RabbitMQ.Client.Impl.Channel.BasicConsumeAsync(String queue, Boolean autoAck, String consumerTag, Boolean noLocal, Boolean exclusive, IDictionary`2 arguments, IAsyncBasicConsumer consumer, CancellationToken cancellationToken) at RabbitMQ.Client.Impl.AutorecoveringChannel.BasicConsumeAsync(String queue, Boolean autoAck, String consumerTag, Boolean noLocal, Boolean exclusive, IDictionary`2 arguments, IAsyncBasicConsumer consumer, CancellationToken cancellationToken) at NServiceBus.Transport.RabbitMQ.MessagePump.ConnectToBroker(CancellationToken cancellationToken) in /_/src/NServiceBus.Transport.RabbitMQ/Receiving/MessagePump.cs:line 148 at NServiceBus.Transport.RabbitMQ.MessagePump.StartReceive(CancellationToken cancellationToken) in /_/src/NServiceBus.Transport.RabbitMQ/Receiving/MessagePump.cs:line 112 at Particular.LicensingComponent.MonitoringThroughput.MonitoringThroughputHostedService.StartAsync(CancellationToken cancellationToken) in /_/src/Particular.LicensingComponent/MonitoringThroughput/MonitoringThroughputHostedService.cs:line 30 at Microsoft.Extensions.Hosting.Internal.Host.<StartAsync>b__15_1(IHostedService service, CancellationToken token) at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[T](IEnumerable`1 services, CancellationToken token, Boolean concurrent, Boolean abortOnFirstException, List`1 exceptions, Func`3 operation) at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken) at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token) at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token) at ServiceControl.Hosting.Commands.RunCommand.Execute(HostArguments args, Settings settings) in /_/src/ServiceControl/Hosting/Commands/RunCommand.cs:line 28 at ServiceControl.Hosting.Commands.CommandRunner.Execute(HostArguments args, Settings settings) in /_/src/ServiceControl/Hosting/Commands/CommandRunner.cs:line 13 at Program.<Main>$(String[] args) in /_/src/ServiceControl/Program.cs:line 40 at Program.<Main>(String[] args)</Data> 
  </EventData>
  </Event>


<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
  <Provider Name="ServiceControl" /> 
  <EventID Qualifiers="0">11</EventID> 
  <Version>0</Version> 
  <Level>2</Level> 
  <Task>0</Task> 
  <Opcode>0</Opcode> 
  <Keywords>0x80000000000000</Keywords> 
  <TimeCreated SystemTime="2026-01-29T10:53:25.0518172Z" /> 
  <EventRecordID>224656</EventRecordID> 
  <Correlation /> 
  <Execution ProcessID="0" ThreadID="0" /> 
  <Channel>Application</Channel> 
  <Computer>psprmq.domain.net</Computer> 
  <Security /> 
  </System>
<EventData>
  <Data>Category: Microsoft.Extensions.Hosting.Internal.Host EventId: 11 Hosting failed to start Exception: System.Threading.Tasks.TaskCanceledException: A task was canceled. at RabbitMQ.Client.Impl.Channel.BasicConsumeAsync(String queue, Boolean autoAck, String consumerTag, Boolean noLocal, Boolean exclusive, IDictionary`2 arguments, IAsyncBasicConsumer consumer, CancellationToken cancellationToken) at RabbitMQ.Client.Impl.AutorecoveringChannel.BasicConsumeAsync(String queue, Boolean autoAck, String consumerTag, Boolean noLocal, Boolean exclusive, IDictionary`2 arguments, IAsyncBasicConsumer consumer, CancellationToken cancellationToken) at NServiceBus.Transport.RabbitMQ.MessagePump.ConnectToBroker(CancellationToken cancellationToken) in /_/src/NServiceBus.Transport.RabbitMQ/Receiving/MessagePump.cs:line 148 at NServiceBus.Transport.RabbitMQ.MessagePump.StartReceive(CancellationToken cancellationToken) in /_/src/NServiceBus.Transport.RabbitMQ/Receiving/MessagePump.cs:line 112 at Particular.LicensingComponent.MonitoringThroughput.MonitoringThroughputHostedService.StartAsync(CancellationToken cancellationToken) in /_/src/Particular.LicensingComponent/MonitoringThroughput/MonitoringThroughputHostedService.cs:line 30 at Microsoft.Extensions.Hosting.Internal.Host.<StartAsync>b__15_1(IHostedService service, CancellationToken token) at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[T](IEnumerable`1 services, CancellationToken token, Boolean concurrent, Boolean abortOnFirstException, List`1 exceptions, Func`3 operation)</Data> 
  </EventData>
  </Event>


<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
  <Provider Name="ServiceControl" /> 
  <EventID Qualifiers="0">0</EventID> 
  <Version>0</Version> 
  <Level>2</Level> 
  <Task>0</Task> 
  <Opcode>0</Opcode> 
  <Keywords>0x80000000000000</Keywords> 
  <TimeCreated SystemTime="2026-01-29T10:53:24.7168236Z" /> 
  <EventRecordID>224655</EventRecordID> 
  <Correlation /> 
  <Execution ProcessID="0" ThreadID="0" /> 
  <Channel>Application</Channel> 
  <Computer>psprmq.domain.net</Computer> 
  <Security /> 
  </System>
<EventData>
  <Data>Category: ServiceControl.Operations.ErrorIngestion EventId: 0 Failed to start infrastructure Exception: System.Threading.Tasks.TaskCanceledException: A task was canceled. at RabbitMQ.Client.Impl.Channel.BasicConsumeAsync(String queue, Boolean autoAck, String consumerTag, Boolean noLocal, Boolean exclusive, IDictionary`2 arguments, IAsyncBasicConsumer consumer, CancellationToken cancellationToken) at RabbitMQ.Client.Impl.AutorecoveringChannel.BasicConsumeAsync(String queue, Boolean autoAck, String consumerTag, Boolean noLocal, Boolean exclusive, IDictionary`2 arguments, IAsyncBasicConsumer consumer, CancellationToken cancellationToken) at NServiceBus.Transport.RabbitMQ.MessagePump.ConnectToBroker(CancellationToken cancellationToken) in /_/src/NServiceBus.Transport.RabbitMQ/Receiving/MessagePump.cs:line 148 at NServiceBus.Transport.RabbitMQ.MessagePump.StartReceive(CancellationToken cancellationToken) in /_/src/NServiceBus.Transport.RabbitMQ/Receiving/MessagePump.cs:line 112 at ServiceControl.Operations.ErrorIngestion.SetUpAndStartInfrastructure(CancellationToken cancellationToken) in /_/src/ServiceControl/Operations/ErrorIngestion.cs:line 221</Data> 
  </EventData>
  </Event>



<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
  <Provider Name="particular.servicecontrol.audit" /> 
  <EventID Qualifiers="0">0</EventID> 
  <Version>0</Version> 
  <Level>4</Level> 
  <Task>0</Task> 
  <Opcode>0</Opcode> 
  <Keywords>0x80000000000000</Keywords> 
  <TimeCreated SystemTime="2026-01-29T10:53:17.4362695Z" /> 
  <EventRecordID>224654</EventRecordID> 
  <Correlation /> 
  <Execution ProcessID="0" ThreadID="0" /> 
  <Channel>Application</Channel> 
  <Computer>psprmq.domain.net</Computer> 
  <Security /> 
  </System>
<EventData>
  <Data>Service started successfully.</Data> 
  </EventData>
  </Event>


<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
  <Provider Name="particular.servicecontrol" /> 
  <EventID Qualifiers="0">0</EventID> 
  <Version>0</Version> 
  <Level>4</Level> 
  <Task>0</Task> 
  <Opcode>0</Opcode> 
  <Keywords>0x80000000000000</Keywords> 
  <TimeCreated SystemTime="2026-01-29T10:52:49.0110245Z" /> 
  <EventRecordID>224653</EventRecordID> 
  <Correlation /> 
  <Execution ProcessID="0" ThreadID="0" /> 
  <Channel>Application</Channel> 
  <Computer>psprmq.domain.net</Computer> 
  <Security /> 
  </System>
<EventData>
  <Data>Service started successfully.</Data> 
  </EventData>
  </Event>

Additional Information

Workarounds

Possible solutions

Additional information

RabbitMQ Log is showing the following
2026-01-29 12:15:30.549457+00:00 [warning] <0.78660274.2> queue 'particular.servicecontrol.staging' in vhost 'vhost1': policy may not have been successfully applied. Error: {timeout,
2026-01-29 12:15:30.549457+00:00 [warning] <0.78660274.2> {'vhost1_particular.servicecontrol.staging',
2026-01-29 12:15:30.549457+00:00 [warning] <0.78660274.2> rabbit@RABBITMQ01}}
2026-01-29 12:15:30.844511+00:00 [warning] <0.116488089.2> queue 'ServiceControl.ThroughputData' in vhost 'vhost1': policy may not have been successfully applied. Error: {timeout,
2026-01-29 12:15:30.844511+00:00 [warning] <0.116488089.2> {'vhost1_ServiceControl.ThroughputData',
2026-01-29 12:15:30.844511+00:00 [warning] <0.116488089.2> rabbit@RABBITMQ01}}
2026-01-29 12:15:37.562472+00:00 [warning] <0.116499669.2> queue 'audit' in vhost 'vhost1': policy may not have been successfully applied. Error: {timeout,
2026-01-29 12:15:37.562472+00:00 [warning] <0.116499669.2> {vhost1_audit,
2026-01-29 12:15:37.562472+00:00 [warning] <0.116499669.2> rabbit@PRVLXAS62RABBITMQ01}}
2026-01-29 12:15:40.522516+00:00 [warning] <0.116504458.2> queue 'particular.servicecontrol.Errors' in vhost 'vhost1': policy may not have been successfully applied. Error: {timeout,
2026-01-29 12:15:40.522516+00:00 [warning] <0.116504458.2> {'vhost1_particular.servicecontrol.Errors',
2026-01-29 12:15:40.522516+00:00 [warning] <0.116504458.2> rabbit@RABBITMQ01}}
2026-01-29 12:16:00.554553+00:00 [warning] <0.116538387.2> queue 'particular.servicecontrol.staging' in vhost 'vhost1': policy may not have been successfully applied. Error: {timeout,
2026-01-29 12:16:00.554553+00:00 [warning] <0.116538387.2> {'vhost1_particular.servicecontrol.staging',
2026-01-29 12:16:00.554553+00:00 [warning] <0.116538387.2> rabbit@RABBITMQ01}}
2026-01-29 12:16:00.849489+00:00 [warning] <0.116529867.2> queue 'ServiceControl.ThroughputData' in vhost 'vhost1': policy may not have been successfully applied. Error: {timeout,
2026-01-29 12:16:00.849489+00:00 [warning] <0.116529867.2> {'vhost1_ServiceControl.ThroughputData',
2026-01-29 12:16:00.849489+00:00 [warning] <0.116529867.2> rabbit@RABBITMQ01}}
2026-01-29 12:16:07.568470+00:00 [warning] <0.116552473.2> queue 'audit' in vhost 'vhost1': policy may not have been successfully applied. Error: {timeout,
2026-01-29 12:16:07.568470+00:00 [warning] <0.116552473.2> {vhost1_audit,
2026-01-29 12:16:07.568470+00:00 [warning] <0.116552473.2> rabbit@RABBITMQ01}}
2026-01-29 12:16:10.528481+00:00 [warning] <0.116558394.2> queue 'particular.servicecontrol.Errors' in vhost 'vhost1': policy may not have been successfully applied. Error: {timeout,
2026-01-29 12:16:10.528481+00:00 [warning] <0.116558394.2> {'vhost1_particular.servicecontrol.Errors',
2026-01-29 12:16:10.528481+00:00 [warning] <0.116558394.2> rabbit@RABBITMQ01}}

[hazel-bohon]

@ags-tombrennan Can you check what you get back when you go tohttps://rmq-psp.domain.net/api and http://localhost:33333/api?

You should get a small json doc with information about the service that looks something like this:

{
  "description": "The management backend for the Particular Service Platform",
  "endpoints_error_url": "http://localhost:33333/api//endpoints/{name}/errors/{?page,per_page,direction,sort}",
  "known_endpoints_url": "/endpoints/known",
  "endpoints_message_search_url": "http://localhost:33333/api//endpoints/{name}/messages/search/{keyword}/{?page,per_page,direction,sort}",
  "endpoints_messages_url": "http://localhost:33333/api//endpoints/{name}/messages/{?page,per_page,direction,sort}",
  "audit_count_url": "http://localhost:33333/api//endpoints/{name}/audit-count",
  "endpoints_url": "http://localhost:33333/api//endpoints",
  "errors_url": "http://localhost:33333/api//errors/{?page,per_page,direction,sort}",
  "configuration": "http://localhost:33333/api//configuration",
  "remote_configuration": "http://localhost:33333/api//configuration/remotes",
  "message_search_url": "http://localhost:33333/api//messages/search/{keyword}/{?page,per_page,direction,sort}",
  "license_status": "valid",
  "license_details": "http://localhost:33333/api//license",
  "name": "ServiceControl",
  "sagas_url": "http://localhost:33333/api//sagas",
  "event_log_items": "http://localhost:33333/api//eventlogitems",
  "archived_groups_url": "http://localhost:33333/api//errors/groups/{classifier?}",
  "get_archive_group": "http://localhost:33333/api//archive/groups/id/{groupId}"
}

The check pulls the URL that is entered, and verifies that the fetch returned without an error, and that it has the X-Particular-Version header present:

async function testServiceControlUrl() {
  if (localServiceControlUrl.value) {
    testingServiceControl.value = true;
    try {
      const response = await fetch(localServiceControlUrl.value);
      serviceControlValid.value = response.ok && response.headers.has("X-Particular-Version");
    } catch {
      serviceControlValid.value = false;
    } finally {
      testingServiceControl.value = false;
    }
  }
}

https://github.com/Particular/ServicePulse/blob/ddf1d8259cbc210d5ad96e105f87402691affcdb/src/Frontend/src/components/configuration/PlatformConnections.vue#L23-L35

[ags-tombrennan]

Thanks for the assist

As its shown, servicecontrol is showing it to be running, but unfortunately not

however the monitoring api endpoint port 33633 is accessible.
Stopping servicecontrol instance and starting it up again crashes