We had a failed Exchange update last night that caused us to roll back. Since then Lync 2013 prompts for credentials and does not populate EWS URL's. Upon further investigation the EWSCtpMonitor on Exchange 2013 CU6 is marked as unhealthy. It points to an authentication error. Where is this probe defined or how can I fix the authentication error? I've aready read these two MS articles:
https://technet.microsoft.com/en-us/library/ms.exch.scom.ews(v=exchg.150).aspx
https://technet.microsoft.com/en-us/library/ms.exch.scom.owa.protocol(v=exchg.150).aspx
They did not help as they do not explain what to do when you have an auth error. All DB's are running, and the MSExchangeServices AppPool is running. The health mailbox for the probe exists, but I wouldn't know what the credentials are or how to even update them if thats possible.
Here is some output from our server.
[PS] C:\Windows\system32>Get-ServerHealth -Identity email -HealthSet EWS
Server State Name TargetResource HealthSetName AlertValue ServerComp
onent
------ ----- ---- -------------- ------------- ---------- ----------
email NotApplicable EWSCtpMonitor EMAIL.domain.com EWS Unhealthy None
email NotApplicable MaintenanceFailur... EWS Healthy None
email NotApplicable MaintenanceTimeou... EWS Healthy None
[PS] C:\Windows\system32>Get-ServerHealth email | ?{$_.HealthSetName -eq "EWS"}
Server State Name TargetResource HealthSetName AlertValue ServerComp
onent
------ ----- ---- -------------- ------------- ---------- ----------
email NotApplicable EWSCtpMonitor EMAIL.domain.com EWS Unhealthy None
email NotApplicable MaintenanceFailur... EWS Healthy None
email NotApplicable MaintenanceTimeou... EWS Healthy None
[PS] C:\Windows\system32>Invoke-MonitoringProbe EWS\EWSCtpProbe -Server email | fl
RunspaceId : 61df8351-f132-4277-94c7-5e5cbafb5ca2
Server : email
MonitorIdentity : EWS\EWSCtpProbe
RequestId : bc9232b3-4e02-4b48-b0a5-ceab5ae67ffc
ExecutionStartTime : 1/23/2015 2:40:53 PM
ExecutionEndTime : 1/23/2015 2:40:53 PM
Error : System.Exception: System.Exception:
>>> PRIMARY ENDPOINT VERIFICATION
EwsUrl=https://localhost/ews/exchange.asmx
UserName/Password=HealthMailboxe1d5f84ec0ea47ae891b52eca728e706@domain.com/N=rq!ODvifVD!tp^A7kx
Owg)fP1yq]hc82o6S0Qv=s}FdLdw&d4X3IHSx/Hjp=_{(KaDgif65vi/4_tyA?QJGt5Gvy!gpm&Nqi]&>p!d_TCuV2]2hx[8?Q
Y;1%K=FD[X
AuthMethod=Negotiate
GetFolder (Attempt #0) Status=The request failed. The remote server returned an error: (401)
Unauthorized.
GetFolder (Attempt #0) Latency=15.6003
(Attempt #0) X-FEServer=EMAIL
GetFolder (Attempt #1) Status=The request failed. The remote server returned an error: (401)
Unauthorized.
GetFolder (Attempt #1) Latency=31.2006
(Attempt #1) X-FEServer=EMAIL
at Microsoft.Exchange.Monitoring.ActiveMonitoring.Ews.Probes.EWSCommon.RetrySoapActionAndThrow(
Action operation, String soapAction, ExchangeServiceBase service, CancellationToken
cancellationToken, Boolean trackLatency)
at Microsoft.Exchange.Monitoring.ActiveMonitoring.Ews.Probes.EWSGenericProbeCommon.PerformEWSOp
eration(ExchangeService service, String operation, Boolean trackLatency, CancellationToken
cancellationToken)
at Microsoft.Exchange.Monitoring.ActiveMonitoring.Ews.Probes.EWSGenericProbeCommon.ExecuteEWSCa
ll(String endPoint, String operation, Boolean verifyAffinity, CancellationToken
cancellationToken, Boolean trackLatency)
at Microsoft.Exchange.Monitoring.ActiveMonitoring.Ews.Probes.EWSGenericProbeCommon.DoWorkIntern
al(CancellationToken cancellationToken)
Exception : System.Exception: System.Exception: System.Exception:
>>> PRIMARY ENDPOINT VERIFICATION
EwsUrl=https://localhost/ews/exchange.asmx
UserName/Password=HealthMailboxe1d5f84ec0ea47ae891b52eca728e706@domain.com/N=rq!ODvifVD!tp^A7kx
Owg)fP1yq]hc82o6S0Qv=s}FdLdw&d4X3IHSx/Hjp=_{(KaDgif65vi/4_tyA?QJGt5Gvy!gpm&Nqi]&>p!d_TCuV2]2hx[8?Q
Y;1%K=FD[X
AuthMethod=Negotiate
GetFolder (Attempt #0) Status=The request failed. The remote server returned an error: (401)
Unauthorized.
GetFolder (Attempt #0) Latency=15.6003
(Attempt #0) X-FEServer=EMAIL
GetFolder (Attempt #1) Status=The request failed. The remote server returned an error: (401)
Unauthorized.
GetFolder (Attempt #1) Latency=31.2006
(Attempt #1) X-FEServer=EMAIL
at Microsoft.Exchange.Monitoring.ActiveMonitoring.Ews.Probes.EWSCommon.RetrySoapActionAndThrow(
Action operation, String soapAction, ExchangeServiceBase service, CancellationToken
cancellationToken, Boolean trackLatency)
at Microsoft.Exchange.Monitoring.ActiveMonitoring.Ews.Probes.EWSGenericProbeCommon.PerformEWSOp
eration(ExchangeService service, String operation, Boolean trackLatency, CancellationToken
cancellationToken)
at Microsoft.Exchange.Monitoring.ActiveMonitoring.Ews.Probes.EWSGenericProbeCommon.ExecuteEWSCa
ll(String endPoint, String operation, Boolean verifyAffinity, CancellationToken
cancellationToken, Boolean trackLatency)
at Microsoft.Exchange.Monitoring.ActiveMonitoring.Ews.Probes.EWSGenericProbeCommon.DoWorkIntern
al(CancellationToken cancellationToken)
at Microsoft.Exchange.Monitoring.ActiveMonitoring.Ews.Probes.EWSCommon.ThrowError(Object key,
Object exceptiondata, String logDetails)
at Microsoft.Exchange.Monitoring.ActiveMonitoring.Ews.Probes.EWSGenericProbeCommon.DoWorkIntern
al(CancellationToken cancellationToken)
at Microsoft.Exchange.Monitoring.ActiveMonitoring.Ews.Probes.EWSGenericProbeCommon.RunEWSGeneri
cProbe(CancellationToken cancellationToken)
at Microsoft.Office.Datacenter.WorkerTaskFramework.WorkItem.Execute(CancellationToken
joinedToken)
at
Microsoft.Office.Datacenter.WorkerTaskFramework.WorkItem.<>c__DisplayClass2.<StartExecuting>b__0()
at System.Threading.Tasks.Task.Execute()
PoisonedCount : 0
ExecutionId : 6999716
SampleValue : 0
ExecutionContext : EWSGenericProbeError:Exception=System.Exception: System.Exception:>>> PRIMARY ENDPOINT VERIFICATION
EwsUrl=https://localhost/ews/exchange.asmx
UserName/Password=HealthMailboxe1d5f84ec0ea47ae891b52eca728e706@domain.com/N=rq!ODvifVD!tp^A7kx
Owg)fP1yq]hc82o6S0Qv=s}FdLdw&d4X3IHSx/Hjp=_{(KaDgif65vi/4_tyA?QJGt5Gvy!gpm&Nqi]&>p!d_TCuV2]2hx[8?Q
Y;1%K=FD[X
AuthMethod=Negotiate
GetFolder (Attempt #0) Status=The request failed. The remote server returned an error: (401)
Unauthorized.
GetFolder (Attempt #0) Latency=15.6003
(Attempt #0) X-FEServer=EMAIL
GetFolder (Attempt #1) Status=The request failed. The remote server returned an error: (401)
Unauthorized.
GetFolder (Attempt #1) Latency=31.2006
(Attempt #1) X-FEServer=EMAIL
at Microsoft.Exchange.Monitoring.ActiveMonitoring.Ews.Probes.EWSCommon.RetrySoapActionAndThrow(
Action operation, String soapAction, ExchangeServiceBase service, CancellationToken
cancellationToken, Boolean trackLatency)
at Microsoft.Exchange.Monitoring.ActiveMonitoring.Ews.Probes.EWSGenericProbeCommon.PerformEWSOp
eration(ExchangeService service, String operation, Boolean trackLatency, CancellationToken
cancellationToken)
at Microsoft.Exchange.Monitoring.ActiveMonitoring.Ews.Probes.EWSGenericProbeCommon.ExecuteEWSCa
ll(String endPoint, String operation, Boolean verifyAffinity, CancellationToken
cancellationToken, Boolean trackLatency)
at Microsoft.Exchange.Monitoring.ActiveMonitoring.Ews.Probes.EWSGenericProbeCommon.DoWorkIntern
al(CancellationToken cancellationToken)
Important:
Latency(failed_tag=EWSGenericProbeError)=5067
FailureContext : EwsResponseHttpHeaders: <Trace Tag="EwsResponseHttpHeaders" Tid="36" Time="2015-01-23 14:40:53Z">
HTTP/1.1 401 Unauthorized
request-id: b89f64e7-e2bc-4f4b-a9b9-f9d0e775f9ec
X-FailureContext: FrontEnd;401;VW5hdXRob3JpemVk;;;;
Server: Microsoft-IIS/7.5
WWW-Authenticate: Negotiate,NTLM
X-Powered-By: ASP.NET
X-FEServer: EMAIL
Date: Fri, 23 Jan 2015 14:40:53 GMT
Connection: close
Content-Length: 0</Trace>
EwsResponseHttpHeaders: <Trace Tag="EwsResponseHttpHeaders" Tid="10" Time="2015-01-23 14:40:58Z">
HTTP/1.1 401 Unauthorized
request-id: 67818386-43ee-486e-afb1-8afa4b1f42dc
X-FailureContext: FrontEnd;401;VW5hdXRob3JpemVk;;;;
Server: Microsoft-IIS/7.5
WWW-Authenticate: Negotiate,NTLM
X-Powered-By: ASP.NET
X-FEServer: EMAIL
Date: Fri, 23 Jan 2015 14:40:57 GMT
Connection: close
Content-Length: 0</Trace>
ExtensionXml :
ResultType : Failed
RetryCount : 0
ResultName : bc9232b34e024b48b0a5ceab5ae67ffc-EWSCtpProbe
IsNotified : False
ResultId : 38300556
ServiceName : InvokeNow
StateAttribute1 : Monitoring
StateAttribute2 : Unauthorized
StateAttribute3 :
StateAttribute4 :
StateAttribute5 : The monitoring account failed to authenticate with Cafe due to AuthFailure error. Monitoring
account is likely misconfigured - contact monitoring team for assistance.
FailurePoint=FrontEnd, HttpStatusCode=401, Error=Unauthorized, Details=, HttpProxySubErrorCode=,
WebExceptionStatus=, LiveIdAuthResult=
StateAttribute6 : 0
StateAttribute7 : 0
StateAttribute8 : 0
StateAttribute9 : 0
StateAttribute10 : 0
StateAttribute11 :
StateAttribute12 : EMAIL
StateAttribute13 :
StateAttribute14 :
StateAttribute15 :
StateAttribute16 : 0
StateAttribute17 : 0
StateAttribute18 : 0
StateAttribute19 : 0
StateAttribute20 : 5067
StateAttribute21 : [000.000] EWSCommon start: 1/23/2015 2:40:53 PM
[000.000] Configuring EWScommon
[000.000] Probe time limit: 60000ms, HTTP timeout: 29500ms, RetryCount: 1
[000.000] using authN: Negotiate HealthMailboxe1d5f84ec0ea47ae891b52eca728e706@domain.com N=rq!
ODvifVD!tp^A7kxOwg)fP1yq]hc82o6S0Qv=s}FdLdw&d4X3IHSx/Hjp=_{(KaDgif65vi/4_tyA?QJGt5Gvy!gpm&Nqi]&>p!
d_TCuV2]2hx[8?QY;1%K=FD[X
[000.000] using HTTP request timeout: 29500 ms
[000.000] Entering RetrySoapActionAndThrow. soapAction = GetFolder.
[000.000] Action iteration 0
[000.000] Starting (total time left 60000 ms)
[000.016] Action threw Microsoft.Exchange.WebServices.Data.ServiceRequestException: The request
failed. The remote server returned an error: (401) Unauthorized.
[005.023] Action iteration 1
[005.023] Starting (total time left 54977 ms)
[005.054] Action threw Microsoft.Exchange.WebServices.Data.ServiceRequestException: The request
failed. The remote server returned an error: (401) Unauthorized.
[005.054] EWS call GetFolder failed after retryin
StateAttribute22 :
StateAttribute23 :
StateAttribute24 :
StateAttribute25 :
Identity : 02d459e54f5e4be1be7e8abace0f0a15
IsValid : True
ObjectState : New
[PS] C:\Windows\system32>Get-Mailbox -Monitoring -Identity HealthMailboxe1d5f84ec0ea47ae891b52eca728e706@domain.com
Name Alias ServerName ProhibitSendQuota
---- ----- ---------- -----------------
HealthMailboxe1d5f84ec... HealthMailboxe1d5... email Unlimited