So you got a call from help-desk stating they have a lot of users complaining they cannot access their mailbox or performance is terribly slow? Ok, What next?I blogged onthe background process involved where exchange processes transactions and commit to the database.

• Microsoft Exchange Rpc Client Access Not Starting Sbs 2011 Remote Control
• Microsoft Exchange Rpc Client Access Not Starting Sbs 2011 Remotely
• Microsoft Exchange Rpc Client Access Not Starting Sbs 2011 Remote Start
• Microsoft Exchange Rpc Client Access Not Starting Sbs 2011 Remote Access

Let’s take a look at troubleshooting this issue.

That is when we discovered that the RPC Client Access service was not starting on a reboot. So, when migrating to SBS 2011, make sure to check the Exchange Services right after a reboot and then make that the first troubleshooting step when something is not cooperating as expected during the migration process. I have inherited a new SBS 2011 Standard server that the client has no plans on using Exchange or Share Point. Since SBS is such a fickle beast I would like to know if there is any way to safely disable Exchange and Share Point without creating problems down the road. Since it was not purchased in our account, I had to get new ones from them and install. It was a bit of a pain, but I was able to get this to work. I was able to confirm this by checking the ssl certificate with a web tool (sslshopper) and also got errors to go away on phones and remote web access users. Microsoft Exchange Rpc Client Access Not Starting Sbs 2011 Remote Posted on by admin This article describes how to troubleshoot issues that may occur in application protection in Data Protection Manager (DPM). Don't confuse an array of Client Access services with an RPC Client Access Server array that was used for RPC over TCP client connections in Exchange 2010. In Exchange 2016 and Exchange 2019, an array of Client Access services simply indicates a group of load-balanced Client Access services on Exchange 2016 or Exchange 2019 servers.

Once you ensure all mailbox stores are mounted, the next step would be to do this.

Start — Run —Perfmon

You see in the figure:

RPC requests –> Clients are requesting for data

RPC Operations/sec –>Server is acknowledging the client request.

Let’s look at how exchange prepares itself to handle client requests.

1. Microsoft Exchange Information Store service (store.exe) starts on a server
   
2. 500 RPC threads is allocated to the service upon registering with the RPC service
   
3. Clients connect and disconnect to individual RPC threads as they perform operations from their outlook like reading and sending e-mail, creating appointments and tasks, and viewing calendars.
   
4. The MSExchangeISRPC Requests performance counter indicates how many threads are currently in use (“owned” by clients).
   
5. The MSExchangeISRPC Operations/sec performance counter reflects the number of operations the server received in the past second.
   
6. If the number of RPC requests increases steadily over time, it is a clear indication that the server cannot process client operations quickly enough.
   
7. When the MSExchangeISRPC Requests performance counter reaches 500, all RPC threads have been exhausted, and clients will be unable to submit new requests to the server until all operations on an existing thread have been completed and that thread is released.
   

Right now, you are facing an issue mentioned in step7, maybe that’s why you’re reading this.

How to troubleshoot?

I am listing down the various conditions the 2 counters will be and what they mean.

MSExchangeISRPC Requests and MSExchangeISRPC Operations/sec are either low or at zero

• Bottleneck might not be on the server.
  
• Something external to the server is preventing information from reaching the server. Review Active Directory performance, network performance, client configuration etc.
  
• Get information about the GC the server is talking to using the NLTest command: nltest /SC_QUERY:<Type Domain name here>
  
• Reset the GC to another one using: nltest /sc_reset:<Type Domain name here>
  
• To get a list of DC’s for the domain, use: nltest /DCLIST:<Type Domain name here>
  

MSExchangeISRPC Requests is increasing rapidly while MSExchangeISRPC Operations/sec remains fairly stable (Horizontal line)

• The server is unable to handle the existing workload.
  
• Verify hardware components like physical memory, storage, and processor capability
• Open task manager and look at “All processes”. On an Exchange server, apart from the Store.exe process, nothing should consume memory or CPU no matter what
  
• Verify that the system drive (usually C Drive) has atleast 2-5GB free space
  
• Decrease the number of users on the server
  
• See if online defrag is getting completed on all database’s. Believe me, it plays a huge role when the DB is too large and has a lot of whitespace in it. The best way to identify this is to check eventvwr for events related to Online defragmentation like 700, 701, 702 etc.
  
• Run ExMon to see if it is a specific user is a top talker. In ExMon, you can again sort client requests by RPC requests
  

MSExchangeISRPC Requests is steadily increasing but MSExchangeISRPC Operations/sec steadily decreases

• Exchange server is the source of the bottleneck
  
• Verify that the system drive (usually C Drive) has atleast 2-5GB free space
• Open task manager and look at “All processes”. On an Exchange server, apart from the Store.exe process, nothing should consume memory or CPU no matter what
• Might be caused as something is preventing the Information Store from completing RPC operations, and the associated RPC threads remain allocated permanently
  
• More and more threads are allocated; the server has fewer available threads for new operations, so the number of new operations decreases.
  
• If the server eventually reaches 500 outstanding RPC requests, new RPC operations will be refused. This is generally caused by either a serious physical resource shortage (memory or disk) or a processing issue within the Information Store or an integrated component (antivirus, journaling, and so on).
  
• Run ExMon to see if it is a specific user is a top talker. In ExMon, you can again sort client requests by RPC requests
  
• See if online defrag is getting completed on all database’s. Believe me, it plays a huge role when the DB is too large and has a lot of whitespace in it. The best way to identify this is to check eventvwr for events related to Online defragmentation like 700, 701, 702 etc.

Microsoft Exchange Rpc Client Access Not Starting Sbs 2011 Remote Control

If ExMon shows you high RPC requests from one user mailbox, open his mailbox and look for corruption in calendar. Sort the calendar items by “list” and then by “size”. Any calendar item which is more than 8-10MB is not considered as a “calendar” item. Delete it and notify the organizer that you had to delete it since the item got corrupted.

Additionally, look for messages stuck in Outbox for that user. There might be a zip file sitting and looping again and again. Use this:

Get-Mailbox -ResultSize Unlimited | Get-MailboxFolderStatistics |Where-Object {$_.Name -eq “Outbox” -and $_.ItemsInFolder -gt ‘0’ } | Select-Object Identity, FolderType, ItemsinFolder, FolderSize | Export-CSV “C:Outbox.csv”

This is a checklist of RPC counters and their expected values:

If you have a monitoring system in place, it is mandatory to tune it to throw alerts when the counters cross the allowed limit.

Ratish Nair
MVP Exchange
Team@ MSExchange

Keywords: Exchange server performance issues, troubleshoot performance issues with exchange, Exchange RPC request are spiking high, RPC operations not processed on exchange

This topic provides information about the network ports that are used by Exchange Server 2016 and Exchange Server 2019 for communication with email clients, internet mail servers, and other services that are external to your local Exchange organization. Before we get into that, understand the following ground rules:

• We do not support restricting or altering network traffic between internal Exchange servers, between internal Exchange servers and internal Lync or Skype for Business servers, or between internal Exchange servers and internal Active Directory domain controllers in any and all types of topologies. If you have firewalls or network devices that could potentially restrict or alter this kind of internal network traffic, you need to configure rules that allow free and unrestricted communication between these servers: rules that allow incoming and outgoing network traffic on any port (including random RPC ports) and any protocol that never alter bits on the wire.

• Edge Transport servers are almost always located in a perimeter network, so it's expected that you'll restrict network traffic between the Edge Transport server and the internet, and between the Edge Transport server and your internal Exchange organization. These network ports are described in this topic.

• It's expected that you'll restrict network traffic between external clients and services and your internal Exchange organization. It's also OK if you decide to restrict network traffic between internal clients and internal Exchange servers. These network ports are described in this topic.

Network ports required for clients and services

The network ports that are required for email clients to access mailboxes and other services in the Exchange organization are described in the following diagram and table.

Notes:

• The destination for these clients and services is the Client Access services on a Mailbox server. In Exchange 2016 and Exchange 2019, Client Access (frontend) and backend services are installed together on the same Mailbox server. For more information, see Client Access protocol architecture.

• Although the diagram shows clients and services from the internet, the concepts are the same for internal clients (for example, clients in an accounts forest accessing Exchange servers in a resource forest). Similarly, the table doesn't have a source column because the source could be any location that's external to the Exchange organization (for example, the internet or an accounts forest).

• Edge Transport servers have no involvement in the network traffic that's associated with these clients and services.

Network ports required for mail flow

How mail is delivered to and from your Exchange organization depends on your Exchange topology. The most important factor is whether you have a subscribed Edge Transport server deployed in your perimeter network.

Network ports required for mail flow (no Edge Transport servers)

The network ports that are required for mail flow in an Exchange organization that has only Mailbox servers are described in the following diagram and table.

Network ports required for mail flow with Edge Transport servers

A subscribed Edge Transport server that's installed in your perimeter network affects mail flow in the following ways:

• Outbound mail from the Exchange organization never flows through the Front End Transport service on Mailbox servers. Mail always flows from the Transport service on a Mailbox server in the subscribed Active Directory site to the Edge Transport server (regardless of the version of Exchange on the Edge Transport server).

• Inbound mail flows from the Edge Transport server to a Mailbox server in the subscribed Active Directory site. Specifically:

  • Mail from an Exchange 2013 or later Edge Transport server first arrives at the Front End Transport service before it flows to the Transport service on an Exchange 2016 or Exchange 2019 Mailbox server.

  • In Exchange 2016, mail from an Exchange 2010 Edge Transport server always delivers mail directly to the Transport service on an Exchange 2016 Mailbox server. Note that coexistance with Exchange 2010 isn't supported in Exchange 2019.

Microsoft Exchange Rpc Client Access Not Starting Sbs 2011 Remotely

For more information, see Mail flow and the transport pipeline.

The network ports that are required for mail flow in Exchange organizations that have Edge Transport servers are described in the following diagram and table.

Name resolution

Microsoft Exchange Rpc Client Access Not Starting Sbs 2011 Remote Start

DNS resolution of the next mail hop is a fundamental part of mail flow in any Exchange organization. Exchange servers that are responsible for receiving inbound mail or delivering outbound mail must be able to resolve both internal and external host names for proper mail routing. And all internal Exchange servers must be able to resolve internal host names for proper mail routing. There are many different ways to design a DNS infrastructure, but the important result is to ensure name resolution for the next hop is working properly for all of your Exchange servers.

Network ports required for hybrid deployments

The network ports that are required for an organization that uses both on-premises Exchange and Microsoft 365 or Office 365 are covered in Hybrid deployment protocols, ports, and endpoints.

Network ports required for Unified Messaging in Exchange 2016

Microsoft Exchange Rpc Client Access Not Starting Sbs 2011 Remote Access

The network ports that are required for Unified Messaging in Exchange 2013 and Exchange 2016 are covered in the topic UM protocols, ports, and services.