Monday, September 30, 2013

One-way audio dropping on Lync Server

Problem
I have Lync Server 2010 with a SIP Trunk provided by Intelepeer and a SonicWall firewall. When calls are made between internal users to the outside (i.e. Lync calls to PSTN), sometimes the audio drops going to the PSTN user. It is a one-way audio problem, because the Lync user can still hear the other person. We have struggled with this problem for the past year (mostly because i haven't had any time to focus on it).


Solution
1. Add a NIC to the server
2. In the Topology Builder I'm going to differentiate between the Primary IP and the PSTN IP

In my 1 NIC environment i have the IP as 15.1.1.82, so this is what i should put as the Primary IP. In my 2 NIC environment my new IP, which is on a different subnet, 16.1.1.1 instead of 15.1.1.1, is going to 16.1.1.16 and assigned as the PSTN IP
  • Edit the Front-End Server properties
  • Select Limit service usage to selected IP addresses. 
  • For the Primary IP i'm putting in 15.1.1.82
  • For the PSTN IP i'm putting in 16.1.1.16
  • Publish the change
3.  On the firewall, in my case it's a SonicWall PRO 2040 Standard, first I'm going to map my PSTN IP to my public IP, then I'm going to add access rules allowing traffic to/from my PSTN SIP Trunk to my Mediation Server's PSTN IP (internal and public). This is basically making the firewall wide open to SIP/RTP traffic from my SIP Trunk provider, essentially putting the Meditation Server outside of the firewall.
  • In Network > One-to-One NAT replace the primary IP (i.e. 15.1.1.82) with the PSTN IP (i.e. 16.1.1.16). 
  • In Firewall > Access Rules add the following rule
    • Allow
    • Service = Any
    • Source = put in the range of your PSTN IP addresses given to you by your provider. For me this is from 66.x.x.59 - 66.x.x.62
    • Destination = put in the public IP to your Mediation Server
    • Repeat this for the internal IP of the Mediation Server
4. Test calls to make sure it works


UPDATE
The problem resurfaced after several weeks of not happening. I'm noticing alot of errors in the event log on the frontend server saying that it had disconnected from the conferencing service on the edge server.
I followed the advice in this article: http://msunified.net/2011/02/09/lync-server-front-end-lost-connection-to-the-web-conferencing-edge-server/ and disabled IPv6 on the Edge (it was already disabled on the front end). Awaiting results. 

Windows Firewall verification
Another issue I was seeing is that we missed alot of ports that need to be opened in Windows Firewall. I was seeing events in the Lync Logging tool such as "The server actively refused the connection." In my experience with our own networking software that either means
1) The port hasn't be opened in Windows Firewall
I went through this list: http://technet.microsoft.com/en-us/library/gg398833%28v=ocs.14%29.aspx and double checked everything. I was missing several ports in the "Front End Servers that also run a Collocated Mediation Server" section. Pretty much every row in that table where i didn't have the port opened was in a section that I was having intermittent problems with, so that's nice to know what was causing the problem.

-or-

2) The port isn't being listened to.
You can use tools like Port Query to figure out if a port is being listened to. This is most likely a configuration problem or as simple as the service not running.

Update 2
We still experienced this problem after all of the steps above. So i ran Wireshark on all endpoints. I saw that packets were leaving the server even when audio was dropped on the other end of the call. Then I looked in SonicWall. First of all, we had the factory firmware on there, 2.2.x. That SonicWall PRO 2040 is EOL so that means they no longer release firmware updates for it. I did find 3.2.x though. After upgrading to that I was able to use the packet trace diagnostic tool. They must have upgraded it between the versions because i finally was able to see something wrong: packets were not being sent from the public IP to Intelepeer. I could see they were being sent from the private IP. But there was no indication of why it was dropping. "Firwall consuming packets without logging it" sort of situation here. So luckily we had a TZ210 being used at an office. That is a 5th gen SonicWall, compared to the Sonicwall PRO 2040 it's much much better, and the best part is the latest firmware is from Oct 2013 (last month as of this writing).

So i went into the office and replaced the TZ210 with a dlink. It's kinda funny how the office had a better firewall than our server network. After hours i then replaced the PRO 2040 with the TZ210. I haven't ran into this problem yet, but if it does happen at least the SonicOS version (5.9.x) will show what's happening, so I can take action.

Note: It took me awhile to learn how to do One-to-One NAT in 5.9, because i was used to the ridiculously simplified version in 2.2. The trick is to configure the inbound rule first, then check the "create reflexive rule". The documentation has you create an outbound rule first, which doesn't create the reflexive rule for you. I also had a difficult time figuring out how to add a second subnet. But there's plenty of documentation for the newest versions of SonicOS, so it turned out to be easy. 
  
Tools
1. WireShark - http://www.wireshark.org/download.html : I used this to discover that I was getting traffic from an IP from Intelepeer (SIP Trunk Provider) that i didn't have explicitly unblocked in my SonicWall (external firewall). The awesome part is you can filter the network traffic for VoIP traffic, so it's pretty easy to track down relevant traffic.

2. Lync Logging Tools - http://www.microsoft.com/en-us/download/details.aspx?id=21165 : I used this to discover that some of the ports weren't opened in the Windows Firewall on the frontend server. I have Deployment, InboundRouting, ManagementCore, MCUInfra, MediaStack_RTP, MediationServer, OutboundRouting, S4, ServerConfiguration, SIPStack, and TranslationApplication checked with all flags.
Leave comments!
Since this problem took me such a long time to solve, and was painful, I'd like to know if it helps anyone else, or if you need help with it. Please feel free to ask questions if you're having the same problem. I've had at least 3 support cases opened with Microsoft, 2 opened with Intelepeer, and multiple questions on support forums, and of course lots and lots of Googling. 


References
1. http://social.technet.microsoft.com/Forums/lync/en-US/dee64e0e-c343-4e60-83d7-9a899bc84b2f/need-recommendations-for-firewall-that-will-work-100-with-lync-voice?forum=ocsvoice

2. http://technet.microsoft.com/library(OCS.14)/ms.lync.tbdep.FrontEndGeneralSettingsExpander.aspx 

3. Lync Firewall Rules Viewer: http://blogs.technet.com/b/nexthop/archive/2012/07/03/lync-firewall-rules-viewer.aspx

4. Frontend Server lost connection to the web conferencing edge http://msunified.net/2011/02/09/lync-server-front-end-lost-connection-to-the-web-conferencing-edge-server/

No comments:

Post a Comment

There was an error in this gadget