Sunday, December 10, 2017

Polycom CX/VVX Phones - Why did my Transfer Fail?

I received an email from a client this week that users at his site could not complete transfers from their Polycom CX or VVX desk phones. Tranfers using the Skype for Business client worked perfectly, but any transfer via the desk phone failed. The client said when they initiated a transfer from a desk phone, it would either transfer to dead air or reply with a busy signal. This happened when the phone tried to transfer to an extension or the full 10 digit DID.

I broke this problem down into several things I wanted to examine. First off, was there something on the phone that wasn't allowing the transfer? I've found Polycom phone configuration files to be a blessing and a curse. You can edit the configuration files to do almost anything, however, searching thru 60-70 pages of a configuration guide to find the one setting you need can be... challenging?

Yes, challenging. We'll go with that. 😏

As it turns out, there is a field that needs to be enabled for transferring. Feel free to copy and paste the line below into a config file (.cfg) and then upload it to a phone to test:

<CONFIG_FILES
call.BlindTransferSpecialInterop="1"
/>

The other side of this issue is in Skype for Business. The cleint's user dial plan allowed for international calling and I specifically configured normalization rules for multiple internal number ranges at his site. I assumed that the phone would use the same dial plan as the user. Dialing out through the phone worked, but when a user tried to transfer a call it would fail. Are the phones using a different dial plan for transfers, somehow? Spoiler Alert: Yes.

To get transferring to work (along with the configuration change above), I had to add the same normalization rules in the user dial plan to the global dial plan. Once the normalization rules were in the global dial plan in Skype for Business, transferring worked like a charm.

I have a small request for this article - if you are someone who is better than me at Polycom phones, please explain if this is normal. Most phones I've worked with have transferring enabled by default. As for using one dial plan for making calls and another one for transferring - why is this a thing?

Comments and questions are always welcome.



Sunday, November 12, 2017

Why Can't I Search Contacts on my Polycom Trio 8800?

I've spent the past several weeks working with a client deploying several different models of Polycom phones. They're using Polycom Trio 8800's for their conference room phone solution. The client is in the UK, so if they have any questions or concerns, I've been getting up a little earlier in the morning to address them first thing.

Two weeks ago, I woke up to the email below (I'm paraphrasing):

"Josh,

We're unable to search any contacts from the Trio 8800's. Can you assist?

Here's what it looks like from the phone:

"

This normally just works, so I was confused. The client was able to login to the phone ok and select the option to search contacts, but nothing showed up. They're using S4B Online, but I didn't see any Support Issues when checking the portal. What gives?

As it turns out, a lot of others were experiencing the same problem. This was a known issue with the 5.4 firmware. Upgrading to 5.5.2 (or above) resolves the issue. I had the client deploy the latest and greatest firmware to their lab (5.6) and confirmed the issue was fixed. They deployed to their production environent shortly thereafter.

Sunday, October 29, 2017

Skype for Business - Why Can't I Escalate to a Conference?

I had a customer a few weeks ago that could not escalate to a conference. They were having a P2P call with someone, tried to add a third user to the conversation and received an error each time. The problem had been going on for about a week and they asked if I could take a look and see what was going on.

I'll admit, I've heard of this issue before, but I had not seen it yet. I started parsing the Lync Event Log on the FE server and noticed a lot of the following SQL Errors:

The 'General' tab included more data, but here is the most pertinent thing we need to focus on:

The transaction log for database 'rtcxds' is full due to 'LOG_BACKUP'.


I'm not a SQL expert, but shouldn't there be a way to increase or decrease the size of the logs? Well, there is, but it turned out to be more convuluted than I had hoped.

I logged onto the BE SQL server where the rtcxds database is located. I right clicked on the database, went to 'Properties' -> 'Files' and then clicked on the '...' button for the log file. Since there was plenty of space on the disk, I tried to increase the size of the log file from 10 GB. I received the following error:

SQL Server Error 9002: Transaction Log is Full

Yeah, understood. That's why I'm trying to increase the size, so work with me here, ok? 😉

After some Googling, I found that I have to take a backup before increasing the size. However, after talking with the customer, they're taking pretty frequent backups already. So, is there a way to take a back without really taking a backup? Yes, of course!
backup log
dname TO DISK = 'NUL:'; (note: just one 'L' is correct)

Entering the above two lines into a sql query (sans my note: on line two) within SQL Server Managment Studio tells SQL to write a backup to 'Null'. As far as SQL is concerned, a write happened, so we're all good.

After running the query above I was able to increase the size of the log file location and the customer confirmed they could escalate to a conference again. I checked the Lync Event Logs on the FE server and noticed the following message:





Monday, October 9, 2017

Skype for Business Online - Why Aren't Agents Receiving Calls in a Call Queue?

I have been helping a client set up Skype for Business Online Call Queues. This morning, they told me they were able to call the service number associated with a queue and hear the greeting, but none of the agents in the queue received the call.

Normally, this just works. I logged into their tenant and didn't see anything suspicious in their queue or group setup. Also, all the users in a queue they were testing with had Cloud PBX licenses applied.

What's going on? 😕

I was grasping for straws, but I decided to check the 'external communications' settings on the tenant. I noticed the client was whitelisting companies to federate with, but they did not include their own domain. Should you have to allow your own domain? Well, if you're whitelisting domains on your tenant, turns out the answer is 'yes' (you'll also need to allow Microsoft as well).

In the Skype for Business Admin Panel on your tenant click 'organization' then 'external communications'. There are three options in the 'external access' dropdown - 'On only for allowed domains,' 'On except for blocked domains,' and 'off completely.' If you're whitelisting domains, the dropdown should be set to, 'On only for allowed domains.'



Under the 'blocked or allowed domains' section, click the '+' button to add both your domain name and Microsoft's domain (microsoft.com). I had the customer add their own domain (microsoft.com was already allowed) and wait approximately one hour. Afterward, they tested several of the queues they had setup and they worked. All agents in each queue received the call in their respective client.

Check yo'self, amirite?  😁

Sunday, September 24, 2017

A Lesson About Toll Fraud via SIP Tracing

I had Skype for Busiess EV customer contact me saying one of their call center agents couldn't call a delear in Guatemala (For reference, the customer has multiple dealrs throughout Centeral America.). I knew the customer has international dialing capabilities through their PSTN Carrier, so not being able to call one country in Central America seemed odd to me. Why was their PSTN Carrier singling out Guatemala?

I started troubleshooting by making sure the user with the failed calls was EV enabled and also had a voice policy in Skype for Business that allowed for international calls. (Yes, I know, it seems kinda basic, but I prefer to start with the easy things and work my way up.) Upon investigation, the user was EV enabled and setup properly in Skype for Buisnesss to allow for international dialing. I asked if other international calls were working and the user said 'yes'.

The problem didn't appear to be in Skype for Business. I also had another user confirm that they couldn't dial any Guatemalan numbers either. Since some international calls were making it out, I needed to get a trace from their gateway to see what was going on.

Luckily for me, they have an AudioCodes SBC. The AudioCodes Syslog tool makes troubleshooting issues like this one much easier. I had the user having issues place a call while I ran the call trace from the Audiocodes SBC. The 'call ladder' diagram in the Syslog tool is excellent for seeing the flow of the SIP messages between Skype for Business, SBC, and Carrier. Here's the call ladder for the failed Guatemala call:



Notice the PSTN Carrier sends a SIP 403 Forbidden Message? A SIP 403 Forbidden message typically means that the user is not permitted to make the call. I've seen this in Skype for Business, for example, when someone with a long distance voice policy tries to make an international call. I didn't know why the customer was getting a SIP 403 from their PSTN Carrier, but I recommended they open a support ticket with their PSTN Carrier and provide them with the trace I took from the SBC.

The PSTN Carrier contacted the customer later that day and said due to high volumes of toll fraud from certain countries in Central America, they were blocking all calls. Wow! I have never seen/heard a carrier proactively do this before, but this does explain the behavior the customer was experiencing.

The customer is going to work with their PSTN Carrier to see what can be done to unblock calls to certain numbers (essentially, white-listing numbers they need to call). This was interesting problem, but I'm glad the resolution was pretty straightforward. Also, I can't recommend the AC Syslog tool enough when you need to take a trace from an AudioCodes SBC.



Monday, August 28, 2017

Change Caller ID for Individual Skype for Business/Lync 2013 Users on Sonus 1k/2k SBC

I've had multiple clients approach me with the same scenario in Skype for Business/Lync 2013: <NAME> in <DEPARTMENT> wants to change the caller id for their individual DID or extension to a different number.  For clients that have a Sonus 1k/2k SBC, this is very easy to do.

Transformation rules on the Sonus 1k/2k SBC allow for manipulating the 'Calling Party Address/Number' field. The 'Calling Party' is who initiates the call. If we change the calling party from the users number to the number they want, that addresses their issue.

First, we create a transformation rule. Below are examples for both an e164 number and an extension:

Extenstion Example

E164 Example

Lastly, we'll also need to include a 'catch all' rule. The 'catch all' rule catches any other number and passes it through without manipulation. Below is an example:





Catch All Example

After our rules are created, we'll need to make sure they are ordered correctly. Transformation rules are evaluated from top to bottom. If our 'catch all' rule was ordered above our e164 rule, for example, then our e164 rule would never be evaluated. Clicking the pencil icon at the top of the page will allow us to use arrows to move our rules up and down the list of tranformation rules. Use the arrows to move any rules you created above the 'catch all' rule and then click save.

Sunday, August 6, 2017

Skype for Business 30 Second Inbound Call Delay

A customer contacted me with an inbound call issue. Inbound calls were taking at least 30 seconds to connect, and some would not connect at all. While I've experienced call delays before, the root cause was different each time I investigated.

Since the issue was happening so frequently, I took a SIP trace from their Audio Codes Gateway. I noticed the following pattern in the SIP trace:

INVITE
SIP 183
INVITE
SIP 183
CANCEL
200 OK

If everything was working correctly, *I should* see the SIP 183 answered with a '200 OK' rather than an additional SIP 183. My initial thought was, "Why isn't the carrier answering my 183 message?" I spoke with an engineer from the carrier (AT&T) and he provided logs from his end showing they were, in fact, answering my SIP 183. The customer side, however, was not sending the SIP 200 back to them.

What's going on here? The carrier's log proved they weren't causing the problem. The other time I've experienced this issue was due to a network issue. Specifically, a customer's router was looping traffic due to misconfigured routes. The traffic appeared to be routed correctly, though. What else could be causing this?

I asked the customer if any changes had been made recently. The customer mentioned they recently shutdown ports 80 & 443 to the outside due to the WannaCrypt Virus.

Bingo! We found our culprit. 😉

During the ICE (Interactive Connectivity Establisment) Negotiation for audio/video, the Edge Server provides a list of candidates to help Skype for Business find the best possible path for media. The Edge Server *must* have ports 80 & 443 open to the outside world for HTTP/HTTPS traffic. Since the customer had shut off these ports, ICE Negotiation was disrupted. Once the ports were opened via the firewall again, the inbound audio delay disappeared.