Tuesday, December 26, 2017

Skype for Business - Hey, Where Did my Response Groups Go?!?

Now, for its next trick, Skype for Business will make Response Groups in the Control Panel magically disappear!

Seriously? You have a lot of nerve, Skype For Business. 😉

A client opened a ticket recently for help recovering his Skype for Business Response Groups. He shared a picture of the empty Control Panel:

Bye, Felicia.

I had the client run several commands to verify the Response Groups were still seen via PowerShell. 'Get-CsRGSWorkflow' and 'Get-CsRGSQueue' both returned all workflows and queues that had been created. At this point, the only other thing I could think of that may prevent him from seeing the Response Groups was his permissions.

I asked to see which RBAC groups he was in and I noticed he was in both the 'CsReponseGroupAdministrator' and 'CsResponseGroupManager' groups. The 'admin' group manages all the Response Groups for a site, while the 'manager' groups allows for management of specific response groups (Technet Article).

Was it possible that somehow the 'manager' group was overwriting the permissions for the 'admin' group? As it turns out, yes, that appears to have been what happened. A search via Google produced a  blog post outlining the exact same problem in Lync 2013. I had the user remove his account from the 'manager' group and leave himself in the 'admin' group. A quick sign out/sign in forced the group change and he was able to see his Response Groups in the Control Panel again.

Sunday, December 10, 2017

Polycom CX/VVX Phones - Why did my Transfer Fail?

I received an email from a client this week that users at his site could not complete transfers from their Polycom CX or VVX desk phones. Tranfers using the Skype for Business client worked perfectly, but any transfer via the desk phone failed. The client said when they initiated a transfer from a desk phone, it would either transfer to dead air or reply with a busy signal. This happened when the phone tried to transfer to an extension or the full 10 digit DID.

I broke this problem down into several things I wanted to examine. First off, was there something on the phone that wasn't allowing the transfer? I've found Polycom phone configuration files to be a blessing and a curse. You can edit the configuration files to do almost anything, however, searching thru 60-70 pages of a configuration guide to find the one setting you need can be... challenging?

Yes, challenging. We'll go with that. 😏

As it turns out, there is a field that needs to be enabled for transferring. Feel free to copy and paste the line below into a config file (.cfg) and then upload it to a phone to test:


The other side of this issue is in Skype for Business. The cleint's user dial plan allowed for international calling and I specifically configured normalization rules for multiple internal number ranges at his site. I assumed that the phone would use the same dial plan as the user. Dialing out through the phone worked, but when a user tried to transfer a call it would fail. Are the phones using a different dial plan for transfers, somehow? Spoiler Alert: Yes.

To get transferring to work (along with the configuration change above), I had to add the same normalization rules in the user dial plan to the global dial plan. Once the normalization rules were in the global dial plan in Skype for Business, transferring worked like a charm.

I have a small request for this article - if you are someone who is better than me at Polycom phones, please explain if this is normal. Most phones I've worked with have transferring enabled by default. As for using one dial plan for making calls and another one for transferring - why is this a thing?

Comments and questions are always welcome.

Sunday, November 12, 2017

Why Can't I Search Contacts on my Polycom Trio 8800?

I've spent the past several weeks working with a client deploying several different models of Polycom phones. They're using Polycom Trio 8800's for their conference room phone solution. The client is in the UK, so if they have any questions or concerns, I've been getting up a little earlier in the morning to address them first thing.

Two weeks ago, I woke up to the email below (I'm paraphrasing):


We're unable to search any contacts from the Trio 8800's. Can you assist?

Here's what it looks like from the phone:


This normally just works, so I was confused. The client was able to login to the phone ok and select the option to search contacts, but nothing showed up. They're using S4B Online, but I didn't see any Support Issues when checking the portal. What gives?

As it turns out, a lot of others were experiencing the same problem. This was a known issue with the 5.4 firmware. Upgrading to 5.5.2 (or above) resolves the issue. I had the client deploy the latest and greatest firmware to their lab (5.6) and confirmed the issue was fixed. They deployed to their production environent shortly thereafter.

Sunday, October 29, 2017

Skype for Business - Why Can't I Escalate to a Conference?

I had a customer a few weeks ago that could not escalate to a conference. They were having a P2P call with someone, tried to add a third user to the conversation and received an error each time. The problem had been going on for about a week and they asked if I could take a look and see what was going on.

I'll admit, I've heard of this issue before, but I had not seen it yet. I started parsing the Lync Event Log on the FE server and noticed a lot of the following SQL Errors:

The 'General' tab included more data, but here is the most pertinent thing we need to focus on:

The transaction log for database 'rtcxds' is full due to 'LOG_BACKUP'.

I'm not a SQL expert, but shouldn't there be a way to increase or decrease the size of the logs? Well, there is, but it turned out to be more convuluted than I had hoped.

I logged onto the BE SQL server where the rtcxds database is located. I right clicked on the database, went to 'Properties' -> 'Files' and then clicked on the '...' button for the log file. Since there was plenty of space on the disk, I tried to increase the size of the log file from 10 GB. I received the following error:

SQL Server Error 9002: Transaction Log is Full

Yeah, understood. That's why I'm trying to increase the size, so work with me here, ok? 😉

After some Googling, I found that I have to take a backup before increasing the size. However, after talking with the customer, they're taking pretty frequent backups already. So, is there a way to take a back without really taking a backup? Yes, of course!
backup log
dname TO DISK = 'NUL:'; (note: just one 'L' is correct)

Entering the above two lines into a sql query (sans my note: on line two) within SQL Server Managment Studio tells SQL to write a backup to 'Null'. As far as SQL is concerned, a write happened, so we're all good.

After running the query above I was able to increase the size of the log file location and the customer confirmed they could escalate to a conference again. I checked the Lync Event Logs on the FE server and noticed the following message:

Monday, October 9, 2017

Skype for Business Online - Why Aren't Agents Receiving Calls in a Call Queue?

I have been helping a client set up Skype for Business Online Call Queues. This morning, they told me they were able to call the service number associated with a queue and hear the greeting, but none of the agents in the queue received the call.

Normally, this just works. I logged into their tenant and didn't see anything suspicious in their queue or group setup. Also, all the users in a queue they were testing with had Cloud PBX licenses applied.

What's going on? 😕

I was grasping for straws, but I decided to check the 'external communications' settings on the tenant. I noticed the client was whitelisting companies to federate with, but they did not include their own domain. Should you have to allow your own domain? Well, if you're whitelisting domains on your tenant, turns out the answer is 'yes' (you'll also need to allow Microsoft as well).

In the Skype for Business Admin Panel on your tenant click 'organization' then 'external communications'. There are three options in the 'external access' dropdown - 'On only for allowed domains,' 'On except for blocked domains,' and 'off completely.' If you're whitelisting domains, the dropdown should be set to, 'On only for allowed domains.'

Under the 'blocked or allowed domains' section, click the '+' button to add both your domain name and Microsoft's domain (microsoft.com). I had the customer add their own domain (microsoft.com was already allowed) and wait approximately one hour. Afterward, they tested several of the queues they had setup and they worked. All agents in each queue received the call in their respective client.

Check yo'self, amirite?  😁

Sunday, September 24, 2017

A Lesson About Toll Fraud via SIP Tracing

I had Skype for Busiess EV customer contact me saying one of their call center agents couldn't call a delear in Guatemala (For reference, the customer has multiple dealrs throughout Centeral America.). I knew the customer has international dialing capabilities through their PSTN Carrier, so not being able to call one country in Central America seemed odd to me. Why was their PSTN Carrier singling out Guatemala?

I started troubleshooting by making sure the user with the failed calls was EV enabled and also had a voice policy in Skype for Business that allowed for international calls. (Yes, I know, it seems kinda basic, but I prefer to start with the easy things and work my way up.) Upon investigation, the user was EV enabled and setup properly in Skype for Buisnesss to allow for international dialing. I asked if other international calls were working and the user said 'yes'.

The problem didn't appear to be in Skype for Business. I also had another user confirm that they couldn't dial any Guatemalan numbers either. Since some international calls were making it out, I needed to get a trace from their gateway to see what was going on.

Luckily for me, they have an AudioCodes SBC. The AudioCodes Syslog tool makes troubleshooting issues like this one much easier. I had the user having issues place a call while I ran the call trace from the Audiocodes SBC. The 'call ladder' diagram in the Syslog tool is excellent for seeing the flow of the SIP messages between Skype for Business, SBC, and Carrier. Here's the call ladder for the failed Guatemala call:

Notice the PSTN Carrier sends a SIP 403 Forbidden Message? A SIP 403 Forbidden message typically means that the user is not permitted to make the call. I've seen this in Skype for Business, for example, when someone with a long distance voice policy tries to make an international call. I didn't know why the customer was getting a SIP 403 from their PSTN Carrier, but I recommended they open a support ticket with their PSTN Carrier and provide them with the trace I took from the SBC.

The PSTN Carrier contacted the customer later that day and said due to high volumes of toll fraud from certain countries in Central America, they were blocking all calls. Wow! I have never seen/heard a carrier proactively do this before, but this does explain the behavior the customer was experiencing.

The customer is going to work with their PSTN Carrier to see what can be done to unblock calls to certain numbers (essentially, white-listing numbers they need to call). This was interesting problem, but I'm glad the resolution was pretty straightforward. Also, I can't recommend the AC Syslog tool enough when you need to take a trace from an AudioCodes SBC.

Monday, August 28, 2017

Change Caller ID for Individual Skype for Business/Lync 2013 Users on Sonus 1k/2k SBC

I've had multiple clients approach me with the same scenario in Skype for Business/Lync 2013: <NAME> in <DEPARTMENT> wants to change the caller id for their individual DID or extension to a different number.  For clients that have a Sonus 1k/2k SBC, this is very easy to do.

Transformation rules on the Sonus 1k/2k SBC allow for manipulating the 'Calling Party Address/Number' field. The 'Calling Party' is who initiates the call. If we change the calling party from the users number to the number they want, that addresses their issue.

First, we create a transformation rule. Below are examples for both an e164 number and an extension:

Extenstion Example

E164 Example

Lastly, we'll also need to include a 'catch all' rule. The 'catch all' rule catches any other number and passes it through without manipulation. Below is an example:

Catch All Example

After our rules are created, we'll need to make sure they are ordered correctly. Transformation rules are evaluated from top to bottom. If our 'catch all' rule was ordered above our e164 rule, for example, then our e164 rule would never be evaluated. Clicking the pencil icon at the top of the page will allow us to use arrows to move our rules up and down the list of tranformation rules. Use the arrows to move any rules you created above the 'catch all' rule and then click save.

Sunday, August 6, 2017

Skype for Business 30 Second Inbound Call Delay

A customer contacted me with an inbound call issue. Inbound calls were taking at least 30 seconds to connect, and some would not connect at all. While I've experienced call delays before, the root cause was different each time I investigated.

Since the issue was happening so frequently, I took a SIP trace from their Audio Codes Gateway. I noticed the following pattern in the SIP trace:

SIP 183
SIP 183
200 OK

If everything was working correctly, *I should* see the SIP 183 answered with a '200 OK' rather than an additional SIP 183. My initial thought was, "Why isn't the carrier answering my 183 message?" I spoke with an engineer from the carrier (AT&T) and he provided logs from his end showing they were, in fact, answering my SIP 183. The customer side, however, was not sending the SIP 200 back to them.

What's going on here? The carrier's log proved they weren't causing the problem. The other time I've experienced this issue was due to a network issue. Specifically, a customer's router was looping traffic due to misconfigured routes. The traffic appeared to be routed correctly, though. What else could be causing this?

I asked the customer if any changes had been made recently. The customer mentioned they recently shutdown ports 80 & 443 to the outside due to the WannaCrypt Virus.

Bingo! We found our culprit. 😉

During the ICE (Interactive Connectivity Establisment) Negotiation for audio/video, the Edge Server provides a list of candidates to help Skype for Business find the best possible path for media. The Edge Server *must* have ports 80 & 443 open to the outside world for HTTP/HTTPS traffic. Since the customer had shut off these ports, ICE Negotiation was disrupted. Once the ports were opened via the firewall again, the inbound audio delay disappeared.

Wednesday, July 19, 2017

Skype for Business Mobile Authentication Issue with IIS ARR Reverse Proxy

I haven't worked much with IIS ARR. Most of my customers prefer either a F5 or Kemp for their load balancing solution. I recently worked with a customer who did not have a solution in place and did not want to spend money on a dedicated appliance.

Necessity is the mother of invention learning something new. 😀

A colleague of mine forwarded me this article he had used previously to setup IIS ARR for another customer. While the article is several years old, it's still an excellent resource and I would recommend bookmarking it. The guide in the website linked above walked me through most of what I needed for the setup.

I finished the configuration steps and signed in on my Android phone via a test account the customer provided me. I added the customer to my contacts list and sent him an IM. Everything was working alright until my account was logged off without warning and I received the error message below:

I wasn't sure what to make of this error. My mobile phone was connected via wifi, so I thought maybe there was an issue with my home wifi network. I shut off the wifi connection on my phone to connect via my mobile network and couldn't sign back in with the same credentials. My mobile client flashed the error below:

Switching from my wifi network to my mobile network didn't help at all. I couldn't login, so if anything, I made it worse. What the heck!?! 😡

The error I received this time was different. Could 'server address' be an issue with the external DNS entries? My intial impression was no, probably not. If the external DNS entries were incorrect/misconfigured then authentication would not have worked the first time. If it's not my mobile phone network or external DNS, could it be the reverse proxy?

I double checked the reverse proxy settings again. The server certificates and port bindings were correct. What was I missing? The answer was in the server farms I created. The customer had two standalone servers and each server had its own external web services address. Each external web services server farm should have only one server listed. I accidently listed both servers in each server farm. This was clearly an oversight and rookie mistake on my part! I removed the additional entry in each farm and the authentication issues disappeared. 

Below is the correction I made on the reverse proxy (skype1 in webext-se, skype2 in webext-se2):

I hope my 'learning experience' saves you from the same mistake I made in the future.

Tuesday, July 11, 2017

Automate Documentation via the Skype for Business Environment Report

Documentation is useful, but in my experience, it's one of the last things that gets done. At my last job, I was on a team of four people who supported over 20,000 Lync 2013 users. My day to day, tactical activities took precedence over writing wiki articles outlining server details, DNS entries, and other details. To be clear: I documented things, but it was often after hours. I spent most of my normal business hours keeping my users content and the environment stable.

If only there was a quick way to produce documentation about a Skype for Business environment...

Spoiler Alert: lucky for us, there is. 😏

A colleague of mine mentioned this super helpful tool. The Skype for Business Environment Report is a collection of PowerShell scripts that pull information from your Skype for Business topology and output the data into a Word Document, Visio Diagram, or both. 

Below is a list of steps to help run the scripts:

1. Download and unzip the file on a front end server in the Skype for Business environment.

2. Open PowerShell and run the 'Get-CsEnvironmentInfo.ps1' script. The script collects the initial information for the Skype for Business environment. Optional: you can supply credentials for your front end servers using the '-InternalCredentials' switch and for your edge servers using the '-EdgeCredentials' switch.

3. The script will run and collect information from your topology. When you are done, a .zip file is produced and resides in the script directory.

4. Note: the scripts to create the Word Document and Visio Diagram must be run on a machine with Word and Visio installed.

Run the 'New-CsEnvReport.ps1' script and use the .zip file you produced in the previous step. A window opens to allow you to select the .zip file from the previous step. Optional: you can supply the .zip file name via the 'envdatafile' switch. 

5. The Word Document is created in the script directory. Open the Word Docment and view all the data from your topology that is organized with a table of contents. Here is an example:

6. The steps to create the Visio Diagram are similar to Word Document creation steps above. You will run the 'New-CsEnvDiagram' script and supply the .zip file similar to Step 4.

Monday, July 3, 2017

Studying for Exam 70-333 (Deploying Enterprise Voice with Skype for Business 2015)?

Taking the 70-333 Exam soon? If so, you'll want to check out this book written by Michael Tressler (aka, flinchbot). I found this book super helpful when preparing for my exam (which I passed in April 2017, by the way, with a 975 😉) and it does a really good job of covering all aspects of Enterprise Voice. Hands on experience is the best teacher, in my opinion, but I would recommend using this book as a study/reference guide.

 Time to study for my Office 365 Exams. MCSE Productivity, here I come!

Thursday, June 29, 2017

Office 365 Oddity - Why Is the S4B Dial-In Conferencing Number Showing Up Twice?

Well, this is odd.

I was helping a customer in an Office 365 Hybrid Deployment. They decided to go with S4B on prem and Exchange in the cloud. The customer tested scheduling a S4B Meeting and noticed the following:

(Dial-In Conferencing Numbers are blacked out, but trust me, they're the same. ;) )

Why is this a thing?

I wasn't for certain, however, my best guess was this was either being caused by the O365 Pro Plus Client or Exchange. Time to troubleshoot!

I checked S4B Technet Forums and found a post from another user regarding this same issue. I followed the advice in the post above and had the customer schedule a meeting through OWA rather than through their client. Low and behold, scheduling the meeting through OWA listed the dial-in conferencing number once. Alright, now I'm getting somewhere, but what's going on with the client?

The customer was running the most recent O365 First Release Deferred Channel Version (v 1701 as of June 2017). There is a more recent release in the O365 Current Channel (v 1705), but it requires switching channels and running what is essentially a beta version.

NOTE: You cannot revert channels when switching from 'Deferred' to 'Current'. I talked it over with the customer and stated the risks. He agreed to switch channels and do the upgrade. Here is a link that walks through switching from Deferred Channel to Current Channel.

After the upgrade, the customer confirmed that the number is no longer listed twice. I let the customer know that his colleagues on Deferred Channel should see the update in approximately four months according to the O365 Channel Release Schedule

I hope this helps. 

Wednesday, June 28, 2017

PowerShell Tip - How to Avoid Truncating Data

Sometimes, PowerShell will truncate your command output:

Uggh... how annoying. Why do you hide things from me, PowerShell?

Thankfully, this is easy to correct. There is an environment variable in PowerShell for 'EnumerationLimit' (i.e. the number of items enumerated to the screen during a session):

Setting this variable to -1 essentially allows for unlimited enumeration.

I hope you find this helpful.