Microsoft Lync Edge Servers Certificate Expiration Issue

Today I had an interesting problem to fix, I’m not a Microsoft Lync expert and to be honest haven’t had much if any dealing with our Lync infrastructure. Unfortunately our Lync guy is away on holiday so the task fell to me.

 

What was the problem?

Users reported that Microsoft Lync or Skype for Business as it’s now called displayed an error and that external contacts did not show correctly or could be messaged. Basically something was up and the finger was pointed at federation.

 

Troubleshooting Steps

To begin with I logged onto our Lync Edge servers to check the services and found some of them were not running.

I tried to start the services however this failed with the following error –

Next I went for a look in the Event Viewer to see whether anything useful could be found in the logs. Thankfully the information was actually pretty useful.

Log Name:      System
Source:        Service Control Manager
Date:          28/07/2015 09:42:47
Event ID:      7024
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      LYNCEDGE01.bytesizedalex.com
Description:
The Lync Server Access Edge service terminated with the following service-specific error: 
A required certificate is not within its validity period when verifying against the current system clock or the timestamp in the signed file.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
   <System>
     <Provider Name="Service Control Manager" Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}" EventSourceName="Service Control Manager" />
     <EventID Qualifiers="49152">7024</EventID>
     <Version>0</Version>
     <Level>2</Level>
     <Task>0</Task>
     <Opcode>0</Opcode>
     <Keywords>0x8080000000000000</Keywords>
     <TimeCreated SystemTime="2015-07-28T08:42:47.789822600Z" />
     <EventRecordID>517880</EventRecordID>
     <Correlation />
     <Execution ProcessID="760" ThreadID="4308" />
     <Channel>System</Channel>
     <Computer>LYNCEDGE01.bytesizedalex.com</Computer>
     <Security />
   </System>
   <EventData>
     <Data Name="param1">Lync Server Access Edge</Data>
     <Data Name="param2">%%2148204801</Data>
     <Binary>5200740063005300720076000000</Binary>
   </EventData>
</Event>

Thanks to the informative event log we know that the issue is probably certificate related, my guess was it had expired. I opened up an MMC and launched the certificate store for the computer.

Well now look at that – we have a certificate which expired on the 25/07/15 – 3 days ago. Thankfully somebody provided text for the ‘Friendly Name’ so we know this certificate is the ‘Internal’ cert. To be fair I can also tell from the issuing authority as it’s our internal Certificate Authority (CA) whereas the external certificate comes from DigiCert.

Now we know there is definitely an expired certificate which needs to be replaced but I’m going to assume even after adding a new certificate, Lync will have to be told to use it so how do I do that?

I decided to browse through the ‘Program Files’ directory and take a look in the Lync folders. The default path for these is C:\Program Files\Microsoft Lync Server 2013.

Within this folder I had a few choices, I figured the ‘Deployment’ folder was a good place to start –

Within the ‘Deployment’ folder there is an executable called ‘Deploy.exe’ – I launched this program to see what options I had from the deployment program.

OK this looks promising – we can see that Step 1 and 2 are both indicated as complete (the wizard ran a check against these which completed before I could screenshot) however Step 3 and 4 do not have a tick.

I launched the ‘Run’ option for Step 3 which presented me with a new popup window –

We can see that Lync has two certificates listed. one of them is the External Edge certificate and is in date and displayed with a green tick. We also see an Internal Edge certificate however this one has a warning against it.

The next step is for us to create a new certificate, import it to the server and assign it in Lync. There are quite a few ways of doing this but I decided to use the Lync Certificate Wizard tool to ensure the request was valid. I could have looked on TechNet or somewhere like that to get the certificate requirements but up to this point I hadn’t needed help so why start now?

 

Certificate Request and Creation

I clicked on the ‘Request’ button in the Certificate Wizard tool, this opened a new window for me.

I clicked ‘Next’ and selected the second radio button to prepare a Certificate Signing Request (CSR).

The next window asked me to provide a file name and location.

I gave a path and then clicked ‘Next’.

The wizard gives us the option of specifying a different CA certificate template. By default it will use the built in WebServer template. I decided to leave it with this, again I could have researched what the certificate requirements were but it was easier to go with the default choice.

Now we are asked to provide a ‘Friendly’ name for the certificate and choose a key size/bit length. As a minimum we want to select a 2048 bit key size, anything less is insecure. I also chose to make the private key exportable, the reason for this is we have multiple Lync Edge servers. I want to be able to copy the same certificate to each server and to do so I’ll need the private key.

Next we need to provide some organisation details, this is just information used to complete the CSR and if you’ve ever created a certificate I’m sure you are familiar with these values.

Again we are asked for further details as part of the CSR creation, in this case our Country, State and Locality.

Now it is time to configure the certificate subject name.

You will have seen that there was a box for Subject Alternative Names (SANs), we now get a chance to configure them. In my case I don’t need any SAN attributes so I left this blank and proceeded.

Finally we are presented with a summary of our choices.

I quite like this next screen as it gives me the commands used by this wizard and a quick way to access the log file. Just click ‘Next’ to continue.

Commands –

Request-CSCertificate -New -Type Internal -Output "C:\Users\Administrator\Documents\lyncedgepool.req" -Country GB -State "Some State" -City "Some City" -FriendlyName "Lync Edge Internal Certificate" -KeySize 2048 -PrivateKeyExportable $True -Organization "Some Organisation Name" -OU "IT Services" -AllSipDomain -Verbose -Report "C:\Users\Administrator\AppData\Local\Temp\1\Request-CSCertificate-[2015_07_28][10_27_27].html"
Creating new log file "C:\Users\Administrator\AppData\Local\Temp\1\Request-CSCertificate-188944ef-2316-48f1-84a4-82092452b880.xml".
Create a certificate request based on Lync Server configuration for this computer.
Offline request generated for use "Internal".
No changes were made to the Central Management Store.
Creating new log file "C:\Users\Administrator\AppData\Local\Temp\1\Request-CSCertificate-[2015_07_28][10_27_27].html".
"Request-CSCertificate" processing has completed successfully.
Detailed results can be found at "C:\Users\Administrator\AppData\Local\Temp\1\Request-CSCertificate-[2015_07_28][10_27_27].html".

We now have access to the CSR file, click on ‘View’ to open up the request.

Now that we have our CSR we can submit this to a CA. I’m going to assume that like me you have your own internal CA and will either use the web interface or the certreq command.

 

Certificate Import

Once you have submitted the CSR and generated a certificate file it’s time to import it back onto the server.

From the Lync Certificate Wizard menu click on the ‘Import Certificate button.

This will open a new window and from here we can select the certificate file to import. At this point it is worth mentioning that you could of course have created the certificate through other means. You may have then exported to a .pfx with a password and if so you can enter the password here. It all depends how you decided to create the certificate.

Select your certificate file.

Click ‘Next’.

A summary screen is presented, click ‘Next’.

Again the system is kind enough to present us with a page detailing the commands to be run. Something to note – if you are importing a file with the private key and have entered a password it will be displayed here in CLEAR TEXT. If you intend to document or blog this screen be mindful.

Commands –

Import-CSCertificate -Path "C:\certnew.cer" -PrivateKeyExportable $False -Verbose -Report "C:\Users\Administrator\AppData\Local\Temp\1\Import-CSCertificate-[2015_07_28][10_30_25].html"
Creating new log file "C:\Users\Administrator\AppData\Local\Temp\1\Import-CSCertificate-7d3d3709-13f9-462f-a816-e62b9345bab9.xml".
Import the certificate response or read the certificate from file.
No changes were made to the Central Management Store.
Creating new log file "C:\Users\Administrator\AppData\Local\Temp\1\Import-CSCertificate-[2015_07_28][10_30_25].html".
"Import-CSCertificate" processing has completed successfully.
Detailed results can be found at "C:\Users\Administrator\AppData\Local\Temp\1\Import-CSCertificate-[2015_07_28][10_30_25].html".

We can check the log file using the helpful link.

 

Assign New Lync Certificate

Now that we have imported the certificate it’s time to assign it to the Lync service.

Once again we start with the Lync Certificate Wizard menu. This time we will click on the ‘Assign’ button to start the process.

This will open a new wizard, click ‘Next’ to start.

The wizard will now display the certificates found in the computers certificate store.

If we check the MMC we will see the same data as presented by this tool.

Select the newly imported certificate – you can either check the data stamp for when it was issued or look at the SHA1 thumbprint and match it that way.

Here we have my new certificate file and it’s thumbprint.

After clicking ‘Next’ in the wizard I can see the certificate details, the thumbprint is a match. Click ‘Next’ to continue.

Again we are given a breakdown on the commands executed. In my case a number of warnings were generated however after checking everything I was happy to proceed.

If we check the log we get a good breakdown  of what happened and what the warning is about.

Now for the best part – close the wizard by clicking ‘Finish’ and if you’ve done everything correctly you should now find a nice green tick has replaced the warning symbol.

The Lync services have also started and after restarting my Lync/Skype for Business client application everything was back to normal.

 

 

---

 

It’s always nice when you figure something out by yourself, especially when it’s something you’ve never dealt with before.