During an In-Place upgrade on a Lync 2013 Edge Server the upgrade failed with the following Error

The MSI InstallCsDatabase.exe returned error code 1, log file location C:\users\Administrator\Appdata\Temp\2\

This is helpful lol better check the folder for some clues then. Here there was a number of log files. I checked them and an Install-csdatabase-Attach-CentralManagementStore log file had something interesting.

I ran through this and found that the XDS database was MISSING! WTF where has that gone walkies to 😦

Information:Attaching database xds from Data Path c:\Program Files\Microsoft SQL Server\MSSQL11.RTCLOCAL\MSSQL\DATA\, Log Path c:\Program Files\Microsoft SQL Server\MSSQL11.RTCLOCAL\MSSQL\DATA\.
Error:The operation failed because of missing file ‘c:\Program Files\Microsoft SQL Server\MSSQL11.RTCLOCAL\MSSQL\DATA\xds.mdf’
Following Exception ocurred while creating or updating the database
Microsoft.Rtc.Common.Data.DbSetupMissingFileException: The operation failed because of missing file ‘c:\Program Files\Microsoft SQL Server\MSSQL11.RTCLOCAL\MSSQL\DATA\xds.mdf’
   at Microsoft.Rtc.Common.Data.DatabaseManager.Attach(String dbFileBase, String dbPath, String logPath)
   at Microsoft.Rtc.Common.Data.DbSetupBase.AttachDatabase()
   at Microsoft.Rtc.Common.Data.InstallCsDatabase.Execute()
   at Microsoft.Rtc.Common.Data.Program.Main(String[] args): The operation failed because of missing file ‘c:\Program Files\Microsoft SQL Server\MSSQL11.RTCLOCAL\MSSQL\DATA\xds.mdf’

So i checked the location on the edge servers and it was indeed missing 😦

Well that cant be good! as the XDS is read only and on each Lync 2013 server i wondered (a very Long shot) if i could copy this from the other Edge Server and rerun the in-place setup upgrade.

I stopped the Lync services on the second Edge and copied the database to the missing location and reran the Skype for Business In-Place setup and it finished successfully. I thought great BUT no there were errors starting the Skype for Business services from the event log i saw lots of errors relating to SQL permissions with local back end databases 😦 it was a long shot after all but live and learn. So plan to plan B remove SfB and do a fresh install.

Copying the XDS from another server was a long shot to start and indeed it doesn’t work as its most likely not attached in SQL and got the correct permissions etc in SQL Express. Scrap that!

Next i wanted to do a removal on the Edge server and reinstall Skype for Business and SQL Express and start a fresh installation.

I first used the Skype for Business Management Shell and ran

Disable-CsComputer –scorch

Next visited programs and features and removed all software

• Skype for Business Edge Role
• Skype for Business Core Components
• SQL 2014
• All SQL related programs left
• Windows fabric
• All other software on the server as well.

I then exported a fresh topology file from SfB management Shell on an FE

Export-csconfiguration –filename  c:\temp\edgetopo.zip

copied this to the edge server in preparation for the new installation.

I ran a new installation of SfB and this time got a different error! 😦 again i checked the log file location and found this time there was a problem with the rtc local database !

Attaching database rtc from Data Path \\lyncedge.domain.local\C$\CsData\RtcDatabaseStore\rtclocal\DbPath, Log Path \\lyncedge.domain.local\C$\CsData\RtcDatabaseStore\rtclocal\LogPath.
Attaching Database rtc. Data Path \\lyncedge.domain.local\C$\CsData\RtcDatabaseStore\rtclocal\DbPath\rtc.mdf, Log Path \\lyncedge.domain.local\C$\CsData\RtcDatabaseStore\rtclocal\LogPath\rtc.ldf.
Failed to attach database rtc from Data Path \\lyncedge.domain.local\C$\CsData\RtcDatabaseStore\rtclocal\DbPath\rtc.mdf, Log Path \\lyncedge.domain.local\C$\CsData\RtcDatabaseStore\rtclocal\LogPath\rtc.ldf.
Microsoft.SqlServer.Management.Smo.FailedOperationException: Attach database failed for Server ‘lyncedge.domain.local\rtclocal’. —> Microsoft.SqlServer.Management.Common.ExecutionFailureException: An exception occurred while executing a Transact-SQL statement or batch. —> System.Data.SqlClient.SqlException: Directory lookup for the file "\\lyncedge.domain.local\C$\CsData\RtcDatabaseStore\rtclocal\DbPath\rtc.mdf" failed with the operating system error 5(Access is denied.).
   at Microsoft.SqlServer.Management.Common.ConnectionManager.ExecuteTSql(ExecuteTSqlAction action, Object execObject, DataSet fillDataSet, Boolean catchException)

So this time even with a fresh install i have problems 😦

What i remembered and seen before is after removing SQL and SfB from the server i didn’t delete any of the left over crap folders from program files and programs file x86 so i thought perhaps this was the case.

Again i removed all the software so had a clean server and this time deleted any reference to Lync Server 2013, SfB and SQL Server and for luck a restart of the server.

Again i tried a fresh install and again the same error !

I checked the logs and it was the same error. I then noticed that i hadn’t removed the csData folder it was referencing in the log file on the c drive. I checked the dates on the csData folder and the rtc database and it was august 2014! hmmmm i wonder if its trying to attach an old Lync 2013 database as it already exists.

So again i removed SfB, SQL and all software from server, deleted an references in programs to SfB and SQL Server and also deleted the csData folder on the c drive.

I did a fresh installation and it installed successfully worked !!! it had created a new csData folder with new rtc db with today’s date.

I started the SfB services and all started, checked the event log and all was good.

Summary

So some how the loss of the XDS database on the edge server during the in-place upgrade seemed to cause an issue which broke the in-place upgrade and also removing SfB and SQL Express doesn’t mean you have removed everything as the old databases, folder etc are left behind.

So this in-place upgrade was painful and needed a new SfB install. I know to expect issues but on the other hand the other three edge servers in-place upgraded were fine so its was just this server that had the issues.