Recently one of our sites began having some issues with domain joined devices losing their trust relationship with Active Directory. While some users were able to log in with cached credentials, we had no easy way to get admin credentials to repair the domain trust. I’m going to show how Configmgr Run Scripts and Microsoft Quick Assist helped us get admin access to the devices to perform troubleshooting and remediation. The purpose of this post is to show how we can leverage various tools to solve challenging problems in production. We will likely automate this using Johnny Radeck’s solution at the end, but I wanted to use this post to show highlight ways to use these tools in a real-world scenario.
- Devices randomly lose domain trust and unable to repair without admin intervention
- Active Directory domain joined devices
- Healthy ConfigMgr client installed
- Local Administrator accounts are have been renamed and disabled by Group Policy
- LAPS is used to scramble the local Administrator password but since the account is disabled, it’s not helpful here.
- No cached domain account with Administrator rights
Enable Local Administrator with ConfigMgr Run Scripts
One of the most useful new features of ConfigMgr is Run Scripts. It allows you to deploy PowerShell scripts from the ConfigMgr console immediately to any online device. Because the ConfigMgr client is running under the System account of the device being targeted, these scripts are also run in the System context which has local admin rights to the device.
For this example I’ve written a simple PowerShell script that can enable or disable a local account (using the account SID as a parameter). In production, we would only need to enable the account then retrieve the password from LAPS. While the script has the code commented out to set the password, I would advise against using it since the password will get logged in the local client logs. Additionally, you could use NET commands (included in the script as an example) in Run Scripts but I wanted to have a script that was flexible and could output results to the console and could catch errors.
If you aren’t familiar with Run Scripts, check out the the docs to learn everything about Run Scripts then come back here for the rest of the story. Follow the steps to add this script to your console. Create and run scripts - Configuration Manager | Microsoft Docs
I’ve submitted the script to the Community Hub but it is still pending at the time of this writing. Check out more about the Community Hub here: Community hub and GitHub - Configuration Manager | Microsoft Docs. You can check the status of my PR here https://github.com/Microsoft/configmgr-hub/pull/64.
Once the script has been added to your console, select the impacted device(s) and run the script. You can monitor the status of the script using Scripts.log in the C:\Windows\CCM\Logs folder.
Getting Connected with Microsoft Quick Assist
Once the local administrator account has been enabled, you can remotely connect to the device using Quick Assist (or before - Quick Assist is not dependent on Admin rights to launch). If you haven’t used this tool yet, be sure to give it a test drive. I have been using it for the past year and it has been THE MOST RELIABE remote management tool I’ve ever used! Hands down! I have even used it to (begrudgingly) work on my parents & in-laws computers with ease and it even works from the Windows OOBE screen (Shift+F10 then type QuickAssist) to allow you to troubleshoot device provisioning issues when using Windows Autopilot. I was going to write a nice post on Quick Assist, but MVP Oliver Kieselbach has an excellent write-up on it. Quick Assist the built-in Remote Control in Windows 10 – Modern IT – Cloud – Workplace (oliverkieselbach.com) plus Microsoft does as well Solve PC problems over a remote connection (microsoft.com).
Simply have the user with the problem computer launch QuickAssist.exe from a command prompt or Windows search box so you can connect to them. Follow the instructions on the screen and you can connect within minutes. Oh, and did I mention Quick Assist even handles reconnecting after reboots (it at least tries to reconnect).
If you’re not already using Microsoft LAPS, you should really take a look. Gary Blok has a great write-up on the Recast Software blog Overview of Microsoft LAPS (Local Administrator Password Solution) (recastsoftware.com)
Since we are using LAPS, we can simply used the LAPS console integrated in the ConfigMgr console to retrieve the password for the broken device. If we didn’t have LAPS enabled or if the expected LAPS password wasn’t working, we could have edited our script from above to set the password manually (last resort).
Repair the Domain Trust
For years, any time a device lost it’s domain trust, I would disjoin the device from the domain then re-join it. Then I discovered that you could skip the disjoin and simply change from the short Netbios domain to the FQDN or the other way around and it would do the same thing. Then one day I discovered
Test-ComputerSecureChannel and it changed my life!
Once logged onto the broken device, we were able to launch PowerShell using the local Administrator account then run the following command.
Get-Credential will pop up a dialog asking for your domain credentials. You could also use Get-Credential and store the credentials into a variable and pass them in like this.
As I mentioned at the top, Johnny Radeck wrote this great blog post and script to that uses a ConfigMgr Configuration Item to automatically detect and repair the broken trust. In Johnny’s solution, he encrypts domain credentials and uses those to reset the password using an special domain account that only has rights to reset passwords. My plan is to use this as a foundation for detecting, reporting and remediating this issue automatically moving forward.
I would like to say thanks to Nickolaj, Maurice and Johnny for helping me out here. Johnny wrote the post above 2 years ago and it got lost during the MSEndpointMgr site migration. I reached out to them and they were able to recover the post (I heard Johnny had to re-write it!). You guys are top notch! Thank you for all you do for the community.
So I know this post is mostly just links to other posts, but sometimes that’s just what a day in our IT lives is like - putting together all of the pieces of information we can to attack a problem - so I felt like it was worth sharing.
To wrap up the story, the Server team has been working to replace the domain controller at the site where the issue is occurring. I haven’t heard anything from local support in a while, so either it’s working or they are all offline and can’t send me a message 🙂 Now that we have a handle on the issue, putting some automation in to detect and report on future issues is going to be useful and having the option to auto-remediate will be the icing on the cake. Well, that’s all for now. Wait for the next post where I tell you how bad I am at IT. It’s gonna be a rollercoaster!!