Debugging “Failed to compile rule” client errors after SCCM Application Catalog Role Removal

In my post Removing the Application Catalog Role after Upgrading to ConfigMgr 1806 I mentioned the error Failed to compile rule "{Rule_WRC10000}": 0x8000ffff and that we were unsure of it’s impact on the client. The product group assured us that the error was nothing to worry about and they were right – I have seen no ill effects on my clients, even though I continually see this error whenever the clients process machine policies. However, it has continued to bother me so I decided to get to the bottom of the issue. Here’s what I found.

WRC10000

Most of my research up to this point was done on my production environment AFTER uninstalling the Application Catalog role. As such, I found it difficult to track the error to it’s source. Today I tested in my lab where I had never install the Application Catalog role and came up empty as well, sort of. I discovered that the Policy WRC10000 doesn’t even exist in the Policy table in the SCCM Database until AFTER you install the Application Catalog role. I then spun up a second lab instance where I installed the Application Catalog role and immediately saw the WRC10000 policy/rule being processed on the client in PolicyAgent.log.

Compiling policy 'WRC10000' version '1.00' hash 'SHA256:0C11C0B2C324A0D760B768D33475CE5F2B1493CF7AE43390035E2D4A33BE47BB' from 'SMS:CB1' (2019-07-10 02:40:06.663)

Raising event:
instance of CCM_PolicyAgent_AssignmentEnabled
{
	AssignmentConditionID = "{45841A79-1208-48D9-8BA1-1213E30CF049}";
	AssignmentID = "{83607b32-abea-4a20-af9f-6e24d6dbd991}";
	ClientID = "GUID:74D9FF90-9FB5-4B7D-BF30-E7160580642F";
	DateTime = "20190710024112.148000+000";
	PolicyID = "WRC10000";
	PolicyNamespace = "\\\\.\\ROOT\\ccm\\Policy\\Machine\\RequestedConfig";
	PolicySource = "SMS:CB1";
	PolicyVersion = "1.00";
	ProcessID = 2792;
	ThreadID = 3340;
};

Initializing download of policy 'CCM_Policy_Policy5.PolicyID="WRC10000",PolicySource="SMS:CB1",PolicyVersion="1.00"' from 'http://CMCB1-CM1.asd.lab/SMS_MP/.sms_pol?WRC10000.SHA256:0C11C0B2C324A0D760B768D33475CE5F2B1493CF7AE43390035E2D4A33BE47BB'

Download of policy started - DTS job ID: '{C7664B63-4C60-4673-ACE2-2D7ECBEC3CE9}'

Raising event:
instance of CCM_PolicyAgent_PolicyDownloadStarted
{
	ClientID = "GUID:74D9FF90-9FB5-4B7D-BF30-E7160580642F";
	DateTime = "20190710024112.287000+000";
	DownloadMethod = "BITS";
	DownloadSource = "http://CMCB1-CM1.asd.lab/SMS_MP/.sms_pol?WRC10000.SHA256:0C11C0B2C324A0D760B768D33475CE5F2B1493CF7AE43390035E2D4A33BE47BB";
	PolicyNamespace = "\\\\.\\ROOT\\ccm\\Policy\\Machine\\RequestedConfig";
	PolicyPath = "CCM_Policy_Policy5.PolicyID=\"WRC10000\",PolicySource=\"SMS:CB1\",PolicyVersion=\"1.00\"";
	ProcessID = 2792;
	ThreadID = 3340;
};

Download of policy CCM_Policy_Policy5.PolicyID="WRC10000",PolicySource="SMS:CB1",PolicyVersion="1.00" completed (DTS Job ID: {C7664B63-4C60-4673-ACE2-2D7ECBEC3CE9})

Raising event:
instance of CCM_PolicyAgent_PolicyDownloadSucceeded
{
	ClientID = "GUID:74D9FF90-9FB5-4B7D-BF30-E7160580642F";
	DateTime = "20190710024116.685000+000";
	DownloadMethod = "BITS";
	DownloadSource = "http://CMCB1-CM1.asd.lab/SMS_MP/.sms_pol?WRC10000.SHA256:0C11C0B2C324A0D760B768D33475CE5F2B1493CF7AE43390035E2D4A33BE47BB";
	PolicyNamespace = "\\\\.\\ROOT\\ccm\\Policy\\Machine\\RequestedConfig";
	PolicyPath = "CCM_Policy_Policy5.PolicyID=\"WRC10000\",PolicySource=\"SMS:CB1\",PolicyVersion=\"1.00\"";
	ProcessID = 2792;
	ThreadID = 5740;
};

Then I used WMIExplorer (written by Vinay Pamnani) to quickly browse into the locations listed in the log ROOT\ccm\Policy\Machine\RequestedConfig. You could also use the new Configuration Manager Support Center tool to load and view policies. After a bit of digging, I found the location under CCM_Policy_Policy5 and CCM_Policy_Rules.

Namespace:ROOT\ccm\Policy\Machine\RequestedConfig
SELECT * FROM CCM_Policy_Policy5 WHERE PolicyID='WRC10000'
SELECT * FROM CCM_Policy_Rules WHERE PolicyID='WRC10000'

Up until now, I had been unable to link the WRC10000 policy to the Application Catalog role. The WebServiceSettingsConfig category is tied to the Application Catalog role and includes a few additional WMI classes that get created when the requested config becomes the actual config.

Namespace: ROOT\ccm\Policy\Machine\ActualConfig
SELECT * FROM CCM_WebService_Settings

I put together some quick SQL queries to find the newly created policy values in the SCCM Database. *Note – some screenshots may show different versions for the policy due to multiple iterations of testing.

DECLARE @PolicyID nvarchar(10)
SET @PolicyID = 'WRC10000'

SELECT * FROM Policy WHERE PolicyID = @PolicyID

SELECT * FROM PolicyAssignment WHERE PolicyID = @PolicyID

SELECT * FROM ResPolicyMap WHERE PADBID = (SELECT PADBID FROM PolicyAssignment WHERE PolicyID = @PolicyID)
  1. Each time the Application Catalog role is uninstalled/reinstalled, the Version is incremented. In my testing, an odd Version indicates that the role is installed and an even number indicates that the role has been uninstalled.
  2. The MachineID column from the ResPolicyMap table seems to indicate that this policy is applied to all devices without requiring deployment to a collection.
  3. The Tombstone fields are for policies that have been deleted or decomissioned
    1. IsTombStoned – 0 = Active, 1 = Inactive/Deleted
    2. TombstoneBody/ TombstoneHash/ TombstoneSignature – Backup copies of the Body/ BodyHash/ BodySignature fields that clients will reference if the policy has been tombstoned to ensure they delete the correct version of the policy.

The “Bug”

I don’t know if this is a bug or if it’s by design, but I’m calling it a bug. Whenever you uninstall the Application Catalog role, IsTombStoned flag never gets set to 1 and when clients download and evaluate machine policies, PolicyAgent.log will show these events:

Failed to compile rule "{Rule_WRC10000}": 0x8000ffff
Deleting policy rules of 'WRC10000' version '2.00'
Raising event:
instance of CCM_PolicyAgent_PolicyCompileFailed
{
	ClientID = "GUID:74D9FF90-9FB5-4B7D-BF30-E7160580642F";
	DateTime = "20190710033018.242000+000";
	PolicyNamespace = "\\\\.\\ROOT\\ccm\\policy\\Machine\\RequestedConfig";
	PolicyPath = "CCM_Policy_Policy5.PolicyID=\"WRC10000\",PolicyVersion=\"2.00\",PolicySource=\"SMS:CB1\"";
	ProcessID = 2792;
	ThreadID = 5740;
	Win32ErrorCode = 2147549183;
};
Policy state for CCM_Policy_Policy5.PolicyID="WRC10000",PolicySource="SMS:CB1",PolicyVersion="2.00" was successfully reset. Policy download will be retried at next evaluation cycle.

Additionally, a partial record will be created in WMI for the newly created version of the policy, however the old version(s) will not be purged, as they should whenever the version is incremented. Since the client sees the new policy as corrupt instead of tombstoned, it never gets notified that the old versions need to be deleted as well. As you can see below, version 5.00 is still listed as Active, while 6 hasn’t been processed and is missing the PolicyState field.

Additionally, the CCM_WebService_Settings Class stays behind with the original settings.

The “Fix”

STOP! Do not perform these steps in your production environment. You should NEVER directly edit the SCCM Database unless directed to do so by the SCCM support team (even then, I’d question them about alternatives before doing so!). You’ve been warned.

07/10/2019 – UPDATE – I have updated the section below to reflect a less destructive approach using UPDATE instead of DELETE as previously posted.

After digging through countless stored procedures and functions, I finally checked the PolicyAssignment table for triggers and was excited to find that it has just what we need, an INSERT/UPDATE/DELETE trigger!

If you aren’t familiar with table triggers, join the club! 🙂
Actually, a table trigger is function that is attached to a database table that is launched (triggered) whenever a specific type of action (INSERT/UPDATE/DELETE – etc) is taken on that table. In this case, the PolicyAssignment table has a trigger that captures the ID of the affected record(s) then updates related records in other tables as needed.

The PolicyAssignment table also has triggers for DELETE as well. I tested deleting the record from this table then letting the triggers clean up everything else, however I found that the clients don’t get notified of a change since the policy is completely gone, so the local policies would get orphaned. *Fun note, if you delete from PolicyAssignment, then reinstall the Application Catalog role, the policy version resets to 1.00. It also loses the tombstoned versions of the policy which prevent the client from removing old entries as well. Just don’t do this!! If you do, the only workaround is to reset client policies.

So, to “Fix” the problem in a way that allows the client to clean off old references (be sure that you’ve already uninstalled the Application Catalog role from your site), run this:

UPDATE PolicyAssignment SET IsTombstoned = 1 WHERE PolicyID = 'WRC10000'

Once updated, you should be able to re-run the previous queries and see that the policy is tombstoned.

Now download and evaluate client policies and you should see all WRC10000 policies get removed.

Deleting policy assignment '{cedd84fc-ad71-40af-abc9-9426424ecb7e}'
Revoking policy 'WRC10000' version '1.00'
Raising event:
instance of CCM_PolicyAgent_PolicyRuleRevoked
{
	ClientID = "GUID:74D9FF90-9FB5-4B7D-BF30-E7160580642F";
	DateTime = "20190710065211.025000+000";
	PolicyID = "WRC10000";
	PolicyNamespace = "\\\\.\\ROOT\\ccm\\Policy\\Machine\\RequestedConfig";
	PolicySource = "SMS:CB1";
	PolicyVersion = "1.00";
	ProcessID = 12412;
	RuleCondition = "";
	RuleID = "{Rule_WRC10000}";
	ThreadID = 13012;
};
Deleting policy 'WRC10000' version '1.00'

After you tombstone the policy, if your clients continue to see the error or have issues with seeing software in Software Center, you can reset client policies to purge any orphaned policies as well.

Summary

I haven’t worked with support on this issue and the fix above may be completely incorrect, so please call someone before attempting this. I really wrote this for myself so that I don’t have to remember what all I did. I’d love to hear from anyone else who’s “fixed” this error some other way. Also, this may be completely unnecessary, since I’ve never seen any adverse affects for clients, however I have seen posts from others who have. Your mileage may vary. Proceed with caution!

You Might Also Like

6 Comments

  • Reply
    Matt K
    July 10, 2019 at 5:00 am

    Hi Adam, thank you so much for this post, it address my issue almost completely! The issue I have post the “disaster” of uninstalling the Application Catalog website is that “User” deployed Applications still don’t display in the Software Center!
    Your solution has certainly cleared up the WRC10000 error I was receiving (0x8000FFFF) but the User deployed applications within SCCM never return.
    Any other thoughts on the matter?

    Cheers, Matt

    • Reply
      Glen C
      July 10, 2019 at 8:11 am

      I saw the same thing when I first removed the App Catalog role. Check the Client Agent settings, as per the documentation at https://docs.microsoft.com/en-us/sccm/apps/plan-design/plan-for-and-configure-application-management#bkmk_remove-appcat

      “Review the default and any custom client settings. In the Computer Agent group, make sure the Default Application Catalog website point is (none).”

      Once that setting had been corrected, User-targeted apps show up correctly.

      • Reply
        Matt K
        July 10, 2019 at 4:26 pm

        Thanks Glen,
        I have followed the instructions (a number of times) but I see the following error in the [email protected]_2.log

        GetApplicationsAsync: The HTTP request is unauthorized with client authentication scheme ‘Negotiate’. The authentication header received from the server was ‘Negotiate oWowaKADCgEBomEEX2BdBgkqhkiG9xIBAgIDAH5OMEygAwIBBaEDAgEepBEYDzIwMTkwNzEwMjIxNjQyWqUFAgMGgzemAwIBKakNGwtVUkJJUy5MT0NBTKoSMBCgAwIBAaEJMAcbBWNtMDIk’.. Unable to fetch user categories, unknown communication problem. (Microsoft.SoftwareCenter.Client.ViewModels.SoftwareListViewModel+d__163 at MoveNext)

        I can “force” or change the error is I make changes to the IIS Site “CMUserService_WindowsAuth” authentication type, but it still looks like it’s having some trouble with IIS Authentication providers.

        It’s driving me mad! I think I’ll just have to concede defeat and contact Microsoft as Adam suggests

        Thanks for your response.

        Matt

        • Reply
          Matt K
          July 11, 2019 at 1:47 pm

          Just an update to my issue that might help other affected by this issue, and the subsequent issue of “User based” applications not appearing in ConfigMgr Software Center.

          My Issue was actually related to an SPN entry associated to an MBAM account (AD account) that was created by a colleague. A fellow worker was apparently “testing” MBAM and ConfigMgr integration on the quiet. The SPN entry created for an IIS AppPool account looked to cause the issue.

          Good luck folks!

          Matt K

    • Reply
      Adam Gross
      July 10, 2019 at 9:26 am

      Other’s who have seen the issue where user deployed apps don’t return, generally need to wait up to 60 minutes for it to resolve itself. If that doesn’t work, I would contact Premier Support.

  • Reply
    Umair Khan
    July 12, 2019 at 9:18 am

    Umair Khan [MSFT]: Thanks for documenting this. Yeah it has been around for quite a while and even though the error is benign but kind of frustrating this to have and clients trying to download and fail.
    As part of 1906 upgrade, this policy will be removed from DB if we don’t have AppCat role.

  • Comment

    This site uses Akismet to reduce spam. Learn how your comment data is processed.

    507