Debugging “Failed to compile rule” client errors after SCCM Application Catalog Role Removal

In my post Removing the Application Catalog Role after Upgrading to ConfigMgr 1806 I mentioned the error Failed to compile rule "{Rule_WRC10000}": 0x8000ffff and that we were unsure of it’s impact on the client. The product group assured us that the error was nothing to worry about and they were right - I have seen no ill effects on my clients, even though I continually see this error whenever the clients process machine policies. However, it has continued to bother me so I decided to get to the bottom of the issue. Here’s what I found.

WRC10000

Most of my research up to this point was done on my production environment AFTER uninstalling the Application Catalog role. As such, I found it difficult to track the error to it’s source. Today I tested in my lab where I had never install the Application Catalog role and came up empty as well, sort of. I discovered that the Policy WRC10000 doesn’t even exist in the Policy table in the SCCM Database until AFTER you install the Application Catalog role. I then spun up a second lab instance where I installed the Application Catalog role and immediately saw the WRC10000 policy/rule being processed on the client in PolicyAgent.log.

A Square Dozen Image

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
Compiling policy 'WRC10000' version '1.00' hash 'SHA256:0C11C0B2C324A0D760B768D33475CE5F2B1493CF7AE43390035E2D4A33BE47BB' from 'SMS:CB1' (2019-07-10 02:40:06.663)

Raising event:
instance of CCM_PolicyAgent_AssignmentEnabled
{
	AssignmentConditionID = "{45841A79-1208-48D9-8BA1-1213E30CF049}";
	AssignmentID = "{83607b32-abea-4a20-af9f-6e24d6dbd991}";
	ClientID = "GUID:74D9FF90-9FB5-4B7D-BF30-E7160580642F";
	DateTime = "20190710024112.148000+000";
	PolicyID = "WRC10000";
	PolicyNamespace = "\\\\.\\ROOT\\ccm\\Policy\\Machine\\RequestedConfig";
	PolicySource = "SMS:CB1";
	PolicyVersion = "1.00";
	ProcessID = 2792;
	ThreadID = 3340;
};

Initializing download of policy 'CCM_Policy_Policy5.PolicyID="WRC10000",PolicySource="SMS:CB1",PolicyVersion="1.00"' from 'http://CMCB1-CM1.asd.lab/SMS_MP/.sms_pol?WRC10000.SHA256:0C11C0B2C324A0D760B768D33475CE5F2B1493CF7AE43390035E2D4A33BE47BB'

Download of policy started - DTS job ID: '{C7664B63-4C60-4673-ACE2-2D7ECBEC3CE9}'

Raising event:
instance of CCM_PolicyAgent_PolicyDownloadStarted
{
	ClientID = "GUID:74D9FF90-9FB5-4B7D-BF30-E7160580642F";
	DateTime = "20190710024112.287000+000";
	DownloadMethod = "BITS";
	DownloadSource = "http://CMCB1-CM1.asd.lab/SMS_MP/.sms_pol?WRC10000.SHA256:0C11C0B2C324A0D760B768D33475CE5F2B1493CF7AE43390035E2D4A33BE47BB";
	PolicyNamespace = "\\\\.\\ROOT\\ccm\\Policy\\Machine\\RequestedConfig";
	PolicyPath = "CCM_Policy_Policy5.PolicyID=\"WRC10000\",PolicySource=\"SMS:CB1\",PolicyVersion=\"1.00\"";
	ProcessID = 2792;
	ThreadID = 3340;
};

Download of policy CCM_Policy_Policy5.PolicyID="WRC10000",PolicySource="SMS:CB1",PolicyVersion="1.00" completed (DTS Job ID: {C7664B63-4C60-4673-ACE2-2D7ECBEC3CE9})

Raising event:
instance of CCM_PolicyAgent_PolicyDownloadSucceeded
{
	ClientID = "GUID:74D9FF90-9FB5-4B7D-BF30-E7160580642F";
	DateTime = "20190710024116.685000+000";
	DownloadMethod = "BITS";
	DownloadSource = "http://CMCB1-CM1.asd.lab/SMS_MP/.sms_pol?WRC10000.SHA256:0C11C0B2C324A0D760B768D33475CE5F2B1493CF7AE43390035E2D4A33BE47BB";
	PolicyNamespace = "\\\\.\\ROOT\\ccm\\Policy\\Machine\\RequestedConfig";
	PolicyPath = "CCM_Policy_Policy5.PolicyID=\"WRC10000\",PolicySource=\"SMS:CB1\",PolicyVersion=\"1.00\"";
	ProcessID = 2792;
	ThreadID = 5740;
};

Then I used WMIExplorer (written by Vinay Pamnani) to quickly browse into the locations listed in the log ROOT\ccm\Policy\Machine\RequestedConfig. You could also use the new Configuration Manager Support Center tool to load and view policies. After a bit of digging, I found the location under CCM_Policy_Policy5 and CCM_Policy_Rules.

1
2
3
Namespace:ROOT\ccm\Policy\Machine\RequestedConfig
SELECT * FROM CCM_Policy_Policy5 WHERE PolicyID='WRC10000'
SELECT * FROM CCM_Policy_Rules WHERE PolicyID='WRC10000'

A Square Dozen Image

Up until now, I had been unable to link the WRC10000 policy to the Application Catalog role. The WebServiceSettingsConfig ****category is tied to the Application Catalog role and includes a few additional WMI classes that get created when the requested config becomes the actual config.

1
2
Namespace: ROOT\ccm\Policy\Machine\ActualConfig
SELECT * FROM CCM_WebService_Settings

A Square Dozen Image

I put together some quick SQL queries to find the newly created policy values in the SCCM Database. *Note - some screenshots may show different versions for the policy due to multiple iterations of testing.

1
2
3
4
5
6
7
8
DECLARE @PolicyID nvarchar(10)
SET @PolicyID = 'WRC10000'

SELECT * FROM Policy WHERE PolicyID = @PolicyID

SELECT * FROM PolicyAssignment WHERE PolicyID = @PolicyID

SELECT * FROM ResPolicyMap WHERE PADBID = (SELECT PADBID FROM PolicyAssignment WHERE PolicyID = @PolicyID)

A Square Dozen Image

  1. Each time the Application Catalog role is uninstalled/reinstalled, the Version ****is incremented. In my testing, an odd Version indicates that the role is installed and an even number indicates that the role has been uninstalled.
  2. The MachineID column from the ResPolicyMap table seems to indicate that this policy is applied to all devices without requiring deployment to a collection.
  3. The Tombstone ****fields are for policies that have been deleted or decomissioned
    1. IsTombStoned - 0 = Active, 1 = Inactive/Deleted
    2. TombstoneBody/ TombstoneHash/ TombstoneSignature - Backup copies of the Body/ BodyHash/ BodySignature fields that clients will reference if the policy has been tombstoned to ensure they delete the correct version of the policy.

The ‘Bug’

I don’t know if this is a bug or if it’s by design, but I’m calling it a bug. Whenever you uninstall the Application Catalog role, IsTombStoned ****flag never gets set to 1 and when clients download and evaluate machine policies, PolicyAgent.log will show these events:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
Failed to compile rule "{Rule_WRC10000}": 0x8000ffff
Deleting policy rules of 'WRC10000' version '2.00'
Raising event:
instance of CCM_PolicyAgent_PolicyCompileFailed
{
	ClientID = "GUID:74D9FF90-9FB5-4B7D-BF30-E7160580642F";
	DateTime = "20190710033018.242000+000";
	PolicyNamespace = "\\\\.\\ROOT\\ccm\\policy\\Machine\\RequestedConfig";
	PolicyPath = "CCM_Policy_Policy5.PolicyID=\"WRC10000\",PolicyVersion=\"2.00\",PolicySource=\"SMS:CB1\"";
	ProcessID = 2792;
	ThreadID = 5740;
	Win32ErrorCode = 2147549183;
};
Policy state for CCM_Policy_Policy5.PolicyID="WRC10000",PolicySource="SMS:CB1",PolicyVersion="2.00" was successfully reset. Policy download will be retried at next evaluation cycle.

A Square Dozen Image

Additionally, a partial record will be created in WMI for the newly created version of the policy, however the old version(s) will not be purged, as they should whenever the version is incremented. Since the client sees the new policy as corrupt instead of tombstoned, it never gets notified that the old versions need to be deleted as well. As you can see below, version 5.00 is still listed as Active, while 6 hasn’t been processed and is missing the PolicyState field.

A Square Dozen Image

A Square Dozen Image

A Square Dozen Image

Additionally, the CCM_WebService_Settings Class stays behind with the original settings.

The ‘Fix’


STOP! Do not perform these steps in your production environment. You should NEVER directly edit the SCCM Database unless directed to do so by the SCCM support team (even then, I’d question them about alternatives before doing so!). You’ve been warned.

07/10/2019 - UPDATE - I have updated the section below to reflect a less destructive approach using UPDATE instead of DELETE as previously posted.


After digging through countless stored procedures and functions, I finally checked the PolicyAssignment table for triggers and was excited to find that it has just what we need, an INSERT/UPDATE/DELETE trigger!

A Square Dozen Image

If you aren’t familiar with table triggers, join the club! 🙂
Actually, a table trigger is function that is attached to a database table that is launched (triggered) whenever a specific type of action (INSERT/UPDATE/DELETE - etc) is taken on that table. In this case, the PolicyAssignment table has a trigger that captures the ID of the affected record(s) then updates related records in other tables as needed.

The PolicyAssignment table also has triggers for DELETE as well. I tested deleting the record from this table then letting the triggers clean up everything else, however I found that the clients don’t get notified of a change since the policy is completely gone, so the local policies would get orphaned. *Fun note, if you delete from PolicyAssignment, then reinstall the Application Catalog role, the policy version resets to 1.00. It also loses the tombstoned versions of the policy which prevent the client from removing old entries as well. Just don’t do this!! If you do, the only workaround is to reset client policies.

So, to ‘Fix’ the problem in a way that allows the client to clean off old references (be sure that you’ve already uninstalled the Application Catalog role from your site), run this:

1
UPDATE PolicyAssignment SET IsTombstoned = 1 WHERE PolicyID = 'WRC10000'

Once updated, you should be able to re-run the previous queries and see that the policy is tombstoned.

A Square Dozen Image

Now download and evaluate client policies and you should see all WRC10000 policies get removed.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
Deleting policy assignment '{cedd84fc-ad71-40af-abc9-9426424ecb7e}'
Revoking policy 'WRC10000' version '1.00'
Raising event:
instance of CCM_PolicyAgent_PolicyRuleRevoked
{
	ClientID = "GUID:74D9FF90-9FB5-4B7D-BF30-E7160580642F";
	DateTime = "20190710065211.025000+000";
	PolicyID = "WRC10000";
	PolicyNamespace = "\\\\.\\ROOT\\ccm\\Policy\\Machine\\RequestedConfig";
	PolicySource = "SMS:CB1";
	PolicyVersion = "1.00";
	ProcessID = 12412;
	RuleCondition = "";
	RuleID = "{Rule_WRC10000}";
	ThreadID = 13012;
};
Deleting policy 'WRC10000' version '1.00'

A Square Dozen Image

After you tombstone the policy, if your clients continue to see the error or have issues with seeing software in Software Center, you can reset client policies to purge any orphaned policies as well.

Summary

I haven’t worked with support on this issue and the fix above may be completely incorrect, so please call someone before attempting this. I really wrote this for myself so that I don’t have to remember what all I did. I’d love to hear from anyone else who’s ‘fixed’ this error some other way. Also, this may be completely unnecessary, since I’ve never seen any adverse affects for clients, however I have seen posts from others who have. Your mileage may vary. Proceed with caution!