Windows 10 Feature Updates – The Challenge of Servicing in the Enterprise

I hope I never have to build another Task Sequence again. Seriously. Wouldn’t that be nice? I know that’s not truly feasible, but can’t a guy dream? It’s not that I dislike Task Sequences, it’s just that I strongly believe that Modern Desktop Management (AutoPilot/Device Reset/Feature Update servicing/etc.) is going to continue to improve and we should continue to attempt to adopt these tools with as much effort as we spend on customizing Task Sequences.

I continue to have conversations with people who tried Feature Update servicing and had a bad experience so they went back to Task Sequences because they are easier to manage. I totally understand the feeling, we even did the same thing in our organization. I also understand that there are plenty of organizations that simply cannot use Feature Update servicing due to many factors including bandwidth, infrastructure and 3rd party software (which are very valid reasons). I’m hoping to challenge you to reconsider your opinion on Feature Update servicing without Task Sequences and give them another go. I’m still learning and Microsoft is still improving, so I can only hope that this becomes easier to adopt as time goes by.

The Enterprise Challenge

My organization, along with many others, are just coming to the Feature Update table after dealing with a slow and agonizing migration off of Windows 7, while others have been managing Windows 10 for several releases now and dealing with the challenges involved with ensuring their clients are on a supported Windows 10 version. I’ve been watching many others deal with the many missteps and challenges that were prevalent in early Windows 10 releases and feel like this is one of those times where being late to the game may actually save have saved us some headaches.

SCCM/MDT Task Sequences continue to be the recommended approach for Feature Updates for several reasons, including flexibility, user visibility, reporting and more. However, my hope is that there are organizations with simple requirements that can benefit from Windows 10 Servicing without Task Sequences. There are several obvious areas that make servicing of Feature Updates more difficult to adopt than Task Sequences but I if you stick with me, I’ll share what we are doing and what’s working.

Before continuing, I’ll mention that I’ve recently made some contacts at Microsoft who are eager to hear feedback on the process and help improve things. I’m hopeful that we will see this space continue to get better!

Some of the obvious challenges I’ve seen with Feature Update servicing include:

  • Limited documentation from Microsoft – There a just a few pages in the docs for Windows 10 that cover options for managing Feature Updates via servicing. The examples are very high level and basic and since most people are using Task Sequences, it has been hard to find (m)any sources to help navigate enterprise management of Windows 10 Features Update servicing.
  • Reporting capability is lacking when compare to Task Sequences
  • Poor reliability track record – previous releases have stumbled and many people who tried Feature Update servicing previously have given up and gone with Task Sequences instead. As of 1803, Custom Actions were added, which many people haven’t looked into.
  • Dynamic Updates – We have been fighting to stay on top of Dynamic Updates for the past year or so and now they have been moved out of WSUS and are pulled directly from the internet – this just doesn’t work for many orgs with bandwidth limitations.
  • Features On Demand and Language Packs – Some people have seen where they don’t get preserved during Feature Updates and are somewhat difficult to manage.

For the above reasons alone, the safe bet is Task Sequences for Feature Updates but I also know that the time I’ve spent on building and managing Task Sequences compared to the work required for Feature Updates is enough motivation to give up on one of those things and move forward.

Real World Implementation

We just spent the past 6 months upgrading from Windows 7 to Windows 10 1803 (1803 because 1809 kept having issues initially and 1803 was a huge improvement over 1709). We knew that we’d be up against the 18-month support window for 1803, so as soon as we could, we began working on a plan for 1809. As we began testing Feature Updates, we ran into 4 main challenges that I built workable solutions around:

  • Non-Blocking Driver & App Compatibility Blocks
  • 802.1x/NAC whitelisting
  • Error Handling
  • Logs/Reporting

The solution and scripts provided in this series are copies of what we deployed to our enterprise. We deployed the update as available for about 2 weeks with a required deployment set for a Thursday night, just before a holiday weekend (maybe could have had a bit better timing). Within a week of the update deadlining over 90% of our 1803 devices have been upgraded to 1809. As we began looking at the remaining devices, we found a few devices that had upgraded, but SCCM/WSUS weren’t updating the status and kept triggering a reboot every 1.5hrs. These were easily identified and remediated by resetting SCCM client policies. We didn’t encounter a single instance of a user’s machine failing outright or even rolling back due to an error.

Note: My organization doesn’t support additional language packs and we haven’t had to deal with major driver issues, so I haven’t spent any time working with them at this point, though I likely will do some testing in the future.

A Bit About Application Compatibility

One of the major areas that causes organizations issues is application compatibility. We have been leveraging Upgrade Readiness and now Desktop Analytics to manage our upgrades since we began our Windows 7 to Windows 10 migration. It gave us the insight into our application compatibility which greatly reduced our application testing efforts and allowed us to rapidly move to 1809 with no application issues.

I don’t mean to imply that we don’t have any application compatibility issues. We do. However, we were able to use these amazing tools to help drive our testing and remediation efforts and most importantly, identify devices which SHOULD NOT be upgraded yet. This is why we have been able to user Feature Update servicing with confidence. In fact, we built up our confidence in Upgrade Readiness while we were building our 7 to 10 Task Sequences and were able to In-Place upgrade over 50% of our environment (the remainder were hardware lifecycle replacements) without having to perform any device-based pre-assessments.

By shining a light on Application Compatibility, we have been able to build up confidence in the toolset and worry less about applications causing issues with deployments.

A Note About Drivers

Since I started this post, I have done some basic driver analysis and found that existing drivers from 1803 were migrated to 1809 without issue. If you know that you will have driver issues for specific models, you should either pre-upgrade the drivers if possible OR pre-stage the new drivers and add a line to setupconfig.ini to reference the staged drivers. You just need to get the drivers onto a known location on the device before the feature update AND point setupconfig.ini to that location.

Marius Wyss, a Microsoft PFE, built a solution for managing his setupconfig.ini using a silent Task Sequence. It has great examples for how to manage Drivers and Language Packs. I’m not saying that you need to use a Task Sequence to manage it, just want to share ONE possible option, which is pretty clever. You can find more info here: https://github.com/MrWyss-MSFT/PrepareIPUTaskSequence

The Enterprise Solution

We identified 4 main hurdles to overcome when deploying Feature Updates so I built a set of scripts and other components to address each of them. Combining them, we now have a relatively robust solution that has streamlined our Feature Update servicing. I built each piece to be as modular as possible, so only use what’s required for your organization. The list below matches up the challenge with a detailed post about that topic and associated scripts.

  1. Driver Compatibility Blocks = SetupConfig.ini
  2. 802.1x/NAC whitelisting = Custom Actions – precommit.cmd
  3. Error Handling = Failure.cmd/SetupDiag
  4. Logs/Reporting = Setupcomplete.cmd/Failure.cmd/SetupDiag/Custom Inventory

I will also included a guide in the repo (soon-ish) for how to manage the included scripts if they aren’t intuitive. At a minimum, to get started with Feature Update servicing, you should deploy Setupconfig.ini to your clients with the Compat option set to Ignore Dismissable Warnings. This setting alone will reduce your Feature Update issues greatly. Everything else just adds nice-to-have features to be more like the Task Sequence experience.

[Setupconfig]
Compat=IgnoreWarning

Looking Back

It has been over a month since I built the solutions above (and wrote the post honestly). In that time, I’ve had time to fine tune some things in our environment and assess the effectiveness/necessity of the toolset. (I have some updates to the scripts that I need to publish to the repo – nothing major, just a few minor bug fixes/tweaks. You can find everything here – if you Watch the repo, you will get alerted when I publish updated scripts.).

  • Currently, we run the copy logs routine for success and failure. I think I may update it to only copy logs on failure since I don’t think I even looked at any logs that were copied.
  • I debated whether running setupdiag, whether or not there was a failure, and I am glad that I did. I’ve actually added a baseline to SCCM now that will run setupdiag on any machine that hasn’t run it (it doesn’t report anything for new builds, just upgrades). This gives me a more complete picture of deployment when we run reports.
  • I’m not entirely happy with how each solution needs to be managed. I need to combine them a bit more to make changes easier to handle. I know, I should just use a Task Sequence right???
  • Drivers – I still need to figure out how I want to deal with these. I plan to add some things for managing them for the next feature update (I hope).

Summary

I hope that you are inspired to give Feature Updates another go. I know I’ve covered a lot of ground in these posts, but there is even more that needs to be said and plenty of room for Microsoft to close the gaps needed to make Feature Updates more accessible to everyone.

15 Comments

  • Reply
    Carl
    October 5, 2019 at 12:30 pm

    Interesting approach. I’m curious how much of InTune/Autopilot you’ve considered as part of your getting modern effort?

    • Reply
      Adam Gross
      October 5, 2019 at 12:37 pm

      We will be moving into Co-Management/Intune/AutoPilot more within the next few months I believe. They have been on our roadmap and we have done some intermittent testing with them over the past few years. We just have some larger enterprise hurdles to deal with before we are able to begin to fully leverage them.

      However, I definitely had Intune in mind as I was building out the solutions. Basically, we should be able to use PowerShell Scripts and Win32 applications to stage the needed content onto the boxes before feature updates and just use the native Intune Feature Updates capabilities.

      I don’t see us moving to full AAD Joined devices any time soon, so we will still be able to leverage the ConfigMgr integrations with things like Desktop Analytics and inventory/reporting to provide a robust experience for us.

      Ultimately – I haven’t tested these solutions with Intune managed devices, but I believe they should be pretty easy to deploy from Intune with a few modifications.

  • Reply
    jandreacola
    October 6, 2019 at 7:38 am

    Great article – one of the biggest reasons to move to Feature Updates is user experience. Feature Updates greatly decrease the amount of time a user cannot use their machine as they can continue to use the machine during the entire Downlevel phase.

  • Reply
    MarcoB
    October 7, 2019 at 8:12 am

    Thanks Adam. We (sadly) still have quite a few 1511’s and 1607’s. Is there a way to use Feature Update Servicing for those builds or has that ship sailed? I’ve tried to do so and it just reports as compliant even though the update was never applied.

  • Reply
    Jason Gibson
    January 31, 2020 at 10:21 pm

    My main question is how do you inject SSU, LCU, etc into the feature update before it runs on each machine? Can you service it like you can service the original WIM? If so, where does it live so I can start tinkering with it?

    • Reply
      Adam Gross
      January 31, 2020 at 10:34 pm

      Unfortunately, you can’t. It will get dynamically updates with the LCU, SSU and Dynamic updates at install time – currently downloaded from the internet, per device. This should change soon with UUP in ConfigMgr, but it’s not here yet.

      The main issue is that WSUS only gets the first release of a particular FU but never gets an updated one. Home PCs use UUP and they will get an updated FU package with updates baked in. That’s what we are hoping to see when that gets rolled out with ConfigMgr.

  • Reply
    Dominick
    July 13, 2020 at 11:02 am

    Hi Adam, great article. We are now in July 2020 – would I be in pretty much the same boat as when this was written when it comes time to using Feature Updates? We are soon going from 1803 -> 1909 later this summer and I am now at ConfigMgr version 2002. I am wondering if I still should plan to use your various modular steps and/or be concerned with figuring out Dynamic Updates when it comes time to deployment. With COVID-19 still rampant, my endpoint mixture is something like 60% at-home laptop workers on VPN and the rest desktops sitting in [mostly vacant for now] offices but powered on. We don’t have CMG or anything else fancy right now; just straight up BranchCache and on-prem distribution points.

  • Reply
    Jason Gibson
    July 13, 2020 at 5:56 pm

    Dominick – I don’t have cmg or branch cache either – only on-prem DPs, and a similar ratio of home/office machines. I’m using Adam’s method for feature update to 1909 with great success. Just be CERTAIN that the XML and ini files, as well as any scripts and supporting files you need are copied to the machine before you put it in the collection that gets the software update.

    • Reply
      Dominick
      July 14, 2020 at 8:07 am

      Much appreciate Jason, glad to know I’m not alone. I will most definitely plan to deploy the INI and anything else something like a week prior to the actual feature update. Did you have to worry about drivers, working around reestablishing VPN connectivity, or anything else weird and did you leave Dynamic Updates enabled or disabled? The part I am struggling to understand is does the Feature update require Dynamic updates or will be it ok to remain disabled? Unfortunately our VPN (user-initiated) also does not have split-tunneling so ALL traffic both internal and external goes through one VPN tunnel. Thank you.

  • Reply
    Jason Gibson
    July 14, 2020 at 8:39 am

    Dominick, you must work for my company’s twin or something lol… Our vpn is also user-initiated (because 2fa) and not split tunnel. I have a “prep” task sequence which copies all the files I need including the latest drivers for that model (if not hyperv), then it adds the machine to the collection that gets the sw update (reboot not forced, apply asap) and removes it from the “prep” collection. Everything else I need done is handled in setup complete.cmd and uses files I copied local beforehand, so it doesn’t matter how many reboots the machines does, it doesn’t need to talk to cfgmgr, so it works great no matter where the machine is physically located, and doesn’t force a reboot on anyone, so user acceptance is MUCH higher, and so is the success rate.

    Yes I left dynamic update enabled so the machine will be fully patched as soon as the upgrade finishes so the user doesn’t have to reboot AGAIN right after they just had to wait 30-45 minutes for the main upgrade and all its reboots. We have about 3k clients, half of which are laptops that right now are all at home. The network guys haven’t come after us, so I guess we’re ok 😁

    • Reply
      Dominick
      July 14, 2020 at 9:19 am

      Haha, yes definitely twin companies… VPN 2fA and 3000 clients spread out like you! Did you use those task sequences and setup complete.cmd files that Adam lists above in his “Enterprise Solution” section or did you find task sequences by other folks from your research (Adam lists a few people’s different sequences)? Also, did you use the INI file or the SCCM Client Settings to set the Dynamic Update setting? Thank you.

      • Reply
        Jason Gibson
        July 15, 2020 at 9:35 am

        I made my own, along with some fairly simple scripts. The bulk of my scripts are just to apply branding, since MS really likes changing it back to the default wallpaper and usericon.

        I would have to take screenshots of my task sequence (singular) to better explain it. if i have time today i’ll see if i can put together a blog for you.

        • Reply
          Dominick
          July 20, 2020 at 8:23 am

          That’s great Jason, I really appreciate that. If you wanted also, you could export your Task Sequence (without dependencies) to .ZIP and I’m sure I could make out what you did. Whenever you get the time feel free to my address (Velardo @ gmail .com)!

        • Reply
          Kevin
          September 11, 2020 at 2:58 pm

          Maybe you can help me out as well. I pre-Cache OS and a few other Apps using TS and need a way for those VPN folks. Not to need TS to check-in after reboot.

          • Jason Gibson
            September 14, 2020 at 1:08 pm

            Sorry guys, i haven’t had time to properly document my process, but basically i have 2 collections. when i’m ready to upgrade a machine, i put it in the first collection.

            The first collection has a required asap deployment of a TS. the TS will delete any existing c:\~featureupdatetemp, then recreate it so it’s empty and doesn’t have any files from a previous upgrade, or in case an upgrade was attempted and failed for some reason, and things have been updated since then (usually drivers). then the TS copies in my drivers, wallpapers, user icons and scripts, and copies a couple other files that windows needs during the upgrade (like setupcomplete.cmd and setupconfig.ini). Then the TS puts the machine into the second collection, waits 2 minutes, triggers CfgMgr hardware inventory and machine policy update (so it will realize it’s in another collection and see the deployments to that collection), then removes it from the first collection.

            The second collection has 2 different things deployed to it. First it has the 1909 feature update deployed as required, asap, and SUPPRESS reboot. This will make it start downloading the content asap and then begin applying it in the background. Eventually the user will get the little green reboot icon (that everyone ignores), and next time they reboot, it’ll upgrade windows. My setupcomplete.cmd calls a powershell script which will inject our company wallpaper and user icons, add a couple reg entries, then remove it from the 2nd collection. if they’re on VPN (which for us is NOT always-on), then obviously it can’t remove it from the 2nd collection, but the rest of it (copying in wallpaper/usericon/etc) will still work fine because all those files were previously copied to c:\~featureupdatetemp. At least this way if they’re physically in one of our offices or they have a home VPN gateway device (which IS always on), the upgrade will finish more efficiently. the add/remove collection script is written so that it won’t throw an error if it can’t talk to the CM server, or if the machine isn’t in the collection you tried to remove it from.

            The second thing deployed to the 2nd collection is a different TS, and it’s set to run every 30 minutes no matter what happened the previous try. This is mostly for the VPN machines, but it doesn’t hurt to run it on everything. The first thing in this TS is a folder that contains the entire TS. It has a condition (check the build of windows via registry) to see if the machine is running 1909, and if it’s NOT, the TS just quits without error and the machine waits another 30 minutes to see if it’s been upgraded during that time. If it upgraded OK and is now on 1909, it actually runs what’s in that folder, which basically does everything my setupcomplete.cmd does – just to be sure – since i’d seen a few that didn’t apply wallpaper or usericon for some reason. the TS also removes the machine from the 2nd collection and triggers inventory again, just to help get it into the right collections in our environment, and then finally it deletes ~featureupdatetemp to free that disk space (drivers are bulky!). Once it’s out of that collection, it will quit trying to run that TS, so i don’t have to worry about keeping track of them afterwards.

            the only wrench in the gears are machines with low disk space, broken/flaky CM clients (which sometimes start jobs but sometimes don’t). Worst of all is Citrix VDA, which is on all of our VDIs. Seems VDA doesn’t play nice with CM and feature updates. quite often, the FU goes just fine, but in the process, VDA does something to break the CM client, and the only fix at that point is to manually remote the machine (which thankfully still works), uninstall VDA, reboot, remote again, reinstall VDA, and reboot one more time. I’ve tried making a monster TS that will uninstall VDA, reboot, do the windows FU, then reinstall VDA, but VDA doesn’t play nice with automation. it’s a miracle i can actually install VDA during the clean build of a new VDI. the best part is that citrix doesn’t seem to care – they want us to pay for premium support to fix their bug. but that’s a story for another day…

    Comment

    This site uses Akismet to reduce spam. Learn how your comment data is processed.

    6,379