Owen Reynolds Personal blog

I most recently presented this topic at E2EVC (org link) in Montreal on July 26, 2024

Here is a link to my slide deck

I recently returned as a consultant for a client that I was working for in 2020, and it’s inspired this piece. The reason being, I realized I’d left some items in place that should have been removed and had confused the local staff.

This is a common scenario; where you are the trusted expert for an implementation as an SME, but don’t fully hand off to other people on your team so they can continue to support your implementation when you’re gone (or on vacation).

The BUS TEST

Around 2014, I was working for a large financial company that was going full ITIL for their change management process. I was already familiar with change records, but the full ITIL compliance introduced a lot of extra steps that I found cumbersome. At one point, I got annoyed by having my change records rejected by a change management person, so we decided to talk on the phone rather than bounce emails back and forth. He told me why my change records weren’t being approved: they didn’t pass the “BUS TEST.” He said: 

“Owen, if you were hit by a BUS before the implementation start date, and were laid up in the hospital, would anyone else on your team be able to complete the change on your behalf?”  

I had to answer “NO.” The instructions I included were written by me, for me, and were certainly not usable by anyone else. I went back, and re-wrote the instructions from an outsider’s perspective, and my change was approved. To this day, I follow the “BUS TEST” model for writing change MGMT records as well as documentation for clients / co-workers / blog posts. 

With that MORBID intro out of the way, here are some common items of consideration for a successful & sustainable Citrix implementation:

The dirty dozen (and then some)

• Scheduled tasks that run under a non-service account: Don’t put in your OWN DOMAIN ID! I had to do this once, but I set a reminder for a future date to re-visit and set to a proper shared group service account.
  
• Scheduled tasks that don’t contain any info on what they do via the description field, or link to external doc. When I used to work for a big financial institution years back, I’d host the accompanying docs on SharePoint, and link them in the description field.
  
  
• PowerShell code: Other than the mighty Guy Leech, there are few of us who are formally trained in programming, but, we are writing code whenever we open Powershell ISE or MS Visual Studio Code to create PowerShell code.
  
  I’d consider myself “intermediate level” with PowerShell. One thing I’m good at, is documenting! Document your code inside the script, or in greater detail via the README on github. Don’t assume everyone understands your code. People will run into an error, not report it (especially if they don’t know you) and never try to use it again. This happened to me with a health check script I wrote last year. V1 required more exit statements at the start for restricted environments, and I only caught it when someone working for the same company as me reported it. The Internet is like space: no one can hear you scream. People won’t report issues and will move on to the next thing they find by search engine search. Generally speaking, you should assume that any code you write for a client, is code that you “own” and ultimately have some level of responsibility should it stop working as expected for the clients who continue to use it
  
• Related to the above point ; scripts that don’t include a synopsis / header to explain their function, or, for more complex scripts, no accompanying documentation in wiki/SharePoint/OneNote/etc. Create a README on your git hub or at least include what the script does in contents of the script
  
• If you’ve got the time, it’s a good idea to leave the client with a basic means to health check the Citrix / hosting resources you setup for them: Shameless plug, I’ve got two scripts that work for this exact purpose: One is fully documented on my personal blog here, and another for health checking vCenter ESXi environments is on my git hub here . Both of these scripts require very little config to run, and give you a quick overview of your environment that you can once, or as a scheduled task 
• GPO’s and OU’s that were used for testing called “TEST,” “TST” that don’t contain any active items. If they ain’t used, delete ’em!
  
• A GPO for each setting/settings group: This comes down to preference, I suppose, but for me, I like to have large sets of common EUC-related settings in just two GPOs: One for computer, one for user-side settings. Having lots of GPOs can be confusing if they aren’t labeled, and can adversely impact logon times as each GPO must be processed at user logon / computer start-up
  
• Not commenting your GPOs/GPP entries: With Citrix, there are obscure registry keys that affect user experience that aren’t well documented, CTX hooks for instance! Always use the comments field in your GPO/GPP settings to link to the blog/KB where you got it, and explain why you are using it. If it’s only supposed to be in place temporarily , make a note that it’s to be removed at date “MM/DD/YYYY”
  
• Use of un-supported settings/registry keys/etc. that were in place to resolve an issue, that were later resolved by a patch from the vendor. I’ll admit, this one isn’t easy to track. I’m open to ideas on how you’ve re-visited old registry keys/GPO settings for clients you’ve set up.
  
• For work-arounds you provide to fix symptomatic issues, keep in mind, “quick and dirty” can get dirtier as time goes on. Ever forgotten a banana peel behind your trash can? I had a job once years ago, where PRPs (problem resolution procedures) that had beginning / end date! When the end date came up, you had to evaluate if the work-around was still valid, and justify renewal of the PRP. This was a great process, as you really shouldn’t be using a work-around for more than a few months, right!?
  
• Citrix products that introduce extra infra components / agents: WEM is a popular component that provides an alternative to the normal method of applying user settings to VDA sessions: GPO/GPPs. However, if you aren’t using the performance optimization features of the WEM agent, the use of WEM to control your user experience such as resetting desktop icons/mapped drives/printers introduces two caveats: 
  
  1: On-site staff who are already familiar with GPOs/GPPs will need to learn the new WEM console mgmt interface.
  
  2: The WEM agent is another binary to update in your golden image / platform layer. There is no LTSR for WEM, and each release introduces a risk of a new “bug” that can be user experience impacting.
  
  3: If you’re using WEM on-prem, you’ll have to maintain an SQL instance as well as the windows server instance where your WEM instance is installed. If/when either of these components suffers an outage, the WEM agents will go into “cache” mode, and you won’t be able to apply new settings to your WEM agents.
  
  For my clients that have engaged me in 2020/2021, and already have GPOs in place to control the base required user experience, I’m recommending they only use WEM via Citrix Cloud for it’s “killer feature,” which is its CPU/memory management.
  
• App-layering: The inspiration for this presentation came from a project I was leading in 2019 to migrate a client on a Win 7 VDI with PvD to Win 10. The original plan was to deploy app-layering + user layers as “like for like” replacement for the depreciated PvD feature. The POC using Citrix App layering was successful, but process docs we had written for the client were so complex / long, I knew the client would be challenged to continue to support the implementation once I was gone. Instead, we went with a solution that had ALL the key apps installed on one golden image, and used Microsoft FSLogix app masking to selectively hide/un-hide apps based on AD group membership. The project was a complete success. We were able to complete on-time, despite staff shortages due to covid budget restraints. 
   
• PVS vs MCS: I started using PVS in 2016, and MCS in 2019. I’m familiar enough with both products, and am happy to work with either, but the administrative overhead of PVS is significantly higher with PVS. If your client’s local staff aren’t dedicated to Citrix, which is often the case, PVS is not going to be the right choice for them
  
• Hosting review: I’ve got a related blog post on using a PowerCLI based script to provide rapid over-views of vSphere based environments. This script will help catch the following items:
  
  1) Check that the NTP service is set to ca.ntp.org or pool.ntp.org and to set to “start/stop with host”
  2) Check that the power management settings for the ESXi host is set to HIGH PERFORMANCE , else the max performance the ESXi server CPU will run at, will be 80%
  3) Scans all VMs to ID which ones are still using Intel E1000 NIC types, which are less performant than the VMware VMNet 3 type NICs. The suggestion would be to change the NIC from E1000 to VMnet 3. Note: This will require re-adding static IP info, if a MAC address was used for firewall rules or DHCP reservation, that will need to be reset as well4
  4) Scan all VMs to ID which ones are using LSI logic SCSI adapters, which are less performant than the VMware Paravirtual SCSI controller types (see this independent blog post. Where possible change the type to VMware paravirtual, you will first need to add the VMware paravirtual controller on the related shell, have the OS detect it, then power off the VM shell and amend the SCSI settings to point to the new VMware paravirtual controller5
  5) Scan all VMs to ID which are are running older versions of VMware tools. The suggestion would be to update the ESXi host VMware tools repo with the latest version, then set VMs to upgrade to the latest version on reboot. Ensure this is tested first and done in a maintenance window
  6) Scan all VMs to ID which are on older VM hardware types on their shell. The suggestion would be to schedule upgrades/reboots of the shells in a maintenance window
  7) Scan all VMs to ID which ESXi hosts are running older BIOS (EFI) levels. The suggestion would be to update where required in a maintenance window
  8) Checks vCPU to Physical CPU ratio. It’s suggested not to exceed a ratio of 5 for Citrix environments , but other workloads can run fine at higher ratios
• Collects CPU ready time to determine if you’ve over or under-provisioned vCPU count for your various VMs. Remember, a high CPU ready time means the VM wants to queue a thread to the physical CPU on the associated host, but can’t, it’s like raising your hand in a meeting or class, and not being answered right away
  
• Azure / Cloud cost management: TBD screenshot
  
  I’ve created two scenarios, one where you have lots of time towards the end of a project for documentation, the other where you’ve got very little time
  
  Short amount of time
  -AD Group > Delivery group mapping
  -Golden image virtual machine names > AD machine account mapping
  -Inventory of on-prem Citrix components and service account names
  -Storefront internal / external URLs
  
  Greater amounts of time
  -All of the above items
  -MCS / PVS / App Layering image update process
  -Full documentation on scheduled tasks
  -Full documentation on user migration scripts
  -Architecture diagrams / connection flows
  -Problem resolution procedures for the helpdesk

The above list is culled from my work as a Citrix consultant here in Montreal, Quebec, Canada from 2017 to early 2024, and from my past experience working for large financial companies. Your own experience will vary.

For me, the key to a successful implementation isn’t just about checking off the all the project requirements, but ensuring that the platform you deliver will continue to work a day after you’re gone, and a year later!

Here are some good litmus tests you can use to gauge staff acceptance during the course of the project implementation:

• If you’ve already written process docs, have the local staff read/reviewed/understood what’s written? Are they capable of completing basic changes when you’re not gone? Sick days, holidays, etc.
  
• How many emails do you get asking questions you had previously answered?  
  
• Have you ever been called on a day off for assistance on your implementation? 
  
• Have you requested feedback on any client / operational documentation that you’ve written? For instance, you might be training ops staff on the MCS image update process , which that might not prior experience with. On a recent implementation that I needed to hand off once it was completed, I booked meetings with the ops staff, and we went through the MCS image update process, and completed live edits with lots of screenshots based on the process. in the end, the ops staff were more comfortable to take over the process from me

If any of the above come up with any regularity, you might have created a monster that only YOU can tame. The client will call you back, your sales folks might be happy as you’ve increased billable hours, but your professional rep (and your company brand) may take a hit

thanks for reading and have a nice day, Owen