I’m a fan of Latin. No, not the music; the so-called dead language that was recently praised by a novelist and Oxford professor as “useless.” So when I set out to write about the benefits of a group managed service account in Windows, I headed to my trusty resource, the Oxford English Dictionary (OED), to replace the “g” and “M” in “gMSA.” (I’m also a fan of etymology, and one of the most entertaining reads that I love to recommend is Mark Forsyth’s “The Etymologicon,” which I describe as the OED meets 60 degrees of Kevin Bacon.) Getting the “g” was easy, but the “m” took some thought, but the result — from the Latin, melior, meaning better — sparks joy for me.
The past few years have brought some exciting news in the world of gMSAs that might have escaped notice by AD administrators: support for gMSAs on Linux using a popular AD bridging software product, and support for a non-domain joined host for launching domain-joined containers. That means everyone should be using gMSAs everywhere they can.
And then the year 2022 rolled around and spoiled everyone’s fun…
I asked for a service account and all I got was this lousy AD user object
Active Directory administrators will tell you that service accounts, created as standard domain accounts, are the bane of their existence:
- They’re a pain to track.
- They’re annoying to secure.
- They’re difficult to manage by account holders.
Sean Metcalf of adsecurity.org maintains a list of the service account hall of shame: vendors and software that require real domain accounts, sometimes with absurdly privileged accesses attached to them, almost always with no good reason. As he says: “Rarely does a service account actually require Domain Admin level rights.”
Trying to constrain a service account is a Herculean task. As a standard domain account, you get quite a few things right out of the box by virtue of membership in Domain Users. The list of user rights assignments, as a starting point, is extensive: interactive logins, access to network resources, etc. Then there’s all of those places where admins made use of the Domain Users or Authenticated Users security principal to control access: great for end users, not so much for service accounts.
Beyond that is the problem of the humble password. If you hand out a service account to a non-admin, but block interactive logons, how does your non-admin account holder change the password? Let’s say the account is used for a production system with high criticality and an aggressive SLA. How do you minimize downtime while you cycle the password due to password expiration policies? (The usual answer: disable password expiration, either directly on the account or through the use of AD password policies.)
Tracking their use is also a challenge. Because these are regular AD user accounts, they function like regular users, whose activity can vary widely — think web-based services leveraging LDAP or enterprise and virtual desktops making use of roaming profiles and Profile Containers. Service account credentials used for application A, however, might end up also being used for application B out of convenience; yet the account name may specifically cite app A, leading an AD admin to miss its use in app B. (Incidentally, this is where your SIEM comes to the rescue.) And what about recording the owners of the service account? That’s a task typically left to the random attributes of an AD account, or an IT request/ticketing system, or an asset and user management COTS product. (See: CIS controls #5, Account Management.)
We’ve actually had nice things [since 2009] [when it comes to service accounts]
It may come as a surprise to some AD admins that when it comes to running services on a server, there’s actually been a supported method for service accounts since the release of Windows Server 2008 R2. Seasoned AD admins will immediately balk — because the original supported method was, to put it nicely, not that useful.
Windows in the modern era (meaning 2000/XP) has always had local accounts for various functions, the two most notable security principals being NETWORK SERVICE and SYSTEM. The latter is the highest-privileged account on any Windows system, and generally only accessible through means like services, scheduled tasks, and tools like Sysinternals PsExec. But these accounts are local — they only exist on the “domain” that is the computer itself.
The release of Windows Server 2008 R2 bridged the gap between those local accounts and a fully-provisioned domain account in Active Directory. They called it the Managed Service Account, or MSA. (Redmond has since rewritten history, and now refers to them as Standalone Managed Service Accounts, to distinguish them from the aforementioned Group Managed Service Accounts.) MSAs had two major advantages over the traditional AD domain user account:
- MSAs have a long, pseudorandom password
- MSA passwords are rotated every 30 days (unless you’ve changed certain defaults in the AD domain)
Because MSAs are domain accounts, they have access to network resources. Just as with a domain user account, MSAs can be added to NTFS ACLs and share permissions. And MSAs can be used with Windows services and scheduled tasks, as long as the underlying applications support the use of a MSA.
Some things are better in a group
In response to the gotchas and caveats of Managed Service Accounts, Microsoft improved MSAs with the next release of Windows. Enter Windows Server 2012 and the gMSA: group managed service account.
As cleverly described by Doug Symalla on Microsoft Technet, the new operating system came “to the rescue” of Windows admins in a number of ways:
- gMSAs can be used on multiple systems.
- gMSAs can be used to run scheduled tasks.
- gMSAs support applications like SQL Server and IIS.
Now you have a much more flexible use case, especially with #2 and #3. You still get the benefits of a rolling, pseudorandom, lengthy password and an account that plays nice with ACLs and other things that support AD objects.
The advantage of tightly-coupled secrets management
sMSAs and gMSAs are great for secrets management — rotating and federating access to account credentials — because of their tightly-coupled nature. When a MSA is configured, you have to specify which computers have access to the account. Only those computers and Domain Admins have permissions in AD to obtain the MSA’s password. That restricts the MSA credentials to those servers, unlike a standard AD user account used as a service account.
[Sidebar: There’s a surprisingly convenient button in AD Users and Computers for user objects called “Log on to” under the Accounts tab. It pops open a sub-property sheet where you can specify that the user can log on to all computers, or just the specified computers. That sheet links to the AD attribute called “userWorkstations.” Microsoft now advises that you don’t use the attribute and instead implement user rights assignments, i.e. log on/deny log on locally and log on/deny logon through Remote Desktop Services. If you’ve been using this setting, the 2000s called: they want their LDAP attribute back.]
This reflects the current trend of vendors like CyberArk, Thycotic (since acquired by Centrify/Delinea), and Hashicorp, and tools like Netflix’s Lemur and AWS’s Secrets Manager. For privileged services and platforms, it’s better to keep credentials out of the hands of developers and admins, and rotate those credentials often.
Growing adoption of MSAs by vendors
The biggest barrier to the use of MSAs is on the application side. Even Microsoft’s own products don’t always support them. But in recent years, vendors have added support because customers have asked for it.
In 2019, Kubernetes developers started work on enabling Windows containers to use gMSAs. This solved a critical gap in the use of Windows as a container OS: the inability to join a Windows container to an AD domain. In the initial design, the container host still had to be domain-joined, but at least you could leverage features like Integrated Windows Authentication for the first time.
In 2021, Centrify (now Delinea) added support for gMSAs in its popular Active Directory bridging software, DirectControl (now Authentication Service and Privilege Escalation Service), for Linux and UNIX-based clients. This allowed Linux servers, which were joined to AD domains through the use of the DirectControl agent, to also retrieve and store the gMSA account password in the local Kerberos keytab.
The release of Windows Server 2022 improved support for containers by removing an important prerequisite: that the container host be domain-joined. Not only did this simplify management of a container farm, but it reduced the attack surface of a container host and limited the scope of domain credentials. (How many VMware ESXi servers are actually joined to an AD domain?)
And now to spoil the party
It’s clear that gMSAs have advantages, and developers and vendors alike have flocked to the technology. Microsoft even tells you for on-premise service accounts to use gMSAs first, then sMSAs, then computer and user (domain or local) accounts as a last option. What’s not to like about it?
When you first deploy gMSAs, you have to create the Key Distribution Service (KDS) Root Key for the domain. That key is stored in Active Directory as a “msKds-ProvRootKey” object type. And therein lies the problem, according to Semperis security researcher Yuval Gordon.
Published in March 2022, the Golden gMSA attack looks a lot like a Kerberos Golden Ticket attack. The key distinction — pun not intended — is how the secret is managed. For the Kerberos Ticket-Granting Ticket service account (krbtgt), the account password can [and should] be manually cycled, allowing periodic rotation of the secret, as well as invalidation of forged Kerberos tickets. But the KDS root key has no mechanism.
That’s right: the KDS root key can’t be rotated. If it’s compromised, you have to create a brand new key.
Unfortunately, replacing the KDS root key means you also have to replace all of the gMSAs in the domain, because all of them are linked to the same key. Have fun working your way through all of those SQL Server instances and Azure Kubernetes Service containers…
Back to domain user accounts for services?
The natural reaction to the Golden gMSA attack is to run, not walk, away from gMSAs. But is this a good idea?
In my opinion: no.
Let’s be clear that the ability to compromise all of a domain’s (or forest’s) gMSA accounts is a big deal. But so is obtaining Domain Admin. In other words: attacking a gMSA isn’t easy. Combining the mitigations outlined by Microsoft with the traditional recommendations for protecting a domain goes a long way towards protecting the KDS root key. And if someone has managed to grab the KDS root key, your incident response team had better be looking for other domain admin targets, like the krbtgt account, and not just worrying about a global gMSA reset.
The advent of Golden tickets in Kerberos didn’t cause Active Directory admins to throw up their hands in frustration and declare Active Directory dead. (Shameless plug: I do intend to declare AD dead [with caveats] in an upcoming commentary…stay tuned!) We turned to better security controls, logging, and good AD maintenance and hygiene. There’s no reason we can’t do the same here with gMSAs.
One possible mitigation would be to use gMSAs for services that don’t provide lateral access or privilege escalation to an attacker, and use dedicated domain user accounts (with very strong passwords, restrictive attributes, and highly-constrained user rights assignments) for sensitive services. Your low-level containers running on AKS would use a gMSA, but your MECM (SCCM) or SCOM SQL database would use a domain user account.
A more interesting option is a periodic topic of conversation over on the SANS Internet Storm Center (ISC) blog: honeypots. An attractive-looking gMSA — especially in tandem with other low-level gMSAs that perform real functions but don’t provide a functional path for privilege escalation — could serve as a useful canary.
Greatly improved service accounts
For those who are still debating my use of “meliorate” rather than “ameliorate,” I’ll concede that “greatly improved service accounts” has a better ring to it. Perhaps Microsoft could consider replacing “Managed Service Accounts” with “[AD-]Integrated Service Accounts,” or ISAs.
[Sidebar: Those of us in the earlier days of computing might object to raising the acronym ISA from the dead: the days of the Industry Standard Architecture bus, which greatly expanded the market for expansion cards for PCs but was thankfully replaced by the Peripheral Component Interconnect (PCI) bus in the 90s and PCI Express (PCIe) in the 2000s. Yes, ISA slots launched the market for sound cards, video cards, modems, you name it; but OS compatibility was a mess until plug-and-play (PnP) support matured, and by then ISA had been left behind by PCI. Not sure anyone really misses the days of ISA slots…]
Naming aside, Microsoft’s solution for service accounts isn’t perfect, and the shared nature of the KDS root key poses a unique risk. But there are clear benefits to the alternative of domain user accounts, and AD admins can manage the risks involved with gMSAs. It’s not every day that what a business touts as an improvement is, in fact, an improvement. Managed Service Accounts? Definitely an improvement.