The lights dim, and the curtain rises
Sysadmin 1: So I reset all the passwords in Active Directory, like you asked. I still can’t believe our security vendor reset everyone’s password by accident.
Sysadmin 2: Yeah, I know. Do you have all of the individual user passwords stored somewhere?
Sysadmin 1: Huh?
Sysadmin 2: The user passwords. You set a different one for each user. Where are they?
Sysadmin 1: Uh…
Sysadmin 2: You’re kidding.
Sysadmin 1: Um.
Sysadmin 2: You set everyone’s password to be the same.
Sysadmin 1: Yeah…
[end scene]
Time to reset all the passwords
The risk of a mass password reset has to be assessed and planned for by an organization’s security team and the C-suite. Take a look at any of the major reports on cyber breaches from Microsoft, Verizon, or Mandiant (Google), and you’ll find that one of the biggest targets is the identity management system, whether it’s on-premises Active Directory, Entra ID (formerly Azure Active Directory), PingIdentity, or Google Workspace. If someone gains full access to the system, you have to assume that sensitive information like passwords and encryption keys were compromised. That poses a complicated situation: not only do you need to reset passwords for all of the user accounts, but you also have to distribute those passwords.
However you decide to perform a mass password reset, try not to do what this high school outside Chicago, Illinois did, according to reporting from TechCrunch. The school had retained a cybersecurity vendor to perform an audit. In the course of that work, an “unexpected vendor error” occurred and “the system reset every student’s password, preventing students from being able to log in to their Google account.”
The fix was simple. Too simple, in fact: “We have reset your child’s password to Ch@ngeme! so that they can once again access their Google account.”
This, of course, is a Very Bad Idea®.
According to TechCrunch, students began logging in to other students’ accounts, not only to change passwords and lock out their peers, but also to gain access to documents and emails. The school’s IT office eventually shut down all of the students’ access and committed to a more secure password reset process, but the damage was already done.
There’s a long history of bad password practices (see Spaceballs), overly aggressive password attempt limits (see Jurassic Park — the good one), and poor password hygiene (see Internet Relay Chat, also known as IRC) in IT, and this incident will surely end up on the list of events recorded for posterity. But this column isn’t intended to pile on to the existing Internet commentary. Instead, let’s be constructive: how do you solve this kind of problem in a secure manner?
Multi-factor authentication to the rescue
For enterprise environments that have deployed multi-factor authentication (MFA) — especially when it’s been implemented using a hardware token, like an RSA SecureID device or a Yubico YubiKey — it can be really easy to handle password resets. Your users already have a secure second authentication factor, so it’s just a matter of selecting software for a password reset tool and allowing users to reset using their token. (This scenario is where FIDO2 and WebAuthn shine over HOTP and TOTP, because the physical device has to be present for authentication.) All you have to do is reset every user account to a long, random password, and direct users to the password reset tool with their MFA token.
Organizations that haven’t moved to hardware MFA should seriously consider it. The security benefits are well worth the cost, and many applications today can integrate with identity management systems using SAML, OAuth, and OpenID Connect, shifting the authentication and authorization responsibility from applications to the IdAM platform, and gaining the ability to implement MFA and risk-based access controls across the board. FIDO-only YubiKeys are relatively inexpensive, and there’s a wide range of smartphone-based authenticator apps for a no-cost solution.
Distributing passwords is an IT project management challenge
Primary and secondary schools are a great example of organizations that have to plan their password distribution strategy carefully. Each year, they get a significant number of students enrolling in the school system, and those students all need to establish their accounts and passwords. If, for example, a school system gives accounts out starting in sixth grade, roughly 15 to 20 percent of your user accounts are being created each year. (This assumes graduation after twelfth grade, and some variance in class sizes from year to year.)
Unlike a traditional employer-employee relationship, you can’t assume that a student will have home or mobile internet, or a personally-owned smartphone or computer, or even a parent or guardian with email access. Distributing hardware tokens like a YubiKey would easily cost over $10,000 per one-thousand students, overhead that might not be possible on most school budgets, plus the expense of replacement keys. And using challenge questions for a password reset tool would be challenging, not to mention a logistical nightmare — you’d have to start from a known set of student data and configure the questions, while making sure the information isn’t easily guessed, like child’s city of birth.
So how do you distribute passwords for the first time? There’s no easy answer, but there are some good options out there, like having students come to the school before the start of the academic year, or handing them out through a homeroom or the first class of the day once the academic year starts. It comes down to prioritizing “fast” versus “cheap,” in the classic IT project management triangle — you can’t give up “accurate,” because you’re dealing with passwords.
There are also some clearly incorrect answers. Your help desk should never, under any circumstances, have a “default” password that accounts get reset to; this includes constructed passwords, where a person’s information is used to change the “default.” (For instance, avoiding “Password” plus the last four of a person’s Social Security Number or their two-digit month and day of birth — information that might be easy to look up — or “Password” plus the current month and day — though this is less risky if you’re talking with the user and they change the password while you’re communicating with them.) Sending cleartext passwords via SMS also isn’t a good idea, given the lack of message encryption.
Keep calm and don’t rush to a solution
In the IT world, it’s easy to focus on the technical problem at hand and start to come up with engineering solutions. When the problem is the result of a crisis, it’s even easier to rush the process and look for a quick fix. What you should do instead is follow the first principle of a talk I give, titled “The Three Principles of Troubleshooting”:
Size up Goliath first.
Robert J. Funches, “The Three Principles of Troubleshooting”
Take the opportunity to figure out what your technical problem is. You’ll be surprised how many times someone misstates the problem or has made an assumption about the problem, and when you go to fix it, it turns out the problem you thought existed really doesn’t — the problem is something else entirely. Ask questions, pose hypotheticals, pull out the whiteboard markers or whatever else you need to orient yourself. [In this specific example: all of the students’ passwords have been reset.]
Next, understand what led up to the problem. Context can be incredibly helpful in your search for a fix, even if you don’t have all of the answers. It’s also important to take note of information that might be inaccurate or wrong. [In this specific example: A cybersecurity vendor was performing an audit; while the details aren’t publicly known, it’s clear the vendor was examining the identity store AND was working with administrative credentials that had permissions to reset passwords.]
Finally, identify the relevant tools that are available to you. This is not the same thing as identifying a technical solution or fix. If your AD group memberships have been completely screwed up, your tools may be log files (meaning Windows Event Viewer), your SIEM (Splunk, Elastic, etc.), Active Directory backup files, or custom/commercial/open-source scripts previously deployed to track group membership. The idea is to take inventory of what’s at your disposal before diving into solution-land. [In this specific example: unknown, as public reporting doesn’t really touch on this, but hopefully something like Google Apps Manager (GAM) to automate management of Google Workspace.]
Ultimately, the goal is to slow down and understand what you’re dealing with. This is how IT gets to be the hero of the story — or at least keeps the bad news to a minimum.