What follows is a take on what I learned as Salesforce moved to require all of its customers to use MFA. There’s plenty more left on the cutting room floor but it will definitely give you a flavor for the experience. If you don’t want to read all this you can check out the version I delivered at Identiverse 2022.
It is an honor and a privilege to be here on the first day of Identiverse. I want to thank Andi and the entire program team for allowing me to speak to you today.
This talk is an unusual one for me. I have had the pleasure and privilege to be here on stage before. But in all the times that i have spoken to you, I have been wearing my IDPro hat. I have never had the opportunity to represent my day job and talk about what my amazing team does. So today I am here to talk to you as a Salesforce employee.
And because of that you’re going to note a different look and feel for this presentation. Very different. I get to use the corporate template and I am leaning in hard to that.
Salesforce is a very different kind of company and that shows in up many different ways. Including the fact that, yes, there’s a squirrel-like thing on this slide. That’s Astro – they are one of our mascots. Let’s just get one thing out of the way up front – yes, they have their own backstories and different pronouns; no, they do not all wear pants. Let’s move on.
So the reason why I am here today is to talk to you about Salesforce’s journey towards complete customer adoption of MFA. There are 2 key words in this: Customer and Journey.
‘Customer’ is a key word here because the journey we are on is to drive our customers’ users to use MFA. This is not going to be a talk about how we enable our workforce to use MFA. Parenthetically we did that a few years ago and got ~95% of all employees enrolled in MFA in under 48 hours. Different talk another time. We are focused on raising the security posture of our customers with their help.
Journey is the other key word here. The reason why I want to focus on the Journey is because I believe there is something for everyone to take away and apply in their own situations. And I want to tell this Journey as a way of sharing the lessons I have learned, my team has learned, to help avoid the mistakes we made along the way.
The Journey Begins
So the Journey towards complete customer MFA.
It starts in the Fall of 2019. Our CISO at the time makes a pronouncement. Because MFA is the single most effective way our customers could protect themselves and their customers, we wanted to drive more use of MFA. So the pronouncement was simple: in 3 months time every one of our at the time 10 product groups (known as our Clouds) will adopt a common MFA service (which was still in development at the time btw) and by February 1 of 2021, 100% of end users of our well over 1500,000 customers will use MFA or SSO on every log in. Again this is Salesforce changing all of our customers’ and all of their users’ behavior across all of our products in roughly a year’s time.
That means in a year’s time we are going change the way every single person who logs into a Salesforce product. And let’s be honest with ourselves fellow identity nerds, this is what people think of MFA:
100% service adoption in 3 months. 100% user penetration within about 1 year.
Of all end users.
All of them.
There’s laughing from the audience. There’s some whispering to neighbors. I assume this is your reaction to the low bar that the CISO set for us… a trivial thing to achieve.
Oh wait no… the opposite. You, like I did at the time reacted to the absolute batshit nutsery of that goal. What the CISO is proposing is to tell customers, WHO PAY US, here is the minimum bar for your users’ security posture and you must change behaviors and potentially technologies if you don’t currently meet that bar and want to use our services.
100%… oh hell no.
I reacted like most 5 year olds would. I stomped my feet. I pulled the covers over my head thinking if the monsters couldn’t see me, they couldn’t get me. If I just didn’t acknowledge the CISO’s decree, it would somehow not apply. Super mature response. Lasted for about a week. Then I learned that the CISO committed all this to our Board of Directors. So… the chances of ignoring this were zero. But still, I fought against the tide. I was difficult. I was difficult to the program team and to my peer in Security. That was immature and just wasted time. I spent time rebuilding those relationships during the first 6 months of the program.
Step 0: Get a writer and a data person
What would you do hotshot?
If you got this decree what should be the first thing you’d do. Come on – shout them out! (Audience shout out answers.) All good ideas… but the first thing you should do is hire the best tech writer you can. Trust me you are going to need that person in the 2nd and 3rd acts and its gonna take them a bit of time to come up to speed… so get going, hire a writer!
(It’s also not a bad idea to get data people on the team. If you are going to target 100% rollout then you need good ways to measure your progress. And you’ll want to slice and dice that to better understand where you need more customer outreach and which regions or business are doing well.)
Step 1: Form a program with non-SMEs
Ok probably the next thing you’d do is get a program running which is what we did. That program was and is run by non-identity people. Honestly, my first reaction was that this was going to be a problem. What I foresaw was a lot of explaining the “basics” of identity and MFA and SSO to the program team and not a lot of time left to do the work.
I was right and I was wrong. I was correct in that I and my team did spend a lot of time explaining identity concepts to the program team. I was wrong in that the work of explaining was actually the work that needed to be done. The program team were not identity people and we were asking them to do identity stuff and this was just like the admins at our customers. They were not identity people and we were now asking them to do identity stuff.
So having a program team of non-subject matter experts was a great feature not a bug. As the SMEs, my team spent hours explaining so many things to the program team and it turned out that the time we spent there was a glimpse of what the entire program would need to do with our customers.
Not only did we have a program team staffed with non-subject matter experts, we also formed a steering committee staffed, in part, with non-subject matter experts. The Steerco was headed by a representative from Security, Customer Success, and Product. This triumvirate helped us to balance the desires of Security with the realities of the customers with our ability to deliver needed features.
Step 2: Find the highest ranking exec you can and use them as persuaders as needed
Next up – if we needed all of our clouds to use MFA, we need to actually get their commitment to do so. The program dutifully relayed the CISO’s decree to the general managers of all the clouds. Understand that Salesforce’s financial year starts Feb 1, so we were just entering Q4 and here comes the program team telling the GMs, “yeah on top of all your revenue goals for the year, you need to essentially drop everything and integrate with the MFA service,” which again wasn’t GA yet.
We were asking the GMs to change their Q4 and next fiscal year plans by adding a significant Trust-related program. And at Salesforce Trust is our number 1 value which means that this program had to go to the top of every cloud’s backlog. As a product manager, if someone told me “hey Ian, this thing that you really had no plans to do now has to be done immediately” I would take it poorly. Luckily, we have our CISO with the support of our Co-CEOs and Board to persuade the GMs.
Step 3: Get, maintain, and measure alignment using cultural and operational norms
So we got GM commitments but needed a way to keep them committed in the forthcoming years of the program. We used our execs to help do this and we relied on a standard Salesforce planning mechanism: the V2MOM. V2MOM stands for Vision, Values, Methods, Obstacles, and Measures. Essentially, where do you want to go, what is important to you in that journey, what are the things you are going to do get to that destination, what roadblocks do you expect, and how will you measure your progress. V2MOMs are ingrained in Salesforce culture and operations. Specific to MFA, we made sure that service adoption and customer MFA adoption measures were in the very first Method of every Cloud’s V2MOM and we used the regular review processes within Salesforce to monitor our progress.
Do not create someting new! Find whatever your organization uses to gain and monitor alignment and progress and use it!
Lesson 1: Service delivery without adoption is the same thing as no service delivery
Round about this time I made the first of many mistakes. We had just GA’ed the new MFA service and I wanted to publish a congratulatory note and get all the execs to pile on. Keep in mind that the release was an MVP release and exactly zero clouds had adopted it. My boss stoped me from sending the note. Instead of a congratulatory piling on from the execs, I got a piling on from the CISO for the lack of features and adoption.
I am a product manager and live in a product org… not an internal IT org, not the Security org. My world is about shipping features… my world was about to get rocked. I had lost sight of the most important thing, especially to the execs: adoption.
Thus service delivery without adoption is the same thing as no service delivery.
Lesson 2: Plan to replan
At this point it is roughly February 2020 and no clouds had adopted the MFA service and we had just started to get metrics from the clouds as to their existing MFA and SSO penetration. It wasn’t pretty but at least we knew where we stood. And where we stood made it pretty clear to see that we were not going to be in a position to drive customer adoption of MFA and certainly not achieve 100% user coverage within the original year’s time.
We needed to reset our timeline and in doing so we had to draw up a two new sets of plans: one for our clouds adopting the MFA service and one for our customer adoption. In that process, we moved the dates out for both. We gave our clouds more time to adopt the MFA service and moved the date for 100% customer end-user adoption to February 1 2022.
No matter how prepared you are at the beginning of a program like this, there will always be externalities that force you to adapt.
So with our new plans in hand, a reasonably well-oiled program in place, we began to roll out communications to customers in April of 2020. We explained what we wanted them to do 100% MFA usage and why – MFA is the single best control they could employ to protect themselves, their customers, and their data against things like credential stuffing and password reuse. And we let them know about the deadline of February 1 2022. We did this in the clearest ways we knew how to express ourselves. We did it in multiple formats, languages, and media. We had teams of people calling customers and making them aware of the MFA requirements.
Remember when I said hire a writer early… yeah, that. Clear comms is crucial. Clear comms about identity stuff to non-identity people is really difficult and crucial to get as right as possible (and then iterate… a lot.)
Gain traction; get feedback
The program team we had formed was based on a template for a feature adoption team. Years ago, Salesforce released a fundament change to its UX tier which had profound impact to how our customers built and interacted with apps on our platform. To drive adoption for the new UX tier, we put together an adoption team… and we lifted heavily from that team and their approach.
Using the wisdom of those people, we knew that we were going to have to meet our customers where they were. First and foremost, we need a variety of ways to get the MFA message out. We used both email and in-app messages along with good ole’ fashion phone calls – yes we called our customer admins. Besides a microsite, we build ebooks and an FAQ. We put on multiple webinars and found space in our in-person events to spread the word. We even build some specialized apps in some of our products to drive MFA awareness.
And we listened… our Customer Success Group brought back copious notes from their interactions with customers. We opened a dedicated forum in our Trailblazer Community. We trained a small army of people to respond to customer questions. We tracked customer escalations and sentiment and reported all of this to the CISO and other senior execs.
In our leadership development courses at Salesforce, we do a business simulation. This simulation puts attendees in the shoes of executives of a mythical company and they are asked to make resource allocation and other decisions. Over the course of the classes, you compete with fellow attendees and get to see the impact of your decisions. It’s a lot of fun.
One consistent thing in all of the simulations is “The Wobbler.” The Wobbler is an externality thrown at you and your teammates. They can be intense; they can definitely knock a winning team out of contention. And so you can say to a colleague, “We were doing great until this wobbler” and they totally know what you mean.
Predictably, the MFA program was due for a wobbler. This one came in from a discrepancy in what we were communicating and the CISO noticed it first. Despite the many status briefings. Despite having one of his trusted deputies as part of the steering committee for the MFA program. There was a big disconnect. The MFA Program was telling our customers “By February 1 2022 you need to be doing MFA or SSO.” The CISO thought we were telling customers “MFA or SSO with MFA.”
There are probably a few MBA classes on executive communication that could be written about this “little” disconnect. There was going to be no changing the CISO’s mind; the program team simply needed to start communicating the requirement of MFA or SSO with MFA.
From our customers perspective, Salesforce was moving the goal posts. They were stressed enough as it is and this eroded trust. Our poor lead writer had a very bad week. The customer success teams doing outreach and talking to customers had very bad weeks. My teams had to redo their release plans to pull forward instrumentation to log and surface whether an SSO login used MFA upstream.
A word from our speaker
And now a word from a kinda popular SaaS Service Provider: “Hi, are you like me? Are you a service provider just trying to make the internet a safer place and increase the security posture of your customers but are thwarted by the lack of insight into the upstream authentication event? Isn’t that frustrating? But don’t worry we have standards and things like AuthNContext in SAML and AMR claims in OIDC. Now if only on-prem and IDaaS IDPs would populate those claims consistently as well as consistently use the same values in those claims. If we could do that, it would make the world a better place. Don’t let this guy down.”
Ok I know this isn’t sexy stuff but please please please! It is damn hard as an SP to consistently get any insight into the upstream user authentication event. I know my own services can do better here when we act as an IDP. Please, industry peers, please please make this data available to downstream SPs. And, standards nerds, I know it ain’t sexy but can we please standardize or at least normalize not only the values in those claims but the order and meaning of the order of values within those claims. Pretty please? (include scrolling spreadsheet of all the amr values we’ve seen)
Step 4: Accommodate the hard use case
The wheels had begun to gain traction so to speak. We heard from customer CISOs who were thrilled about our MFA requirements – it gave them the justification they were looking for to go much bigger with MFA. But we also heard from customers with hard use cases for whom there aren’t always great answers. For example, we have customers who use 3rd parties to administer and modify their Salesforce environments. Getting MFA into those peoples’ hands is tricky. Another example, people doing robotic process automation or have UX test suites struggle to meet the MFA requirements of MFA on every UI-based login. Those users look like “regular” human users and have access to customer data. They need MFA. And yet the support for MFA in those areas is spotty at best.
We had another source of challenging use cases – brought to us by our ISV and OEM partners. These vital parts of our business have a unique relationship with our products and our customers and the challenges that our customers feel are amplified for our ISVs and OEMs.
What we learned was that there are going to be use cases that are just damn hard to deal with. 3rd party call centers. RPA tools. Managed service providers. The lesson here is – it’s okay. Your teams are comprised of smart people and even still there is no way to know at the onset of such a program these use cases. Find the flexibility to meet the customers where they are… and that includes negotiated empathy with your executives and stakeholders. I truly believe there is always a path forward but it does require flexibility.
At this point we have clouds mostly adopted, people are rolling out MFA controls in their products. Customer adoption of MFA and SSO are climbing and we are feeling good. And, predictably, the universe decided to take us down a peg. Enter Wobbler #2 – outages.
Raise your hand if you know the people that maintain the DNS infrastructure at your company… if you don’t know them, find them. Bring them chocolate, whiskey, aspirin… DNS is hard. And when DNS goes squirrely it tends to have a massive blast radius. Salesforce had a DNS-related outage and the MFA service that most of our clouds had just adopted was impacted.
And a few weeks after we recovered from that, the MFA service suffered a second outage due to a regional failover process not failing over in a predicted manner.
We recovered, we learned, we strengthened the service, we strengthened ourselves.
So when things are going well, just assume that Admiral Ackbar is going to appear in your life… “It’s a trap.”
Step 5: Address the long tail
So where are we today? Well, while we found lots of MFA and SSO adoption in our largest customers – especially SSO, we have a lot of customers with less than 100 users and their adoption rates were low. One concerning thing about these customers is that the ratio of general users to those with admin rights is very high. Where privileged users might make up less than low single digits of the total user population in larger tenants, it was much much higher in smaller ones. Although we had a great outreach program there are literally tens of thousands of tenants and thus tens of thousands of customers whose login configurations and behaviors we had to change.
And here is where we learned that we had to enlist automation and that is where our teams are focused today. Building ways to ensure that new tenants have MFA turned on by default, customers have ways of opting out specific users such as their UX testing users, and means to turn on MFA for all customers, not just new ones without breaking those that put in the effort to do MFA previously. That takes time but it is well spent time – we are going to automatically change the behavior of the system which directly impacts our customers users – it is not something ones does lightly (one does not simply turn on mfa meme)
Lesson 3: Loving 100% percent
Standing here today, I can say that I really like the 100% goal. As I wrote this talk, I looked back at some of my email from the beginning of the project… and I am a little ashamed. I really fought the 100% goal hard… it wasn’t a good look. It wasn’t the right thing to do. The reason I like the goal is that although we are at roughly 80% of our monthly active users using mfa or sso, had we not made 100% the goal then we’d have achieved less and been fine with where we are. Without that goal we wouldn’t have pushed to address the long tail of customers; we would not have innovated to find better solutions for both our customers and ourselves. Would I have liked our CISO to deliver the goal in a different way? Sure. But I have become a fan of a seemingly impossible goal… so long as it is expressed with empathy and care.
Step 6: Re-remember the goal
We ended last fiscal year with about 14 million monthly active users of MFA or SSO with MFA. They represent 14M people who are habituated; the identity ceremonies they perform include MFA.
And that has a huge knock on effect. They bring that ceremony inclusive of MFA home with them. They bring the awareness of MFA to their families and friends. And this helps keep them safer in their business and their personal lives. The growth of MFA use in a business context is a huge deal professionally speaking. As I tell the extended team, what they have done and are doing is resume-building work: they rolled out and drove adoption of MFA across multiple lines of business at a 200 billion dollar company. That is no small feat!
But that knock on effect – that those same users are going to bring MFA home with them and look to use it in their family lives… that, as an identity practitioner, is just as big of a deal. That makes the journey worth it.