Breaking Prod is a Part of the Process

Oh, it seems so simple. Create a widget on the Service Portal and add some current catalog items and record producers within that catalog. Seems easy enough. After further investigating, I create the catalog and create a category. I add the catalog items and record producers in the category, which places them in the catalog. Next, I add the widget to the portal page. 

I test my work. Everything looks amazing. 

Things are working and functioning on Dev.

Now, onto the Test environment. I commit the update set and test to see if my changes applied correctly. Everything is functioning properly, and I am closer to getting the QA Tester on the team to confirm my work. 

A little note about the QA Tester on my team: She is really good at what she does, which can be a slightly tedious thing if you are on the receiving end of one of her reviews. She always finds a way to look beyond my test cases and find creative ways of breaking them. 

But this time things went smoothly in QA. No defects! 

Whew!

Finally, it’s time to push to the Production instance. My update set is pulled from Test to Production and everything looks exactly like it did in Dev and Test. Proud of my work, I confirm that what I worked on is working and functioning in Production. 

The next day, the tickets are rolling in. 

One after another.

My colleague is working the queue and he writes me:

‘‘You know it’s a problem when you get a message at 8 AM.”

He asks, “Didn’t you have that Widget-Service Portal story last sprint?”

“Yeah, I did. What’s up?”

“So… yeah… I didn’t want to write you this early but your story is breaking stuff in Prod.”

“Umm, what do you mean?”

“The widget you created looks good; your catalog items and record producers you added in the catalog are showing. But when you go in, all the record producers are missing the Submit button and the catalog items are completely blank.”

I’m panicking at this point. 

Holy crap! I messed up big time! Everything was working just fine. 

WHAT THE HECK HAPPENED!?!

I begin to internalize my mistake as an utter failure. I begin to call myself names in my head. I begin to worry about how many other people (about 26,000 people work here, btw) have attempted to use these producers and items and saw an absolute blank. And i begin to worry about what my colleagues will think.

This is the largest mistake I’ve ever made at my job.

I reach out to my team’s Senior Developer and tell her what happened. She can tell I’m frazzled by the way I type. I’ve looked up to her since I’ve been here and hold her in high regard. As I’m continuing to write, I’m imagining what she’s thinking:

She should know better. How did she let this go through? I can’t believe she made that big of a screw up. I should not have said she was a good fit for the team. Look at what she did and now I have to fix this.

She never says any of this, of course. I just don’t want to let her down. The perfectionist in me is truly bothered by the mistake. 

Next, I reach out to the QA tester. I hold her in just as high regard as the Senior Developer. Feeling like a nervous child that just received a terrible grade, I began writing her.

“Hi [QA tester], I messed up big time. I’m so sorry.”

At this point, I’m on the verge of tears and it’s not even 10:00am! It’s getting closer to our daily scrum, which starts every morning at 10:15am, and the pressure in my head is pounding. The tension in my shoulders feels like I am carrying boulders. And did I mention the red flush on my face due to absolute embarrassment?

Microsoft Teams begins to chime on my computer, letting me know it’s time I let the entire team know how much of a screw-up I am. Our QA tester, who happens to be our Scrum Master, calls us one by one to talk about the work we did the previous day, any meetings/work we are currently doing, and any blockers (the usual). Minutes begin to seem like hours as I wait nervously for my name to be called.

“Ebony?”

My heart beats a mile a minute, my mouth is parched, and my face beet-red. The words I utter next feel like a synopsis of my whole existence in life.

“I’m sorry, I let the team down. I pushed something into Production that is breaking the Service Portal. I don't know what to do. I told our Senior Developer and she will back out the update sets. My blocker is my mistake.”

As I am talking, my manager writes to me and he tells me to breathe. Everything is OK. He then tells me he’s done so much worse.

After the Scrum meeting is over, I get a call from the Senior Developer. I’m getting anxious again. I just want to go somewhere and not come back, but I hesitantly answer.

“Ebony?”

“Yes, ma’am.”

“Are you okay?”

“No, I screwed up. I don’t make mistakes like this.”

The words she says next really change my perception.

“Breaking Prod is a part of the process. If you didn’t break it, you could never learn from that mistake. You’re making mistakes, but the work that you took on was pretty advanced for a Sys Admin.”

This makes me feel slightly better, but I still can’t get over my mistake. I spend most of the day reflecting on the mistake that happened earlier that morning that carried on throughout the day.

Before calling it a day, the Senior Developer calls me again. I nervously answer.

“Hey Senior Developer.”

“Ebony, I know you’re still thinking about this morning. We backed out the update sets, and everything is fine. I ran through the same thing you did, and I didn’t see anything wrong with what you did. I would’ve done the same thing.”

I am now confused and slightly intrigued. 

“Wait, what?”

“I did exactly what you did, and everything was fine.”

“What happened?”

“I don’t know.”

“You don’t know?”

“In the world of ServiceNow, things will work, and we sometimes don’t know why. Also, things will break, and we won’t know why.”

At this point, I am eight minutes away from clock out time and I feel everything that has been building up since 8:00 AM fade away. I am no longer upset, discouraged, or depressed. I don’t feel stupid or question my abilities. I realize that certain things will never have an explanation.

Breaking Prod is your initiation to the most elite club in ServiceNow. A place where things will rarely make sense, you’ll feel out of your depth a good 70% of the time, but that doesn’t matter as long as you have a great team supporting you.