Sponsored by Rezilion, the purpose of this research is to understand the state of organizations’ DevSecOps efforts to manage vulnerabilities throughout the software attack surface. Ponemon Institute surveyed 634 IT and IT security practitioners who are knowledgeable about their organizations’ attack surface and effectiveness in managing vulnerabilities.

All organizations have adopted DevSecOps or are in the process of adopting a DevSecOps approach. According to the research, the lack of the right security tools is the primary barrier to having an effective DevSecOps. This challenge is followed by a lack of workflow integration and the growing vulnerability backlog.

In this research, we have defined DevSecOps (short for development, security and operations) as the automation of the integration of security at every phase of the software development lifecycle from initial design through integration, testing, deployment and software delivery.

At the heart of having a successful vulnerability management program is alignment between DevSecOps and the development team in being able to achieve both innovation and security when delivering products. Only 47 percent of respondents say their organizations’ development team delivers both an enhanced customer experience and secure applications and 53 percent of respondents are concerned that the lack of visibility and prioritization in DevOps security practices puts product security at risk.

Fifty-five percent of respondents say their development engineers, product security teams and compliance teams are aligned to understand their organizations’ security posture and each other’s area of responsibilities to deliver secure products.

The following are key takeaways from the research.

 The two primary reasons to adopt DevSecOps are to improve the collaboration between development, security and operations and reduce the time to patch vulnerabilities, according to 45 percent of respondents. In addition to improving collaboration and reducing time to patch, 41 percent of respondents say it automates the delivery of secure software without slowing the software development cycle (SDLC).

 Almost half of respondents say their organizations have a vulnerability backlog. Forty-seven percent of respondents say in the past 12 months organizations had applications that have been identified as vulnerable but not remediated. On average, 1.1 million individual vulnerabilities were in this backlog in the past 12 months and an average of 46 percent were remediated. However, respondents say their organizations would e satisfied if 29 percent of vulnerabilities in a year were remediated.

“This is a significant loss of time and dollars spent just trying to get through the massive vulnerability backlogs that organizations possess,” said Liran Tancman, CEO of Rezilion, which sponsored the research. ”If you have more than 100,000 vulnerabilities in a backlog, and consider the number of minutes that are spent manually detecting, prioritizing, and remediating these vulnerabilities, that represents thousands of hours spent on vulnerability backlog management each year. These numbers make it clear that it is impossible to effectively manage a backlog without the proper tools to automate detection, prioritization, and remediation.”

 The inability to prioritize what needs to be fixed is the primary reason vulnerability backlogs exist, according to 47 percent of respondents. A primary reason for the existence of backlogs is not having enough information about risks that would exploit vulnerabilities (45 percent of respondents) and the lack of effective tools (43 percent of respondents).

Forty-seven percent of respondents say their organizations have adopted a shift right strategy, which enables continuous feedback from users. Fifty-one percent of respondents believe the benefit of a shift right strategy empowers engineers to test more, test on time and test late.

Organizations are slightly more effective in prioritizing their most critical vulnerabilities than patching vulnerabilities. Fifty-two percent of respondents say their organizations’ prioritization of critical vulnerabilities is very effective but only 43 percent of respondents say timely patching is highly effective.

Vulnerability patching is mostly delayed because of the difficulty in tracking whether vulnerabilities are being patched in a timely manner. Difficulty in tracking (51 percent of respondents) is followed by the inability to take critical applications and systems off-line so they can be patched quickly (49 percent of respondents).

Automation significantly shortens the time to remediate vulnerabilities. Fifty-six percent of respondents say their organizations use automation to assist with vulnerability management. Of these respondents, 59 percent say their organizations automate patching, 47 percent say prioritization is automated and 41 percent say reporting is automated. Each week, the IT security team spends most of its time on the remediation of vulnerabilities. Sixty percent of respondents with automation say it significantly shortens the time to remediate vulnerabilities (43 percent) or slightly shortens the time (17 percent).

DevOps is an approach based on lean and agile principles to quickly deliver software that enables organizations to quickly seize market opportunities. Fifty-one percent of respondents say they have some involvement in their organization’s DevOps activities. As shown Fifty-two percent of these respondents say they are involved in vulnerability management and 49 percent of these respondents say they are involved in application security.

Certain features are important to creating secure applications or services. Sixty-five percent of respondents say the ability to perform tests as part of the workflow instead of stopping, testing, fixing and restarting development is very important and 61 percent of respondents say automating vulnerability, scanning and remediation at every stage of the SDLC is very important.

The inability to quickly detect vulnerabilities and threats is the number one reason vulnerabilities are difficult to remediate in applications. Sixty-one percent of respondents say it is very difficult or difficult to remediate vulnerabilities in applications. Why it is so difficult is because of the inability to quickly detect vulnerabilities and threats (55 percent of respondents), the inability to quickly perform patches on applications in production (49 percent of respondents) followed by the lack of enabling security tools (43 percent of respondents).

More than half of organizations focus only on those vulnerabilities that pose the most risk. Fifty-three percent of respondents believe it is important to focus on only those vulnerabilities that pose the most risk and not on remediating all vulnerabilities. Forty-nine percent of respondents say their organization remediates all vulnerabilities because it does not know which ones pose the most risk.

Testing applications and keeping an inventory of business-critical applications are steps that have been fully or partially implemented. To manage vulnerabilities, 45 percent of respondents say their organizations test the application for vulnerabilities using automation and 44 percent of respondents say their organizations have created and maintained an inventory of applications and assess their business criticality.

Software Bill of Materials (SBOM) is a list of components in a piece of software. Software vendors often create products by assembling open source and commercial components. The SBOM describes the components in the product. A dynamic SBOM is updated automatically whenever a release or change occurs. Forty-one percent of respondents say their organizations use SBOM. Risk assessment and compliance with regulations are the top two features of these organizations’ SBOMs. While 70 percent of respondents say continuous automatic updates are important or very important, only 47 percent say their SBOM features continuous updates.

 The growing software attack surface is a high concern.  Seventy-one percent of respondents say their organizations are very or highly concerned about risks created by the growing software attack surface. A higher percentage of respondents (77 percent) believe it is very or highly important.

Despite the concerns, most organizations are not effective in both knowing the attack surface and securing it. Only 43 percent of respondents say their organizations’ effectiveness is very high and only 45 percent of respondents say their organizations are effective in knowing the attack surface.

 Elimination of complexity and eliminate vulnerabilities that are exploitable are the most important steps to safeguard the attack surface. Sixty percent of respondents say the elimination of complexity in the software attack surface vulnerabilities that are exploitable (56 percent of respondents) will reduce threats to the attack surface. This is followed by knowledge of all software components (51 percent of respondents). Only 26 percent of respondents say regular network scans reduce threats.

To read the complete results of the survey, visit the Rezilion website.

Why am I holding this odd-looking sign in what looks a lot like a mug shot? Because my cell phone was stolen recently.  And the worst part of that experience has been dealing with …. Instagram.  As I’ve written before, poor customer service is actually a massive security vulnerability, and I think my story will illustrate that.  But if you don’t care for those details, at least scroll down to watch me and my dog struggle to submit a selfie video so I could attempt to regain access to @RustyDogFriendly. It’s worth the price of admission. (And, sadly, it did not work. Many, many times)

Many years ago I was scared straight on two-factor authentication when I was working on a documentary podcast about Russian hackers and I received notification from Facebook that someone in St. Petersburg, Russia, had tried to hack into my Instagram account.  I was already pretty careful with my work and banking accounts, but now I put two-factor on everything I could.

And I didn’t opt for the less-secure SMS text-message-code style two-factor. I went with the stronger token-based model.  I installed Google Authenticator on my phone and used its mathematically-generated codes as my second step when logging into all my various accounts. Even my @bobsulli Instagram, used mainly for my photography hobby, and @rustydogfriendly, where fans of my beloved golden retriever could get their fix of Rusty. (Long-time readers know Rusty has enjoyed his own time in the media spotlight from a story I wrote for the Today show).

That worked well until my cellphone was stolen while traveling last week.  Everyone understands the hassle that usually brings. I was actually fortunate. I have insurance, so after a $230 deductible, I received a replacement phone and all my data is backed up, so I didn’t really lose anything.

Except my sanity as I tried to log back into sites where I had employed Google Authenticator.  You see, there is no way to restore that. When your phone gets stolen, your token math is gone. There’s no way to import the old math into the new phone with out access to the old phone; at least none I am aware of. So every site which required an Authenticator code now required an alternative sign-in process.  The good news is: None of them were particularly easy. I wouldn’t want that! After all, what good is two-factor authentication if someone can just say, “I forgot my password” and sign in with a new one.

So I went through various alternative means of logging in…many involved using other gadgets or laptops where  I was already logged into these accounts and answering various questions.  Most sites that use tokens offer the chance to download a series of one-time backup codes designed for such an emergency, which I had done in most cases. Of course, many of the codes date back to those frantic moments five years ago when I was preparing for a possible Russian hack, so they weren’t necessarily easy to find.  But, I muddled through.

Until I got to Instagram.

I’ve written a lot recently about the problems Instagram is having with hackers. Well — in my view — the real problem is Instagram’s customer service failures. It’s easy to find horror stories about Instagram users who’ve had their accounts hijacked — then, those impersonator accounts are used for ongoing crimes, like crypto scams — and the victims are unable to even get the accounts turned off, let alone restored to the rightful owner.

So I shouldn’t have been surprised when attempts to log into my Instagram accounts with a brand new phone — and without my Authenticator — ran into roadblocks worthy of Fort Knox.  Let me be clear: I am glad Instagram makes it hard to log into my accounts from a strange new cell phone. Kudos to them for making this challenging.  But…when challenging becomes impossible, something else becomes clear. Their security implementation is a failure. And as a result, I can no longer recommend that Instagram users employ strong two-factor authentication, because you may very well be signing your account’s death warrant that way.

My @BobSulli account is much older, and I occasionally use it for professional purposes — I was among the first Instagram users, relatively speaking — so I dived into that problem first. I asked for a password reset at the email address on file. That worked. I tried to log in. I couldn’t without an Authenticator code.  I asked for an alternative.  I entered the backup codes I had. That didn’t work. I felt desperate.

I should note that every one of these interactions with Instagram came with a subject line “Hi BobSulli — we’ve made it easy to get back on Instagram.”

So I asked the software one more time — isn’t there any alternative?

I was then asked if pictures of myself were in the account. Of course!  So then I was told to make a “selfie video.”  Great!  Someone — or  something — was going to look at this video, compare it to the 500 other pictures of me, and override whatever system was blocking me from the account. Perhaps (hopefully?) after a phone call or some other final check of who I was.  Following the instructions, I looked right, looked up, looked left….and then submitted it.  I was told it might take three or four days. Bummer, but worth it to have this piece of the puzzle solved!

Within minutes, I had my answer.

“We weren’t able to confirm your identity from the video you submitted. You can submit a new video and we’ll review it again,” a sad email said.

That was fast, I thought. It’s probably a machine.  So I set about running around my home trying to find the same lighting as my profile photo. Even the same suit jacket I wore.  I submitted several selfie videos over the next few hours.  All of them were rejected.

The only other alternative I was offered was ….no alternative.  Just a link to a help center page that, predictably, offered no help at all.  I was at a dead end.

But then I had one more thought.  I had an old iPod touch. Perhaps I had logged into my account from that device and Instagram would recognize it through device fingerprinting or something similar and I’d at least get a different alternative.  Bang!  This time, when I entered my password, failed the Authenticator test, and begged for help, I was presented with a form to fill out. I did so, and received a hopeful — if odd — response in email.

“Thanks for contacting us. Before we can help, we need you to confirm that you own this account. Please reply to this message and attach a photo of yourself holding a hand-written copy of the code below…. 6XXXX…Please make sure that the photo you send includes the above code hand-written on a clean sheet of paper, followed by your full name and username …Clearly shows both the code and your face.”

So, that’s where the mugshot photo above comes in. I sent it in.  To shorten the story a bit, that worked.  Withing a few hours, I had access to @BobSulli!  That was the hard part, I thought.  There was just one more thing to do to recover from the awful experience of having my smartphone stolen — recover my dog’s account, @rustydogfriendly.

And that, dear readers, has proven to be my Waterloo.

Because when the time came to submit a selfie video for his account…that didn’t go quite as well, as you can see in the embedded YouTube video below.

I know what you’re thinking: I’ve already tried a selfie video of just me. That didn’t work either. I’ve tried about 10 different variations. Each time, the video is rejected. I actually had a reasonable dialog with the folks who helped me log into @BobSulli over email. I pleaded with them to look at @rustydogfriendly. The accounts are linked in both their bios!  It’s obvious we are connected! I sent a list of pictures with both me and him together! The person(s) on the other end of the keyboard kept telling me they could only help with my @BobSulli account. I begged for an alternative. The “line” went dead.

So I am stuck in a perpetual loop, as the geeks say. I log in, I’m asked for an Authenticator code, I say I don’t have it, I’m asked for a backup code, I try it, it fails, I ask for an alternative, I’m asked to make a selfie video, it fails, and then my only option is to make another selfie video.

The backup codes I have for @rustydogfriendly were downloaded the day I opened the account. Why don’t they work? I don’t know. Perhaps they have expired.  I don’t recall ever using them, but who knows. It was four years ago.

So I am stuck. Rusty is usually a big hit on Halloween, but I’ve now written that holiday off.  Perhaps there is some other route to logging in that I’ve missed, and for that I’m sorry, but I believe I’ve taken every step a reasonable consumer would take in my situation — and a few that only a cybersecurity journalist would take — and I have nothing but a zombie account to show for it.  And a belabored blog post that many of you probably have not finished reading.

But I belabor the point because it’s important: when security isn’t accompanied by customer service, it’s a failure. Poor customer service is, I believe, our greatest security vulnerability. Two-factor authentication is ESSENTIAL.  Many people use text-message-based authentication, and I’ve been telling anyone who will listen that it’s now failed. It’s too easy for criminals to intercept those texts or obtain them in other ways.  So I’ve been urgings banks and other institutions to force consumers into Authenticator or other software-based tokens instead. They are much safer.

But I can no longer do this in good conscience. Because if there is no plan for consumers who lose access to their phones, there is no plan.  I can’t tell you how much of the weekend I spent explaining to various websites — “No, I can’t verify myself via text because….I don’t have texts right now.” And when there is no alternative, the implementation is broken.

Thanks to Instgram, I will forever be gun-shy about two-factor authentication now. And the more stories like this that you hear, the more you will be inclined to turn it off, too.  After all, which risk is higher — a criminal hacking your account or a corporation blocking your access to the account?

Note that I *was* able to restore my other accounts, so clearly the problem is fixable. I will continue to use two-factor everywhere I can. And you should too.  But know this: If Facebook were required to answer the phone — virtually or otherwise — these situations would not arise. Poor customer service gives security a bad name, and that puts us all at risk.

Meanwhile, if anyone has any suggestions for getting back into my dog’s account, I’m all ears.