➡ On July 20, 2023, the PRCA’s PROCOM experienced a ransomware hack resulting in a complete shutdown of its computer systems, including member accounts and online entry systems, leaving contestants and contract personnel unable to access critical data and communication with headquarters.
➡ The ransomware attackers demanded more than $1 million to unlock the PRCA’s systems.
➡ Rodeo secretaries have resorted to manual methods and social media to manage rodeo operations and ensure events’ success.
Thursday, July 20, 2023, probably started normally enough for most members of the Pro Rodeo Cowboys Association (PRCA). For contestants, there were horses to saddle, drives to make, fees to pay and a slice of nearly $3 million to win during one of rodeo’s busiest weeks; for contract personnel, there were rodeos to produce, stock to feed, draws to do, and results to process.
But very quickly, everyone in ProRodeo was hit with the news that the PRCA’s computer systems, including member accounts through ProRodeo.org, online entries through PROCOM and the PRCA’s Secretary System, were down and unavailable for use due to a virus.
Even email and phone systems, using Voice over Internet Protocol (VoIP), were affected initially and the rodeo world collectively began to scramble to ensure seamless production of scheduled rodeo performances while operating without any communication to rodeo headquarters.
A Demand for Payment
Matt Reeves, PRCA board member and steer wrestling event rep, knew something was wrong on Wednesday night.
Reeves had attempted to log in to his member account at ProRodeo.org to pay entry fees but was getting nowhere.
“I was going to call the next morning but I had an email before I could call saying the system was down,” Reeves noted.
More than a random system outage, the paralyzation was coordinated.
“We were attacked by ransomware,” PRCA Director of Rodeo Administration Steve Knowles said.
Ransomware is exactly what it sounds like: malware (at its most basic level, a computer virus) designed to hold an organization’s systems and data hostage until hackers’ financial demands are met. In this case, the hackers wanted more than $1 million to give back the key to unlock the PRCA systems.
According to numerous surveys done by groups like Statista and Barracuda, just over 70% of businesses worldwide have been affected by ransomware. One of the most notorious ransomware attacks was the 2021 attack on the Colonial Pipeline Company in the Southeastern United States, which shut down all activities and cost the company more than $2 million.
While many corporations have dealt with the growing threat in escalating terms since COVID, it may seem odd that hackers would target a non-profit but stats show that non-profits are a surprisingly common target. One reason: their security protocols are often less strident than their for-profit counterparts.
The truth is any company possessing data necessary to its operations is a potential target because the overriding goal of hackers is to extort money from organizations that must have access to their data in order to stay in business.
The New Built on Old
ProRodeo’s IT systems are actually several systems built to interface with one another. Membership information is housed on the AS/400, a system built back in the 1980s.
New systems are the member portal found on ProRodeo.org, which also includes the online entry system (the new PROCOM, replacing the former call-in system), and the Secretary System, which processes rodeos. Both were built to interface with the AS/400—in laymen’s terms, these systems talk to the AS/400 to give it information and to receive its outputs.
PRCA servers, the systems which manage all the network resources, are in Phoenix, which as Knowles points out, are a long way away.
When all systems are working properly, contestants can log into their member portal, enter rodeos, pay entry fees and see callback information and trades lists. Rodeo secretaries can download rodeos they are working through the Secretary System and use the information and software to complete stock draws, process results and figure payouts and advancements to short rounds, amongst other things.
All contestants, PRCA and barrel racers and breakaway ropers who are Women’s Pro Rodeo Association (WPRA) members, pay $21 per rodeo for PROCOM-related services. All PRCA members, including secretaries, also support tech systems through their annual membership dues.
This was the second disruption of service in the last month. The member portal and PROCOM services also briefly went down during the Fourth of July. Whether that issue was related to the most recent attack has not been stated.
When the ransomware hit on July 20, the PRCA members’ portal was frozen, meaning no one could see their own accounts or perform any PROCOM functions. No turning out, no finding and processing trades, no entering. At the same time, secretaries could only see rodeos they had previously downloaded but didn’t have the ability to upload, meaning results weren’t going to ProRodeo.com and money wasn’t being added to standings.
“It’s terrible,” Reeves noted, “but we’re all going to live through it.”
He noted that vulnerabilities in the system had been brought to the PRCA Board in recent meetings and steps had begun to close known loopholes in the systems. Ransomware attacks can commence through several fronts including phishing emails and drive-by downloading, when a user unknowingly visits an infected website and malware downloads without their knowledge, and also through third-party vendors.
“The hackers were just faster than us,” Reeves said. “But honestly, I don’t know that this could have been prevented.”
Complicating matters further, PRCA Chief Technology Officer Gordon Knopp recently underwent open heart surgery.
Social Media and Secretaries to the Rescue
With PROCOM unable to handle even phone calls, at least initially, rodeo secretaries were directed to begin accepting releases and turn-outs. In Salt Lake, where ground rules allow the rodeo secretary to fulfill 30-hour replacements, secretary Eva Chadwick and her staff took on that duty.
“When they just gave us the responsibility of taking our own turnouts and notifications, and handed the whole process of calling for replacements over, it was less frustrating than not being able to get through to the central entry office,” Chadwick noted. “Contestants were great and figured out how to reach the secretaries. Dozens and dozens of calls for replacements made our event a tremendous success, with our troubles hopefully invisible to the public.”
@teamropingjournal Gather round kids, and let Jake Barnes & Clay O’Brien Cooper tell the tale of entering @PRCA ProRodeos before the days of the online portal, and even before the invention of the cell phone. #payphones #rodeoing #oldschool #cowboys #prorodeo #jakebarnes #clayobriencooper ♬ Western Music: Western Ballad – Luis Bacalov
However, it felt like chaos compared to the conveniences upon which most have come to rely and added to an already busy workload for those processing the record-keeping function of the rodeos.
“Honestly, probably who was most stressed, or should have been, was the secretaries,” Reeves agreed. “It was harder for them to get their releases and turnouts and they couldn’t upload their rodeos.”
Contestants, many of whom feel they were paying for services they suddenly weren’t receiving, began calling secretaries looking for information.
“I think just having to answer so many more phone calls was really inconvenient for the secretaries too,” Reeves added.
“It’s been difficult,” ProRodeo Hall of Famer Sunni Deb Backstrom said. In addition to many years as secretary of the Wrangler National Finals Rodeo, Backstrom also currently represents secretaries and timers to the PRCA Contract Personnel Executive Council. She was in Nampa when the attack hit.
“I know how to secretary by hand, I’ve been doing it all my life, but we had more time and fewer requirements then,” she noted. “It was really tough on my secretaries who have never done a rodeo by hand.”
The fact that the attack came in the midst of huge rodeos in Nampa (Idaho), Ogden, Spanish Fork and Salt Lake (Utah), Cheyenne (Wyo.) and Salinas (Calif.) complicated things further with short go rounds for which the rulebook allows contestants conflicted to be held over. Without the help of the central headquarters, secretaries began using social media to inform competitors who was making short rounds and working together to share information allowing secretaries to hold and move contestants as rules allowed.
“I truly feel bad for the secretaries,” Backstrom said. “I feel bad for me, for goodness sake. I’ve had a rodeo every day since July 5 and got two and three a week coming up.”
“But I have to say, the secretaries have been rockstars,” she continued. “They didn’t just throw their hands in the air and say, ‘you figure it out.’ I’m really proud of all of them.”
Re-Opening Entries
By Monday, July 24, PROCOM’s phone service was restored and rodeo entries were prioritized with call-in service only, returning to what was normal practice just a few short years ago. As staff was flooded with calls, many contestants found themselves on hold for long periods of time. Social media memes began to surface, poking fun of what is now seen as an antiquated method.
“We were running PROCOM like we did in the 1970s and 1980s except back then we had 25 or 30 people working,” Knowles said. “It’s like 10 to 12 now so half the staff we had back then.”
Lack of staff was just one issue. Another was that the queue for keeping people on hold was set up for 40 people.
“Obviously, we had more than 40 people calling in at once,” Reeves noted, explaining a common complaint that contestants were listening to messages noting how many callers were ahead of them only to be apparently kicked back to the end of the line.
Even with entries being accepted, there was still only one system working, the AS/400, into which the entries could be input and processed.
“We were taking entries on paper,” Knowles said. “It’s been a big challenge for us, keeping up with the rodeos.”
“I’ve been taking entries,” he added with a laugh. “People can’t get through so the ones that had my number called me. So I just take their info and card number and walk it down there.”
Entries that were open online at the point of the attack were re-opened and competitors were directed to enter again.
“Some of that information started to show back up but we knew there could be lapses in the data that we didn’t know about so we just had everybody enter again,” Knowles said.
All callback information and day sheets were pushed to RodeoSetUp.com, a website whose use was abandoned several months ago but thankfully is still operational. While the solution proved a decent workaround for most, no breakaway roping draws were posted for several days, again providing fodder—some of it quite funny—for social media.
Paying Out (and Getting Paid)
Somewhere in the midst of the busy weekend, competitors began to worry about getting paid their winnings and about personal data being leaked.
One silver lining is that the end game for most groups behind ransomware is the extortion of funds from the organization, either by holding data hostage or threatening to leak it. A smaller percentage are looking to resell information.
“This was a data hostage outfit,” Knowles confirmed. “We have no evidence at this time that personal information was compromised though it’s always a good idea to keep an eye on your financial information.”
As far as payouts go, the PRCA uses RodeoPay, a third-party vendor who was unaffected by the attack. The problem was getting results from the secretaries and getting the information to RodeoPay.
Another concern? With the portal down, members were no longer paying fees online, which is much more common than the practice of paying the rodeo secretary. A big chunk of rodeo payouts come from entry fees.
“We had enough cash reserves to make it work,” Knowles said. “We had to figure a workaround to figure out how to get money into people’s accounts.”
“The problem is that secretaries can’t upload so we’re getting paper sheets and we’re relying on that data to pay off,” he said.
Normal auditing processes were shortened to ensure that payouts for major rodeos could go out as expected on Tuesday and Wednesday.
Coming Back
With PROCOM using old-school methods to take entries for high-priority rodeos (those happening soon), work was being done to get other systems operational.
Knowles noted that the member portal was reopened Friday, July 28 at noon, making a small group of rodeos available for online entries again.
“Our IT team has done a fabulous job,” Knowles said. “They’ve rebuilt our systems, building them all over again and we’ve been able to recapture a lot of our data.”
The turnaround is not typical in ransomware attacks. Most surveys cite 22 days as the average recovery time. The 2023 North America Executive Summary data notes that 19% of organizations who paid ransom still couldn’t recover their data and that trends point to more than half of all organizations facing subsequent attacks in the future.
“We’ve built better security systems into it this time, there are extra security measures,” Knowles noted. And the reopening is scheduled to start slow with only a handful of rodeos opening Friday and closing Monday and those being mostly one or two-perf rodeos with uncomplicated entries.
“We hope it won’t crash this weekend,” Knowles said. “But we’re excited to see the light at the end of the tunnel.”
At Least One More Week of Manual Processes
Though the member portal and online entries are coming back, the final weekend of July will still be rough as the Secretary System is not functional yet. Until it is, results and standings may not be updated on ProRodeo.com.
“They got Nampa uploaded from my system so we’re getting very, very close,” Backstrom said.
Entry information configured through the AS/400 is being sent to secretaries working this weekend and the PRCA’s IT department is finding workaround solutions to get the information where it needs to be. In many cases, this is the same as the information (long sheets) posted to RodeoSetUp.com and the secretaries are going from there in order to manage their rodeos, either just writing in information by hand or creating spreadsheets to better manage the data.
Because many tournament-style rodeos don’t process accurately through the system anyway, Backstrom noted she has been able to provide some help to ease the workload.
“Because I do Denver and San Antonio and all of those, I have a packet of forms I use and I’ve been sharing that,” she said, adding she’s had an extremely high volume of calls from secretaries across the country. “I’ve told them that’s all I can do but call me with any questions and just do your best.”
For now, social media is still the go-to for many competitors looking for slack orders and draws. In addition, no entries taken via phone are going to appear in the member portal.
“We can’t get those callbacks on the portal,” Knowles said. “They’re still going to have to go to the other site [RodeoSetUp.com].”
Lessons Learned and an Old School Champion
As it turned out, the AS/400 system was the saving grace as the only system not affected by the attack.
“It’s the oldest but it was also the most secure,” Knowles said.
Whether a testament to the lasting power of a relatively old technology or not, the PRCA seems to have dodged a major landmine in this case.
Industry standards for nearly all businesses include a focus on redundancy and security as well as business resumption and contingency/continuity planning (BRCP) precisely because of the growing threat not only of ransomware but all cyberattacks and system failures. One of the most common BRCPs is to ghost or mirror an organization’s entire system to a separate server, which is regularly tested so that the organization could flip to the backup in the case of an attack.
Certainly, the PRCA Board and leadership team will have plenty to discuss as they examine this incident and plan for future security.
Just as many on social media posted about the downfalls of modern conveniences in light of the experience, Knowles noted that while some more experienced folks, like himself, remember the old ways of calling in to enter, including from pay phones and keeping track of callbacks on a paper calendar, many competitors, and PRCA employees, were a little lost without the benefit of computers.
“We found out a lot of people didn’t even know the PROCOM phone number.”
“This has been an eye-opening experience,” Knowles said, “about how easy we probably have it every day.”
But Knowles adds a bit of his perspective.
“This is definitely the worst disaster to hit PRCA systems,” he said. “I say disaster but nobody died. It’s just things we have to work through.” TRJ
