RICHMOND, Va — Leaders at Richmond's Department of Public Utilities (DPU) raised "urgent" concerns amid the flawed rollout of its new billing system but seemingly struggled to get help from a vendor, according to internal communications reviewed by CBS 6.
Over the summer, many DPU customers expressed frustrations with their utility bills— reporting issues such as inaccurate balances, confusing account information, and the inability to access payment portals.
PAST COVERAGE: Richmond restaurant owner says a DPU billing error automatically pulled $60,000 from her account
The issues came as DPU was transitioning to a modernized cloud-based billing system. The city partnered with vendors Oracle and Kubra on the nearly two-year long multi-million-dollar project called "Project Synergy."
To try and better understand what went wrong, CBS 6 submitted Freedom of Information Act (FOIA) requests to the city seeking communications between DPU leadership and its vendors from May through August.
Here's what we found based on the hundreds of pages of internal emails that were released.
DPU escalated 'urgent' concerns but struggled to get help
Just before the new system was planned to go live, emails showed DPU had learned its bill print and payment services contractor Kubra would not be available to help ahead of launch day.
It appeared to be a shock to the department.
"We must have Kubra available to support our implementation on Monday May, 25th as was agreed upon and assured to us well in advance of today," a DPU employee wrote in an email to a Kubra representative on May 21.
A Kuba representative responded the next day saying she was able to "secure an exception approval to support the Monday, May 25 go-live activities as originally planned."
DPU spokesperson Rhonda Johnson told CBS 6 Kubra ultimately provided adequate resources to support the launch.
On the launch date, May 26, error messages started pouring in. As of 10 a.m. that day, DPU Senior Deputy Director Jamie Porter said she had received 80 such alerts.
But the problems seemed to grow over the coming days and weeks.
MORE: Richmond DPU misses its own deadline to fix billing system problems
By June, at least 18 defects were documented in a list of billing issues; many of them rated "high" in terms of impact.
Among the defects: multiple meter reads appeared on the same bill, inaccurate information on bills, missing bills, and autopay profiles not being activated.
On June 11, DPU staff told a Kubra representative that DPU Director Scott Morris wanted to speak with them about "billing items we have been seeing since go-live."
On June 16, a DPU deputy director told a Kubra representative that the call was "urgent" and needed to happen that same day.
On June 22, Director Morris emailed the Kubra representative, highlighting what he said appeared to be a "communication constraint." He said he needed to speak with Kubra's "executive leadership" about "ongoing issues."
Then again on June 24, DPU Senior Deputy Director Jamie Porter wrote that DPU was still trying to get in touch with Kubra.
Kubra President Rick Watkin responded later that evening to Porter's email saying he would make himself available "immediately."
The email communications showed it took 13 days for DPU to get someone at Kubra to agree to discuss the "urgent" concerns.
On June 26, Kubra's Senior Vice President of Client Experience Steve Ryan apologized to DPU leadership and acknowledged the scope of the problems DPU was facing.
PAST COVERAGE: Richmond sent this man a $20,000 water bill: 'It’s stressing me out'
"I sincerely apologize that these issues have reached this level of severity and have impacted both your team and your customers so significantly. We deeply value our partnership and appreciate you giving us the opportunity to step in, align the necessary resources, and resolve these matters as quickly as possible," Ryan said.
According to his email, 25% of DPU's customer base had been affected by disruptions.
Ryan said Kubra would prioritize tackling critical issues such as account lookup failures, billing data integrity, payment posting discrepancies, and bill print delays.
CBS 6 asked both the city and Kubra about the apparent communication issues.
DPU spokesperson Rhonda Johnson told CBS 6 the department needed to speak with Kubra related to "ongoing issues that needed escalation" and that the "resources available" to DPU were "not adequate to address the concerns." She confirmed the "communication constraint" Morris referenced in his email was related to how long it took to get someone from Kubra to agree to speak with DPU about its concerns.
Kubra Vice President of Marketing Alison Copeland told CBS 6 in a statement, "Once the need for additional resources was identified and committed from our executive leadership, we moved quickly, standing up daily working sessions with the City and our project partners, as well as a shared tracker to manage the work across vendors and the City."
Billing issues continued to mount
Over the weeks that followed, the emails revealed mounting problems the teams worked to address including:
- Adjustments not reflecting on bills
- Customers seeing confusing balances on their old accounts
- Customers being incorrectly shown pre-collection notices
- Payments being sent to a Canadian server instead of a U.S. server
On July 21, DPU requested a pause on customer billing until bills could be validated. Johnson told CBS 6 the pause lasted for one week.
At the end of July, the emails showed Richmond Mayor Danny Avula, Chief Administrative Officer Odie Donald, and DPU Director Scott Morris wanted to put out a public news release addressing the billing failures.
DPU solicited input from Oracle and Kubra regarding the messaging of the news release.
A representative from Oracle responded that his understanding was that the Oracle team "has been on the periphery of these issues" and that the city and Kubra were the ones leading the troubleshooting.
Morris appeared to push back on that, responding, "For clarity, the last issue was directly related to Oracle and impacted over 4,000 customers."
When asked to identify the Oracle-related issue, Johnson told CBS 6 that a "final product change was pushed to production prior to verification."
CBS 6 reached out to Oracle for comment on this exchange, but we have not heard back.
DPU accepted statements from both vendors and published them in its press release. Emails showed some issues continued into August.
On August 6, a representative from Oracle said more than 5,700 customers still had not received a bill.
That same email said DPU had, at that time, about 10,000 billing issue items on its to-do list, which was down from at least 15,000 items the previous week.
On August 12, a new issue was flagged as a "recurring pain point": customers with closed accounts were not able to pay their bills online. DPU was able to implement a change fairly quickly to prevent future similar problems, the emails showed.
The timeframe for CBS 6's records request was May 1 through August 12.
DPU: 'These records reflect DPU’s commitments to our customers'
Johnson told CBS 6 that a majority of the issues referenced in this story were related to the system transition, but not all of them.
WATCH: Richmond city leaders outline new DPU billing error fixes
She said no customers were charged late fees associated with the transition or disconnected during it. She said all meter readings remained accurate, and if any customers have concerns, their accounts can be audited for accuracy.
As for the payments routed to a Canadian server instead of a U.S. server, Johnson said that issue did not cause any payments to be misapplied.
When asked which parties were mostly responsible for the rollout problems, Johnson said "it was a combination of all parties" but that the city is ultimately responsible for its customers.
Alison Copeland from Kubra said she could not assign responsibility of the issues with specificity as "responsibility on a project of this scope is rarely attributable to a single party."
"Implementations of this scale and complexity, involving multiple partners, can surface real challenges, and this one did," Copeland said.
CBS 6 asked the same question to Oracle, but we have not heard back.
Johnson said of the documents referenced in this story, "These records reflect DPU’s commitments to our customers and demonstrates our commitment to working together to enhance system performance, improve reliability, and work toward providing the modern, dependable utility service Richmond customers expect and deserve."
This is a developing story, so check back with WTVR.com for updates. Email the CBS 6 Newsroom if you have info, photos or videos to share.
