Big data's biggest obstacles – and how to overcome them

In 2016, global technology analyst Gartner pessimistically predicted that 60% of the world’s data initiatives would fail. By the end of last year, this number had been revised up to 85%.

While it’s important to fail fast, no one wants to fail often. How can local governments overcome the many obstacles they face as they become data-driven organisations?

Building a better back-end
Most councils face a similar core challenge: back-end systems. Improving those back-end systems can be stymied by the data quality and complicated by the vast array of services that any council undertakes.

“Local governments have limited experience working with big data, since most of our traditional datasets are largely limited to the number of properties and residents,” says Dr. Adam Mowlam, who manages Wyndham City’s Smart Cities Office.

“Traffic volumes, people movements, video analytics and social media sentiment analysis are all key big data sets which could inform policies and advocacy strategies. But without growing skills in big data and using tools such as artificial intelligence, we won’t realise the full benefits.”

Chris O’Connor, Manager of Digital and Data at the City of Casey in Victoria, says the first step is to address what he calls “the mechanics of data” – how to capture it, where to store it, how to refine it and how to analyse it.

“The mechanics are relatively straightforward to address, as long as you have the technical knowledge to identify the products that really solve your problems,” O'Connor says.

“Straightforward doesn’t mean cheap, quick and easy – it is extremely hard to bring together data from different source systems and build data hubs to enable analysis at scale – but there are proven market solutions that can give us a head start,” Chris adds.

Breaking down the silos
While breaking down information silos “may not be the most exciting or shiny aspect of the data world," it's nonetheless essential, says Darryl Ellis, Logan City Council’s Business Solution Architect.

“Our experience so far has shown us we need to build a solid foundation of information management, and to treat information as a strategic asset,” he says.

Ellis and his team are working hard to “strike a balance between identifying the gaps in our information management and also delivering working prototypes of advanced analytic techniques."

This approach, he hopes, will inspire people with the possibilities for big data, while also ensuring “knowledge transfer and a shared understanding of what is necessary from both people and processes."

O’Connor agrees that cultural change is the hardest obstacle to overcome.

His team faces several cultural change challenges, including “building appreciation of data governance responsibilities as a part of, rather than an addition to, one’s job” and “getting people to think outside of the service verticals and understand the benefits of data collection to other use cases,” he says.

“We are working with people to change our processes and practices, so data can be incorporated into all our decision making. We are helping people to choose the right projects that deliver real benefits – and to prioritise useful data projects ahead of interesting projects. And we are also trying to shift perceptions of the legal framework as an enabler of data sharing and use, rather than as a blocker. But it’s all a work in progress.”

Connecting the dots
Northern Beaches Council is collecting a large amount of data to provide better services and adapt to changing needs, says Chief Information Officer Nathan Rogers.

Rogers says one of the greatest challenges for councils is identity verification of people who sign up for online services, and points to the work of the Australian Government’s Digital Transformation Agency as an important step forward.

“Without access to identify verification data from sites like MyGov, it’s very difficult for us to do. We see there is an opportunity to link services with state and federal agencies through a data transfer agreement. This would reduce the number of times customers have to interact with governments at all levels.

“Imagine, for example, automatically receiving a parking permit replacement when you register a new vehicle – this is what linking up services could do.”

Shrinking the skills gap
The local government sector is beginning to grasp the potential of data analytics, but it’s early days, say our smart cities chroniclers.

“Executive level decision makers are beginning to see the value of data analytics, and understand how it can unlock digital transformation programs,” says Dr. Mowlam.

“But a lack of skills and experience with emerging technologies can limit innovation,” he adds..

“Animal patrol, for example, may advocate for digital forms to monitor community issues related to a dog barking, whereas a basic sound sensor would be a more robust solution in terms of true digital transformation,” he explains.

We also need to grow the pool of talented data analysts. “The pay and organisational structures of local government are generally not designed to cope with the starting salaries of data experts,” Dr. Mowlam  says.

Chris O’Connor agrees. Building the capability of the organisation across the whole value chain from collection to use, is a huge challenge, but "not impossible."

The secret? “Executive support and funding to bring in or build experience, creative thinking, commitment to improved outcomes, plus a willingness of the IT and information management teams to modernise systems and processes.”