Supporting residents with a modern data platform

Challenge

Hackney Council wanted to better harness the power behind its data to modernise and improve services for residents.

Hackney’s data was in multiple sources, including hidden in legacy technology and modern SaaS applications. They were finding it challenging to use it for analysis. Working with their team we delivered a cloud platform. Designed to help the council maximise its data-driven insights and deliver the best results for residents.

Creating a modern data platform allowed the council to store and analyse its data all in one place. Through this new holistic view of its residents, Hackney could help them live healthier and happier lives.

Hackney council area (Image: Google earth)

Our approach

The council wanted to focus on collaboration, sharing information and awareness of the available data. We worked with them to build a data lake, sorting captured data into 4 tiers: landing, raw, refined and trusted.

We didn’t want Hackney’s engineers to spend all their time maintaining the platform after we left. So we avoided doing anything overly complicated or bespoke.

Hackney, like any local authority, is responsible for delivering a wide range of services. It uses a range of applications to run them. This means it has many different data sources: databases, APIs, CSV files and Google Sheets, amongst others.

The main challenge was the sheer volume, with some services holding huge amounts of data. For example, Hackney’s parking system alone holds millions of rows of data and is updated daily.

Alongside this modern data platform, we consolidated their housing repair data. This allowed further insight and analysis and was used by Hackney’s Link Work team to reach out to support residents at higher risk.

The council wanted the new platform to encourage re-use across the public sector. Hackney’s strong vision of helping other authorities use their own data in better ways was key. We therefore built the platform using AWS infrastructure and open-source technologies. Other councils could then easily view and aim to replicate Hackney’s insights.

Results we achieved together

For the data platform to be sustainable we knew that Hackney’s staff needed to be confident using it. That’s why, as well as the infrastructure of the data platform, we focused on training and documentation. When we left, Hackney’s analysts were able to work independently, adding data to the platform and processing it according to their needs. This was key to make sure it didn’t become another piece of legacy technology.

Wanting to build a culture that recognises the value of data, we trained Hackney’s analysts in Spark, a data processing tool used by the core team. One of our engineers spent time working with the analysts to develop machine learning solutions to some sticky issues. This included Entity Resolution (for example, detecting that “10 Downing Street” is the same as “10 DS”), which we hope will have a huge impact. We also created an in-depth playbook that captured how to use the new platform which they could use for future projects.

Stepping away from legacy technology, the new system helps the team to make sure all data is accessible. This gives them the best chance of reaching residents using more than one service. We combined this with robust data governance to make sure that data can only be accessed by the people that need it – and for the right purpose.

One large benefit of this new data-driven design is cross-collaboration. For example, by combining data on property repairs with a vulnerability index built during the pandemic, Hackney could predict cases where vulnerable people needed repairs. They could then contact elderly residents who hadn’t been heard from in properties that likely had repairs due. A third of the residents contacted by the team were found to have unmet needs. They were offered support, advice and referrals to other vital services including adult social care, financial inclusion and voluntary organisations.