Supporting residents with a modern data platform
Encouraging collaboration, better visibility of data and sharing information through a new platform at Hackney Council.
The project
In its mission to modernise and continue to improve services for residents, Hackney Council wanted to harness the power behind its data.
The data was held in multiple sources, including legacy and modern SaaS applications. We worked closely with the team to design and deliver a cloud platform. This would help the council maximise its data-driven insights to deliver the best results for residents.
Creating a new platform would allow the council to store its data, consolidate and analyse it all in one place. Hackney could then develop a holistic understanding of residents’ needs. Which would in turn help residents to live healthier and happier lives.
Our approach
The council wanted to focus on collaboration, sharing information and awareness of the data available. And so we helped to build a data lake, sorting captured data into 4 tiers: landing, raw, refined and trusted. On top of this we used Apache Spark to add a computation layer, using AWS’s Glue serverless Hadoop capability.
We tried to avoid doing anything overly complicated or bespoke. We wanted Hackney’s engineers to not have to spend 5 days a week working on the platform once we’d stepped away.
Hackney, like any local authority, is responsible for delivering a wide range of services. It therefore uses a range of applications to collect its data, which meant there were many sources from which to pull data into the new platform. This included databases, APIs, CSV files, Google Sheets and various other bits and pieces.
The major challenge was the sheer volume, with some services holding huge amounts of data. For example, the parking system alone holds millions of rows of data and is updated daily.
We also did a lot of work with Hackney to consolidate its housing repair data, which was passed for additional processing and analysis. This dataset was then used by Hackney’s Link Work team so they could reach out to support residents at higher risk.
Hackney wanted to make sure the new data platform would enable reuse across the public sector. This was done with the vision of helping other authorities use their own data to deliver excellent public services. So we used AWS infrastructure and open source technologies to build it.
The results
We also built infrastructure and documentation. This would allow any Hackney analyst to put data into the new platform and process it for whatever needs they have. This was essential to the scalability and sustainability of the platform.
The new system supports the team to make sure all data is accessible, giving them the best chance of reaching residents using more than one service. This is combined with robust data governance to make sure data can only be accessed by the people that need it – and for the right purpose.
One large benefit we saw in this new data-driven design was cross-collaboration. Namely through joining consolidated repairs data with a vulnerability index built during the pandemic. This allowed the team to contact elderly residents that hadn’t been heard from in properties that probably had repairs due.
The result was that a third of the residents the team contacted were found to have unmet needs. They were offered support, advice and referrals to other vital services. These included adult social care, financial inclusion and voluntary organisations. This was a high success rate compared to other outbound campaigns. Even those who didn’t need support at the time were grateful that the council was checking in on them.
We wanted to continue to build a culture that recognises the value of data. And so we upskilled Hackney’s analysts on Spark, a data processing tool that the core team uses heavily. One of our engineers spent time working with the analysts to develop machine learning solutions to some sticky issues. This included Entity Resolution (for example, detecting that “10 Downing Street” is the same as “10 DS”), which we hope will have a huge impact. Finally, we also designed an in-depth playbook to support the team to use the new platform going forward.
Image: Google Earth
Case studies
Building a housing repairs service for easy reuse across the UK
Building a simpler, faster service to report housing repair needs using the GOV.UK Design System in the Microsoft Azure Cloud.
Making NHS Gloucestershire’s mental health services more accessible to children and young people
Made Tech worked closely with the NHS in Gloucester to digitalise access to over 100 mental health support services.
