Supporting residents with a modern data platform
Encouraging collaboration, better visibility of data and sharing information through a new platform at Hackney Council.
The project
In its mission to modernise and continue to improve services for residents, Hackney Council wanted to harness the power behind its data. The challenge was that Hackney’s data was in multiple sources, including legacy and modern SaaS applications. We worked closely with the team to design and deliver a cloud platform to help the council maximise its data-driven insights and deliver the best results for residents.
Creating a new platform would allow the council to store, consolidate and analyse its data all in one place. Through this new holistic view of its residents,’ Hackney could help them live healthier and happier lives.
Our approach
The council wanted to focus on collaboration, sharing information and awareness of the data available. And so we helped to build a data lake, sorting captured data into 4 tiers: landing, raw, refined and trusted. On top of this we used Apache Spark to add a computation layer, using AWS’s Glue serverless Hadoop capability.
We tried to avoid doing anything overly complicated or bespoke. We wanted Hackney engineers to not have to spend 5 days a week working on the platform once we’d stepped away.
Hackney, like any local authority, is responsible for delivering a wide range of services. It therefore uses a range of applications to collect its data, which meant there were many sources from which to pull from. This included databases, APIs, CSV files, Google Sheets and various other bits and pieces.
The major challenge was the sheer volume, with some services holding huge amounts of data. For example, Hackney’s parking system alone holds millions of rows of data and is updated daily.
We also did a lot of work with the team to consolidate housing repair data, which was passed for even more processing and analysis. This dataset in particular was then used by Hackney’s Link Work team so they could reach out to support residents at higher risk.
Hackney wanted to make sure the new data platform could encourage reuse across the public sector. This was done with the vision of helping other authorities use their own data to deliver excellent public services across the UK. So we used AWS infrastructure and open source technologies to build it.
The results
Finally, we built infrastructure and documentation. This would allow any Hackney analyst to put data into the new platform and process it for whatever need they had – essential to the scalability and sustainability of the data platform.
The new system supports the team to make sure all data is accessible, giving them the best chance of reaching residents using more than one service. This combined with robust data governance makes sure data can only be accessed by the people that need it – and for the right purpose.
One large benefit in this new data-driven design is cross-collaboration. Namely through joining consolidated repairs data with a vulnerability index built during the pandemic. This allowed the team to contact elderly residents that hadn’t been heard from in properties that probably had repairs due.
The result was that a third of the residents the team contacted were found to have unmet needs. They were offered support, advice and referrals to other vital services. These included adult social care, financial inclusion and voluntary organisations. This was a high success rate compared to other outbound campaigns. Even those who didn’t need support at the time were grateful that the council was checking in on them.
Wanting to build a culture that recognises the value of data, we upskilled Hackney’s analysts on Spark – a data processing tool the core team uses. One of our engineers spent time working with the analysts to develop machine learning solutions to some sticky issues. This included Entity Resolution (for example, detecting that “10 Downing Street” is the same as “10 DS”), which we hope will have a huge impact. Lastly, we designed an in-depth playbook to support the team to use the new platform going forward.
Case studies
Building a housing repairs service for easy reuse across the UK
Building a simpler, faster service to report housing repair needs using the GOV.UK Design System in the Microsoft Azure Cloud.
Making NHS Gloucestershire’s mental health services more accessible to children and young people
Made Tech worked closely with the NHS in Gloucester to digitalise access to over 100 mental health support services.
