Supporting residents with a modern data platform
Encouraging collaboration, better visibility of data and sharing information through a new platform at Hackney Council.
The project
In its mission to modernise the council and continually improve services for residents, Hackney Council wanted to harness the power behind their data.
The data was held in multiple sources, including legacy and modern SaaS applications. We worked closely with the team to design and deliver a cloud platform to help the council maximise the use of data to deliver the best possible results for residents.
Creating a new platform would allow the council to store their data, consolidate and analyse it all in one place. This would support Hackney to develop a holistic understanding of residents’ needs, helping them to live healthier and happier lives.
Our approach
Hackney was keen to encourage collaboration amongst their teams, share information and raise awareness of the data available.
We helped to build a data lake, sorting captured data into 4 tiers: landing, raw, refined and trusted. On top of this we used Apache Spark to add a computation layer, using AWS’s Glue serverless Hadoop capability.
We tried to avoid doing anything overly complicated or bespoke because we wanted Hackney’s engineers to not have to spend 5 days a week working on the platform once we’d stepped away.
Hackney, like any local authority, is responsible for delivering a wide range of services and therefore uses a wide range of applications to collect its data. This meant there were many sources from which to pull data into the new platform, including databases, APIs, CSV files, Google Sheets and various other bits and pieces.
The major challenge was the sheer volume, with some services holding huge amounts of data. For example, the parking system alone holds millions of rows of data and is updated daily.
We also did a lot of work with Hackney to consolidate their housing repair data, which was then passed for additional processing and analysis. This dataset was then used by Hackney’s Link Work team so they could reach out to support residents at higher risk.
Hackney wanted to make sure the new data platform was designed to enable reuse across the public sector to help other authorities use their data to deliver excellent public services, so we used AWS infrastructure and open source technologies to build it.
The results
We also built infrastructure and documentation to allow any analyst in Hackney to put data into the new platform and process it for whatever needs they have. This was essential to the scalability and sustainability of the platform.
This system supports the team to make sure all data is accessible, giving them the best chance of reaching residents that use more than one service. This is combined with robust governance to make sure that data can only be accessed by the people that need it, and for the right purpose.
One of the first benefits was through joining the consolidated repairs data with a vulnerability index built during the pandemic. This allowed the council’s team to proactively contact elderly residents that hadn’t been heard from, in properties that probably had repairs due.
The result was that a third of the residents the team contacted were found to have unmet needs and could be offered support, advice, and referrals to other support such as adult social care, financial inclusion, or voluntary organisations. This was a high success rate compared to other outbound campaigns and even those who didn’t need support at the time were really grateful that the council was checking in on them.
To continue to build a culture that recognises the value of data, we also upskilled a number of Hackney’s analysts on Spark, a data processing tool that the core team uses heavily. Towards the end of the project, one of our ML Engineers spent time working with the analysts to develop machine learning solutions to some sticky issues, such as Entity Resolution (for example, detecting that “10 Downing Street” is the same as “10 DS”), which we hope will have a huge impact. Finally, we also designed an in-depth playbook to support the team to use the new platform going forward.
Image: Google Earth
Want to know more? Get in touch!
Case studies
Building a housing repairs service for easy reuse across the UK
Building a simpler, faster service to report housing repair needs using the GOV.UK Design System in the Microsoft Azure Cloud.
Making NHS Gloucestershire’s mental health services more accessible to children and young people
Made Tech worked closely with the NHS in Gloucester to digitalise access to over 100 mental health support services.
