The challenge
A government-funded national education programme required complex data and sample files to be uploaded by a large number of contributors across multiple regions, each within a set timeframe. Every submission needed to pass strict validation rules covering sample quality, file structure, and compliance requirements before it could be accepted into the programme's research pipeline.
Most contributors were non-technical. They needed to do something technically demanding - produce and upload files that met a detailed specification - and they needed clear guidance when something went wrong. The existing process relied on programme staff manually checking every incoming file against the validation rules, then chasing contributors individually when submissions failed or hadn't arrived.
The overhead was unsustainable. Staff time was consumed by repetitive QA tasks, and compliance enforcement was fragmented across the team. Problems with individual submissions were only discovered after manual review, by which point they'd already caused delays downstream.
The solution
Kicktag built a bespoke process management platform that handled the entire upload and validation workflow. Each contributor had their own web portal where they logged in, uploaded their files, and received immediate feedback on whether their submission had passed or failed the validation rules.
The platform applied a comprehensive set of business rules automatically on every upload - checking sample quality, file structure, and compliance requirements without any manual intervention. When a submission failed, the platform told the contributor exactly which rules had been broken so they could correct the issue and re-submit. This replaced the previous process where programme staff had to review files by hand and chase contributors individually.
Programme managers received a periodic funnel report showing progress across four stages: total contributors expected to upload, those who had logged in but not yet submitted, those who had attempted but failed validation, and those who had successfully completed their transfer. This gave managers a clear picture of where the process stood and where targeted intervention was needed.
Zero data persistence
A critical architectural requirement: no contributor data was persisted on Kicktag's infrastructure. The platform processed and validated incoming files and applied governance rules, but the data itself was passed through rather than stored. This kept the compliance footprint minimal and meant the programme didn't need to extend its data processing agreements to cover Kicktag as a long-term data holder.
How the rules engine works
Agencies typically receive sample files via secure FTP, then spend time manually fixing formatting issues, checking for errors, and validating against business rules before the data can be used. Problems are discovered late, fixes are slow, and bad data can contaminate downstream processes like survey send-out and analysis.
The Kicktag rules engine works differently. Contributors upload files through a self-serve web portal and the engine validates each upload in real time, before the file is accepted. Rules are categorised as hard or soft:
- Hard rules block the upload entirely until the file is fixed. The contributor gets immediate feedback on what's wrong.
- Soft rules flag issues but allow the upload to proceed if the contributor adds a note explaining why the exception is acceptable.
The rules library goes well beyond basic format checks and referential integrity:
- Checking values against external lists and APIs to catch invalid entries before they reach the destination database
- Validating sample distributions to confirm the sample is representative
- Blocking PII uploads before a data breach can occur
- Custom rules built and configured per project to match the specific quality requirements of the programme
Every rule is configurable. New rules can be added without code changes to the platform.
For every hour spent configuring rules, the engine saves thousands of pounds of downstream effort in data processing at the agency - and, more importantly, prevents bad data from breaking survey send-out and analysis phases where the cost of errors is much higher.
The result was a platform that managed a complex, time-bound, multi-contributor data collection process end-to-end - handling the enforcement and contributor guidance automatically so the programme team didn't have to.
The results
The platform replaced hours of manual checking per submission with automated validation that ran in seconds. Staff who had been spending their time on repetitive QA tasks were freed up to focus on the programme itself - helping contributors who were genuinely stuck rather than reviewing every file by hand.
Compliance enforcement was centralised for the first time. Instead of fragmented manual oversight across the team, the platform applied the same rules consistently to every submission. The periodic funnel report gave programme managers a clear picture of overall progress: how many contributors had successfully uploaded, how many were stuck at the validation stage, and how many hadn't yet engaged.
The zero-persistence architecture meant the programme avoided the need for additional data processing agreements or security assessments covering Kicktag's infrastructure - a significant reduction in compliance overhead for a government programme with strict data handling requirements.
Why this matters for agencies
Not every project is a dashboard. Sometimes the hardest problem is getting a large number of non-technical people to do something technically demanding - upload complex files that meet a strict specification, correctly and on time. That's a process management problem, and it needs a purpose-built platform to solve it.
This project shows Kicktag's ability to build bespoke platforms that go well beyond standard data work. The same approach applies to any organisation managing large-scale data collection, sample transfers, or multi-contributor processes where the real challenges are compliance, quality enforcement, and keeping the whole thing on track.
If your client needs a platform that doesn't exist off the shelf - something that automates validation, guides non-technical users through a complex process, and enforces compliance without adding overhead - talk to us. That's exactly the kind of problem we solve.