Industry overview
What is shaping information services
Data, ratings and information providers need to understand what users actually need, keep their data fresh and size new opportunities accurately.
- Users want timely, verified data in their workflow
- Data collection at scale needs strong quality control
- New products need validation before launch
Client background
A market data company, working in India. Client identity is withheld to protect confidentiality.
- Organisation
- A market data company
- Sector
- Information services
- Team engaged
- IT & Development
- Services
- Data analytics & dashboards, Cloud & DevOps services, AI & automation solutions
Client challenge
Manual processing delayed daily data delivery. Analysts spent hours each morning cleaning files from different sources, and errors occasionally reached clients. Analysts spent most of their time cleaning data from multiple sources by hand, delaying client deliveries.
Research objectives
- Deliver on the brief: automating a data processing pipeline.
- Define requirements and success criteria the solution would be tested against.
- Deliver a reliable, secure solution the team can run and extend.
Target audience
Scope: Daily data feeds.
Audiences we typically reach in information services
Data users and analystsSubscribers and enterprise buyersData contributorsProspectsPartners
Geographic coverage
Coverage: India. Market: India.
- India
Research methodology
An iterative delivery approach: discovery, design, build and test in short cycles, with the client reviewing each step.
- 1Brief
- 2Pipeline automation
- 3Quality checks
- 4Insight
We automated ingestion, cleaning and validation with monitoring. We automated ingestion from each source, added validation rules and alerts for missing or unusual values, and moved the pipeline to a monitored cloud setup. Analysts helped define validation rules based on their knowledge of common data errors.
Data analytics & dashboardsCloud & DevOps servicesAI & automation solutions
Sampling strategy
- Users and stakeholders from every affected team took part in discovery workshops and acceptance testing.
Data collection
- Requirements were gathered through workshops, process walk-throughs and a review of existing systems and data.
Data quality & validation
- Peer code review on every change
- Automated and manual testing
- User-acceptance testing with the client
- Release checklist and rollback plan
Data analysis
- Usage and performance monitoring after launch
- Defect and support-ticket trends
- Process time before and after
Key findings
- Data delivery became faster and more reliable.
Automation reduced errors.
Recommendations
- The company also reduced the time to add new data sources.
- Track adoption and support tickets, and plan the next release from real usage.
What we recommended, and the client put into practice
Business impact
- Analysts spent more time on insight rather than data cleaning, and client complaints about data errors fell.
Deliverables & next steps
- Interactive dashboard
- Deployed solution with documentation and handover
- Support and maintenance plan
- Repeat the measure after changes are in place to track progress.
- A natural follow-on question: how accurate and fresh is the data?
Client identity is withheld to protect confidentiality.