All projects
CESI · 3rd year
Big Data architecture
An end-to-end decision-support pipeline for health data.
CompletedMar 2024 – Apr 2024

A fictitious health group, the CHU (Cloud Health Unit), holds a large amount of data and wants to use it. We had to deliver a complete decision-support system: data integration, history, security and visualisation, with scalability and privacy (GDPR) in mind.
Architecture
- 1Local FTP server collecting the data supplied by the CHU
- 2Talend jobs for extraction, transformation and loading
- 3Loading into a data warehouse on a Hadoop cluster managed by Cloudera
- 4External Hive tables over HDFS, then optimised internal tables (partitioning and buckets)
- 5HiveQL views answering each client need
- 6Power BI dashboards built on those views

