Skip to content
All projects

CESI · 3rd year

Big Data architecture

An end-to-end decision-support pipeline for health data.

CompletedMar 2024 – Apr 2024
Big Data architecture

A fictitious health group, the CHU (Cloud Health Unit), holds a large amount of data and wants to use it. We had to deliver a complete decision-support system: data integration, history, security and visualisation, with scalability and privacy (GDPR) in mind.

Architecture

  1. 1Local FTP server collecting the data supplied by the CHU
  2. 2Talend jobs for extraction, transformation and loading
  3. 3Loading into a data warehouse on a Hadoop cluster managed by Cloudera
  4. 4External Hive tables over HDFS, then optimised internal tables (partitioning and buckets)
  5. 5HiveQL views answering each client need
  6. 6Power BI dashboards built on those views
Architecture diagram
Architecture diagram
Power BI dashboard
Power BI dashboard