How I work

An end-to-end data pipeline, piece by piece

This is the architecture I apply in my projects: from sources to dashboard, with data quality, orchestration and infrastructure as code. Click a piece to see what it does and where I have built it. Then remove it to see what breaks downstream.

Pick a version

The platform I would build today on Google Cloud for a mid-sized company: change capture from operational databases, APIs and files landing in Cloud Storage, events through Pub/Sub, Dataflow in batch and streaming, layered BigQuery with Dataform, and consumption from Looker Studio, Vertex AI, Sheets and an API. Underneath, what keeps it alive: Composer, Dataplex, IAM, Terraform and observability.

Try a scenario

Removed piece
No data or stale
Incomplete data
Unreliable data
Latent risk

The journey of one record

Follow a single change from start to finish. Each step lights up its piece in the diagram.

1
A record changes
HR updates an employee salary in Cloud SQL.
2
Datastream captures it
The change is read from the transaction log and written as Avro to Cloud Storage in under a minute, without querying the database.
3
It lands untouched
It sits in a dated path. Nobody ever modifies it: if anything goes wrong later, it is reprocessed from here.
4
Dataflow validates it
That night the batch job parses the Avro, checks the schema and writes the row to BigQuery bronze. If it fails, it goes to the rejects table with the reason.
5
Dataform models it
Silver closes the previous salary version and opens the new one with an effective date: full history, no duplicates.
6
It passes the assertions
Unique key, salary within range, employee present in the dimension. If anything fails, gold is left alone and an alert fires.
7
It becomes a KPI
The fact table recomputes the gap against the Eurostat market salary for that role.
8
Vertex AI re-scores
The monthly prediction updates the employee's flight risk and SHAP explains that the salary gap now weighs less than before.
9
The manager sees it
In Looker Studio the employee drops off the risk list. Same figure in the finance sheet, because they read the same gold view.

The layers, one by one

What each stage is, why it exists, which services build it and where I have built it.

1 · Sources

What it is: Operational databases in Cloud SQL, SaaS behind APIs, app and sensor events, and business files.
Why it exists: Each source fails differently: a database saturates, an API changes format, a file arrives late. That is why each gets its own ingestion method.
Services: Cloud SQLAlloyDBAPIs RESTSFTPDrive
Where I have done it: HR, Slack, Eurostat and Adzuna in the thesis; GPS, weather and air in CloudRISK; public catalogues in the price monitor.

2 · Ingestion

What it is: Extractors that move data without transforming it: Datastream captures changes, Cloud Functions call APIs, Pub/Sub receives events and Cloud Run Jobs collect files.
Why it exists: Separating extraction from transformation lets you retry without losing anything and swap a source without touching the rest.
Services: DatastreamCloud FunctionsPub/SubCloud Run JobsSecret Manager
Where I have done it: Walker as a Cloud Run Job and three Pub/Sub topics in CloudRISK; Eurostat and Adzuna extractors in the thesis.

3 · Landing and processing

What it is: Everything lands raw and immutable in Cloud Storage. Dataflow batch parses and validates; Dataflow streaming applies rules continuously; Dataproc Serverless reprocesses history with PySpark.
Why it exists: Storing raw data before touching it is what allows reprocessing when something goes wrong, and processing separately allows scaling without touching the warehouse.
Services: Cloud StorageDataflowApache BeamDataproc Serverless
Where I have done it: Stateful Beam pipeline with anti-cheat in CloudRISK; Cloud Storage landing in the thesis.

4 · Bronze and state

What it is: BigQuery bronze keeps untransformed history; Firestore keeps the present that apps read; a service enriches text with Vertex AI.
Why it exists: Analytics and operations have opposite needs: cheap bulk volume versus instant reads. One store for each.
Services: BigQueryFirestoreCloud RunVertex AI
Where I have done it: Normalised bronze and Slack sentiment in the thesis; balances and map in Firestore in CloudRISK.

5 · Silver and gold with Dataform

What it is: Versioned SQL models: silver cleans and keys, assertions block what fails, gold serves facts and KPIs, and feature tables feed the model.
Why it exists: This is the layer where a single truth is defined. Every metric has a written owner, formula and threshold, and tests guarantee it arrives right.
Services: DataformBigQueryBigQuery ML
Where I have done it: Gold views, KPI catalogue and assertions in the thesis; dbt models every five minutes in Air quality.

6 · Consumption

What it is: Looker Studio to decide, Vertex AI to anticipate, Connected Sheets for finance and a Cloud Run API for the product.
Why it exists: Everyone reads the same gold layer. There are not three versions of the KPI depending on who asks.
Services: Looker StudioVertex AIConnected SheetsCloud Run
Where I have done it: Command centre and SHAP model in the thesis; FastAPI backend in CloudRISK; Gemini agent in the Student Hub.

Cross-cutting

What it is: Composer orchestrates, Dataplex catalogues, IAM and Secret Manager protect, Terraform and Cloud Build deploy, Logging and Billing watch.
Why it exists: They move no data at all, but they are the difference between a demo and a platform that survives its first month.
Services: Cloud ComposerDataplexIAMTerraformCloud BuildCloud Monitoring
Where I have done it: Single-apply Terraform and CI/CD in CloudRISK and the Student Hub; catalogue and PII in Dataplex, runbook and costs in the thesis.

The brand-free version: the same layers and rules apply whatever the cloud. It is the mental model I apply before picking services.

Try a scenario

Removed piece
No data or stale
Incomplete data
Unreliable data
Latent risk

The journey of one record

Follow a single change from start to finish. Each step lights up its piece in the diagram.

1
A record is born
HR updates an employee record in the internal system.
2
It gets extracted
At six in the morning a job reads the table and drops a raw copy into the lake.
3
It lands untouched
The copy is partitioned by date. If anything goes wrong later, it is reprocessed from here.
4
It gets cleaned
A SQL model normalises types, removes duplicates and defines what "active employee" means.
5
It passes the tests
Uniqueness, nulls and ranges. If anything fails the pipeline stops and alerts before reaching gold.
6
It becomes a KPI
It enters the fact table and updates the churn rate with its written definition.
7
Someone decides
The dashboard crosses the threshold, the manager sees it and schedules a conversation before it is too late.

The layers, one by one

What each stage is, why it exists, which services build it and where I have built it.

Sources

What it is: Where data is born: internal systems, external APIs, events and files.
Why it exists: Each kind of source fails differently and needs a different ingestion method.
Services: PostgreSQLRESTPub/SubExcel
Where I have done it: HR and Slack in the thesis; GPS, weather and air in CloudRISK; public catalogues in the price monitor.

Ingestion

What it is: Move data without transforming it: batch on a schedule or continuous streaming.
Why it exists: Separating extraction from transformation lets you retry without losing anything.
Services: AirflowCloud Run JobsPub/SubDataflow
Where I have done it: Daily CronJob for the price monitor and a stateful Beam pipeline in CloudRISK.

Lake · bronze

What it is: Data as it arrived, immutable and partitioned by date.
Why it exists: It is what lets you reprocess when a transformation goes wrong.
Services: Cloud StorageBigQuery
Where I have done it: Normalised bronze in the thesis; daily snapshot table in the price monitor.

Warehouse · silver and gold

What it is: Versioned SQL models: cleaning, tests and marts with defined KPIs.
Why it exists: This is where every metric is defined once, with owner, formula and threshold.
Services: dbtDataformBigQuery
Where I have done it: Gold views and retention KPI catalogue; dbt models every five minutes in Air quality.

Consumption

What it is: Dashboards, models and APIs reading the same gold layer.
Why it exists: One source for every consumer: there are not three versions of the KPI.
Services: Looker StudioVertex AIFastAPI
Where I have done it: Looker Studio command centre, Dash dashboard and the CloudRISK FastAPI backend.

Cross-cutting

What it is: Orchestration, governance, infrastructure as code and monitoring.
Why it exists: They move no data, but without them the pipeline does not survive its first month.
Services: AirflowDataplexTerraformCloud Logging
Where I have done it: Terraform and CI/CD in CloudRISK and the Student Hub; Telegram alerts in Air quality.