Move your cursor. The portrait is made of data points.
How I think about data
From noise to signal.
Every platform I work on follows the same arc. Scroll to fly through it.
01/ 04
Capture
Everything starts as noise.
Databases, events and clickstreams never stop talking. I like the part where change data capture and event buses turn that chatter into a stream you can actually reason about.
Streaming and batch compute that reshapes events at scale. Underneath it all sits plain systems work, so I enjoy pulling containers, CPU limits and autoscaling apart to see how they behave.
courseDE MasteryMy course: Spark, Kafka, Flink and lakehouse for analytics engineers
reposimple_dockerA Docker-like container runtime in Python
03/ 04
Store & Model
Shape it so questions are cheap.
Lakehouse layouts, open table formats and a catalog that engines agree on. Good modelling is what makes the next ten questions fast and the next ten dashboards boring, in the good way.
repoDataCatalogViewGlue multi-dialect views across Athena and Redshift
04/ 04
Operate
Run it like you will be paged for it.
A pipeline is only as good as the infrastructure it runs on. Infrastructure as code, least-privilege access across accounts, and a close eye on the bill.
repoaws-sapMy AWS Solutions Architect Professional study guide
01 Capture02 Process03 Store & Model04 Operate
Four AWS certifications.
I came to data engineering through computer vision and machine learning, so I care a lot about pipelines that are observable and trustworthy. The certifications are the cloud half of that.
Solutions Architect
Professional
Data Engineer
Associate
Developer
Associate
SysOps Administrator
Associate
Notes from the cloud.
Hands-on write-ups of things I built or finally understood. Mostly AWS, containers and data infrastructure.