The data lifecycle

From a click to a decision: the journey

It is 8:02 on a Tuesday morning in Taipei. A customer opens a coffee-shop app called BeanCount, taps “buy a latte”, and pays NT$120. To her, that is the end of the story — coffee is on the way. To a data team, that single tap is the beginning of a long, mostly invisible journey. By lunchtime that one tap, joined with millions of others, might help answer a question like “does our new morning promotion actually make money, or does it just give away coffee we would have sold anyway?”

This guide is the map of that journey. We will follow one tap from the moment it is created, through the systems that move it, store it, and reshape it, all the way to a chart on a manager’s screen and a decision that changes the app — which then produces new taps, and the whole thing goes around again. That repeating circle is what people mean by the data lifecycle, and the connected machinery that carries data along it is called a data pipeline.

Before we zoom into any one piece, it helps to see the whole arc at once. Most data, in most companies, travels through the same six stages, in roughly this order:

1. Generation — something happens (a tap, a payment, a sensor reading) and a record of it is created.
2. Ingestion — that record is moved from the live app into a place built for analysis.
3. Storage — it lands in a warehouse or lake, alongside billions of its cousins.
4. Transformation — raw, messy records are cleaned, joined, and reshaped into trustworthy tables.
5. Analysis & modeling — humans explore the tables, dashboards summarize them, and machine-learning models learn from them.
6. Decision — a person or a system acts, the product changes, and fresh data starts flowing again.

Where data is generated

Before data can be analyzed, it has to be born somewhere. In a modern company it is born in many places at once. The most visible source is the product itself: every screen the customer sees can emit events — small records that say what happened and when. Our latte tap is one such event. There are also transactions (the actual NT$120 payment), the operational databases that run the business (customer accounts, the menu, store inventory), background logs from servers, signals from devices and sensors, and data bought or pulled from third parties such as a payment processor, a maps provider, or an advertising platform.

An event is just a timestamped record of one thing that happened, usually written as a small bundle of fields. You do not need to read code to understand it; think of it as a line in a diary. Here is roughly what our latte tap might look like the instant it is created:

event = {
  'user_id':   'u_88213',
  'event':     'purchase',
  'item':      'latte',
  'price_twd': 120,
  'store':     'taipei-da-an',
  'ts':        '2026-06-27T08:02:14+08:00'
}

Some of this data is structured — neat rows and columns, like the event above. Some is unstructured — product photos, customer-review text, support-call audio — which is harder to put in a table but increasingly valuable. Either way, one rule dominates everything downstream: garbage in, garbage out. If the price is logged wrong here, no clever analysis later can fix it. The cheapest place to ensure quality is at the source, which is why teams care so much about how events are defined and logged.

OLTP vs OLAP: two very different jobs

Here is the first idea that really shapes the modern data stack: the database that runs your app and the database that answers your questions want opposite things, so we usually keep them apart. The split has two ugly acronyms worth learning once, because they explain the whole layout that follows.

OLTP stands for online transaction processing. This is the database doing the live work of the business. When our customer pays, an OLTP system must, in a few milliseconds, charge exactly NT$120, mark one latte as sold, and reduce the store’s milk inventory — touching just a handful of rows, perfectly, right now, for thousands of customers at once. It is optimized to read and write tiny slices of data extremely fast and never lose a transaction.

OLAP stands for online analytical processing. This is the database built for questions, not sales. A question like “what were average latte sales, per store, per hour, across the last three months?” has to scan millions of rows and add them up. You do not need it answered in a millisecond — a few seconds is fine — but it must chew through enormous amounts of history at once. OLAP systems are optimized for exactly that: big reads over huge tables.

Ingestion and storage

Ingestion is the unglamorous work of moving data from where it is born (the app, the OLTP database, third-party services) to where it will be analyzed. There are two broad styles. Batch ingestion gathers data in chunks on a schedule — for example, copy all of yesterday’s sales every night at 2 a.m. It is simple, cheap, and good enough for most reporting. Streaming ingestion moves each event continuously, within seconds of it happening — needed when minutes matter, such as catching a fraudulent card before the next charge.

It helps to feel the scale, because that is why this needs real engineering rather than a shared spreadsheet. Suppose BeanCount has 500,000 daily active users, and an average user generates about 40 events a day (opening the app, browsing, tapping, paying). Then the daily and per-second volumes are roughly:

Twenty million events a day — about 231 every second, all day, every day. That volume is why ingestion and storage are engineered systems, not a folder of files.

Where does all of it land? Usually one of three kinds of home. A data warehouse is like a well-organized pantry: data arrives cleaned and structured into tables, and it is wonderful for fast SQL analytics — but you have to tidy things before they go in. A data lake is more like a giant garage: you dump raw data of any shape (logs, images, JSON) cheaply, and worry about organizing it later. A lakehouse is the increasingly popular blend that tries to give you the garage’s cheap, flexible storage together with the pantry’s fast, reliable querying.

Transformation and modeling

Raw data is almost never ready to answer a question. Our twenty-million-events-a-day pile is full of duplicates, half-finished sessions, prices in different currencies, and store codes that changed last spring. Transformation is the stage where raw records become trustworthy tables: cleaning bad values, joining events to customer and store information, and aggregating millions of taps into tidy summaries. The goal is the tidy tables — one row per thing, one column per attribute — that every later step quietly depends on.

What language does this transformation speak? Overwhelmingly, SQL — the decades-old query language that remains the lingua franca of data work. A modern tool called dbt lets teams write these transformations as version-controlled, testable SQL files, a practice often called analytics engineering. Here is a tiny taste: turning the raw events into a daily sales summary for each store.

SELECT
  store,
  date_trunc('day', ts) AS day,
  count(*)         AS orders,
  sum(price_twd)   AS revenue_twd
FROM sales
WHERE event = 'purchase'
GROUP BY store, day
ORDER BY day;

Analysis, BI, and ML

Now we finally reach the payoff. Once the data is clean and tidy, three kinds of people-and-machines put it to work, and most companies do all three. The first is analysis: a human asking open-ended questions and exploring — what statisticians call exploratory data analysis, the craft of looking before you leap. This is where the tools of statistics live.

The second is business intelligence: turning those tidy tables into dashboards and reports the whole company can read at a glance, built with BI tools. A store manager does not run SQL; she opens a dashboard each morning that shows yesterday’s revenue for each store and flags anything unusual. Good BI makes the same trustworthy numbers available to everyone, ending the daily argument about whose spreadsheet is right.

The third is machine learning: instead of a human reading the table, a model learns patterns from it to make predictions or decisions automatically — which customers are about to stop buying, which pastry to recommend next to that latte. Models that run for real customers usually pull their inputs from a shared feature store so the numbers used to train a model match the numbers used live.

Whatever the tool, the output is supposed to lead to a decision — “the morning promotion lifts revenue 6%, so roll it out everywhere.” Two honesty notes before you act on a chart. First, a dashboard is only as trustworthy as the transformed data behind it; a beautiful chart built on a broken join is confidently wrong. Second, a pattern on a dashboard — promotion days had higher sales — is a correlation, not proof the promotion caused the sales; perhaps those were simply payday weekends. Establishing cause is hard enough to deserve its own track. With that caution, the decision changes the app, the changed app produces new taps, and the lifecycle loop closes.

Who does what: the data team

You have now seen every stage of the journey, so you can finally make sense of the job titles. Roughly, each role owns a stretch of the pipeline you just walked. Titles vary wildly between companies — and at a small startup one tired person may wear all of these hats — so treat the labels loosely. What matters is the work, not the badge.

1. Data engineer — builds the plumbing: ingestion, storage, and the pipelines that move data reliably (the generation-to-storage stretch).
2. Analytics engineer — owns transformation: turning raw tables into clean, documented, tested models with SQL and dbt.
3. Data analyst — lives in analysis and BI: answering business questions and building the dashboards people rely on.
4. Data scientist — applies statistics, experiments, and machine learning to harder questions about cause, prediction, and uncertainty.
5. ML engineer — takes promising models the last mile into reliable, monitored production software.