What a feature actually is
By now you know that a model learns from a dataset. But a model never touches the raw world — it only ever sees a row of numbers. Each measurable property in that row is a feature: a single column the model can read. House price prediction might use square meters, number of bedrooms, and distance to the nearest station. Each is a feature; together they form the input vector the model actually consumes.
Crucially, the world does not arrive as numbers. A timestamp, a city name, a paragraph of text, a photograph — none of these are directly a feature. Someone has to decide how to turn them into numbers, and that decision quietly determines what the model is even capable of noticing. A model can only find a pattern that survives the translation into its features. Anything you throw away during that translation is gone forever, no matter how clever the algorithm.
Shaping features: encoding and scaling
The first real job is turning messy reality into clean columns. A category like "city = Tokyo / Osaka / Kyoto" is not a number — and if you just label them 1, 2, 3 the model wrongly assumes Kyoto is three times Tokyo. One-hot encoding fixes this by giving each category its own yes/no column. Knowing the difference between categorical and numerical features is what tells you which trick a column needs in the first place.
Numerical features bring a subtler problem: scale. Suppose one column is age (0–100) and another is annual income (0–1,000,000). To a distance-based method, income dwarfs age purely because its numbers are bigger — not because it matters more. Feature scaling, a form of normalization and standardization, puts every feature on a comparable footing, so the model weighs them by relevance rather than by accident of units.
# standardize a column: center, then divide by spread z = (x - mean(x)) / std(x) # fit mean/std on TRAIN only, then reuse on test mean, std = fit(x_train) x_train = (x_train - mean) / std x_test = (x_test - mean) / std # same numbers, not refit
The curse of dimensionality
It is tempting to think more features is always better — just give the model everything and let it sort things out. Reality pushes back hard. As the number of features (dimensions) grows, the space the data lives in expands explosively, and your fixed number of examples becomes ever more sparse inside it. This is the curse of dimensionality.
Picture 100 points spread along a 1-meter line: cozy, about one per centimeter. Spread the same 100 points across a 1-meter square and they thin out; across a 1-meter cube, they are nearly alone. In high dimensions almost every point is far from every other, and "nearby" loses meaning — which is exactly what distance- and neighbor-based methods rely on. More columns can mean less signal, not more.
There is a practical consequence too. Each extra feature gives the model another knob to twiddle, another way to fit noise instead of signal — feeding straight into overfitting. The rough rule of thumb: you need dramatically more examples to fill a high-dimensional space, and you rarely have them. Fewer, better features usually beat many weak ones.
Dimensionality reduction: keeping what matters
If too many dimensions hurt, the natural response is to compress them. Dimensionality reduction squeezes many features into a few, trying to keep as much of the meaningful variation as possible while throwing away redundancy. The best-known technique is PCA (principal component analysis): it finds the directions along which the data varies most and re-describes each point using just those few directions.
Think of photographing a flat sheet of paper floating in 3D space. The paper is really two-dimensional; the third dimension carries almost no information. PCA finds that flat plane automatically and lets you store two numbers per point instead of three, losing almost nothing. Real data is rarely that clean, so reduction is a trade: you accept a little lost detail in exchange for a smaller, less noisy, faster-to-train representation.
Hand-crafted features vs. learned representations
For decades, everything above was *the* job: skilled people hand-crafted features, and the difference between a winning and losing model was usually the features, not the algorithm. Then deep learning changed the deal. Given enough data, a deep network can learn its own features directly from raw inputs — pixels, audio samples, characters — through representation learning. Early layers discover edges, later layers combine them into shapes and objects, all without a human naming a single feature.
Those learned features often live as an embedding: a dense vector where similar things land near each other, discovered rather than designed. This is genuinely powerful and has automated away much hand-engineering for images, audio, and text. But do not mistake it for a free lunch. Learned features need large data and compute, are hard to interpret, and can quietly latch onto the wrong cue — like classifying "wolf" by spotting snow in the background.
So which do you use? On tabular, small-to-medium data — most business problems — careful feature engineering with classic models still wins, and it remains the heart of data-centric AI. On raw perceptual data at scale, learned representations dominate. The honest summary: deep learning moved the feature work rather than abolishing it. Choosing the inputs, cleaning them, and deciding what the model should be allowed to see is still human judgment — and still where most of the real gains hide.