tensor representation

A tensor is just a multi-dimensional array of numbers, the universal data structure that flows through every neural network. A single number is a 0-D tensor (scalar), a list is 1-D (vector), a table is 2-D (matrix), and stacking those gives 3-D, 4-D, and higher. In vision, images and the intermediate activations inside a network are naturally tensors: a color image is a 3-D block of numbers (height by width by color channels), and a batch of images is a 4-D block. Everything a vision model computes is a transformation of one tensor into another.

A batch of feature maps is conventionally a rank-4 tensor with axes N by C by H by W: N is the batch size (how many images processed together), C is the number of channels (3 for an RGB input; tens to thousands for internal feature maps, one per learned filter), and H and W are spatial height and width. Each axis has a size, and the product is the element count; the tensor also has a dtype (float32, float16 or bfloat16, int8) and a device (CPU or GPU). PyTorch defaults to this channels-first NCHW order; TensorFlow defaults to channels-last NHWC, and hardware kernels (and quantized or inference paths) often prefer NHWC, a frequent source of confusion when porting models.

Under the hood a tensor is a flat block of memory plus shape and strides; the strides say how many elements to step to move one unit along each axis, which lets operations like transpose, reshape or view, slicing, and broadcasting be cheap views that reinterpret the same memory without copying. This is why a transpose can be free but a following operation may force a copy to make the data contiguous. Broadcasting (automatically expanding a smaller tensor's shape to match a larger one, for example adding a per-channel bias of shape C to an N by C by H by W tensor) is the rule that makes elementwise math concise; mismatched shapes that silently broadcast are a common bug.

Tracking tensor shapes is the day-to-day work of building vision models. A convolution maps N by C_in by H by W to N by C_out by H' by W'; pooling shrinks H and W; flattening or global pooling collapses spatial axes before a dense head; a Vision Transformer reshapes an image into a sequence of patch tokens, an N by L by D tensor (L tokens of dimension D), to feed attention. Beyond grids, point clouds, meshes, NeRF samples, and video (adding a time axis, N by C by T by H by W) are all tensors too. Mastering shapes, and the GPU memory their product implies, is the most practical tensor skill.