a backreference
Imagine a rule like 'a word, then later the exact same word again'. Plain regular expressions cannot say this, because they have no memory of what an earlier part actually matched, only of the pattern's shape. A backreference is a feature that programming regex engines add to break that limit: it lets a pattern refer back to the literal text some earlier group captured.
In most engines you capture a piece of text with parentheses and then refer to it later with a numbered reference. For instance the pattern (.+)\1 means 'some nonempty text, immediately followed by that very same text again', so it matches abab and hellohello but not abcd. The \1 does not re-match the pattern (.+); it demands the identical characters that the first group actually captured this time. That is a fundamentally different power: the engine must remember and compare concrete substrings, not just track which state it is in.
This is the precise place where engineering regex leaves the mathematical theory behind. Languages like 'ww for some string w' (a word repeated) are provably not regular, and indeed not even context-free, yet a single backreference expresses one. So a pattern using backreferences is no longer describing a regular language, and Kleene's theorem and the automaton constructions no longer apply to it. Backreferences are useful (matching repeated words, balanced quotes of the same kind), but they are also the feature most responsible for regex matching becoming slow or, in the worst case, NP-hard.
(.+)\1 matches a string that is some text repeated twice: abcabc matches (with the group capturing abc), but abcabd does not. The language 'ww' this expresses is not regular, so no finite automaton can recognize it.
Backreferences demand identical repeated text, which is beyond regular.
A backreference makes a pattern non-regular: it can match languages no finite automaton can, so such a pattern is not a regular expression in the mathematical sense even though the engine calls it one.