I take the fact that the derivative of the exponential function is the exponential function itself as defining feature and make the educated guess that there could be some kind of polynomial \(\sum_{k=0}^n a_k x^k \) with this feature (after all, they do give us smooth curves and being able to add arbitrarily many terms makes them pretty flexible).
What would such a polynomial look like? Well, the limit of a sum is the sum of the limits of its terms, so I'll look at the derivative of an individual term \(a_k x^k\). According to the power rule, the derivative of this term is \( a_k k x^{k-1}.\) I'd like to make sure that the derivative is the same as the initial function, so there are two problems here.
To solve the first problem, it would be convenient if the new factor canceled out.
This can be achieved if \(a_k = \frac{1}{k!}\), so that \(\frac{1}{k!} k x^{k-1} = \frac{1}{(k-1)!} x^{k-1}.\)
With this little trick the kth term becomes the (k-1)th term after differentiation.
The first term \(\frac{x^0}{0!} = 1\) is just a constant and disappears.
To not lose any terms, we need an infinite number of them.
So the polynomial
is actually a power series.
$$
e^x = \sum_{k=0}^\infty \frac{x^k}{k!}
$$