First, we claim that any quartic with real coefficients can be written as the product of two quadratic polynomials with real coefficients. Let $ z $ be a complex root of the quartic. If $ z $ is not real, then its complex conjugate $ \overline{z} $ is also a root. Then the quadratic $ (x - z)(x - \overline{z}) $ has real coefficients, and when we factor out this quadratic, we are left with a quadratic that also has real coefficients. If $ z $ is real, then we can factor out $ x - z, $ leaving us with a cubic with real coefficients. Every cubic with real coefficients has at least one real roots, say $ w $. Then we can factor out $ x - w, $ leaving us with a quadratic with real coefficients. The product of this quadratic and $ (x - z)(x - w) $ is the original quartic. So, let \[x^4 + ax^3 + 3x^2 + bx + 1 = (x^2 + px + r) \left( x^2 + qx + \frac{1}{r} \right), \quad (*)\]where $ p, $ $ q, $ and $ r $ are real. Suppose one quadratic factor has distinct real roots, say $ z $ and $ w $. Then the only way that the quartic can be nonnegative for all real numbers $ x $ is if the roots of the other quadratic are also $ z $ and $ w $. Thus, we can write the quadratic as \[(x - z)^2 (x - w)^2.\]Thus, we can assume that for each quadratic factor, the quadratic does not have real, distinct roots. This implies that the discriminant of each quadratic is at most 0. Thus, \[p^2 \le 4r \quad \text{and} \quad q^2 \le \frac{4}{r}.\]It follows that $ r > 0 $. Multiplying these inequalities, we get \[p^2 q^2 \le 16,\]so $ |pq| \le 4 $. Expanding $ (*) $ and matching coefficients, we get \begin{align*} p + q &= a, \\ pq + r + \frac{1}{r} &= 3, \\ \frac{p}{r} + qr &= b.\end{align*}Therefore, \begin{align*} a^2 + b^2 &= (p + q)^2 + \left( \frac{p}{r} + qr \right)^2 \\ &= p^2 + 2pq + q^2 + \frac{p^2}{r^2} + 2pq + q^2 r^2 \\ &= p^2 + 4pq + q^2 + \frac{p^2}{r^2} + q^2 r^2 \\ &\le 4r + 4pq + \frac{4}{r} + \frac{4r}{r^2} + \frac{4}{r} \cdot r^2 \\ &= 4pq + 8r + \frac{8}{r}.\end{align*}From the equation $ pq + r + \frac{1}{r} = 3, $ \[r + \frac{1}{r} = 3 - pq,\]so \[a^2 + b^2 \le 4pq + 8(3 - pq) = 24 - 4pq \le 40.\]To obtain equality, we must have $ pq = -4 $ and $ r + \frac{1}{r} = 7 $. This leads to $ r^2 - 7r + 1 = 0, $ whose roots are real and positive. For either root $ r, $ we can set $ p = \sqrt{4r} $ and $ q = -\sqrt{\frac{4}{r}}, $ which shows that equality is possible. For example, we can obtain the quartic \[\left( x - \frac{3 + \sqrt{5}}{2} \right)^2 \left( x + \frac{3 - \sqrt{5}}{2} \right)^2 = x^4 - 2x^3 \sqrt{5} + 3x^2 + 2x \sqrt{5} + 1.\]Hence, the maximum value of $ a^2 + b^2 $ is $ \boxed{40} $.