Let $ \theta $ be the angle between $ \mathbf{a} $ and $ \mathbf{c}, $ so \[\|\mathbf{c} \times \mathbf{a}\| = \|\mathbf{a}\| \|\mathbf{c}\| \sin \theta.\]Then $ 3 = 2 \|\mathbf{c}\| \sin \theta, $ so $ \|\mathbf{c}\| = \frac{3}{2 \sin \theta} $. Hence, \begin{align*} \|\mathbf{c} - \mathbf{a}\|^2 &= \|\mathbf{c}\|^2 - 2 \mathbf{a} \cdot \mathbf{c} + \|\mathbf{a}\|^2 \\ &= \frac{9}{4 \sin^2 \theta} - 2 \|\mathbf{a}\| \|\mathbf{c}\| \cos \theta + 4 \\ &= \frac{9}{4 \sin^2 \theta} - 2 \cdot 2 \cdot \frac{3}{2 \sin \theta} \cdot \cos \theta + 4 \\ &= \frac{9}{4 \sin^2 \theta} - \frac{6 \cos \theta}{\sin \theta} + 4.\end{align*}We can express this in terms of $ \cot \theta $: \begin{align*} \frac{9}{4 \sin^2 \theta} - \frac{6 \cos \theta}{\sin \theta} + 4 &= \frac{9 (\sin^2 \theta + \cos^2 \theta)}{4 \sin^2 \theta} - 6 \cot \theta + 4 \\ &= \frac{9}{4} + \frac{9}{4} \cot^2 \theta - 6 \cot \theta + 4 \\ &= \frac{9}{4} \cot^2 \theta - 6 \cot \theta + \frac{25}{4}.\end{align*}Completing the square in $ \cot \theta, $ we get \[\|\mathbf{c} - \mathbf{a}\|^2 = \left( \frac{3}{2} \cot \theta - 2 \right)^2 + \frac{9}{4}.\]Hence, the smallest possible value of $ \|\mathbf{c} - \mathbf{a}\| $ is $ \boxed{\frac{3}{2}}, $ which is achieved when $ \cot \theta = \frac{4}{3}, $ or $ \tan \theta = \frac{3}{4} $.