Since \begin{align*} p(p(1)) &= p(1) = 1, \\ p(p(2)) &= p(3) = 2, \\ p(p(3)) &= p(2) = 3, \end{align*}three of the four solutions to $ p(p(x)) $ are $ x = 1, $ 2, and 3. Also, the quadratic equation $ p(x) = x $ has $ x = 1 $ as a root. Let $ r $ be the other root. Then \[p(p(r)) = p(r) = r,\]so $ r $ must be the fourth root we seek. Since $ p(x) - x = 0 $ for $ x = 1 $ and $ x = r, $ \[p(x) - x = c(x - 1)(x - r)\]for some constant $ c $. Setting $ x = 2 $ and $ x = 3, $ we get \begin{align*} 1 &= c(2 - r), \\ -1 &= 2c(3 - r).\end{align*}Dividing these equations, we get \[-1 = \frac{2(3 - r)}{2 - r}.\]Solving for $ r, $ we find $ r = \boxed{\frac{8}{3}} $.