Since $ \{a_n\} $ is an arithmetic sequence, we may let $ a_n = a + (n-1)d $ for some $ a $ and $ d $. Since $ \{g_n\} $ is a geometric sequence, we may let $ g_n = cr^{n-1} $ for some $ c $ and $ r $. Then we have \[\begin{aligned} a + c &= 0 \\ a + d + cr &= 0 \\ a + 2d + cr^2 &= 1 \\ a + 3d + cr^3 &= 0.\end{aligned}\]The first equation gives $ c = -a, $ so the remaining equations become \[\begin{aligned} a + d - ar &= 0 \\ a + 2d - ar^2 &= 1 \\ a + 3d - ar^3 &=0.\end{aligned}\]From the equation $ a+d-ar=0, $ we get $ d=ar-a, $ and substituting in the remaining two equations gives \[\begin{aligned} -a + 2ar - ar^2 &= 1 \\ -2a + 3ar - ar^3 &= 0.\end{aligned}\]The equation $ -2a + 3ar - ar^3 = 0 $ factors as \[a(r-1)^2(r+2) = 0.\]Having $ a=0 $ would contradict the equation $ -a+2ar-ar^2=1, $ so either $ r=1 $ or $ r=-2 $. But if $ r=1, $ then $ \{g_n\} $ is a constant sequence, which means that $ \{a_n + g_n\} $ is itself an arithmetic sequence; this is clearly impossible, because its first four terms are $ 0, 0, 1, 0 $. Thus, $ r = -2 $. Then we have \[-a + 2a(-2) - a(-2)^2 = 1,\]or $ -9a = 1, $ so $ a = -\frac{1}{9} $. Then $ c = -a = \frac{1}{9} $ and $ d = ar - a = -3a = \frac{1}{3} $. We conclude that \[\begin{aligned} a_n &= -\frac19 + (n-1)\frac13, \\ g_n &= \frac19(-2)^n \end{aligned}\]for all $ n $. Then \[a_{5} + g_{5} = -\frac19 + 4 \cdot \frac13 + \frac19 (-2)^{4} = \boxed{3}.\]