By the Remainder Theorem, $ p(1) = 3, $ and $ p(3) = 5 $. Let $ q(x) $ and $ ax + b $ be the quotient and remainder, respectively, when the polynomial $ p(x) $ is divided by $ (x - 1)(x - 3), $ so \[p(x) = (x - 1)(x - 3) q(x) + ax + b.\]Setting $ x = 1, $ we get $ p(1) = a + b, $ so $ a + b = 3 $. Setting $ x = 3, $ we get $ p(3) = 3a + b, $ so $ 3a + b = 5 $. Solving, we find $ a = 1 $ and $ b = 2 $. Therefore, the remainder is $ \boxed{x + 2} $.