"And my mom would always side with him."