Common Law Separation -- Child Custody Dilemma

Just a quick question.. My common law husband and I have periods in our relationship when arguments go beyond typical exchanging of heated words and turn into full blown insults and put downs which could be considered as abusive. I have a 2 month old daughter and while he loves her and would never ever put her in harms way I don't feel like I want to raise my daughter in an environment where her mother is being put down.. As she gets older and this behavior continues I don't want her to think this is an acceptable way for a woman to be treated. During my pregnancy (although he denies ever saying it) jokes and comments were made about him (and his mother.. Whom I've previously had issues with regarding her access to my daughter) taking my daughter away from me. This topic is a hot issue during arguments.
We are in the process of moving into a new apartment and I am very hesitant on whether or not I want to continue the relationship or just leave and move in with an older relative. The alternative would be a stable environment for both myself and my child and would be until I was able to secure my own personal residence. I am perfectly willing to sit down and work with him to come up with a parenting plan..however this older relative lives an hour away from where we are living now and I fear that the move would fuel the fire with regards to him taking steps to either file for sole custody or if we were able to agree on a parenting plan outside of court my fear would then be that he wouldn't bring her back from a scheduled visit and then there would be nothing I or the police could do to get her back aside from actually going to court after all. This whole thing has left me wondering whether or not leaving is even worth it.. Just to save myself and my daughter all of the potential hassle and stress that may or may not occur. My question is basically what am I legally allowed to do in this situation? Are there any steps I can take aside from filing for sole custody that would ensure I had protection against her taking and keeping my child during a weekend visit?