Should I be with him?

My boyfriend and I are having problems with our relationship. He's very insecure and thinks I am going to cheat on him.He was married in the past, not too long before we started dating. She moved to california when they got a divorce, later on, he and I moved out there. 2 months into our relationship (in california at the time) I would find out he is still talking to her, and hed deny it. He put her number in his phone under a different name. He says they'd talk about the divorce. Now and then, she tries to contact him. I just feel like he asked me to move out there with him, thinking he would end up seeing her again but he was infatuated by me, and was confused. I know she was his first love, but I just feel like everything is so unfair. I am in love with him. I love his family and we have been dating now for almost a year. I have been so faithful to him, I am always there for him, and have never done anything to hurt him intentionally. We argue a lot, and its very pathetic, because I don't even remember when I was happy. I have a lot of negative otherwise going on in my life and I want to be with him. He tells me he loves me, and that he really wants me in his life. Its just that, whenever we go out with friends, he gets upset and starts a fight with me for no reason. Also, I try to do small stuff for him, I painted his room to make it more comfortable to live in, and all he does is complain that there is no money, no car. I just want his love. And I want to give him mine. I am just so confused and I think he is too. I feel like I need to focus on myself, I would love for the ex wife to just be out. I cannot help but think he will back stab me again. I don't want to catch him talking to her again in another room, whispering.. Seems obvious that he still loves her. I am very bothered and emotional. What should I do?