Can my past be used against me?

I have a 7 month old daughter by my ex-boyfriend. One day he woke up, kissed her goodbye, cashed his check and left. He didn't tell me anything, he didn't leave any money for rent, or even diapers. He just walked away. I had 2 children from a previous marriage, my ex-husband was a bad guy, and I was not very smart. I never abused my children in ANY way, shape or form, and I didn't drink, and I didn't do drugs. but because of the way I dealt with the situation, I was asked to relinquish my rights for my children. I did that, and they were adopted almost a year later. I see them all the time, and I have grown up a lot since then. I have mild OCD due to Anxiety issues. With medication I am able to do as much as a normal person, but without, I am still able to do as much. I just get a little more organized and detail oriented. Anyways, me and my current boyfriend broke up a few days ago. I am concerned that if this gets to a judge that they will use my past children against me. I was told they can't because it is a different father, and entirely diff rent situation. But I am still concerned. Can they take her because of that?