DO NOT GIVE IN OR UP!!!!!!
You are amazing and awe inspiring to put up with this as a mother!
Now that we have this figured out- you need to document evidence. Hire a lawyer and write down everything, get affidivits from those who will write one for you and build up a strong case against the whole family.
If he is irresponsible not to pay support- he can't be responsible enough to care for your munchkins.