HOA problems - what is going to happen?

I rent out a house I used to live in. Because my interest rates have gone up, I can't afford to rent it out to a family, so for the last six years, I've been renting out the rooms to individuals, and sometimes relatives. It barely pays, but I've been hanging on. I live minutes away, so am constantly over there, fixing up & monitoring things, nearly every day. It's quiet, no parties, on end of cul de sac. Sometimes the tenants are doctors & only live there a couple of months. My next door neighbor has always been mean. He has complained to the HOA, saying I run a "hotel". That's not at all true, 3 of the tenants have lived there 5 yrs. But HOA says the CCRs state "no multiple families" so they want to fine me. I don't have multiple families. In fact my house has very few people & cars. The neighbor is racist & a bigot, he hates gays & one tenant is gay. I don't know what to do. If I rent to a family, I will have to foreclose. If I rent to individuals, I get fined.