A freelance accountant or computer programmer or graphics designer working from home is not controversial in one can do so without the neighbor noticing anything different.
If you have to receive many clients/customers into your “home office,” it gets trickier.
Imagine a piano teacher teaching from home next door to you with students and their parents streaming in and out all day. You the neighbor essentially see lots of “strangers” visiting day in and day out. Ditto a tax accountant or lawyer “working from home” receiving clients day in and day out. Not all neighbors/neighborhoods are willing to accept that.
Then we get into zoning. I’d call it fair and a win if legalized sex work is zoned the same way as music teachers and accountants and lawyers who have to receive comparable volume of visiting clients.