Your brand counts.
Your brand is yours, but it isn't you.
A brand separate to you means your staff can use it, and your client's are still being looked after by your brand. It also means that it can grow to be bigger than you: serving many more people than you alone could. It can become a property that you can sell when the time comes. It can be stronger than you - by hiring more staff and using outsourced services your brand can have skills that you cannot, because one person cannot do everything.
Your brand can rise above the general impression of your category.
This last point is so important right now: the general impression of anyone in the financial services sector has not been great since the global financial crisis. The general impression of people that give insurance advice is not great: people do not like to think about the need for insurance, like a visit to the dentist.
But your brand can rise above the general impression. Your business can have a different reputation. By having a brand that breaks out of the generic category name "Insurance adviser" or "financial adviser" you can fill it with meaning and create an impression amongst your target group of clients that is very different, much more positive, and owned entirely by your business.
But it will take investment. It takes money, of course, but less than you think. What it really takes is time and effort. The more you work to understand your market the better you will build your brand.
If you are telling yourself something like "I work to be a better adviser, I do not waste time on advertising as I get most of my business by referrals" you are conflating "brand" with "promotion." Try using the word "reputation" as a substitute for "brand" and hopefully the doors will open for you.