Do you think there is a time in your life when you shouldn't be afraid of anyone?

As children, some people were afraid to stand up for themselves and speak their minds. Do you think we should dismiss our fear of people once we are older? I'm talking about regular people not rapists, murderers or robbers (we should be wary of them). All perspectives welcome.