Cowardice, or introspection?
I think considering the real, and potentially deadly, consequences of riding is just about a requirement for any serious rider. What I am talking about is being brutally, honestly aware of the risks. Once you are there you can look at the risks and try to mitigate them with gear, training, and proper maintenance. Sure, a lot of people live very superficially and don't give the likelihood of their demise on any particular day one whit of thought. They are not going to understand where you are coming from if you do. Typically, they retort with macho posturing and go back to grunting and hooting about whatever has their attention at the time. They are the ones who may call you a coward, but I wouldn't worry myself with their typically ill-informed, shallow opinions. We aren't talking about them riding. This is about YOU.
I had some friends who knew I had a sort of "fear of heights". If I am in a building, an aircraft, or behind a very solid barrier, I'm okay. Put me near the lip of a cliff or the edge of a roof and uh-uh... No. I am not a happy guy. My friends got me started rappelling and rock climbing, and I became technically proficient as a beginner. I found that "in an emergency" I could rappel or climb, but do it for fun? No. It always required me pushing myself in ways that were never easy, nor fun. I didn't have a need to continue it, so I didn't.
So, why are you riding? Be honest. It's perfectly cool that you wanted to try it and then had doubts. If you find out it's like rock climbing was for me (you CAN do it, but aren't enjoying it) then is there a reason to continue?
What's your mental state? There have been times where I have had enough stress or emotional agitation that I didn't feel it was sensible for me to ride my bike to the store. Riding a motorcycle safely does take more focus and concentration than driving a car... but honestly, a driver in a car should use the same level of focus and concentration. But if you make a little mistake in the car you aren't likely to be as severely injured.
Where do you live? I have a lot of rather rural roads to ride, with light to moderate traffic. I like that. I have ridden in moderately busy towns and, while I manage, I wouldn't qualify it as "fun". Are there roads you can ride that give you a better sense of comfort? I started out riding rural roads and quiet suburban streets, worked toward busier streets, and then eventually highways.
If you know you are riding for yourself, good. If you weigh the risks, counter them as best as you can, and set unreasonable concerns aside, good. If you can find a place to ride to improve skill and confidence, great! Have you taken an MSF class? Knowledge can be a great antidote for fear.
People die doing things from taking showers to walking across streets. Once again, maybe it's better to realize that all people die, but their deaths are not your own. Live your own life.
If these things don't work out, hey, maybe riding isn't for you. Recreational rock climbing seems to be a great source of satisfaction for some, but it's just work with little reward for me. There is a huge array of pursuits to try; maybe another one is a better fit for you. Even if motorcycling isn't the right thing for you right now, depending on who you are, where you are at and such it may become a thing that is right for you in another time. Don't attach that "hang it up for good" label on it. There is no need to be absolute.
Maybe you have some issues elsewhere in your life that are affecting your ability to enjoy something that you feel you WANT to enjoy? Work on those issues, and matters like this fear may drop away.
And no, you are no coward, in spite of what someone may have told you. A coward wouldn't voice their inner fears and concerns. I hope you find your answers. The quest for self-discovery is one of the boldest journeys anyone can partake of in this life.