Fear always exists when there is a sense of separation. What is happening is that you are living a dual life. Where there is duality there is always going to be fear. Try an experiment:
You said yourself that you know that your day to day life and family will help push you along your path. Starting now and for the next couple days, drop the spiritual stuff (trying to meditate more, yoga, etc.) you can come back to that, as it will always be there.
Instead, fall into your life completely and consider whatever your life is presenting as your spiritual path (just for a couple days). The truth is part of you is pushing it away in the hopes to find something deeper, higher, etc. Maybe “what is” is all you need for growth, and when you give 100% of your attention to it with the conviction that you are going to serve the moment whatever it is, you will integrate your normal life which becomes your custom tailored path.
All the lessons you need to learn are right there within your closest relationships. Try for just two days to believe that meeting your day to day completely is the highest spiritual practice, not anything outside of that.