GNU Make Skipping Straight to Linking

Yeah, this gets messy / difficult. The problem you have is that you can specify prerequisite lists - that can work in order, but as soon as you start to use -j then make can start processing prerequisites in any old order. So bin_file requires $(OBJ_FILES) which require prepare. Then %.o requires the same named %.cpp file - which it can do for main.o, but not the filex.o since they don't exist yet - but it tries anyway and fails - in the mean time make (in parallel) is potentially starting to generate the .cpp files, but by this time its too late...etc...

My Prerequisites Build Pattern

I use a very specific prerequisites pattern of my own design - some might frown upon - but I have carefully considered this over the years and found it to be optimal for me.

I create a rule called build or something - which requires build_prerequisites target and then calls make to do the actual build once this is complete:

.PHONY: build
build: build_prerequisites
build:
    @echo "start_build"
    @$(MAKE) bin_file

This means that build_prerequisites is always run first before the recipe runs. You cant seem to achieve the same forcing of order (at least not easily) using just dependencies. I.e. a list of dependencies can be run in any order with -j, but the rule recipe is always run last.

Now we have this pattern we can fill in the rest. First the build_prerequisites target which does your file generation - I am using echo in my example because I don't have your python script:

.PHONY: build_prerequisites
build_prerequisites:
    @echo "build_prerequisites"
    echo "create file1" > file1.cpp
    echo "create file2" > file2.cpp
    echo "create file3" > file3.cpp

Finally add in the c++ compile and link stages - these will be run with the single recursive make call from build - i.e. $(MAKE) bin_file (again I am using echo to create the files in my example):

%.o : %.cpp
    @echo "compiling: $<"
    @#echo "$(CXX) $(SRC_INCLUDES) $(LIB_INCLUDES) $(CXXFLAGS) -c $< -o $@"
    @echo "touch" > $@

bin_file : $(OBJ_FILES) 
    @echo "linking: $<"
    @echo $(CXX) $(SRC_INCLUDES) $^ $(LIB_INCLUDES) $(LDFLAGS) -o $@
    @echo "touch" > $@

Output

Here is the output from my test program (using echo) and main.cpp already exists usingn -j10:

make -j10
build_prerequisites
echo "create file1" > file1.cpp
echo "create file2" > file2.cpp
echo "create file3" > file3.cpp
start_build
make[1]: Entering directory '/mnt/d/software/ubuntu/make'
compile: bin_main.cpp
compile: file1.cpp
compile: file2.cpp
compile: file3.cpp
link: bin_main.o
g++ bin_main.o file1.o file2.o file3.o -o bin_file
make[1]: Leaving directory '/mnt/d/software/ubuntu/make'

Note: if I put a sleep 1 in the "compile" rule - this still takes only 1 second for all 4 files to compile.

Put it all together

GEN_FILES := file1.cpp file2.cpp file3.cpp
CXX_FILES := bin_main.cpp $(GEN_FILES)
OBJ_FILES := $(patsubst %.cpp,%.o,$(CXX_FILES))

###### STAGE 1

.PHONY: build
build: build_prerequisites
build:
    @echo "start_build"
    @$(MAKE) bin_file

.PHONY: build_prerequisites
build_prerequisites:
    @echo "build_prerequisites"
    copy_and_pp_files.py $(CXX_FILES) $(SEARCH_DIRS) .
    copy_and_pp_files.py $(CFG_FILES) $(SEARCH_DIRS) .

###### STAGE 2

%.o : %.cpp
    @echo "compiling: $<"
    @$(CXX) $(SRC_INCLUDES) $(LIB_INCLUDES) $(CXXFLAGS) -c $< -o $@

bin_file : $(OBJ_FILES) 
    @echo "linking: $<"
    @$(CXX) $(SRC_INCLUDES) $^ $(LIB_INCLUDES) $(LDFLAGS) -o $@

###### OTHER RULES

.PHONY: clean
clean :
    @$(RM) *.o
    @$(RM) file*

I have attempted to use your actual code, but I have no way to test this so there may be a bug in there. I split it up into 2 "stages" for clarity. Stage 1 is done in your makeor make build call, then state 2 is done in the recursive make call in the build recipe.