It is not your CNN occupying the GPU, it is Tensorflow. By default, Tensorflow occupies all available GPU memory before your code is run, even if that memory is not required.
You can make Tensorflow occupy only as much memory as required by using the following code snippet (GPU 0, TF2.*):
import tensorflow as tf
gpu_devices = tf.config.experimental.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(gpu_devices[0], True)
The set_memory_growth option prevents Tensorflow from reserving all memory.