I am beginner in python, deep learning and neural network. I had made custom activation function. What i want to know when i am making custom activation function that root from sigmoid, where should i define the derivative for my custom activation function?
I've tried reading about automatic differentation. but i am not sure does keras automatically derivative my custom sigmoid?
my custom activation function in keras/activation.py
def tempsigmoid(x, temp=1.0):
return K.sigmoid(x/temp)
my model
def baseline_model():
# create model
model = Sequential()
model.add(Conv2D(101, (5, 5), input_shape=(1, 28, 28), activation='relu'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.2))
model.add(Flatten())
model.add(Dense(128, activation='relu'))
model.add(Dense(num_classes, activation='tempsigmoid'))
# Compile model
model.compile(loss='mse', optimizer='adam', metrics=['accuracy'])
return model
Yes, Keras uses automatic differentiation, as it only supports backends with this feature (like TensorFlow).
So you do not need to define the gradient or derivative at all, it will be computed for you automatically.
thank you for your response. Wow , do you know resource good reading for automatic differentiation? I knew that other DL framework such as chainer dont have this thing. And i want to know how about pytorch.
Is there a reference to any documentation confirming that Keras automatically takes the derivative?