I created a reproducible example and encountered no issue :
import random
import string
import pandas
def gen_random_word(word_length=30):
return ''.join((random.choice(string.ascii_letters) for _ in range(word_length)))
# I create a list of 300000 labels (but only 150k distinct labels) of 30 characters
labels = [gen_random_word() for _ in range(150000)]
labels = labels + labels
random.shuffle(labels)
# A dataframe here is useless but I try to get close to your own example
df = pandas.DataFrame(
{'labels': labels}
)
class A:
def __init__(self):
self.direct = {}
self.inverse = {}
self.progressive = 0
def fromDataset(self, column):
# create direct map
for label in column:
# skip if present
if str(label) in self.direct:
continue
# else add to direct
self.direct[str(label)] = self.progressive
self.inverse[self.progressive] = str(label)
self.progressive += 1
class B:
def __init__(self):
self.direct = {}
self.inverse = {}
self.progressive = 0
def fromDataset(self, column):
# reset map
direct = {}
inverse = {}
# progressive
progressive = 0
# create direct map
for label in column:
# skip if present
if str(label) in direct:
continue
# else add to direct
direct[str(label)] = progressive
inverse[progressive] = str(label)
progressive += 1
self.direct = direct
self.inverse = inverse
self.progressive = progressive
First test :
%%time # remove that if you are not using jupyter and use another timing solution
direct = {}
inverse = {}
# progressive
progressive = 0
# create direct map
for label in df['labels']:
# skip if present
if str(label) in direct:
continue
# else add to direct
direct[str(label)] = progressive
inverse[progressive] = str(label)
progressive += 1
Wall time : 267 ms
Second test :
%%time
a = A()
a.fromDataset(df['labels'])
Wall time : 249 ms
Third test :
%%time
b = B()
b.fromDataset(df['labels'])
Wall time : 220 ms
So... nothing significant.
Yeour reproduction is right. I've removed anything from the class (i mean other functions and even anchestor classes) and execution now is normal. By testing each single super class i've found one in the project that is inheriting Tf.keras.Model - it is possible that keras model modifies memory management on non @tf functions? (When I inherit it again the behaviour changes to as described by me)
well of course tf.keras.Model can do a lot of stuff that affects the memory and execution time.
If you know more about it I would like you to update your answer, thank you.