JamesWang
2021-02-22 07:45:07
Here is my solution. I used PySpark's accumulators to collect the metrics(number of http calls, payload sent per call, etc.) at each partitions, at the driver node, assign these accumulators' value to statsD gauge variable, and send these metrics to Graphite server and eventually visualized them in Grafana dashboard. It works so far so good.
did you find an efficient way to extract counts from dataframes? like number of loaded or saved ones?
yes, like what I mentioned, I used Spark's accumulator to accumulate the metrics at each partition (executors) and then at driver node I can assign these accumulated metrics to
statsDgaugevariable.