rebalance: introduce go-carbon health check with better sync rate control

The original sync mechanism is a bit too simple, as it is controllyed by only workers.
It is hard to balance both efficiency and reliability. Setting the worker count too low,
rebalancing becomes too slow; setting it too high, buckyd might take away too much
resources.

To meet both criteria, this changes introduced a go-carbon health check and a automatic
sync rate adjustments based on metrics per second per node.

  bucky rebalance -f -offload -ignore404 \
                  -graphite-ip-to-hostname \
                  -graphite-metrics-prefix carbon.minutely.buckytool.rebalance.$cluster.dst.$to_location.src.$from_location \
                  -graphite-endpoint 127.0.0.1:3002 \
                  -go-carbon-health-check=$enable_go_carbon_health_check \
                  -go-carbon-health-check-interval 5 \
                  -go-carbon-port 8080 \
                  -go-carbon-protocol http \
                  -go-carbon-cache-threshold 0.5 \
                  -sync-speed-up-interval $sync_speed_up_interval \
                  -metrics-per-second $sync_metrics_per_second \
                  -h ${seed_node}:4242 \
                  -workers $workers \
                  -timeout $timeout

Using the above example for explanation:
  -go-carbon-health-check=true: asks buckytools to check go-carbon cache usage, which indicates if there is capacity issues.
  -sync-speed-up-interval: means increased sync rate (metrics per second) every specified seconds.
  -metrics-per-second: means the initial sync rate. Should be set to a lower value like 5 - 10 initially while using -sync-speed-up-interval.

Whats more, with go-carbon health check, buckytools also does automatic sync rate easing. Could be disabled by `-no-random-easing=false`.

Full flags could be found with `bucky rebalance -help`.

Besides go-carbon health check, this commits also introduce exporting some internal sync rate metrics to graphite
for monitoring. Could be enabled by -graphite-metrics-prefix.

By default -go-carbon-health-check is disabled, which means buckytools fallback to the original sync behaviour.

Relevant go-carbon PR: go-graphite/go-carbon#433

Author: bom-d-van

Makefile

@@ -37,6 +37,15 @@ test: clean bucky buckyd
 	go run -mod vendor testing/rebalance.go $(REBALANCE_FLAGS)
 	go run -mod vendor testing/copy.go $(COPY_FLAGS)
 
+# only works on linux
+test_rebalance_health_check_setup:
+	sudo ip addr add 10.0.1.7 dev lo
+	sudo ip addr add 10.0.1.8 dev lo
+	sudo ip addr add 10.0.1.9 dev lo
+
+test_rebalance_health_check: clean bucky buckyd
+# 	go run -mod vendor testing/rebalance_health_check.go $(REBALANCE_FLAGS)
+
 clean_test:
 	rm -rf testdata_rebalance_*
 	rm -rf testdata_copy_*

cmd/bucky/common.go

@@ -157,7 +157,9 @@ func DeleteMetric(server, metric string) error {
 
 	switch resp.StatusCode {
 	case 200:
-		log.Printf("DELETED: %s", metric)
+		if msFlags.printDeletedMetrics {
+			log.Printf("DELETED: %s", metric)
+		}
 	case 404:
 		log.Printf("Not found / Not deleted: %s", metric)
 		return fmt.Errorf("Metric not found.")