Regarding Chronos-Bolt's Full Inference Duration Across 100 FEV Datasets

[blisky-li]

Hello, I am running the fev-benchmark inference locally using Chronos-Bolt-small. The entire pipeline appears to be relatively slow during data preprocessing, taking approximately 3-4 hours to infer 100 tasks.

I would like to inquire about the official, accurate runtime for the fev benchmark. Thanks~

[shchur]

Hi @blisky-li, the inference reported in the leaderboard corresponds only to the inference time of the model, excluding the data loading/preprocessing/scoring.

fev/examples/chronos/evaluate_model.py

Lines 33 to 57 in 80b3a05

	inference_time = 0.0
	quantile_levels = task.quantile_levels.copy()
	if 0.5 not in quantile_levels:
	quantile_levels.append(0.5)
	predictions_per_window = []
	for window in task.iter_windows():
	past_data, _ = fev.convert_input_data(window, adapter="datasets", as_univariate=True)
	past_data = past_data.with_format("torch").cast_column("target", datasets.Sequence(datasets.Value("float32")))
	quantiles_all = []
	mean_all = []
	start_time = time.monotonic()
	for batch in batchify(past_data["target"], batch_size=batch_size):
	quantiles, mean = pipeline.predict_quantiles(
	context=batch,
	prediction_length=task.horizon,
	limit_prediction_length=False,
	quantile_levels=quantile_levels,
	)
	quantiles_all.append(quantiles.numpy())
	mean_all.append(mean.numpy())
	inference_time += time.monotonic() - start_time

Thanks for reporting the issues with the overall runtime of the full benchmark - this indeed is suboptimal, but we didn't catch this issue earlier because of running the evaluation in a distributed setup over multiple machines.

We have merged a series of PRs (#83, #85, #86) that reduce overhead. This should reduce the end-to-end runtime of the full benchmark on a single machine to several minutes instead of several hours. We will release a new version of the fev package (0.7.0) sometime this week enabling you to take advantage of these changes.

[blisky-li]

Thank you for your efforts. I have incorporated tqdm into my workflow to monitor the FEV benchmark process.

benchmark = Benchmark.from_yaml(
    "/local_path/fev_bench/tasks.yaml"
)

tasks = benchmark.tasks if num_tasks is None else benchmark.tasks[:num_tasks]

summaries = []

for task in tqdm(
    tasks,
    desc="Evaluating tasks",
    total=len(tasks),
):
    predictions, inference_time, extra_info = predict_with_model( ....

My current progress is as follows:
Evaluating tasks: 35%|████████████████████████▏ | 35/100 [1:17:44<2:25:11, 134.02s/it

I hope this helps with your work. Thank you again for your response, and I look forward to a faster benchmarking process.

[shchur]

Hi @blisky-li, we just released fev-0.7.0 on PyPI. Please try running the benchmark with the new version and let me know if this resolves the problem for you.

[blisky-li]

Hello! @shchur , My local inference seems to have sped up considerably, though it may not yet meet the ideal of completing all tasks within minutes—likely constrained by my hardware resources. Thank you for your efforts!

Evaluating tasks: 15%|██████████▎ | 15/100 [11:07<30:03, 21.21s/it

[shchur]

Can you please check what fraction of the runtime is due to the inference speed of the model vs. the overhead from fev (splitting and metrics computation)? You can check the model inference time in the summaries produced by the model. It would also be helpful to know your specific hardware.