[WIP] add benchmarking scripts

[radekosmulski]

No description provided.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[bschifferer]

This line gave me an errer because I do not have this directoy

Reply via ReviewNB

[radekosmulski]

this should be now fixed

[bschifferer]

timeit is nice - but we should store the output to a file, that we can save it

We need a warmup phase as well (first requests are normally slow). We need something like

import time
MODEL_NAME_PT = "t4r_pytorch_pt"
WarmUp
for _ in range(200):

    payload = cudf.DataFrame(data={'sess_pid_seq': np.random.randint(0, 390001, 20), 'id': 0}).groupby('id').agg({'sess_pid_seq': list})

    with grpcclient.InferenceServerClient("localhost:8001") as client:

        col_names = ['sess_pid_seq']

        inputs = nvt_triton.convert_df_to_triton_input(col_names, payload, grpcclient.InferInput)

        response = client.infer(MODEL_NAME_PT, inputs)

    end_time = time.time()
Collecting
out = []

for _ in range(200):

    payload = cudf.DataFrame(data={'sess_pid_seq': np.random.randint(0, 390001, 20), 'id': 0}).groupby('id').agg({'sess_pid_seq': list})
    start_time = time.time()

    with grpcclient.InferenceServerClient("localhost:8001") as client:

        col_names = ['sess_pid_seq']

        inputs = nvt_triton.convert_df_to_triton_input(col_names, payload, grpcclient.InferInput)

        response = client.infer(MODEL_NAME_PT, inputs)

    end_time = time.time()

    out.append(end_time-start_time)

Reply via ReviewNB

[rnyak]

@radekosmulski can you add some explanations in the notebook what's the purpose of this notebook? what data is being downloaded/used? thanks.

[rnyak]

Line #15.    apt-get install unzip -y

Can you add some explanations here for user who dont know what's going on here:

• what data is used?
• From where does it take the trained models? Who trained the models?  what script we should use to train the model and export it?
• what does this rees46_ecom_dataset_small_for_ci.zip file include? and did you generate it? does it have the trained models in it as exported?

Thanks

Reply via ReviewNB

[radekosmulski]

Added a whole new notebook for documenting training! 🙂 Will continue to keep information as I go.

[rnyak]

/transformers4rec/TF4Rec/models/ --> when and how these models were exported to that folder?  can you add some explanations?

Reply via ReviewNB

[radekosmulski]

absolutely! added a notebook with steps for training and exporting models