Refactor SequentialRecommender and add Sequential models: SASRec, BERT4Rec, GPT2Rec, FPMC

[hieuddo]

Description

• Refactor NextItemRecommender to SequentialRecommender with two settings: session-based and session-aware.
• Rename NextItemEvaluation to SequentialEvaluation
• Add new sequential models: SASRec, BERT4Rec, GPT2Rec (similar to BERT4Rec, but with GPT2 architecture)
• Refactor existing GRU4Rec compatible with session-aware setting

Quick run with 10 epochs from examples/sequential_diginetica.py:

Model	AUC	MRR	NDCG@10	NDCG@50	Recall@10	Recall@50	Train (s)	Test (s)
FPMC-sbr	0.7579	0.1254	0.1430	0.1686	0.2242	0.3398	2.8742	0.4422
GRU4Rec-sbr	0.8051	0.0611	0.0632	0.1035	0.1068	0.2907	6.0934	0.7539
SASRec-sbr	0.7902	0.3884	0.4216	0.4295	0.5289	0.5657	2.2551	1.4375
BERT4Rec-sbr	0.7880	0.3812	0.4140	0.4221	0.5201	0.5569	6.3330	1.4493
GPT2Rec-sbr	0.8111	0.3702	0.4021	0.4139	0.5096	0.5604	1.3811	1.4056
FPMC-sar	0.7480	0.1195	0.1341	0.1614	0.2067	0.3275	0.3568	0.3948
GRU4Rec-sar	0.8029	0.0754	0.0820	0.1146	0.1349	0.2837	3.4652	1.9199
SASRec-sar	0.7890	0.3891	0.4208	0.4297	0.5236	0.5639	1.6576	1.3925
BERT4Rec-sar	0.7910	0.3768	0.4080	0.4184	0.5114	0.5552	1.4889	1.3984
GPT2Rec-sar	0.8243	0.3790	0.4084	0.4217	0.5079	0.5674	1.3703	1.3644

Related Issues

Checklist:

• I have added tests.
• I have updated README.md (if you are adding a new model).
• I have updated examples/README.md (if you are adding a new example).

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[qtuantruong]

holy moly, really long PR

[hieuddo]

holy moly, really long PR

Haha. Most of these are from https://github.com/PreferredAI/CoVE/tree/main/models in our joint work with @lthoang

[qtuantruong]

Things to check carefully:

SequentialEvaluation breaks SIT / SITJson session-based evaluation

ranking_eval always iterates with test_set.usi_iter(...) and evaluates only session_items[-1]. For SIT formats, every row is mapped to uid=None, so all sessions become one pseudo-user. The updated Yoochoose examples will score only the last test session instead of all sessions.

References:

• cornac/eval_methods/sequential_evaluation.py:64
• cornac/data/dataset.py:1176

Session-aware evaluation never includes training history

history_items is built only from sessions present in the test or validation split, while train_set is only used for item masks. With a common one-held-out-session-per-user split, session-aware models are evaluated with only the current test prefix, or no history for first / any, so the cross-session behavior being added is not actually measured.

Reference:

• eval_methods/sequential_evaluation.py:97

user_based is ignored

The function accepts user_based, but aggregation always averages over users via user_sessions.keys() with no session-level branch. Experiment(user_based=False) therefore cannot reproduce the old session-averaged next-item behavior.

Reference:

• cornac/eval_methods/sequential_evaluation.py:123

FPMC cold-start user fallback does not trigger for unseen eval users

Because the shared uid_map is mutated while building test and validation sets, self.total_users includes users that were never trained. score() compares against total_users, so those users get random untrained embeddings instead of the padding row. Compare against self.num_users or explicitly track train-known users.

Reference:

• cornac/models/fpmc/recom_fpmc.py:253

[qtuantruong]

If you don't mind, I would recommend to break it down into multiple small PRs so the verification is manageable.

[hieuddo]

If you don't mind, I would recommend to break it down into multiple small PRs so the verification is manageable.

Yeah. I'm closing this PR.

Will discuss with @lthoang further about the session-based vs. session-aware design.

Currently, all the models we're adding are session-based models. We don't have any session-aware model yet. I'm trying to anticipate and future-proof the design for both session-based and session-aware settings, and it turns out to be more complicated than necessary. The reason for this is because https://github.com/PreferredAI/CoVE are meant for session-aware setting.

Would you recommend dropping all session-aware support and keeping it simple with session-based only, then work on that later (because maybe YAGNI lol)?

[qtuantruong]

Yes, I’d recommend dropping session-aware support from this round and keeping the first PR session-based only. We can rename NextItemEvaluation to SequentialEvaluation, but preserve the existing NextItemEvaluation behavior, including SIT/SITJson examples and evaluation semantics. Then we can add the session-based models in separate PRs. Finally, we can open a dedicated design PR/RFC for session-aware support after we agree on the evaluation contract.