Skip to content

Commit 3542ce4

Browse files
ninggaoninggao
authored
update entity numbers, remove multilingual entity models (#6215)
1 parent 516f8d2 commit 3542ce4

File tree

2 files changed

+5
-45
lines changed

2 files changed

+5
-45
lines changed

‎Orchestrator/docs/NLRModels.md‎

Lines changed: 5 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -52,16 +52,6 @@ This is a yet another high quality EN-only base model for entity extraction.
5252
It is a 12-layer pretrained pretrained [Transformer][7] model optimized for conversation.
5353
Its architecture is pretrained for example-based use ([KNN][3]), thus it can be used out of box.
5454

55-
### pretrained.20210105.microsoft.dte.00.12.bert_example_ner_multilingual.onnx (experimental)
56-
This is a high quality multilingual base model for entity extraction.
57-
It is a 12-layer pretrained pretrained [Transformer][7] model optimized for conversation.
58-
Its architecture is pretrained for example-based use ([KNN][3]), thus it can be used out of box.
59-
60-
### pretrained.20210105.microsoft.dte.00.12.tulr_example_ner_multilingual.onnx (experimental)
61-
This is a high quality multilingual base model for entity extraction.
62-
It is a 12-layer pretrained pretrained [Transformer][7] model optimized for conversation.
63-
Its architecture is pretrained for example-based use ([KNN][3]), thus it can be used out of box.
64-
6555
### pretrained.20210205.microsoft.dte.00.06.bert_example_ner.en.onnx (experimental)
6656
This is a high quality EN-only base model for entity extraction. It's smaller and faster than its 12-layer alternative.
6757
It is a 6-layer pretrained pretrained [Transformer][7] model optimized for conversation.
@@ -72,16 +62,6 @@ This is a high quality EN-only base model for entity extraction. It's smaller an
7262
It is a 6-layer pretrained pretrained [Transformer][7] model optimized for conversation.
7363
Its architecture is pretrained for example-based use ([KNN][3]), thus it can be used out of box.
7464

75-
### pretrained.20210205.microsoft.dte.00.06.bert_example_ner_multilingual.onnx (experimental)
76-
This is a high quality multilingual base model for entity extraction. It's smaller and faster than its 12-layer alternative.
77-
It is a 6-layer pretrained pretrained [Transformer][7] model optimized for conversation.
78-
Its architecture is pretrained for example-based use ([KNN][3]), thus it can be used out of box.
79-
80-
### pretrained.20210205.microsoft.dte.00.06.tulr_example_ner_multilingual.onnx (experimental)
81-
This is a high quality multilingual base model for entity extraction. It's smaller and faster than its 12-layer alternative.
82-
It is a 6-layer pretrained pretrained [Transformer][7] model optimized for conversation.
83-
Its architecture is pretrained for example-based use ([KNN][3]), thus it can be used out of box.
84-
8565
## Models Evaluation
8666
For a more quantitative comparison analysis of the different models see the following performance characteristics.
8767

@@ -136,13 +116,17 @@ For a more quantitative comparison analysis of the different models see the foll
136116
| ------------------------------------------------------------ | ---------- | ------ | ----------------------- | --------------- |
137117
| pretrained.20210205.microsoft.dte.00.06.bert_example_ner.en.onnx | BERT | 6 | ~ 23 ms | 259M |
138118
| pretrained.20210205.microsoft.dte.00.12.bert_example_ner.en.onnx | BERT | 12 | ~ 40 ms | 427M |
119+
| pretrained.20210218.microsoft.dte.00.06.bert_example_ner.en.onnx | BERT | 6 | ~ 23 ms | 259M |
120+
| pretrained.20210218.microsoft.dte.00.12.bert_example_ner.en.onnx | BERT | 12 | ~ 40 ms | 425M |
139121

140122
- The following table shows how accurate is each model relative to provided training sample size using [Snips NLU][4] system, evaluated by **macro-average-F1**.
141123

142124
| Training samples per entity type | 10 | 20 | 50 | 100 | 200 |
143125
| ------------------------------------------------------------ | ----- | ----- | ----- | ----- | ----- |
144-
| pretrained.20210205.microsoft.dte.00.06.bert_example_ner.en.onnx | 0.662 | 0.678 | 0.680 | 0.684 | 0.674 |
126+
| pretrained.20210205.microsoft.dte.00.06.bert_example_ner.en.onnx | 0.615 | 0.636 | 0.647 | 0.661 | 0.665 |
145127
| pretrained.20210205.microsoft.dte.00.12.bert_example_ner.en.onnx | 0.637 | 0.658 | 0.684 | 0.698 | 0.702 |
128+
| pretrained.20210218.microsoft.dte.00.06.bert_example_ner.en.onnx | 0.637 | 0.658 | 0.673 | 0.686 | 0.684 |
129+
| pretrained.20210218.microsoft.dte.00.12.bert_example_ner.en.onnx | 0.661 | 0.664 | 0.670 | 0.685 | 0.681 |
146130

147131

148132

‎Orchestrator/v0.2/nlr_versions.json‎

Lines changed: 0 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -41,18 +41,6 @@
4141
"description": "Bot Framework SDK release 4.10 - Multilingual ONNX V1.4 12-layer per-token intent base model",
4242
"minSDKVersion": "4.10.0"
4343
},
44-
"pretrained.20210105.microsoft.dte.00.12.bert_example_ner_multilingual.onnx": {
45-
"releaseDate": "01/05/2021",
46-
"modelUri": "https://models.botframework.com/models/dte/onnx/pretrained.20210105.microsoft.dte.00.12.bert_example_ner_multilingual.onnx.zip",
47-
"description": "(experimental) Bot Framework SDK release 4.10 - Multilingual ONNX V1.4 12-layer per-token entity base model",
48-
"minSDKVersion": "4.10.0"
49-
},
50-
"pretrained.20210105.microsoft.dte.00.12.tulr_example_ner_multilingual.onnx": {
51-
"releaseDate": "01/05/2021",
52-
"modelUri": "https://models.botframework.com/models/dte/onnx/pretrained.20210105.microsoft.dte.00.12.tulr_example_ner_multilingual.onnx.zip",
53-
"description": "(experimental) Bot Framework SDK release 4.10 - Multilingual ONNX V1.4 12-layer per-token entity base model",
54-
"minSDKVersion": "4.10.0"
55-
},
5644
"pretrained.20210205.microsoft.dte.00.06.bert_example_ner.en.onnx": {
5745
"releaseDate": "02/05/2021",
5846
"modelUri": "https://models.botframework.com/models/dte/onnx/pretrained.20210205.microsoft.dte.00.06.bert_example_ner.en.onnx.zip",
@@ -70,18 +58,6 @@
7058
"modelUri": "https://models.botframework.com/models/dte/onnx/pretrained.20210205.microsoft.dte.00.06.unicoder_multilingual.onnx.zip",
7159
"description": "Bot Framework SDK release 4.10 - Multilingual ONNX V1.4 6-layer per-token intent base model",
7260
"minSDKVersion": "4.10.0"
73-
},
74-
"pretrained.20210205.microsoft.dte.00.06.bert_example_ner_multilingual.onnx": {
75-
"releaseDate": "02/05/2021",
76-
"modelUri": "https://models.botframework.com/models/dte/onnx/pretrained.20210205.microsoft.dte.00.06.bert_example_ner_multilingual.onnx.zip",
77-
"description": "(experimental) Bot Framework SDK release 4.10 - Multilingual ONNX V1.4 6-layer per-token entity base model",
78-
"minSDKVersion": "4.10.0"
79-
},
80-
"pretrained.20210205.microsoft.dte.00.06.tulr_example_ner_multilingual.onnx": {
81-
"releaseDate": "02/05/2021",
82-
"modelUri": "https://models.botframework.com/models/dte/onnx/pretrained.20210205.microsoft.dte.00.06.tulr_example_ner_multilingual.onnx.zip",
83-
"description": "(experimental) Bot Framework SDK release 4.10 - Multilingual ONNX V1.4 6-layer per-token entity base model",
84-
"minSDKVersion": "4.10.0"
8561
}
8662
}
8763
}

0 commit comments

Comments
 (0)