IS NULL and NOT NULL Filters not working as expected on INT64 and VARCHAR Fields

[rohitreddy1698]

Hi Team,

We are facing an issue where one of our INT64 fields, say field_B is set to be Nullable: true, still the IS NULL and IS NOT NULL filters are not working properly.

for (CreateCollectionReq.FieldSchema fieldSchema : fieldSchemaList) {
            System.out.println(String.format("Field: %s, DataType: %s, IsPrimary: %s, AutoID: %s, Nullable: %s, DefaultValue: %s",
                    fieldSchema.getName(),fieldSchema.getDataType(),fieldSchema.getIsPrimaryKey(),fieldSchema.getAutoID(),
                    fieldSchema.getIsNullable(),fieldSchema.getDefaultValue()));
        }
.
.

Field: field_B, DataType: Int64, IsPrimary: false, AutoID: false, Nullable: true, DefaultValue: null


String filter = "field_B IS NOT NULL";
QueryReq queryReq = QueryReq.builder()
                .collectionName(collection)
                .filter(filter)
                .outputFields(List.of("field_A_PrimaryKey","field_B"))
                .limit(10)
                .build();
        QueryResp queryResp = client.query(queryReq);

{field_B=23534553, field_A_PrimaryKey=00000109-27f9-4186-9ad3-b2dc5500bd5b_0}
{field_B=null, field_A_PrimaryKey=0000018f-d8a7-44e6-ab93-4b48ea4b12b4_0}
{field_B=null, field_A_PrimaryKey=0000018f-d8a7-44e6-ab93-4b48ea4b12b4_1}
{field_B=null, field_A_PrimaryKey=00000522-0ff2-4846-bfd1-c59030a6c3d3_0}
{field_B=null, field_A_PrimaryKey=00000587-5bb8-44e2-9b6f-1cba884eb4fe_0}
{field_B=null, field_A_PrimaryKey=000006cc-2a32-4184-9ea7-4ff46eb58130_0}
{field_B=null, field_A_PrimaryKey=000006e4-8095-4c90-a150-914b5a65abf7_0}
{field_B=null, field_A_PrimaryKey=000006e4-8095-4c90-a150-914b5a65abf7_1}
{field_B=null, field_A_PrimaryKey=00000a30-e4f5-4702-b711-ffc26f569553_0}
{field_B=null, field_A_PrimaryKey=00000a30-e4f5-4702-b711-ffc26f569553_1}

Same issue for varchar fields as well.

Thanks,
Rohit

[rohitreddy1698]

Also the filter does on VARCHAR fields is throwing this error :
Is it mandatory to have a scalar field for supporting null and not null filters on the VARCHAR fields ?

10:43:55.517 [main] ERROR io.milvus.v2.utils.RpcUtils -- QueryRequest collectionName:test_collection_name, databaseName:null failed, error code: 2, reason: fail to Query on QueryNode 3018: worker(3026) query failed: Operator::GetOutput failed for [Operator:PhyFilterBitsNode, plan node id: 1412950] :  => span() interface is not implemented for variable column at /workspace/source/internal/core/src/mmap/ChunkedColumn.h:322

Exception in thread "main" io.milvus.v2.exception.MilvusClientException: fail to Query on QueryNode 3018: worker(3026) query failed: Operator::GetOutput failed for [Operator:PhyFilterBitsNode, plan node id: 1412950] :  => span() interface is not implemented for variable column at /workspace/source/internal/core/src/mmap/ChunkedColumn.h:322

	at io.milvus.v2.utils.RpcUtils.handleResponse(RpcUtils.java:63)
	at io.milvus.v2.service.vector.VectorService.query(VectorService.java:194)
	at io.milvus.v2.client.MilvusClientV2.lambda$query$31(MilvusClientV2.java:506)
	at io.milvus.v2.utils.RpcUtils.retry(RpcUtils.java:100)
	at io.milvus.v2.client.MilvusClientV2.query(MilvusClientV2.java:506)
	at com.apple.milvus.examples.MilvusNullTest.executeQuery(MilvusNullTest.java:27)
	at com.apple.milvus.examples.MilvusNullTest.main(MilvusNullTest.java:70)

Process finished with exit code 1

Even after having a scalar index on this field the query is still failing.

DescribeIndexResp.IndexDesc(fieldName=var_char_field, indexName=var_char_field, id=458340402841824382, indexType=INVERTED, metricType=INVALID, extraParams={}, indexedRows=21982592, totalRows=21982592, pendingIndexRows=0, indexState=Finished, indexFailedReason=, properties={})

Can you please help us with this.

Thanks,
Rohit

[rohitreddy1698]

Hi @xiaofan-luan

Can you please help us on this ?

Thanks,
Rohit

[xiaofan-luan]

might be milvus-io/milvus#42554

could you try with the latest 2.5.13?

you might need to drop the interved index and rebuild