Skip to content

INGOREME: CBloomfilter integration (STEP 2) #23565

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
cpegeric wants to merge 126 commits into
base: main
Choose a base branch
from
Draft

INGOREME: CBloomfilter integration (STEP 2) #23565

cpegeric wants to merge 126 commits into from

Conversation

@cpegeric
Copy link
Contributor

@cpegeric cpegeric commented Jan 20, 2026 •
edited
Loading

What type of PR is this?

  • API-change
  • BUG
  • Improvement
  • Documentation
  • Feature
  • Test and CI
  • Code Refactoring

Which issue(s) this PR fixes:

issue #23551

What this PR does / why we need it:

  1. CBloomFilter in C
  2. integration with readutil.
  3. HNSW filtered search with bloom filter
  4. move WaitBloomFilter to sqlexec and wait bloom filter right before run SQL so that centroids computation and bloomfilter build task can run in parallel.
  5. change the build bloomfilter location to ivf search. hashbuild only send unique join keys.
  6. introduce variable ivf_preload_entries to enable preload centroid bloomfilters
  7. introduce variable ivf_small_centroid_threshold so that when finding closest centroids, centroid with size smaller than threshold will still be included in selected lists and total number of selected centroid will be increased by one, i.e. probe_limit + 1.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@XuPeng-SH XuPeng-SH Awaiting requested review from XuPeng-SH XuPeng-SH is a code owner

@aunjgr aunjgr Awaiting requested review from aunjgr aunjgr is a code owner

@fengttt fengttt Awaiting requested review from fengttt fengttt is a code owner

@iamlinjunhong iamlinjunhong Awaiting requested review from iamlinjunhong

@gouhongshen gouhongshen Awaiting requested review from gouhongshen gouhongshen will be requested when the pull request is marked ready for review gouhongshen is a code owner

@ouyuanning ouyuanning Awaiting requested review from ouyuanning ouyuanning will be requested when the pull request is marked ready for review ouyuanning is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

kind/feature size/XXL Denotes a PR that changes 2000+ lines

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@cpegeric @matrix-meow @XuPeng-SH