Skip to content

Conversation

@dimitri-yatsenko
Copy link
Member

Summary

Removes text from core DataJoint types. It remains available as a native SQL passthrough type (with portability warning).

Rationale

Core types should encourage structured, bounded data:

Need Better Alternative
Bounded strings varchar(n) with explicit limit
Structured text json
Large text files <object>
Notes/descriptions varchar(4000) typically sufficient

Problems with text as a core type:

  • No size constraint = unpredictable storage
  • Behavior varies across MySQL/PostgreSQL
  • Encourages dumping unstructured data
  • Hurts query performance with unbounded fields

Changes

declare.py:

  • Remove text from CORE_TYPES
  • Update NATIVE_TEXT pattern: r"(tiny|small|medium|long)?text$" (now includes plain text)

Documentation:

  • Update archive docs to note text is native-only
  • Add guidance on alternatives (varchar, json, object)

Migration

Users who need text can:

# Option 1: Bounded varchar (preferred)
notes : varchar(4000)

# Option 2: JSON for structured content  
config : json

# Option 3: Object storage for large text
document : <object>

# Option 4: Native text (with warning)
description : text  # Works but shows portability warning

Related

🤖 Generated with Claude Code

`text` is no longer a core DataJoint type. It remains available as a
native SQL passthrough type (with portability warning).

Rationale:
- Core types should encourage structured, bounded data
- varchar(n) covers most legitimate text needs with explicit bounds
- json handles structured text better
- <object> is better for large/unbounded text (files, sequences, docs)
- text behavior varies across databases, hurting portability

Changes:
- Remove `text` from CORE_TYPES in declare.py
- Update NATIVE_TEXT pattern to match plain `text` (in addition to
  tinytext, mediumtext, longtext)
- Update archive docs to note text is native-only

Users who need unlimited text can:
- Use varchar(n) with generous limit
- Use json for structured content
- Use <object> for large text files
- Use native text types with portability warning

Co-Authored-By: Claude Opus 4.5 <[email protected]>
@github-actions github-actions bot added bug Indicates an unexpected problem or unintended behavior enhancement Indicates new improvements documentation Issues related to documentation labels Jan 9, 2026
@dimitri-yatsenko dimitri-yatsenko merged commit d38a670 into pre/v2.0 Jan 9, 2026
8 checks passed
@dimitri-yatsenko dimitri-yatsenko deleted the fix/remove-text-core-type branch January 9, 2026 21:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Indicates an unexpected problem or unintended behavior documentation Issues related to documentation enhancement Indicates new improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants