IMPR: Modernize Project Configuration Management

[dimitri-yatsenko]

Improvement Request

Problem

The current configuration system is nonstandard and lacks common features such as validation. Secrets and regular settings are managed jointly, which is not a best practice.

Requirements

• Load settings hieararchically from project specific locations (. and up), user specific, and system wide.
• Use pydantic to strictly validate all loaded settings against a predefined schema on startup, providing clear errors for any misconfiguration.
• Supports secure management of sensitive data through a .gitignore'd secrets file and environment variables.
• Ensure that environment variables have highest precedence, overriding configuration files
• Clearly documented, explaining loading order, file structure, and validation rules.

[github-actions]

This issue is stale because it has been open for 45 days with no activity.

[dimitri-yatsenko]

Implemented in DataJoint 2.0

Configuration management has been modernized with Pydantic:

Features implemented:

• Pydantic validation: Type-safe settings with automatic validation
• Hierarchical loading: Project → User → System → Defaults
• Secrets separation: .secrets/datajoint.json for credentials (gitignored)
• Environment variables: DJ_HOST, DJ_USER, DJ_PASS, etc. with highest precedence
• Context manager: dj.config.override() for temporary settings
• Template generation: dj.config.save_template() creates starter config

Configuration files:

project/
├── datajoint.json          # Non-sensitive settings (commit to git)
└── .secrets/
    └── datajoint.json      # Credentials (gitignored)

Precedence (highest to lowest):

1. Programmatic settings
2. Environment variables
3. .secrets/datajoint.json
4. datajoint.json
5. Defaults

See src/datajoint/settings.py for implementation.