Skip to content

[EPIC] Complete Span (source location) information / feature #1548

Open

Description

@alamb

This ticket tracks the work remaining to complete adding source location information into sqlparser

Background

  • @Nyrox and @iffyio introduced the foundations for storing source location information in the AST nodes in Implement Spanned to retrieve source locations on AST nodes #1435. This information can be used to provide more specific error messages, and potentially syntax highlighting among other great things.
  • In order to 1. minimize the disruption to downstream projects that use sqlparser-rs and 2. avoid a single massive PR and 3. work together as a community, we are implementing this feature incrementally over several releases.

Let's use this ticket to organize needed / remaining work. If you find additional features are needed / issues, please leave a comment on this ticket

Source Span Contributing Guidelines

For contributing source spans improvement in addition to the general
contribution guidelines, please make sure to pay attention to the
following:

  • Ident always have correct source spans

  • We try to minimize downstream breaking changes

  • Consider using Span::union in favor of storing spans on all nodes

  • Any metadata added to compute spans must not change semantics (Eq, Ord, Hash, etc.). See AttachedToken for more information.

When adding support for source spans on a type, consider the impact to consumers of that type and whether your change would require a consumer to do non-trivial changes to their code.

Example of a trivial change

match node {  
   ast::Query { 
     field1,
     field2, 
     location: _,  // add a new line to ignored location
 }

If adding source spans to a type would require a significant change like wrapping the type, please open an issue to discuss. 

# AST Node Equality and Hashes

When adding tokens to AST nodes, make sure to store them using the [AttachedToken](https://docs.rs/sqlparser/latest/sqlparser/ast/helpers/struct.AttachedToken.html) (TODO UPDATE SOURCE REFERENCE)to ensure that semantically equivalent AST nodes compare as equal and hash to the same value. i.e. `select 5` and `SELECT 5` would compare as different `Select` nodes, if the select token was stored directly. f.e.

```rust
struct Select {
     select_token: AttachedToken, // only used for spans
     /// remaining fields
     field1,
     field2,
     ...
 }

Some high level work (list from #1435)

  • Store keyword TokenWithLocation for expressions that currently don't have them
  • Implement spans for the rest of the AST, namely Statements

Tasks

Activity

  1. alamb commented on Nov 26, 2024

    @alamb
    ContributorAuthor

    I think the initial thing we should do to kick this project is get a test pattern setup:

    Then we should be good to start cranking out location information

  2. eliaperantoni commented on Jun 5, 2025

    @eliaperantoni
    Contributor

    I noticed that CASE .. END expressions don't have correct spans because the locations of the heading and trailing keywords are not considered. ie. the following test fails:

    #[test]
    fn test_case_expr_span() {
        let sql = "CASE 1 WHEN 2 THEN 3 ELSE 4 END";
        let mut parser = Parser::new(&GenericDialect {}).try_with_sql(sql).unwrap();
        assert_eq!(
            parser.parse_expr().unwrap().span(),
            Span::new(Location::new(1, 1), Location::new(1, sql.len() as u64 + 1))
        );
    }

    I'm about to open a PR that fixes the issue following the footsteps of what was done for Select: including the AttachedTokens for the keywords in the AST node and using them in the Spanned impl.

    Should I make a subticket for this? @alamb

  3. agis commented on Jun 5, 2025

    @agis

    #1858 is also related to this

  4. alamb commented on Jun 6, 2025

    @alamb
    ContributorAuthor

    Should I make a subticket for this? @alamb

    Sure! I always think tickets help the process (and you never know who might pick them up!)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions