Skip to content

feat(dataframe): expose withColumn and unnestColumns #41

Description

@andygrove

Is your feature request related to a problem or challenge?

The DataFrame API gained dropColumns and withColumnRenamed in #30
but the most common column-shaping primitive — adding or replacing a
column with an expression — is still missing. unnestColumns for
struct / list flattening is in the same family.

Describe the solution you'd like

  • DataFrame.withColumn(String name, String expr). The expression is
    a SQL fragment parsed via DataFusion's parse_sql_expr, matching the
    convention already used by DataFrame.filter(String). Behaviour
    mirrors DataFusion::DataFrame::with_column — replaces a column of
    the same name in place, otherwise appends.
  • DataFrame.unnestColumns(String... columns) and an overload
    accepting an UnnestOptions value (preserve-nulls, recursions). Maps
    to unnest_columns / unnest_columns_with_options.
  • Tests in DataFrameTransformationsTest (round-trip on a small
    fixture, schema check, error on unknown column).

Describe alternatives you've considered

SELECT a + b AS c, * via ctx.sql(...). Works for withColumn, but
requires the DataFrame to be registered as a table first; awkward for
intermediate transformations.

Additional context

Follows the same JNI shape as #19's filter(String)parse_sql_expr
on the native side, no Java-side Expr model required. Keeps this
issue independent of #N (joins issue) where an Expr builder may
eventually be needed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions