Add shared backend test fixtures and helpers for narwhals-migrated tests - #1045
Merged
Merged
Conversation
…r tests
Tests for narwhals-migrated transformers each defined their own way of
building pandas/polars inputs and reading results back (_to_backend,
_assert_values, _cols, _to_dict, _to_pandas, ...), which makes the test
suite hard to maintain. Standardise on one structure:
- tests/conftest.py: `make_df` fixture parametrized over pd.DataFrame and
pl.DataFrame (ids "pandas"/"polars"). Tests that request it run once per
backend; pandas-only tests simply don't request it.
- tests/backend_helpers.py: `to_dict` (contents as {column: values}, NaN
normalised to None), `null_count`, and `make_series` (target on the same
backend as X).
- tests/test_encoding/conftest.py: data shared by the encoder tests, as
fixtures returning plain dicts (missing values written as None) that tests
build with make_df(data).
CountEncoder tests are migrated to this structure: they check the output is
of the input backend with isinstance(X, make_df) and compare contents with
to_dict().
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Shared by the OneHotEncoder, RareLabelEncoder and StringSimilarityEncoder tests, which each defined their own copy of this data. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Sep 15, 2026
solegalli
added a commit
that referenced
this pull request
Sep 15, 2026
The shared helper was renamed from to_dict to frame_to_dict in #1045. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
solegalli
added a commit
that referenced
this pull request
Sep 15, 2026
…gData output type (#1046) * Use shared backend test fixtures and helpers in imputation tests Move the data duplicated across the imputer test files into tests/test_imputation/conftest.py (data_na, and data_na_dob for the two transformers that need a never-null datetime column), and replace the file-local helpers (_cols, _null_count, _values, _same_values, assert_df_equal, _missing_count, _to_list, _make_series) with the shared ones: make_df fixture, to_dict, null_count and make_series. Every transform output is now also checked to be of the input backend with isinstance(X, make_df). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Fix DropMissingData returning a narwhals frame and skipping its pandas path _select_rows() receives the narwhals frame returned by check_X, but still dispatched with nwd.is_pandas_dataframe(X), which is never True for a narwhals frame (narwhals warns about it). As a result: - when no variable was selected (e.g. missing_only=True on a clean training set), transform() returned the narwhals frame itself instead of a pandas/polars dataframe; - the benchmarked pandas fast path never ran, so pandas input silently went through the slower narwhals expression path. Branch on X.implementation.is_pandas() instead, and always return the native frame. Caught by the new isinstance(X, make_df) output checks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use frame_to_dict in imputation tests The shared helper was renamed from to_dict to frame_to_dict in #1045. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * update conftest * update arbitrary imputer tests * update categorical imputer * reorder tests in drop missing data * refactor end tail tests * refactor mean median imputer tests * refactor random sampler and improve seeding procedure --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
solegalli
added a commit
that referenced
this pull request
Sep 15, 2026
The shared helper was renamed from to_dict to frame_to_dict in #1045. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
solegalli
added a commit
that referenced
this pull request
Sep 15, 2026
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
solegalli
added a commit
that referenced
this pull request
Sep 15, 2026
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
solegalli
added a commit
that referenced
this pull request
Sep 15, 2026
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
solegalli
added a commit
that referenced
this pull request
Sep 15, 2026
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
solegalli
added a commit
that referenced
this pull request
Sep 15, 2026
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
solegalli
added a commit
that referenced
this pull request
Sep 15, 2026
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
solegalli
added a commit
that referenced
this pull request
Sep 15, 2026
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
solegalli
added a commit
that referenced
this pull request
Sep 15, 2026
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
solegalli
added a commit
to ojassharma7/feature_engine
that referenced
this pull request
Sep 15, 2026
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
solegalli
added a commit
that referenced
this pull request
Sep 15, 2026
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
solegalli
added a commit
that referenced
this pull request
Sep 15, 2026
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
solegalli
added a commit
that referenced
this pull request
Sep 15, 2026
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
solegalli
added a commit
that referenced
this pull request
Sep 15, 2026
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
solegalli
added a commit
that referenced
this pull request
Sep 15, 2026
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
solegalli
added a commit
that referenced
this pull request
Sep 15, 2026
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
solegalli
added a commit
that referenced
this pull request
Sep 15, 2026
* Migrate MeanEncoder.fit() to narwhals, add polars support
fit() computes, per variable, the mean of y per category (and, with
smoothing="auto", the target variance per category), blended with the
overall target mean via a weight that increases with category count.
transform() and inverse_transform() already came dataframe-agnostic
for free from CategoricalMethodsMixin (base_encoder.py, merged
separately).
Benchmarked a pure-narwhals fit() (group_by/agg for count+mean(+var))
against pandas-native (value_counts + groupby) at 10k-100k rows x
1-10 cols x 5-50 categories: narwhals-on-pandas ran ~1.5x-2.9x slower,
worst at the most common shape (1-2 columns, 50k-100k rows), crossing
the ~1.7x real-loss threshold; narwhals-on-polars was competitive to
faster than pandas-native throughout. Per the benchmark-driven
merge-vs-split rule, and matching what the OrdinalEncoder sibling
migration found for the same y-groupby-by-category shape of fit(),
this splits on `is_pandas = nwd.is_pandas_dataframe(X)`: pandas keeps
a close variant of its original value_counts/groupby code, while
polars (and other narwhals backends) goes through group_by()/agg().
Bug fixed (pre-existing, confirmed against the unmodified file): the
old fit() always called `y.groupby(X[var])`, which raises
AttributeError whenever y is a numpy array rather than a Series -
e.g. list/array-like y input, which sklearn's check_X_y machinery
converts to numpy. This is the exact same bug the OrdinalEncoder
sibling found and fixed in its own fit(). Confirmed failing against
the unmodified file (tests/test_encoding/test_mean_encoder.py::
test_inverse_transform_when_no_unseen, ::test_inverse_transform_when_
ignore_unseen, ::test_inverse_transform_when_encode_unseen, plus
test_check_estimator_encoders.py::test_encoders_when_x_pandas_y_numpy
[encoder1] for MeanEncoder) and now passing. Fixed on the pandas
branch by pairing X[var] with y via `.assign()` when y isn't a Series
(aligns a numpy y positionally, matching how `y.groupby(X[var])`
aligned a Series y by index), and on the narwhals branch via
`nw.new_series` for a numpy y. Unlike OrdinalEncoder, no cross-backend
tie-break fix was needed: MeanEncoder's encoder_dict_ is a
category-to-target-mean mapping (a dict), not a rank-ordered list, so
backend-dependent group order doesn't affect the result - verified
pandas and polars produce identical dicts across smoothing=0.0/100/
"auto" and all three `unseen` settings.
Rewrote every test in test_mean_encoder.py as one
@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) case
per behavior (41 tests, up from 20), using a narwhals-based, NaN-aware
comparison helper; y is passed as a plain list in most tests, which
also exercises the numpy-y bug fix on every parametrized case.
test_variables_cast_as_category stays pandas-only - it exercises
pandas Categorical dtype, which polars has no direct equivalent for.
Verified: tests/test_encoding/test_mean_encoder.py 41 passed (was 20,
3 failing). tests/test_encoding full suite: 345 passed, 13 failed -
same failing test IDs as the unmodified base minus the 4 MeanEncoder-
specific ones fixed here (unmodified base: 17 failed/326 passed);
remaining 13 are pre-existing and unrelated (numpy-X rejection per the
narwhals check_X() contract, affecting every encoder; OrdinalEncoder's
and WoEEncoder's own unmigrated fit() bugs on other in-progress
branches). flake8 and mypy clean. Module imports with pandas blocked
(verified in isolation from unmigrated sibling modules in the
encoding package, which still import pandas on this per-file
migration branch). sphinx -W build clean (only the pre-existing
linkcode_resolve warning). Verified the class docstring example and
every code example in docs/user_guide/encoding/MeanEncoder.rst that
doesn't require the Titanic dataset against real output, and added a
"With polars" section verified the same way; the Titanic-dataset
examples could not be re-run in this sandbox (no network access to
openml.org) but are untouched by this change. Downstream consumers
(feature_engine/_prediction/base_predictor.py and
target_mean_selection.py, which construct MeanEncoder internally)
verified via their test suites: 66 passed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Adapt MeanEncoder to narwhals-returning check_X
check_X_y now returns a narwhals frame, so bind that to nw_X and keep the
original native X for _check_or_select_variables, _check_na,
_get_feature_names_in and the nwd.is_pandas_dataframe(X) fast-path check
(those helpers still expect native input, matching the CategoricalImputer
migration on narwhals-migration). The pandas value_counts/groupby fast
path is unchanged - X stays native so no rehydration is needed. The
narwhals branch reuses nw_X from check_X_y instead of nw.from_native(X).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Use shared backend test fixtures and helpers in MeanEncoder tests
Replace the file-local _to_backend/_assert_values helpers with the shared
test structure: make_df and data_enc* fixtures, y built with make_series on
the backend under test, isinstance(X, make_df) plus to_dict() checks, and
pytest.raises/warns(match=...). Add a test passing the target as a list and
as a numpy array, which take a different code path than a Series.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Use frame_to_dict in MeanEncoder tests
The shared helper was renamed from to_dict to frame_to_dict in #1045.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor mean encoder
* Simplify MeanEncoder narwhals fit and drop init asserts from fit tests
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor mean enc tests
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
solegalli
added a commit
that referenced
this pull request
Sep 15, 2026
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
solegalli
added a commit
that referenced
this pull request
Sep 15, 2026
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
solegalli
added a commit
that referenced
this pull request
Sep 15, 2026
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
solegalli
added a commit
that referenced
this pull request
Sep 15, 2026
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
solegalli
added a commit
that referenced
this pull request
Sep 15, 2026
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
solegalli
added a commit
that referenced
this pull request
Sep 15, 2026
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
solegalli
added a commit
that referenced
this pull request
Sep 15, 2026
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
solegalli
added a commit
that referenced
this pull request
Sep 15, 2026
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
solegalli
added a commit
that referenced
this pull request
Sep 15, 2026
* Migrate OrdinalEncoder.fit() to narwhals, add polars support fit() has two paths: "arbitrary" (X[var].unique()) and "ordered" (target mean per category, via y.groupby(X[var])). transform() and inverse_transform() already came dataframe-agnostic for free from CategoricalMethodsMixin (base_encoder.py, merged separately). Benchmarked a pure-narwhals fit() (group_by/agg/sort for "ordered", unique() for "arbitrary") at 10k-100k rows x 1-10 cols x 5-50 categories: it ran 5x-18x slower than pandas-native fit() at every size tested - a large, consistent loss, unlike the ~1.1x seen for the encode/transform hot path in base_encoder.py. Per the benchmark-driven merge-vs-split rule, this is a real loss, so fit() splits on `is_pandas = nwd.is_pandas_dataframe(X)`: pandas keeps a close variant of its original groupby/unique code (confirmed via a like-for-like full-class benchmark to run within noise of the old code, ~1.0x), while polars (and any other narwhals backend) goes through group_by()/agg()/sort()/unique(). New pandas branch differs from the old code only in how "ordered" pairs y with X[var] (see bug below) - "arbitrary" is untouched. Two real issues found, confirmed against the unmodified pre-migration file (both predate this migration): 1. Bug (fixed): the old "ordered" fit() always called `y.groupby(X[var])`, which raises AttributeError whenever y is a numpy array rather than a Series - e.g. list/array-like y input, which sklearn's check_X_y machinery converts to numpy. This is exactly the scenario tests/test_encoding/test_check_estimator_encoders.py ::test_encoders_when_x_pandas_y_numpy exercises for OrdinalEncoder (encoder2, added in 2022 for issue #376) - it failed against the unmodified file and now passes. Fixed on both the pandas branch (pair X[var] with y via `.assign()`, which aligns a numpy y positionally and a Series y by index, instead of `y.groupby(X[var])`) and the narwhals branch (`nw.new_series` for a numpy y). 2. Cross-backend ordering hazard (avoided, not a regression since old code was pandas-only): grouping by category then sorting by target mean does not, by itself, guarantee the same tie-break order on ties across backends - verified polars reversed two tied categories relative to pandas without it. Old pandas code effectively tie-broke on the category itself (pandas groupby sorts keys ascending by default, and sort_values() is stable). Reproduced that explicitly with a compound sort `.sort([target_name, var])` in the narwhals branch; verified pandas and polars now produce the same dict for a deliberately tied-mean fixture, matching the old code's order exactly. Rewrote every test in test_ordinal_encoder.py as one @pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) case per behavior (43 tests, up from 26), using a narwhals-based, NaN-aware comparison helper. test_variables_cast_as_category stays pandas-only - it exercises pandas Categorical dtype, which polars has no direct equivalent for. Verified: tests/test_encoding/test_ordinal_encoder.py 43 passed. tests/test_encoding full suite: 344 passed, 16 failed - identical failing test IDs to the unmodified base (17 failures, one of which is the bug fixed above), all pre-existing and unrelated to OrdinalEncoder (numpy-X rejection per the narwhals check_X() contract, and MeanEncoder's own unmigrated fit() bug). flake8 and mypy clean. Module imports with pandas blocked. sphinx -W build clean (only the pre-existing linkcode_resolve warning, confirmed identical on the unmodified base). Verified every code example in docs/user_guide/encoding/OrdinalEncoder.rst against real output (California Housing dataset) and added a "With polars" section, verified the same way; the Titanic-dataset examples in that file could not be re-run in this sandbox (no network access to openml.org) but are untouched by this change and were not touched. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Adapt OrdinalEncoder to narwhals-returning check_X check_X / check_X_y now return a narwhals frame, so bind that to nw_X and keep the original native X for _check_or_select_variables, _check_na, _get_feature_names_in and the nwd.is_pandas_dataframe(X) fast-path check (those helpers still expect native input, matching the CategoricalImputer migration on narwhals-migration). The pandas groupby/unique fast path is unchanged - X stays native so no rehydration is needed. The narwhals branch reuses nw_X from check_X / check_X_y instead of nw.from_native(X). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Use shared backend test fixtures and helpers in OrdinalEncoder tests Replace the file-local _to_backend/_assert_values helpers with the shared test structure: make_df and data_enc* fixtures, y built with make_series on the backend under test, isinstance(X, make_df) plus to_dict() checks, and pytest.raises/warns(match=re.escape(msg)). Add a test passing the target as a list and as a numpy array, which take a different code path than a Series. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use frame_to_dict after the shared helper rename in #1045 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor code * Check encoding_method type, simplify OrdinalEncoder fit, group init tests Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use add_target_to_X in OrdinalEncoder Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
solegalli
added a commit
that referenced
this pull request
Sep 15, 2026
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
solegalli
added a commit
that referenced
this pull request
Sep 15, 2026
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
solegalli
added a commit
that referenced
this pull request
Sep 15, 2026
* Migrate WoEEncoder to narwhals, add polars support fit() splits by backend: pandas keeps _calculate_woe()'s existing two-groupby implementation unchanged (it's directly unit-tested for that exact pandas-Series-with-category-index contract); polars/other narwhals backends use one group_by() instead of two, deriving the negative-class count as the complement of the positive-class count per category - benchmarked competitive with, and often faster than, pandas-native at 50k-100k rows. Zero-count-per-class fill_value handling preserved exactly. Bug fix: _check_fit_input() previously assumed y was always a pandas Series (y.nunique()/y.min()/y.max()), breaking on a numpy y (e.g. a plain list/array-like target, which sklearn's check_X_y machinery converts via column_or_1d). Wrapped numpy y into a narwhals Series aligned to X's backend; for pandas specifically, also had to line the wrapped Series up with X's actual index, since _calculate_woe()'s y.groupby(X[var]) aligns by index and a mismatched default RangeIndex silently drops every row instead of raising, leaving encoder_dict_ empty. Fixes test_encoders_when_x_pandas_y_numpy's WoEEncoder case (was failing on the unmigrated file, confirmed pre-existing). Verified: 44/44 own tests, full encoding suite 342 passed/16 failed (was 17 pre-existing on the narwhals-encoding-base baseline - one less here since this branch's own numpy-y bug is now fixed, rest confirmed unrelated), flake8 and mypy clean, sphinx -W build clean (only the pre-existing unrelated linkcode_resolve warning). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Adapt WoEEncoder to narwhals-returning check_X check_X_y now returns a narwhals frame. In _check_fit_input, bind that to nw_X and keep the original native X: the nwd.is_pandas_dataframe(X) check, the native_y.index = X.index alignment and the returned X all need native input, and fit()'s pandas _calculate_woe fast path and nwd checks are then unchanged (X stays native so no rehydration is needed). Take the y-series backend from nw_X.implementation instead of re-wrapping X. In transform(), bind _check_transform_input_and_state to nw_X, keep native X for _check_contains_na, and pass nw_X to _encode (which now expects narwhals). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Use shared backend test fixtures and helpers in WoEEncoder tests Replace the file-local data dicts and assert_df_equal/_none_to_nan helpers with the shared test structure: make_df and data_enc* fixtures, y built with make_series on the backend under test, isinstance(X, make_df) plus to_dict() checks, and pytest.raises/warns(match=re.escape(msg)). Add a test passing the target as a list and as a numpy array, which take a different code path than a Series. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use frame_to_dict after the shared helper rename in #1045 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use add_target_to_X in WoEEncoder, group init tests, match errors Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Compute WoE with narwhals in _calculate_woe, shared by WoEEncoder and SelectByInformationValue Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Replace zero counts by 0.5 in WoE, remove fill_value, add variables_with_zero_counts_ Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
solegalli
added a commit
that referenced
this pull request
Sep 15, 2026
* Migrate OneHotEncoder to narwhals, add polars support
Uses narwhals' to_dummies() for the actual expansion rather than a manual
numpy/dict loop, since it's a real vectorized one-hot op on both backends.
Handles two edge cases to_dummies() doesn't cover directly: a fixed-length
prefix placeholder ("__ohe_tmp__") swapped back out by slicing rather than
by using the real column name, since to_dummies() only prefixes with the
Series name when it's truthy - a falsy real name (e.g. an int column
literally named 0) would otherwise silently drop the prefix; and learned
categories absent from (or present-but-unlearned in) a given transform
batch, filled with an explicit all-0 column so unseen categories are
encoded as 0 across the board, matching the pre-narwhals behavior exactly.
fit()'s value_counts()/unique() calls and transform()'s reassembly are a
single unified narwhals path - no pandas/polars split needed, verified
directly on both backends (identical dummy columns/values for identical
input).
Rewrote tests/test_encoding/test_onehot_encoder.py to the single
cross-backend-parametrized-test convention: local dict fixtures (dropping
the pandas-only global df_enc_big/df_enc_numeric/df_enc_binary fixtures)
parametrized over make_df in [pd.DataFrame, pl.DataFrame], with narwhals-
based column/sum assertions replacing pd.testing.assert_frame_equal.
test_variables_cast_as_category stays pandas-only (pandas category dtype
has no polars equivalent under test there).
Verified: 43/43 own tests, full encoding suite 340 passed/17 pre-existing
failures (matches the narwhals-encoding-base baseline exactly), flake8
and mypy clean, sphinx -W build clean (only the pre-existing unrelated
linkcode_resolve warning), no pandas import in this file itself.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Adapt OneHotEncoder to narwhals-returning check_X
Bind check_X / _check_transform_input_and_state results to nw_X and keep
the original native X for _check_or_select_variables, _check_contains_na
and _get_feature_names_in (those helpers still expect native input,
matching the CategoricalImputer migration on narwhals-migration). Drop
the now-redundant nw.from_native(X) round-trips in fit() and transform();
they reuse the narwhals frame returned by check_X /
_check_transform_input_and_state.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Use shared backend test fixtures and helpers in OneHotEncoder tests
Replace the file-local data dicts and _columns/_colsum helpers with the
shared test structure: make_df and data_enc* fixtures, isinstance(X, make_df)
plus to_dict() checks (keeping the column-order assertions, which are part of
this encoder's output contract), and pytest.raises(match=re.escape(msg)).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Use frame_to_dict after the shared helper rename in #1045
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Group OneHotEncoder init tests, match errors, shorten comments
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Match the fixed get_feature_names_out error message
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor ohe
* fix code style
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
solegalli
added a commit
that referenced
this pull request
Sep 16, 2026
* Migrate RareLabelEncoder to narwhals, add polars support
fit() replaces pandas .unique()/.value_counts(normalize=True) with
narwhals Series.n_unique() (for the cardinality check - matches pandas'
plain unique() length, which counts a null as its own category, unlike
pandas' nunique() which drops it) and drop_nulls().value_counts(sort=True,
normalize=True) (same drop_nulls()/sort=True reasoning as
CountEncoder.fit(): narwhals' value_counts() has no dropna param and
narwhals' own value_counts default is unsorted).
transform() doesn't reuse CategoricalMethodsMixin._encode() (that's a
dict-based numeric remap; this encoder keeps frequent categories as-is
and only replaces the rest), so it's rewritten from pandas'
.loc[~isin(...), feature] = replace_with onto
nw.when(<Series>).then(<Series>).otherwise(nw.lit(replace_with)).alias(
feature). Passing Series from get_column() (not nw.col()) into
when/then/otherwise keeps this working for pandas integer column names,
same as base_encoder.py's precedent. A pandas Categorical column still
needs its own add_categories(replace_with) step before assignment - kept
as a small is_pandas-gated block (structural, like base_encoder.py's
existing reorder branches), since narwhals has no cross-backend
equivalent and polars has no matching restriction. Unlike the old
pandas-only code, no manual object-dtype fixup is needed before
assignment for the ignore_format + numeric-variable + string
replace_with case: narwhals resolves the common dtype itself (object in
pandas, cast-to-string in polars).
Benchmarked pandas-native vs narwhals-on-pandas vs narwhals-on-polars at
10k/50k/100k rows x 1/2/10 columns x 5/50 categories, warmed up. First
pass (zip_with(col, new_series_filled_with_replace_with)) averaged
2.41x pandas-native at 50k-100k rows - most of that cost was
constructing a full same-length replacement Series every transform()
call (~2.5ms of a ~4.8ms transform at 100k rows, confirmed by isolating
just the Series construction). Switched to nw.when(keep).then(col)
.otherwise(nw.lit(replace_with)), which lets the backend broadcast the
scalar instead of materialising a parallel array: dropped the average
to 1.60x, converging to 1.12x-1.54x at 100k rows/10 columns, the
"realistic size" range. narwhals-on-polars is faster than pandas-native
throughout (0.7x-1.5x, mostly <1x at 50k+ rows). Merged into a single
narwhals path per the established decision rule - no pandas/polars
performance split - the remaining overhead is fixed per-call cost, not
scaling cost, and stays under a few ms in absolute terms even at the
largest sizes tested.
Rewrote test_rare_label_encoder.py to one parametrized test per
behaviour over @pytest.mark.parametrize("make_df", [pd.DataFrame,
pl.DataFrame]), replacing the shared pandas-only module-level fixtures
(df_enc_big, df_enc_big_na, df_enc_numeric, from tests/conftest.py,
still used by other encoder test files) with local dict constants both
backends can build from, per the CountEncoder precedent. Kept
test_when_varnames_are_numbers and the three category-dtype tests
pandas-only (integer column names and pandas Categorical dtype are
backend-specific per AGENTS.md). Split
test_max_n_categories_with_numeric_var into a pandas-only version (the
existing str()-workaround test, unchanged) plus a new polars-only
version documenting the real, expected behavioural difference: polars
can't hold mixed int/str values in one column the way pandas' object
dtype does, so a numeric variable with a string replace_with casts the
whole column to string instead of leaving frequent numeric categories
as numbers.
Verified: tests/test_encoding/test_rare_label_encoder.py - 39 passed
(up from 29, from parametrizing over both backends); full
tests/test_encoding suite - 336 passed, 17 pre-existing failures with
identical test IDs confirmed against the unmodified base_encoder.py
baseline (numpy-array-input rejection checks plus 3 MeanEncoder
inverse_transform failures from mean_encoding.py's still-unmigrated
fit() - predate this change, reproduced identically on the unmodified
rare_label.py too). flake8 clean on feature_engine and tests. mypy
clean. Module imports with pandas blocked (loaded standalone, same
technique as the base_encoder.py migration, since sibling encoder files
in this package still import pandas at module level). sphinx -W build
clean (only the pre-existing linkcode_resolve warning, confirmed
identical against the unmodified baseline). Verified every doc example
in RareLabelEncoder.rst against actual output; fixed a pre-existing,
unrelated value_counts() Series-name drift ("Name: var_A" ->
"Name: count", a pandas version difference, not caused by this
migration) while touching that page, and added a verified "With
polars" section to both the class docstring and the user guide (the
polars value_counts() example needed an explicit .sort() - unlike
pandas, its groupby-based value_counts() order isn't stable run to
run). The Titanic-dataset section of the user guide could not be
re-verified against live output in this sandboxed environment (SSL
cert verification blocks urllib by default here, though curl succeeds)
and was left untouched; a workaround (unverified SSL context) showed
matching encoder_dict_/transform output, with only an unrelated
.unique() repr-formatting difference from a newer pandas version.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Adapt RareLabelEncoder to narwhals-returning check_X
Bind check_X / _check_transform_input_and_state results to nw_X and keep
the original native X for _check_or_select_variables, _check_na and
_check_contains_na (those helpers still expect native input, matching the
CategoricalImputer migration on narwhals-migration). Drop the redundant
nw.from_native(X) round-trips in fit() and transform(). In transform(),
detect the pandas Categorical fix-up path via nw_X.implementation
.is_pandas() and run it on a copy so the user's dataframe is not mutated.
Drop the now-unused narwhals.dependencies import.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Use shared backend test fixtures and helpers in RareLabelEncoder tests
Replace the file-local data dicts and the _to_pandas helper - whose
polars to_pandas() call needs pyarrow, which is not a dependency, so the
polars cases failed - with the shared test structure: make_df and
data_enc_big* / data_enc_numeric fixtures, isinstance(X, make_df) plus
to_dict() checks, and pytest.raises/warns(match=re.escape(msg)).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Use frame_to_dict after the shared helper rename in #1045
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Group RareLabelEncoder init tests, match errors, shorten comments
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Fix typo in RareLabelEncoder replace_with error
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* minor refactor to tests
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
solegalli
added a commit
that referenced
this pull request
Sep 16, 2026
* Migrate StringSimilarityEncoder to narwhals, add polars support
fit() rebuilds encoder_dict_ with narwhals cast(nw.String)/value_counts,
matching the CountEncoder/RareLabelEncoder convention. cast() preserves
nulls as null on both pandas and polars (verified empirically), unlike
pandas' own astype(str) which stringifies NaN to "nan" - this lets
"impute" mode fill_null("") directly and "ignore" mode drop_nulls()
before casting, replacing the old "nan"/"<NA>" text-sentinel workaround
with a real null check (col.is_null()) that can't collide with a
genuine category literally named "nan" or "<NA>" (both edge cases stay
covered by test_string_dtype_with_literal_nan_strings).
transform()'s per-row difflib.SequenceMatcher similarity has no
vectorised narwhals equivalent, so it's computed once per unique value
via numpy broadcasting (np.unique's inverse index fans the small
per-unique-value matrix back out to all rows) and reassembled with
nw.new_series()/with_columns(), same pattern DecisionTreeFeatures uses
for externally-computed new columns.
Benchmarked a pandas-specific fast path (X.join(dict-of-columns), as
DecisionTreeFeatures uses) against the unified narwhals with_columns()
here across 10k-100k rows x 1-10 columns x 5-50 categories: assembly
overhead ranges 0.9x-6.25x depending on shape, but the difflib
computation itself dominates wall time by 1-3 orders of magnitude in
every realistic scenario (e.g. 30ms difflib vs <1ms assembly overhead
at 100k rows/20 categories) - even the worst synthetic case (500 output
columns) only costs ~10ms extra out of an already tens-of-ms-to-seconds
transform. Went with the unified/merged implementation: no is_pandas
split, one code path for both backends.
Rewrote tests as single parametrized cases over
@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]),
keeping only the pandas-NA-sentinel tests (np.nan/pd.NA/None,
StringDtype) pandas-only since polars has no equivalent multi-sentinel
behavior to exercise. All doc examples (including the Titanic worked
example) re-verified against actual output; added a "With polars"
section.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Adapt StringSimilarityEncoder to narwhals-returning check_X
Bind check_X / _check_transform_input_and_state results to nw_X and keep
the original native X for _check_or_select_variables and _check_contains_na
(those helpers still expect native input, matching the CategoricalImputer
migration on narwhals-migration). Drop the redundant nw.from_native(X)
round-trips in fit() and transform(). The empty-variables short-circuit in
transform() now returns nw_X.to_native() so callers still get a native
frame.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Use shared backend test fixtures and helpers in StringSimilarityEncoder tests
Replace the file-local data dicts and _to_pandas/_columns helpers with the
shared test structure: make_df and data_enc* fixtures, isinstance(X, make_df)
plus to_dict() checks, and pytest.raises(match=re.escape(msg)). Tests of
pandas-specific NA sentinels and the nullable string dtype stay pandas-only.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Use frame_to_dict after the shared helper rename in #1045
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Check missing_values type, group StringSimilarityEncoder init tests, match errors
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Match the fixed get_feature_names_out error message
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* refactor enc dict at the end
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Standardises how the tests of narwhals-migrated transformers build pandas/polars inputs and check outputs, so every module follows one structure (modelled on the imputation tests) instead of a different set of helpers per transformer (
_to_backend,_assert_values,_cols,_to_pandas, ...).What this PR adds
tests/conftest.py: amake_dffixture parametrized overpd.DataFrame/pl.DataFrame(test idspandas/polars). Tests that request it run once per backend; pandas-only tests simply don't request it.tests/backend_helpers.py:to_dict(X)(contents as{column: values}, NaN normalised to None),null_count(X, col)andmake_series(make_df, values).tests/test_encoding/conftest.py: data shared by the encoder tests (data_enc,data_enc_rare,data_enc_na,data_enc_numeric,data_enc_big,data_enc_big_na,data_enc_top) as fixtures returning plain dicts, built withmake_df(data).Conventions applied in the PRs stacked on this one
conftest.pyas dict fixtures (missing values written asNone); data used by one file stays in that file.yis built withmake_series(exercises the Series code path), plus one list / numpy array test for transformers that takey.assert isinstance(X, make_df)and ato_dict(X)comparison (pytest.approxfor floats).pytest.raises(..., match=re.escape(msg))/pytest.warns(..., match=...).pd.testing.assert_frame_equal.Stacked on this PR (review this one first): the imputation test PR, encoders #1026-#1032, outliers #1033-#1036, discretisation #1037-#1042 and scaling #1004.
Verification
tests/suite: exactly the same failure set asnarwhals-migration(573 pre-existing failures in not-yet-migrated modules and sklearn estimator checks), nothing new.tests/test_encoding/test_count_frequency_encoder.py: 60 passed. flake8 clean.