Add `addCleanup` to `unittest.subTest`

[encukou]

Feature or enhancement

Proposal:

The unittest.subTest context (i.e. the object you get using with's as clause, currently None) should get methods to manage cleanups:

• addCleanup
• enterContext
• doCleanups

And similar ones for IsolatedAsyncIOTestCase.

They should do the same thing as the same methods on TestCase, but with subtest scope.
Example usage:

for param in 'a', 'b', 'c':
    with self.subTest() as sub:
        tempfile = make_tempfile()
        sub.addCleanup(os.unlink, tempfile)
        do_actual_test(tempfile, param)

See Discuss thread for motivation/discussion.

Has this already been discussed elsewhere?

I have already discussed this feature proposal on Discourse

Links to previous discussion of this feature:

https://discuss.python.org/t/unittest-add-addcleanup-to-subtest/91827

Linked PRs

• gh-134079: Add addCleanup, enterContext and doCleanups to unittest.subTest and tests #134318

[encukou]

(If anyone wants to take it, go ahead! Otherwise I'll get to this eventually.)

[StanFromIreland]

I can get to this in the evening:-) It's suprising that this was not implemented before!

[ArunRawat404]

If it's alright with you guys, I'd love to take on this issue.

[StanFromIreland]

My implementation is not idea since _SubTest is a subclass and would therefore also expose asserts. I'd be interested to see what you have planned.

[ArunRawat404]

I'm thinking of creating a small helper class that only handles cleanup methods — addCleanup, enterContext, and doCleanups — without inheriting from TestCase. That way, it avoids exposing any extra functionality like assert* methods and keeps things more contained. What are your thoughts on this?

[StanFromIreland]

I thought about that, but IMO it is pretty messy with a lot of duplicated code, so I don’t know. It is probably the only option.

[ArunRawat404]

Yes, it will be messy with duplicated code but I guess it's better than exposing unnecessary functionality. I also not able to think of any other way.

[ArunRawat404]

@encukou What are your thoughts on the implementation for this issue?
I’m considering creating a separate helper class _SubTestCleanupHelper with functionality similar to TestCase, but without exposing unnecessary features like _subTest.
The downside is that it would introduce some duplicated code.

Would love to hear your thoughts or suggestions.

[picnixz]

My 2c:

• Leave the interface as is and expose _SubTest as an opaque object with a documented interface. Let's not care about the extra methods that are added, let's only care about documenting those that should be public.
• Use a proxy object with an overridden __getattr__ that only returns a subset of allowed methods. The rest would raise RuntimeError if they are inherited from TestCase and not overloaded. We shouldn't expect _SubTest to be a public class so we shouldn't worry about users subclassing this class and deal with this in our custom __getattr__.

The first solution would at least make any (CPython) code that is currently using _SubTest internally easier to maintain.

[encukou]

I think it's best to leave _SubTest for reporting, and add a new object for the context.

Don't worry about duplicated code until the solution is working and tested. Then it's time to deduplicate.

[serhiy-storchaka]

Three ways to solve this issue was mentioned on the Discuss thread. With examples:

1. Add methods to subTest.

for param in 'a', 'b', 'c':
    with self.subTest() as sub:
        tempfile = make_tempfile()
        sub.addCleanup(os.unlink, tempfile)
        do_actual_test(tempfile, param)

But we need to add asynchronous variants to subTest in IsolatedAsyncIOTestCase. This means two implementations of subTest.
It's unique advantage is that you can add callbacks for the outer scope and break strict LILO order. But I don't know how large a demand for it. The implementation will be complicated.

2. Add new methods with special infix to TestCase.

for param in 'a', 'b', 'c':
    with self.subTest():
        tempfile = make_tempfile()
        self.addSubTestCleanup(os.unlink, tempfile)
        do_actual_test(tempfile, param)

It may be easier to implement for IsolatedAsyncIOTestCase.

3. Add a fencing mechanism independent from subtests.

for param in 'a', 'b', 'c':
    with self.subTest(), self.fence():
        tempfile = make_tempfile()
        self.addCleanup(os.unlink, tempfile)
        do_actual_test(tempfile, param)

addCleanup(), doCleanups(), etc will only work within a fence. All callbacks added in a fence will be called after leaving a fence. It may help to solve also other issues.

But we will be able to fully understand the advantages and disadvantages only when we try to implement all three options.