html.parser produces different output than documented

[mouchen626]

When parsing >>> using html.parser, the actual output differs from the expected behavior as documented.

Run the following code:

from html.parser import HTMLParser
from html.entities import name2codepoint

class MyHTMLParser(HTMLParser):
    def handle_starttag(self, tag, attrs):
        print("Start tag:", tag)
        for attr in attrs:
            print("     attr:", attr)

    def handle_endtag(self, tag):
        print("End tag  :", tag)

    def handle_data(self, data):
        print("Data     :", data)

    def handle_comment(self, data):
        print("Comment  :", data)

    def handle_entityref(self, name):
        c = chr(name2codepoint[name])
        print("Named ent:", c)

    def handle_charref(self, name):
        if name.startswith('x'):
            c = chr(int(name[1:], 16))
        else:
            c = chr(int(name))
        print("Num ent  :", c)

    def handle_decl(self, data):
        print("Decl     :", data)

parser = MyHTMLParser()
parser.feed('>>>')

According to the documentation, the expected output should be:

Named ent: >
Num ent  : >
Num ent  : >

The actual output is:

Data     : >>>

Linked PRs

• gh-131535: Fix stale example in html.parser docs, make examples doctests #131551
• [3.13] gh-131535: Fix stale example in html.parser docs, make examples doctests (GH-131551) #133587
• [3.14] gh-131535: Fix stale example in html.parser docs, make examples doctests (GH-131551) #133589

[brianschubert]

Hi! This is the expected behavior. You need to set convert_charrefs to False in order for handle_entityref / handle_charref to be called (the default is True):

>>> parser = MyHTMLParser(convert_charrefs=False)
>>> parser.feed('>>>')
Named ent: >
Num ent  : >
Num ent  : >

From the docs:

HTMLParser.handle_entityref(name)
This method is called to process a named character reference of the form &name; (e.g. >), where name is a general entity reference (e.g. 'gt'). This method is never called if convert_charrefs is True.

HTMLParser.handle_charref(name)
This method is called to process decimal and hexadecimal numeric character references of the form &#NNN; and &#xNNN;. For example, the decimal equivalent for > is >, whereas the hexadecimal is >; in this case the method will receive '62' or 'x3E'. This method is never called if convert_charrefs is True.

[mouchen626]

Thank you, I understand now. However, the example code at the bottom of this documentation page does not explicitly set convert_charrefs=False, yet it still produces results as if convert_charrefs were True. This has confused me. Would it be better to refine this example for clarity?

[brianschubert]

Ah, in that case I agree the example should be clarified. It looks like it wasn't updated when the default for convert_charref was changed from False to True in Python 3.5.

It would be good to migrate the examples to .. doctest:: blocks at the same time to help catch this sort of discrepancy.