Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mypy crashes with overloads of functools.singledispatch #8356

Open
hauntsaninja opened this issue Feb 2, 2020 · 6 comments
Open

mypy crashes with overloads of functools.singledispatch #8356

hauntsaninja opened this issue Feb 2, 2020 · 6 comments

Comments

@hauntsaninja
Copy link
Contributor

@hauntsaninja hauntsaninja commented Feb 2, 2020

I was trying out the advice I gave in #8354, and encountered a mypy crash:

~/delete λ cat test22.py                                                                                         2 
from typing import *
import functools


@overload
def foo(val: str) -> str: ...
@overload
def foo(val: int) -> int: ...


@functools.singledispatch
def foo(x):
    return x


@foo.register
def _(x: int) -> int:
    return x + 1
~/delete λ mypy test22.py --show-traceback
test22.py:5: error: INTERNAL ERROR -- Please try using mypy master on Github:
https://mypy.rtfd.io/en/latest/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.770+dev.0b9089e9c9be2ad0b345b1d3743f808365d4aefc
Traceback (most recent call last):
  File "/usr/local/bin/mypy", line 11, in <module>
    load_entry_point('mypy', 'console_scripts', 'mypy')()
  File "/Users/shantanu/dev/mypy/mypy/__main__.py", line 8, in console_entry
    main(None, sys.stdout, sys.stderr)
  File "/Users/shantanu/dev/mypy/mypy/main.py", line 89, in main
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 166, in build
    sources, options, alt_lib_path, flush_errors, fscache, stdout, stderr, extra_plugins
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 234, in _build
    graph = dispatch(sources, manager, stdout)
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 2612, in dispatch
    process_graph(graph, manager)
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 2919, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 3018, in process_stale_scc
    graph[id].type_check_first_pass()
  File "/Users/shantanu/dev/mypy/mypy/build.py", line 2111, in type_check_first_pass
    self.type_checker().check_first_pass()
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 293, in check_first_pass
    self.accept(d)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 400, in accept
    stmt.accept(self)
  File "/Users/shantanu/dev/mypy/mypy/nodes.py", line 515, in accept
    return visitor.visit_overloaded_func_def(self)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 433, in visit_overloaded_func_def
    self._visit_overloaded_func_def(defn)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 460, in _visit_overloaded_func_def
    self.check_overlapping_overloads(defn)
  File "/Users/shantanu/dev/mypy/mypy/checker.py", line 481, in check_overlapping_overloads
    assert isinstance(inner_type, CallableType)
AssertionError: 
test22.py:5: : note: use --pdb to drop into pdb
@jeanmonet
Copy link

@jeanmonet jeanmonet commented Feb 4, 2020

Same here:

Trying to combine typing.overload with functools.singledispatch, which gives a mypy internal error. The internal error disappears if I remove functools.singledispatch functionnality.

Using Python 3.7 with mypy 0.750

Python code:

from datetime import datetime, date
import typing
from typing import Union
from functools import singledispatch

from typing_extensions import Literal

DateInType = Union[date, str]

@typing.overload    #    <---- THIS IS ***line 50*** of td.py raising mypy internal error
def date2str(t: DateInType, out_format: Literal['str']) -> str: ...
@typing.overload
def date2str(t: DateInType, out_format: Literal['date']) -> date: ...

@singledispatch
def date2str(t, out_format):
    ''' Converts:
        - datetime.date to string (YYYY-MM-DD)
        - string (YYYY-MM-DD, YYYY/MM/DD, DD-MM-YYYY, ...) to datetime.date
        
        Second argument = 'out_format' can be 'str' or 'date'.
        If specified, returns output in string (YYYY-MM-DD) ['str'] or datetime.date format ['date'].
    '''

@date2str.register(date)
def _date2str_date(t, out_format='str'):
    ''' Input format = datetime.date.
        Output format, if string = YYYY-MM-DD'''
    if out_format == 'str':
        return t.strftime('%Y-%m-%d')
    elif isinstance(t, datetime):
        return t.date()
    else:
        return t

@date2str.register(str)
def _date2str_str(t, out_format='date'):
    ''' Input format = string (YYYY-MM-DD, DD-MM-YYYY, ...)'''
    dformats = ('%Y-%m-%d', '%Y/%m/%d', '%d-%m-%Y', '%d/%m/%Y', '%Y%m%d')
    for i, dformat in enumerate(dformats, start=1):
        try:
            d = datetime.strptime(t, dformat).date()
            break
        except ValueError:
            if i == len(dformats):
                raise
    if out_format == 'date':
        return d
    else:
        return date2str(d, out_format=out_format)


dt = datetime.now()
res = date2str(dt, out_format='date')
print(res, type(res))
assert type(res) == date

res2 = date2str(dt.date(), out_format='str')
print(res2, type(res2))
assert type(res2) == str

res3 = date2str('20061005', out_format='date')
print(res3, type(res3))
assert type(res3) == date

mypy traceback:

======== RESULTS ========
Traceback (most recent call last):
  File "/home/username/anaconda3/envs/py3/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/username/anaconda3/envs/py3/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/username/anaconda3/envs/py3/lib/python3.7/site-packages/mypy/__main__.py", line 12, in <module>
    main(None, sys.stdout, sys.stderr)
  File "/home/username/anaconda3/envs/py3/lib/python3.7/site-packages/mypy/main.py", line 89, in main
    res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
  File "/home/username/anaconda3/envs/py3/lib/python3.7/site-packages/mypy/build.py", line 167, in build
    sources, options, alt_lib_path, flush_errors, fscache, stdout, stderr, extra_plugins
  File "/home/username/anaconda3/envs/py3/lib/python3.7/site-packages/mypy/build.py", line 235, in _build
    graph = dispatch(sources, manager, stdout)
  File "/home/username/anaconda3/envs/py3/lib/python3.7/site-packages/mypy/build.py", line 2620, in dispatch
    process_graph(graph, manager)
  File "/home/username/anaconda3/envs/py3/lib/python3.7/site-packages/mypy/build.py", line 2929, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/home/username/anaconda3/envs/py3/lib/python3.7/site-packages/mypy/build.py", line 3028, in process_stale_scc
    graph[id].type_check_first_pass()
  File "/home/username/anaconda3/envs/py3/lib/python3.7/site-packages/mypy/build.py", line 2119, in type_check_first_pass
    self.type_checker().check_first_pass()
  File "/home/username/anaconda3/envs/py3/lib/python3.7/site-packages/mypy/checker.py", line 292, in check_first_pass
    self.accept(d)
  File "/home/username/anaconda3/envs/py3/lib/python3.7/site-packages/mypy/checker.py", line 399, in accept
    stmt.accept(self)
  File "/home/username/anaconda3/envs/py3/lib/python3.7/site-packages/mypy/nodes.py", line 515, in accept
    return visitor.visit_overloaded_func_def(self)
  File "/home/username/anaconda3/envs/py3/lib/python3.7/site-packages/mypy/checker.py", line 432, in visit_overloaded_func_def
    self._visit_overloaded_func_def(defn)
  File "/home/username/anaconda3/envs/py3/lib/python3.7/site-packages/mypy/checker.py", line 459, in _visit_overloaded_func_def
    self.check_overlapping_overloads(defn)
  File "/home/username/anaconda3/envs/py3/lib/python3.7/site-packages/mypy/checker.py", line 480, in check_overlapping_overloads
    assert isinstance(inner_type, CallableType)
AssertionError: 
typing_test/td.py:50: error: INTERNAL ERROR -- Please try using mypy master on Github:
https://mypy.rtfd.io/en/latest/common_issues.html#using-a-development-mypy-build
Please report a bug at https://github.com/python/mypy/issues
version: 0.750

Am I doing something wrong?

@mattany
Copy link

@mattany mattany commented Apr 7, 2020

Hello, I am an open source noob and would like to try and tackle this issue. Is that possible?

@hauntsaninja
Copy link
Contributor Author

@hauntsaninja hauntsaninja commented Apr 7, 2020

Yes, of course! Welcome. If you get stuck, feel free to ask questions here.

@mattany
Copy link

@mattany mattany commented Apr 8, 2020

I couldn't replicate the bug on my machine. Using Python 3.8.2 and using my forked version of mypy (v0.770 I believe), copying @jeanmonet's code yielded the following output:

2020-04-08 <class 'datetime.date'>
2020-04-08 <class 'str'>
2006-10-05 <class 'datetime.date'>

Process finished with exit code 0

I also tried replicating the error using your (@hauntsaninja ) code but I admit I didn't really understand what was going on there (what does the lambda mean?)

cheers

@hauntsaninja
Copy link
Contributor Author

@hauntsaninja hauntsaninja commented Apr 8, 2020

My repro continues to repro for me (against mypy master). Note it contains the exact command I ran.

The output you copied looks like it would be the result of running the program (with the Python interpreter), which is expected to work — this is a bug in mypy, not in Python itself. Remember mypy is a static checker, it doesn't actually run your code. If you still can't get it to repro, try sharing the exact commands you're running here.

I assume by lambda you mean the def function named _? _ is a valid identifier, like any other, in Python. It doesn't have any special meaning, although it's commonly used as a name when you don't need to refer to the named thing (e.g. to make a list of 1s, [1 for _ in range(10)]).
What is special is the functools.singledispatch decorator applied to it, where use of functions named _ is a common idiom. I'd recommend reading https://docs.python.org/3/library/functools.html#functools.singledispatch and playing around with the foo function my repro defines.

If you need a reference for what overload does (and to understand the code better, since the mypy crash occurs while running some overload checking related code) , I recommend https://mypy.readthedocs.io/en/stable/more_types.html#function-overloading. In general, I've found the mypy docs really helpful.

@graingert
Copy link
Contributor

@graingert graingert commented Apr 26, 2020

see #2904

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants
You can’t perform that action at this time.