Datetime NoneType after calling Py_Finalize and Py_Initialize #71587
Comments
|
After calling Py_Finalize and Py_Initialize I get the message "attribute of type 'NoneType' is not callable" on the datetime.strptime method. Example: from datetime import datetime
s = '20160505 160000'
refdatim = datetime.strptime(s, '%Y%m%d %H%M%S')The first call works fine but it crashes after the re initialization. Workaround: from datetime import datetime
s = '20160505 160000'
try:
refdatim = datetime.strptime(s, '%Y%m%d %H%M%S')
except TypeError:
import time
refdatim = datetime.fromtimestamp(time.mktime(time.strptime(s, '%Y%m%d %H%M%S')))Related Issue: bpo-17408 ("second python execution fails when embedding") |
DennyWeinberg
mannequin
added
interpreter-core
|
Just to be clear: The error happens after these steps:
Now we get the error "attribute of type 'NoneType' is not callable" |
|
Thanks for the report Denny. Looking at https://hg.python.org/cpython/file/30099abdb3a4/Modules/datetimemodule.c#l3929, there's a problematic caching of the "_strptime" module that is almost certainly the cause of the problem - it will attempt to call _strptime._strptime from the already finalized interpreter rather than the new one. It should be possible to adjust that logic to permit a check for _strptime._strptime being set to None, and reimporting _strptime in that case. |
|
Aye, skipping the caching entirely would be an even simpler solution - the only thing it is saving in the typical case is a dictionary lookup in the modules cache. |
|
Nick: Looks like it's quite a bit more work than just a dict lookup. That PyImport_ImportModuleNoBlock call (which seems odd; the implementation of NoBlock is just to wrap the blocking function; guess we don't allow non-blocking imports anymore and this is just to avoid changing all the names elsewhere?) involves a *lot* more work than just a dict lookup (it devolves to a PyImport_Import call https://hg.python.org/cpython/file/3.5/Python/import.c#l1743 , which basically does everything involved in the import process aside from actually reading/parsing the file unconditionally, because of how weird While it's not a perfect comparison, compare: >>> import _strptime # It's now cached
# Cache globals dict for fair comparison without globals() call overhead
>>> g = globals()
# Reimport (this might be *more* expensive at C layer, see notes below)
>>> %timeit -r5 import _strptime
1000000 loops, best of 5: 351 ns per loop
# Dict lookup (should be at least a bit cheaper at C layer if done equivalently, using GetAttrId to avoid temporary str)
>>> %timeit -r5 g['_strptime']
10000000 loops, best of 5: 33.1 ns per loop
# Cached reference (should be *much* cheaper at C layer)
>>> %timeit -r5 _strptime
100000000 loops, best of 5: 19.1 ns per loopNote: I'm a little unclear on whether a Python function implemented in C has its own globals, or whether it's simulated as part of the C module initialization); if it lacks globals, then the work done for PyImport_Import looks like it roughly doubles (it has to do all sorts of work to simulate globals and the like), so that 351 ns per re-import might actually be costlier in C. Either way, it's a >10x increase in cost to reimport compared to a dict lookup, and ~18x speedup over using a cached reference (and like I said, I think the real cost of the cheaper options would be much less in C, so the multiplier is higher). Admittedly, in tests, empty string calls to |
|
Hmm... On checking down some of the code paths and realizing there were some issues in 3.5 (redundant code, and what looked like two memory leaks), I checked tip (to avoid opening bugs on stale code), and discovered that bpo-22557 rewrote the import code, reducing the cost of top level reimport by ~60%, so my microbenchmarks (run on Python 3.5.0) are already out of date for 3.6's faster re-import. Even so, caching wasn't a wholly unreasonable optimization before now, and undoing it now still has a cost, if a smaller one. |
|
PEP-3121 is a big change. Can we use PyModuleDef->m_clear() for a clever hack? |
|
Yes, I think something like the attached patch may do the trick. |
|
Wouldn't it clear strptime_module when a subinterpreter shuts down, too? It's not a big deal because it can't cause a crash. |
|
Having to (re-)fill the cache once per interpreter seems like a reasonable price to pay. Why is 3.5 not included? Did this not cause problems before the import change, or is it just that this bug is small enough that maybe it isn't worth backporting? |
|
Any news here? 3.6.0 is also affected by this bug. |
|
This issue should have a much higher priority as it basically breaks Python embedding unless the user either does not re-initialize the interpreter or avoid the use of _datetime.strptime. We're currently testing with a patch based on Christian and Alexander's idea but using m_free as using m_clear and Py_CLEAR neither makes sense to me nor did it work in conjunction with Py_Finalize when testing it. A version matching the current tip is attached. We did not run into any issues so far (and the only thing I can think of is clearing the static variable too often and thus causing some extra imports). |
|
I've added Steve Dower to the nosy list, as he's done some work recently on making the Windows builds more embedding-friendly, and I believe at least some of that time may have been funded work. Unfortunately, we don't currently have anyone I'm aware of that's being specifically paid to improve or maintain CPython's embedding support in general, so getting attention for these kinds of interpreter reinitialization bugs can be a bit hit-or-miss. |
|
Hi, This issue results in bugs in KODI, see xbmc/xbmc#17311 .
|
|
FYI if anyone wants to fix this, you need to make the following line not static: cpython/Modules/_datetimemodule.c Line 5196 in 5b946d3 Now performance might not suck in this instance if you just change that line, but who knows without testing (last time this was looked at was the 3.5 days and things have obviously changed since then). Otherwise datetime should get updated for multi-phase init or do something such that caching |
|
@brettcannon - there are two patches attached to this issue from 6 years ago. One is mine and another by Christopher Schramm (@cschramm). Let me convert Christopher's patch to a PR so that we can test and review it. |
abalkin
added
extension-modules
|
It looks like embedding the datetime module is completely broken: % ./Programs/_testembed test_repeated_init_exec 'print("import datetime");import datetime'
--- Loop #1 ---
import datetime
--- Loop #2 ---
import datetime
_testembed(90330,0x2093e0100) malloc: *** error for object 0x100db3a50: pointer being freed was not allocated
_testembed(90330,0x2093e0100) malloc: *** set a breakpoint in malloc_error_break to debugHere is the backtrace: |
Yes, it is. The underlying C extension module diff --git a/Modules/_datetimemodule.c b/Modules/_datetimemodule.c
index eda8c5610b..f2e549f525 100644
--- a/Modules/_datetimemodule.c
+++ b/Modules/_datetimemodule.c
@@ -6681,9 +6681,15 @@ datetime_destructor(PyObject *op)
PyMem_Free(ptr);
}
+static int initialized;
+
static int
_datetime_exec(PyObject *module)
{
+ if (initialized) {
+ return 0;
+ }
+
// `&...` is not a constant expression according to a strict reading
// of C standards. Fill tp_base at run-time rather than statically.
// See https://bugs.python.org/issue40777
@@ -6860,6 +6866,7 @@ _datetime_exec(PyObject *module)
if (us_per_hour == NULL || us_per_day == NULL || us_per_week == NULL) {
return -1;
}
+ initialized = 1;
return 0;
}
cc. @kumaraditya303 |
|
@kumaraditya303: I have a proof-of-concept branch for datetime ready at https://github.com/erlend-aasland/cpython/tree/isolate-datetime/poc. There's still some work to do with it, but it is an ok starting point for now. The diff is rather large, so we should consider breaking it up in multiple PRs. Issues:
FWIW, Footnotes
|
You mean there are refleaks? Or the test doesn't like running multiple times? (I assume A quick look at your changes, they seem like what I'd expect. I'm not sure that breaking them up would help anything - apart from the failures, it's pretty mechanical. |
|
Yeah, that's a typo. Multiple test runs result in test failures, not refleaks. Sorry for not specifying that. |
|
As noted in the footnotes of my previous post, I think there is a benefit from converting datetime to Argument Clinic before doing this change. |
SGTM |
Okay but before proceeding to that let's finish #101948 first. |
* Use explicit initialiser for m_base * Add module state stub; establish global state on stack * Put conversion factors in state struct * Move PyDateTime_TimeZone_UTC to state * Move PyDateTime_Epoch to state struct * Fix ref leaks in and clean up initialisation
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
Linked PRs
_datetime#102995_datetime#110475The text was updated successfully, but these errors were encountered: