You are not logged in.

#1 2020-04-25 12:58:20

teckk
Member
Registered: 2013-02-21
Posts: 518

Python module loading.

I did a little test.

Test1.py

#!/usr/bin/python

#Module loading test

def t1():
    import multiprocessing
    
def t2():
    from multiprocessing import Process, Value, Array
    
def t3():
    from PyQt5.QtGui import QFont
    from PyQt5.QtCore import Qt, QUrl, pyqtSignal
    from PyQt5.QtNetwork import QNetworkCookie
    from PyQt5.QtWidgets import QWidget, QHBoxLayout, QApplication
    from PyQt5.QtWebEngineCore import QWebEngineUrlRequestInterceptor
    from PyQt5.QtWebEngineWidgets import (QWebEnginePage, 
                                QWebEngineView, QWebEngineSettings)
                                
def t4():
    import PyQt5

   
Clear RAM before and between loading modules.

sudo sysctl vm.drop_caches=3

python -mtimeit -s 'import Test1' 'Test1.t1()'
1 loop, best of 5: 523 nsec per loop
:0: UserWarning: The test results are likely unreliable. The worst time (2.34 usec) was more than four times slower than the best time (523 nsec).
python -mtimeit -s 'import Test1' 'Test1.t1()'
1000000 loops, best of 5: 293 nsec per loop
python -mtimeit -s 'import Test1' 'Test1.t1()'
1000000 loops, best of 5: 288 nsec per loop
python -mtimeit -s 'import Test1' 'Test1.t1()'
1000000 loops, best of 5: 297 nsec per loop

sudo sysctl vm.drop_caches=3

python -mtimeit -s 'import Test1' 'Test1.t2()'
1 loop, best of 5: 1.94 usec per loop
:0: UserWarning: The test results are likely unreliable. The worst time (7.84 usec) was more than four times slower than the best time (1.94 usec)
python -mtimeit -s 'import Test1' 'Test1.t2()'
200000 loops, best of 5: 1.47 usec per loop
python -mtimeit -s 'import Test1' 'Test1.t2()'
200000 loops, best of 5: 1.47 usec per loop
python -mtimeit -s 'import Test1' 'Test1.t2()'
200000 loops, best of 5: 1.45 usec per loop

sudo sysctl vm.drop_caches=3

python -mtimeit -s 'import Test1' 'Test1.t3()'
1 loop, best of 5: 6.14 usec per loop
:0: UserWarning: The test results are likely unreliable. The worst time (26.3 usec) was more than four times slower than the best time (6.14 usec).
python -mtimeit -s 'import Test1' 'Test1.t3()'
50000 loops, best of 5: 5.82 usec per loop
python -mtimeit -s 'import Test1' 'Test1.t3()'
50000 loops, best of 5: 5.62 usec per loop
python -mtimeit -s 'import Test1' 'Test1.t3()'
50000 loops, best of 5: 5.69 usec per loop

sudo sysctl vm.drop_caches=3

python -mtimeit -s 'import Test1' 'Test1.t4()'
1000000 loops, best of 5: 301 nsec per loop
python -mtimeit -s 'import Test1' 'Test1.t4()'
1000000 loops, best of 5: 292 nsec per loop
python -mtimeit -s 'import Test1' 'Test1.t4()'
1000000 loops, best of 5: 303 nsec per loop

And those result are backwards from what I thought that I would get.
I thought that loading just what you needed was faster.

Someone comment on why that test is invalid.

Offline

#2 2020-04-25 15:02:58

whitie
Member
Registered: 2011-03-13
Posts: 21

Re: Python module loading.

Hello teckk,
you can try it on your own:

Python 3.8.2 (default, Apr  8 2020, 14:31:25) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import dis
>>> dis.dis('import multiprocessing')
  1           0 LOAD_CONST               0 (0)
              2 LOAD_CONST               1 (None)
              4 IMPORT_NAME              0 (multiprocessing)
              6 STORE_NAME               0 (multiprocessing)
              8 LOAD_CONST               1 (None)
             10 RETURN_VALUE
>>> dis.dis('from multiprocessing import Process, Value, Array')
  1           0 LOAD_CONST               0 (0)
              2 LOAD_CONST               1 (('Process', 'Value', 'Array'))
              4 IMPORT_NAME              0 (multiprocessing)
              6 IMPORT_FROM              1 (Process)
              8 STORE_NAME               1 (Process)
             10 IMPORT_FROM              2 (Value)
             12 STORE_NAME               2 (Value)
             14 IMPORT_FROM              3 (Array)
             16 STORE_NAME               3 (Array)
             18 POP_TOP
             20 LOAD_CONST               2 (None)
             22 RETURN_VALUE

There is simply more to do. But if you care about these loading times, take a compiled language.

Greets
Whitie

Offline

#3 2020-04-25 15:46:03

teckk
Member
Registered: 2013-02-21
Posts: 518

Re: Python module loading.

sudo sysctl vm.drop_caches=3

python -X importtime -c "from tkinter import *"
import time: self [us] | cumulative | imported package
import time:      1118 |       1118 | _frozen_importlib_external
import time:      8003 |       8003 |   time
import time:       486 |       8489 | zipimport
import time:       161 |        161 |     _codecs
import time:     43445 |      43606 |   codecs
import time:      8588 |       8588 |   encodings.aliases
import time:     82135 |     134328 | encodings
import time:     12792 |      12792 | encodings.utf_8
import time:       333 |        333 | _signal
import time:      2758 |       2758 | encodings.latin_1
import time:       107 |        107 |     _abc
import time:     22423 |      22530 |   abc
import time:     38396 |      60925 | io
import time:       133 |        133 |       _stat
import time:     10139 |      10271 |     stat
import time:     29884 |      29884 |     _collections_abc
import time:     14903 |      14903 |       genericpath
import time:     15936 |      30839 |     posixpath
import time:     16825 |      87817 |   os
import time:      8787 |       8787 |   _sitebuiltins
import time:       405 |        405 |   sitecustomize
import time:        77 |         77 |   usercustomize
import time:     43197 |     140281 | site
import time:     17778 |      17778 |     types
import time:     22371 |      40149 |   enum
import time:    429193 |     429193 |   _tkinter
import time:     24785 |      24785 |   tkinter.constants
import time:       164 |        164 |       _sre
import time:     15774 |      15774 |         sre_constants
import time:      8786 |      24560 |       sre_parse
import time:     23706 |      48428 |     sre_compile
import time:       282 |        282 |           _operator
import time:     11059 |      11341 |         operator
import time:     20694 |      20694 |         keyword
import time:     11215 |      11215 |           _heapq
import time:     18275 |      29490 |         heapq
import time:       200 |        200 |         itertools
import time:     16630 |      16630 |         reprlib
import time:       146 |        146 |         _collections
import time:     22241 |     100738 |       collections
import time:        90 |         90 |       _functools
import time:     20384 |     121211 |     functools
import time:       153 |        153 |     _locale
import time:      1808 |       1808 |     copyreg
import time:     16394 |     187992 |   re
import time:     51050 |     733167 | tkinter

sudo sysctl vm.drop_caches=3

python -X importtime -c "from tkinter import Frame, Tk, Menu, Label, Scrollbar"
import time: self [us] | cumulative | imported package
import time:      1827 |       1827 | _frozen_importlib_external
import time:      6646 |       6646 |   time
import time:       455 |       7101 | zipimport
import time:       139 |        139 |     _codecs
import time:     35580 |      35718 |   codecs
import time:     16538 |      16538 |   encodings.aliases
import time:     73783 |     126038 | encodings
import time:     12386 |      12386 | encodings.utf_8
import time:       821 |        821 | _signal
import time:     10277 |      10277 | encodings.latin_1
import time:       109 |        109 |     _abc
import time:     22458 |      22566 |   abc
import time:     38658 |      61224 | io
import time:       142 |        142 |       _stat
import time:     18670 |      18812 |     stat
import time:     29582 |      29582 |     _collections_abc
import time:     14877 |      14877 |       genericpath
import time:     15925 |      30802 |     posixpath
import time:     17439 |      96633 |   os
import time:      8233 |       8233 |   _sitebuiltins
import time:      7753 |       7753 |   sitecustomize
import time:       102 |        102 |   usercustomize
import time:     55290 |     168009 | site
import time:     17791 |      17791 |     types
import time:     22409 |      40199 |   enum
import time:    404174 |     404174 |   _tkinter
import time:     24817 |      24817 |   tkinter.constants
import time:       176 |        176 |       _sre
import time:     15777 |      15777 |         sre_constants
import time:      9026 |      24803 |       sre_parse
import time:     24043 |      49020 |     sre_compile
import time:       283 |        283 |           _operator
import time:      5071 |       5354 |         operator
import time:     18302 |      18302 |         keyword
import time:     11206 |      11206 |           _heapq
import time:     18289 |      29494 |         heapq
import time:       231 |        231 |         itertools
import time:     16621 |      16621 |         reprlib
import time:       145 |        145 |         _collections
import time:     22223 |      92368 |       collections
import time:        89 |         89 |       _functools
import time:     19854 |     112310 |     functools
import time:       144 |        144 |     _locale
import time:      1756 |       1756 |     copyreg
import time:     16365 |     179592 |   re
import time:     81546 |     730327 | tkinter

Thanks, I learn something new every other day.

Offline

#4 2020-04-25 16:31:57

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,442
Website

Re: Python module loading.

The import times are not all you should care about.  In fact, they're what you should care about least as they only happen once at startup.  I have not tested it, but I'd not be at all surprised if the (trivial) increase in time in using "from X import Y" over "import X" might result in a (similarly trivial) decrease in time each time you "Y.foo()" versus "X.Y.foo()".

If this was the case, then overall, the "from X import Y" could very well be more efficient as importing happens once, using something from the imported library/namespace is likely to happen many manytimes while a script is running.

EDIT: I just tested comparing the following two scripts and found the second to have a (vanishingly small but consistent) overall speed benefit:

#!/bin/python

import os

for x in range(1000000):
	os.getuid()
#!/bin/python

from os import getuid

for x in range(1000000):
	getuid()

Overall, though, these differences are less than trivial.  There are far better places to focus your optimization goals.

Last edited by Trilby (2020-04-25 16:39:42)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Online

#5 2020-04-26 05:25:08

eschwartz
Fellow
Registered: 2014-08-08
Posts: 4,097

Re: Python module loading.

There should not be any large difference between "from X import Y; Y()" and "import X; X.Y()".

In both cases, the X module is loaded, and inserted into the sys.modules cache and *preserved there*. No matter how many times you re-import X, it is only getting the version stored in memory, so you don't pay the full cost of importing it the first time.

The difference is in the scoping rules. In the former case, the current interpreter context treats X as available, in the latter case, Y is available.

This has two effects:

- it is faster (by a very little bit) to call Y() than to call X.Y() because each invocation only looks up the function, instead of looking up the module and then looking up the function; the more times you invoke Y(), especially in a loop of one million times...
- your namespace doesn't have direct access to the X module, nor to any functions it provides other than Y().

Finally, I don't believe the timeit module is very wise to use for testing imports, since it has access to the global import cache...

Last edited by eschwartz (2020-04-26 05:35:07)


Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

Offline

Board footer

Powered by FluxBB