Arch Linux

shpelda

Member

Registered: 2008-08-07

Posts: 59

python, distutils, data_files - Yay! I know where my data files are.

After a long and painful research i have finally found usable way to make distutils propagate location of data files to installed application.

Python's distutils allow user to specify location for data_files but there's no standard way to acess them.
Yes sure ,there are few workarounds :
1) distributing data_files as package_data
  - Do you really want to have ie. README.txt in /usr/lib/python/... ?
2) Accessing those data_file using $sys.prefix
- hmm. Here you hardcode $sys.prefix relative path to setup.py like this:

 setup(...., data_files=['share/myverycoolpackage/doggie.gif'], ..)

- and access it this way:

 os.path.join(sys.prefix, 'share/myverycoolpackage/doggie.gif')

This works fine, but you're making assertions about layout of users $sys.prefix directory. Not talking about possibility to specify --data-dir during installation.

Both are not really usable. Here goes the solution:

Idea: Install additional 'package.ini' file, that will get modified during install and will hold location of data files.
This package.ini will be installed as package_data, so that it will be accessible relatively to sources.

And this is how i did it:
New command to install process:

from os.path import *
import re
class package_ini(Command):
    """
        Locate package.ini in all installed packages and patch it as requested
        by wildcard references to install process.
    """

    user_options = []
    def initialize_options(self):
        pass
    def finalize_options(self):
        pass

    def visit(self, dirname, names):
        packages = self.distribution.get_command_obj(build_py.__name__).packages
        if basename(dirname) in packages:
            if 'package.ini' in names:
                self.patch(join(dirname, 'package.ini'))

    def patch(self, ini_file):
        print 'patching file' + ini_file
        with open(ini_file,'r') as infile:
            file_data = infile.readlines()
        with open(ini_file,'w') as outfile:
            for line in file_data:
                _line = self.patch_line(line)
                if _line:
                    line = _line
                outfile.write(line)

    def patch_line(self, line):
        """
            Patch an installed package.ini with setup's variables
        """
        match = re.match('(?P<identifier>\w+)\s*=.*##SETUP_PATCH\\((?P<command>.*)\.(?P<variable>.*)\\)', line)
        if not match:
            return line
        print 'Replacing:'+line
        line = match.group('identifier')
        line += ' = '
        data = '(self).distribution.get_command_obj(\''+\
                match.group('command')+'\')'+'.'+\
                match.group('variable')
        line += '\''+eval(data)+'\''
        line += '\n'
        print 'With:' + line
        return line

Apend this package_ini command to install command:

class install(_install):
from distutils.command.install import install as _install
    sub_commands = _install.sub_commands + [
           (package_ini.__name__, None)
        ]

Mention your package.ini in setup():

setup(...,  package_data={'myverycoolpackage':['package.ini']}, data_files='myverycoolpackage','doggie.gif',..)

Write your package.ini like this:

__data_dir__ = '../../data' ##SETUP_PATCH(install_data.install_dir)

Package.ini will get patched properly during install, so if i assume that user specified --data-dir=C:/Program Files/ApplicationData/verycoolpackage, package.ini will be:

__data_dir__ = 'C:/Program Files/Application Data/verycoolpackage'

Notice that you can access any property from setup script, so this might be usable for version string or other stuff.
Also notice that in 'develoment environment' location of data files differs from 'installation environment'. Cute, isn't it?.

Now you just have to evaluate package ini (propably in myverycoolpackage/__init__.py):

with open(inipath) as ini:
    for line in ini.readlines():
        exec(line)

And we're done.