should/can the install script included in source=() list?

imrehg · 2009-10-29 04:44:52

Hi,

It is a good thing that the sources needed to build a package are md5summed for integrity, but in most packages the install script is not. I was thinking why this might be - the PKGBUILD can be expected to modify the script?

So to try, in one of my packages I added the install script to the source=() list. The package builds of course just fine. Then, however, if I want to upload it to AUR, I try to make a source-only package with "makepkg --source", which brakes with something like this:

ln: creating symbolic link `/tmp/foo/srclinks.YjYgQ8XlE/foo/foo.install': File exists

This is because makepkg tries to create a 2 symbolic links with the same filename: one link for the "foo.install" in the source=() list and also for the install=() list....

Is this deliberate, or a bug? Looks more like a bug to me, but thought I better ask first....
Cheers,
Greg

Snowman · 2009-10-29 06:09:10

It's deliberate. The install script shouldn't be in the source array.

imrehg · 2009-10-29 06:32:43

So, what is the rationale for that? I understand that it is not needed to compile the package but it is indeed necessary to use the package. Thus is there a way (besides including it in the sources list, which is a no-go) to have an md5sum for that too?

shining · 2009-10-29 12:36:41

And where is the md5sum for the pkgbuild itself ?

Anyway your first sentence was right. Some PKGBUILDs modifies the install scriptlet. Typically PKGBUILD for kernel modules.

http://repos.archlinux.org/wsvn/package … k/PKGBUILD

# install script
  sed -i -e "s/KERNEL_VERSION='.*'/KERNEL_VERSION='${_kernver}'/" ${startdir}/${pkgname}.install

imrehg · 2009-10-29 15:37:10

shining wrote:

And where is the md5sum for the pkgbuild itself ?

Yeah, where is it? At least for the core/extra/community packages. Other distributions routinely do that (i.e. sign the released packages). Of course, that question is for a whole other day, so just leave it there. But gave me some ideas, so thanks!

Anyway your first sentence was right. Some PKGBUILDs modifies the install scriptlet. Typically PKGBUILD for kernel modules.
http://repos.archlinux.org/wsvn/package … k/PKGBUILD
# install script
  sed -i -e "s/KERNEL_VERSION='.*'/KERNEL_VERSION='${_kernver}'/" ${startdir}/${pkgname}.install

But this could happen at any point, loads of PKGBUILDS modify the supplied files at it is needed by the package. This however does not stop you from md5summing the original source, idea being that if you know the original md5sum and see what the pkgbuild does then you can be fairly sure that it will behave exactly how you expect it.....
To counter your example from core/kernel26 itself:
http://repos.archlinux.org/wsvn/package … k/PKGBUILD

  # set correct depmod command for install
  sed \
    -e  "s/KERNEL_NAME=.*/KERNEL_NAME=${_kernelname}/g" \
    -e  "s/KERNEL_VERSION=.*/KERNEL_VERSION=${_kernver}/g" \
    -i $startdir/kernel26.install
  sed \
    -e "s|source .*|source /etc/mkinitcpio.d/kernel26${_kernelname}.kver|g" \
    -e "s|default_image=.*|default_image=\"/boot/${pkgname}.img\"|g" \
    -e "s|fallback_image=.*|fallback_image=\"/boot/${pkgname}-fallback.img\"|g" \
    -i ${pkgdir}/etc/mkinitcpio.d/${pkgname}.preset

where mkinitcpio.d/${pkgname}.preset is checksummed, but kernel26.install is not... So does it not look like an artificial distinction?

shining · 2009-10-29 16:16:42

imrehg wrote:

shining wrote:
And where is the md5sum for the pkgbuild itself ?
Yeah, where is it? At least for the core/extra/community packages. Other distributions routinely do that (i.e. sign the released packages). Of course, that question is for a whole other day, so just leave it there. But gave me some ideas, so thanks!

You jumped from integrity check of a pkgbuild to signature of packages ?
Those are completely unrelated for two big reasons :
hash != signature
pkgbuild != package

where mkinitcpio.d/${pkgname}.preset is checksummed, but kernel26.install is not... So does it not look like an artificial distinction?

The distinction of the install scriptlet is hardly artificial.
The install scriptlet is VERY special in multiple ways, here are some :
- it is completely internal to pacman
- it never gets installed to your filesystem , unlike the other source files

And when you think/look at the implementation, the install scriptlet has to be handled in a very special way, completely different to a standard source file.

imrehg · 2009-10-30 03:24:33

shining wrote:

You jumped from integrity check of a pkgbuild to signature of packages ?
Those are completely unrelated for two big reasons :
hash != signature
pkgbuild != package

That's why I wrote that "leave it there", since it is a different, although related topic - outside of the scope of my question.

shining wrote:

where mkinitcpio.d/${pkgname}.preset is checksummed, but kernel26.install is not... So does it not look like an artificial distinction?
The distinction of the install scriptlet is hardly artificial.
The install scriptlet is VERY special in multiple ways, here are some :
- it is completely internal to pacman
- it never gets installed to your filesystem , unlike the other source files
And when you think/look at the implementation, the install scriptlet has to be handled in a very special way, completely different to a standard source file.

It might be completely internal to the pacman but defines actions taken on the system - thus whatever is in .install has effects just like the rest of the files, maybe even more: the other files are just copied, the .install is executed.

And while it might never get installed on the system, it's internals are - how else one could define a "post_uninstall()" if it wasn't stored? Being pacman developer you know much more about this than I do, can you give a bit of info on that?

All I'm trying to get to, is there any serious reason (technical, not philosophical) NOT to checksum the install script as well? Because I'm yet to see any valid arguments against it, and there are loads for it....

shining · 2009-10-30 07:04:14

imrehg wrote:

It might be completely internal to the pacman but defines actions taken on the system - thus whatever is in .install has effects just like the rest of the files, maybe even more: the other files are just copied, the .install is executed.
And while it might never get installed on the system, it's internals are - how else one could define a "post_uninstall()" if it wasn't stored? Being pacman developer you know much more about this than I do, can you give a bit of info on that?

Right it is installed to pacman database, but again, it is pacman doing all that.

All I'm trying to get to, is there any serious reason (technical, not philosophical) NOT to checksum the install script as well? Because I'm yet to see any valid arguments against it, and there are loads for it....

Taking your example from kernel26 :

 -i $startdir/kernel26.install
 -i ${pkgdir}/etc/mkinitcpio.d/${pkgname}.preset

Note that the scriptlet edited is the original one in $startdir , because that's the only place where it needs to be, from the packager pov.
While for the preset file, it needs to be copied to $pkgdir and it is edited there.
If you had a checksum for the scriptlet, it would also mean that makepkg -g output could be altered after running makepkg. Very weird..

There might be ways to fix that, though I don't even see any good ones. But why bother ? You said there are loads of valid arguments to have it , but you didn't even give one !

Btw the whole scriptlet file could be created entirely from the pkgbuild build function.

What are you worried about here ? Security ? If you think adding a checksum for the install file is going to give you any security, you are badly fooling yourself. If someone is distributing a malicious pkgbuild, it can contain the md5sum of the malicious scriptlet which is shipped with it.

The only security for pkgbuilds is yourself. When you get one from a strange place, you MUST read the pkgbuild/install files.

imrehg · 2009-10-30 08:37:08

shining wrote:

imrehg wrote:
It might be completely internal to the pacman but defines actions taken on the system - thus whatever is in .install has effects just like the rest of the files, maybe even more: the other files are just copied, the .install is executed.
And while it might never get installed on the system, it's internals are - how else one could define a "post_uninstall()" if it wasn't stored? Being pacman developer you know much more about this than I do, can you give a bit of info on that?
Right it is installed to pacman database, but again, it is pacman doing all that.

Fair enough.... The detail, however, that is not installed does not preempt checksumming. Eg. patches are also: 1) used for the package, 2) not installed, and 3) still in the checksum....

shining wrote:

All I'm trying to get to, is there any serious reason (technical, not philosophical) NOT to checksum the install script as well? Because I'm yet to see any valid arguments against it, and there are loads for it....
Taking your example from kernel26 :
 -i $startdir/kernel26.install
 -i ${pkgdir}/etc/mkinitcpio.d/${pkgname}.preset
Note that the scriptlet edited is the original one in $startdir , because that's the only place where it needs to be, from the packager pov.
While for the preset file, it needs to be copied to $pkgdir and it is edited there.
If you had a checksum for the scriptlet, it would also mean that makepkg -g output could be altered after running makepkg. Very weird..

Ok, this is indeed the case. But if you look at real packages, it seems that after the package is compiled, many times they update the PKGBUILD with the new md5sum values. Again: core/kernel26.

That .install update section in the PKGBUILD is then relegated to a "helper function" role that keep the information in sync when there's a new release, but as the PKGBUILD is distributed, it won't change anything in the .install anymore...

Thus, again, there's no real difference between the .install files and other files used as source.

shining wrote:

Btw the whole scriptlet file could be created entirely from the pkgbuild build function.

Sure, and that's a great thing! I actually wondered why so many packages have separate .install when it can be done in the PKGBUILD. Thought it might be some convention, but probably just habit.... I prefer doing it in the PKGBUILD as well, whenever possible...

shining wrote:

There might be ways to fix that, though I don't even see any good ones. But why bother ? You said there are loads of valid arguments to have it , but you didn't even give one !

On the other hand, I'm yet to find any of your rebuttals that are not rebutted here again...

shining wrote:

What are you worried about here ? Security ? If you think adding a checksum for the install file is going to give you any security, you are badly fooling yourself. If someone is distributing a malicious pkgbuild, it can contain the md5sum of the malicious scriptlet which is shipped with it.

I think you misunderstood me. I'm not complaining, but inquiring. The PKGBUILD system is awesome, and it makes me search out programs that I use but not in the repo/AUR so that I can make a new package. But also, I'm interested in the "how" and "why", and hope to improve things if possible. Now, to me it seems theres an unreasonable distinction to checksum some files needed for a package but not all of them. And thus I set out to ask the elders here on the Forum to see whether someone can shed light on the reasons of the current state. Not trying to piss of anybody, but being interested.

Of course the md5sum is not security, it's a help, nothing more.

As an example: some time ago, there was a package in the community (thus managed by TUs) that had an update. I wanted to check out something in the package, got the PKGBUILD and tried to make it myself - md5sum says that the sources are incorrect.... Obviously something must have gone very bad, because the md5sum field was updated according to the SVN, the package is available on pacman thus it had to build, and the sources used in the package have not been changed for a while so that cannot be a problem....
I've got a package from a trusted source and it is broken, the md5sum told me. So the breakage can be the very same way in the install script, why is the md5sum not allowed to tell me that?

shining wrote:

The only security for pkgbuilds is yourself. When you get one from a strange place, you MUST read the pkgbuild/install files.

Of course, never argued against it. But there are different layers of verbosity....

Anyway....

shining · 2009-10-30 09:07:53

imrehg wrote:

As an example: some time ago, there was a package in the community (thus managed by TUs) that had an update. I wanted to check out something in the package, got the PKGBUILD and tried to make it myself - md5sum says that the sources are incorrect.... Obviously something must have gone very bad, because the md5sum field was updated according to the SVN, the package is available on pacman thus it had to build, and the sources used in the package have not been changed for a while so that cannot be a problem....

And what was the reason of the problem finally ?
Was it a remote source or a local source file ?

I think it makes a lot of sense to have integrity check of remote source files, because these are out of control.
However, the local source files are on the same level than the PKGBUILD, so in control of the one distributing the pkgbuild.

Back to the original problem, install files are not the only special ones. There is also changelog.
Anyway, what would you do ? Include scriptlet both in install= line and in source ? That seems repetitive.

imrehg · 2009-11-10 02:50:27

Never really known the exact problem, but I think it was remote. The maintainer did a few more updates, and finally it was okay...
There was a discrepancy between the files reported by the PKGBUILD and the actual downloaded versions, and this I could see only because of the md5sum fields, so I'm glad it was there.

I think it makes a lot of sense to have integrity check of remote source files, because these are out of control.
However, the local source files are on the same level than the PKGBUILD, so in control of the one distributing the pkgbuild.

Define local.... I think every file related to building a project is remote - I have to download from a (potentially unreliable) server. So In my opinion there's no real difference between the sources and the package "recipe" files....

Back to the original problem, install files are not the only special ones. There is also changelog.
Anyway, what would you do ? Include scriptlet both in install= line and in source ? That seems repetitive.

That is a very fair question, and I had to take some time to think about it. I'm not trying to say I have any solution that is better than other people's, I'm just throwing in ideas, so maybe we can build better things....

So: don't know much about the ChangeLog because not too many packages use it, and it's not on the wiki (that would be a good thing to write about). As much as I understand, the ChangeLog is even more special file, because it has a fixed required name. Is it really so? Because if it is, then how about this idea for the .install:
* Make the default filename for the install file ${pkgname}.install
* If there's such a file (at the end of the build() ), then it does not need to be included in PKGBUILD (can, but not need to) and automatically used as an install script.
* If there's no such file, then no install script for that package
--> This would be very similar behaviour to ChangeLog as I understand
BUT:
* Additionally: the install= variable can define alternative install file if needed, that would override the default, eg. when there's different install for the different architectures
* The install file (and: any and all files related to building a package) need to be included in the sources list, or created by the PKGBUILD itself

Does this make sense? The default name gets rid of the repetitiveness (I completely agree on that). Still it would move the install file to be on equal footing with other sources besides the PKGBUILD. And I think this is minimal change compared to the current settings. Any opinions?

These days I'm working a bit with Ubuntu as well, and their packaging system is very verbose, not sure you've tried it: They have require changelogs for every update, checksumming on all files, signing the file that is equivalent to our PKGBUILD.... They have some different design choices, and not trying to advocate adobting their structure - only the practices that are useful for us too.
Not sure if you ever tried it, their Personal Packaging Archive (where basically you can send a PKGBUILD equivalent and they build it and distribute it for you) is something that we could totally do one day with AUR. But I think the PKGBUILD format needs some rethinking before that.
Please note: this last paragraph is out of the scope of the original question, merely to put it into more perspective and to make it more clear where's the question coming from.

shining · 2009-11-10 09:39:35

imrehg wrote:

Define local.... I think every file related to building a project is remote - I have to download from a (potentially unreliable) server. So In my opinion there's no real difference between the sources and the package "recipe" files....

If you download from a potentially unreliable server, you are FUCKED. You would need a completely different solution to protect you from that.

So why don't you focus on the real problem that concerns you, and propose a real solution, instead of wasting your time and mine on something completely irrelevant and useless.

So: don't know much about the ChangeLog because not too many packages use it, and it's not on the wiki (that would be a good thing to write about). As much as I understand, the ChangeLog is even more special file, because it has a fixed required name. Is it really so? Because if it is, then how about this idea for the .install:
* Make the default filename for the install file ${pkgname}.install
* If there's such a file (at the end of the build() ), then it does not need to be included in PKGBUILD (can, but not need to) and automatically used as an install script.
* If there's no such file, then no install script for that package
--> This would be very similar behaviour to ChangeLog as I understand

We took exactly the opposite way for the next makepkg release. Changelog will be very similar to install there.
This change was needed to deal correctly with split pkgbuilds.

These days I'm working a bit with Ubuntu as well, and their packaging system is very verbose, not sure you've tried it: They have require changelogs for every update, checksumming on all files, signing the file that is equivalent to our PKGBUILD.... They have some different design choices, and not trying to advocate adobting their structure - only the practices that are useful for us too.

Don't you think that would be MUCH MORE useful when downloading from a potentially unreliable server ?

imrehg · 2009-11-10 10:09:20

shining wrote:

imrehg wrote:
Define local.... I think every file related to building a project is remote - I have to download from a (potentially unreliable) server. So In my opinion there's no real difference between the sources and the package "recipe" files....
If you download from a potentially unreliable server, you are FUCKED. You would need a completely different solution to protect you from that.

"Potentially unreliable" meaning any. Please, remember, this package I mentioned was in community... From ftp://ftp.archlinux.org. And I don't trust any of the packages anyway, the ones I care about I build myself - that's how the mentioned problem was actually found - because I cared enough that I rebuilt the official package myself....

So why don't you focus on the real problem that concerns you, and propose a real solution, instead of wasting your time and mine on something completely irrelevant and useless.

I'm sorry, if you feel it is wasting your time, please don't read whatever I answer. For me it is certainly not wasted, irrelevant or useless. What's my real problem? Any remotely reliable way to protect against accidental errors - that's e.g. checksumming all relevant files. The verifiability is a different issue (not discussed here) and on Arch I cannot see any movement in that direction. And I proposed a solution, that's a different question that you don't like it. That's fair.

So: don't know much about the ChangeLog because not too many packages use it, and it's not on the wiki (that would be a good thing to write about). As much as I understand, the ChangeLog is even more special file, because it has a fixed required name. Is it really so? Because if it is, then how about this idea for the .install:
* Make the default filename for the install file ${pkgname}.install
* If there's such a file (at the end of the build() ), then it does not need to be included in PKGBUILD (can, but not need to) and automatically used as an install script.
* If there's no such file, then no install script for that package
--> This would be very similar behaviour to ChangeLog as I understand
We took exactly the opposite way for the next makepkg release. Changelog will be very similar to install there.
This change was needed to deal correctly with split pkgbuilds.

If you have a technical issue (split packages) that can be solved this way that is a fair thing to say. But don't think that that is the only way... To give the example of Ubuntu again (note: only because that's the only alternative I know, not because I want that on Arch or want to have Ubuntu instead), they CAN have multiple packages from single source file, all checksummed. So I'd say what makepkg does is one way, but certainly not the best one.

Proposing solution that would include split packages and other things - I'd have to try them more to have a lot better feeling for them. Then I come and bother the community again with more ideas.

These days I'm working a bit with Ubuntu as well, and their packaging system is very verbose, not sure you've tried it: They have require changelogs for every update, checksumming on all files, signing the file that is equivalent to our PKGBUILD.... They have some different design choices, and not trying to advocate adobting their structure - only the practices that are useful for us too.
Don't you think that would be MUCH MORE useful when downloading from a potentially unreliable server ?

Sure it would! But the two things are complementary... Right now I don't think it is any safer to download a package from ftp://ftp.archlinux.org than from any mirror.... If it was a different conversation, I'd ask you how would you be sure at the moment, but I'm not asking that at all..

shining · 2009-11-10 10:33:26

Going back to your original report.

If you propose a simple patch to fix makepkg --source , it would likely be accepted.
And by fixing , I just mean to allow install scriptlet to be in source array, instead of failing with File exists.

Also it might help to move this discussion to pacman-dev ML, to include more people in the discussion.
Because this is not going anywhere. I don't understand your concerns, and you don't understand mine. But maybe someone else would be able to conciliate us

imrehg · 2009-11-11 03:07:06

shining wrote:

Also it might help to move this discussion to pacman-dev ML, to include more people in the discussion.
Because this is not going anywhere. I don't understand your concerns, and you don't understand mine. But maybe someone else would be able to conciliate us

Yeah, there's certainly a level of misunderstanding. I think I know why you find my comments irrelevant, but that does not help me to make them cleaner.

So probably it is down to coding now, see what comes out, and have something more concrete to discuss. Cheers.

some-guy94 · 2009-11-11 03:11:41

I didn't read the whole thread but you could just md5sum it youself

if [ $(md5sum $startdir/${pkgname}.install | cut -d' ' -f1) != "e0gj86949v8f0whatever" ]; then
return
fi

imrehg · 2009-11-11 03:16:21

some-guy94 wrote:

I didn't read the whole thread but you could just md5sum it youself
if [ $(md5sum $startdir/${pkgname}.install | cut -d' ' -f1) != "e0gj86949v8f0whatever" ]; then
return
fi

that's so dirty
shows the strength (big flexibility) and weakness (too much flexibility) of the PKGBUILD format... I think I'll try to make a non-workaround solution. But thanks for this one, just in case!

Allan · 2009-11-11 04:23:51

We are going to have to do something about install= and changelog= (new in git) within the package functions as these are missed by "makepkg --source" and other tools. Potentially that will allow easy checksuming of these files.

imrehg · 2009-11-11 06:23:53

Allan wrote:

We are going to have to do something about install= and changelog= (new in git) within the package functions as these are missed by "makepkg --source" and other tools. Potentially that will allow easy checksuming of these files.

Nice, straight to the point. I can really see from previous comments that it is indeed work in progress. I'll check out the git repo and see whether I can comment on that one. Cheers!

Arch Linux

#1 2009-10-29 04:44:52

should/can the install script included in source=() list?

#2 2009-10-29 06:09:10

Re: should/can the install script included in source=() list?

#3 2009-10-29 06:32:43

Re: should/can the install script included in source=() list?

#4 2009-10-29 12:36:41

Re: should/can the install script included in source=() list?

#5 2009-10-29 15:37:10

Re: should/can the install script included in source=() list?

#6 2009-10-29 16:16:42

Re: should/can the install script included in source=() list?

#7 2009-10-30 03:24:33

Re: should/can the install script included in source=() list?

#8 2009-10-30 07:04:14

Re: should/can the install script included in source=() list?

#9 2009-10-30 08:37:08

Re: should/can the install script included in source=() list?

#10 2009-10-30 09:07:53

Re: should/can the install script included in source=() list?

#11 2009-11-10 02:50:27

Re: should/can the install script included in source=() list?

#12 2009-11-10 09:39:35

Re: should/can the install script included in source=() list?

#13 2009-11-10 10:09:20

Re: should/can the install script included in source=() list?

#14 2009-11-10 10:33:26

Re: should/can the install script included in source=() list?

#15 2009-11-11 03:07:06

Re: should/can the install script included in source=() list?

#16 2009-11-11 03:11:41

Re: should/can the install script included in source=() list?

#17 2009-11-11 03:16:21

Re: should/can the install script included in source=() list?

#18 2009-11-11 04:23:51

Re: should/can the install script included in source=() list?

#19 2009-11-11 06:23:53

Re: should/can the install script included in source=() list?

Board footer