A discussion about universally/programmatically parsable PKGBUILDs.

iphitus · 2009-07-31 11:57:48

Rather than a whole new format for the metadata, maybe it would be simpler to adopt a "stricter" bashlike syntax such that would make it more parsable outside of bash. The AUR already does this to a somewhat lesser extent.

For a majority of PKGBUILDs, there's no need to use any more exotic bash features in the metadata other than a variable to substitute $pkgver to the source array. Simple things like requiring appropriate quoting within arrays, not using $pkgname and pointlessly declared variables, no bash conditionals and no external commands in the metadata. This would result in a pretty easily parsed syntax.

That said, I'd be interested to see a proof of concept for an alternative format. So long as it generates valid pacman packages, it could easily be used side by side with the existing format

I'm not keen on a whitespace based approach however. The above ideas are much longer/wordier, and somewhat less clear than a 'punctuation' based. approach. p = ( "one" "two" ) is clearly an array, whereas

p:
 one
 two

is not quite so obvious, requires 3 lines, and forced me to use code tags for this post

And as with any whitespace approach, there's the tabs vs 2/4/8 spaces bikeshed and odd ones like me who prefer proportional fonts rather than monospaced.

Last edited by iphitus (2009-07-31 12:17:39)

scio · 2009-07-31 12:17:21

I know this is probably premature, but things like WANTING to write a description on multiple lines (that parse as one line) would need to be thought about, use the standard:

desc
   First line \
   Second line

Or go for something more exotic like:

desc
   First line
   +Second line

Either way, some syntax would have to be considered on the end of a line or beginning of one.

[OT]
This syntax of this is looking more and more like a java based database system I had to write for school once. Can't quite put my finger on what could be possible, but using javacc you could define all the rules for the file format (EBNF style). It would then create all the parsing code for you.

I know java is a bit excessive here, but like you said universally parsable, and that could lead to a self generating parser/interface for PKGBUILDs.
[/OT]

It seems like Square and u_no_hu have the same impression I first had of this being like CMake files for PKGBUILDs, which I think doesn't sound bad.

Xyne · 2009-07-31 12:55:47

@iphitus
Necessity and ability are two very different things. The only way to enforce a strict bash syntax would be through parsing the file, which puts you in the same situation that parsing a non-bash format does.

I've noted your point about whitespace though and the limitations of posting in threads, emails, etc. I'll keep that in mind and see if there is some simple punctuation that could handle it instead. Just to be clear though, tabs and spaces are not an issue because it would only be the relative level of indentation that matters, just as in Python and Haskell.

@scio
Good point. I mentioned the possible need to use quotation marks (or some other delimiiter) for anything with spaces, which might make line breaks a non-issue, but I haven't given it much though. The build function would still remain verbatim though, so shell line breaks with "\" would always work there.

iphitus · 2009-07-31 14:55:47

Xyne wrote:

@iphitus
Necessity and ability are two very different things. The only way to enforce a strict bash syntax would be through parsing the file, which puts you in the same situation that parsing a non-bash format does.

The problem I understand that your solution attempts to solve is that "the current format isn't great to parse".

I suppose I could better word my idea not as a 'strict bash syntax' but merely 'guidelines' such that a PKGBUILD following these 'guidelines' would be easily and globally parsable. In this case, non-makepkg/bash tools could simply reject PKGBUILD's that do not conform. This doesn't quite put using a bash format in the same situation as a non-bash format.

I don't see any need to have makepkg enforce such PKGBUILD guidelines. It's never had any form of enforcement for any packaging guidelines. Such a thing could be built into namcap pretty trivially however.

I've noted your point about whitespace though and the limitations of posting in threads, emails, etc. I'll keep that in mind and see if there is some simple punctuation that could handle it instead. Just to be clear though, tabs and spaces are not an issue because it would only be the relative level of indentation that matters, just as in Python and Haskell.

It was more just how much more verbose it is to post. But if you put it on one line, you'll end up requiring punctuation which just returns you to something akin to the existing syntax.

As for tabs/spaces, it becomes an issue once you start editing someone else's PKGBUILD which is indented differently to how your editor is configured.

Xyne · 2009-07-31 16:43:37

I don't think that would work. Because makepkg is written in bash and the PKGBUILD is sourced, it doesn't care if you use sed, grep, awk, etc to generate the variables in the PKGBUILD. Even if that is not the recommended way of doing things, it would still be a perfectly valid PKGBUILD in the current context and as makepkg is the standard tool, most people don't care. Writing tools that conform to a standard is pointless if the main tools that use it ignore it, so recommendations wouldn't cut it. I see that as the M$ approach to open standards and we all know how well that worked out for browsers.

Just to be clear though, I'm really not saying that makepkg should change. I'm just thinking about how PKGBUILDs could be improved in general. If I decide to implement anything and see that it really works and has the potential to catch on, then I might come back and say "look, this is simple, here's a patch to make makepkg work with this". I'm still just trying to flesh out the ideas, which is why I've rethought and rewritten the example format several times already.

Anyway, I've updated the example format and moved it to its own page. It not longer requires whitespace and I think it's still on the same level of simplicity as the current bash syntax.

Last edited by Xyne (2009-07-31 16:45:14)

Square · 2009-07-31 19:53:55

From the changes you've made, it just looks like you're going to end up back at bash syntax.

Maybe it's this effect happening.
http://www.terminally-incoherent.com/bl … ee-design/

Xyne · 2009-07-31 20:26:03

We'll see.

Xyne · 2009-08-01 01:00:58

I've written a trial script to convert PKGBUILDs to the new format. It can be found at the bottom of the info page. Feel free to let me know where it fails miserably. Also feel free to check that it correctly displays sources and optdepends ("correctly" meaning as specified on the page ).

It doesn't yet convert the architecture... I need to check a few PKGBUILDs which use if-then-else statements for that and see if I can find a way to convert architecture-specific data to the proposed format automatically.

Also, this script will probably be like pacpan... sometimes it will get the job done by itself, sometimes it will get most of it done and leave the user with a good starting point, and sometimes it will fail abysmally but hopefully only rarely.

Square · 2009-08-01 01:17:38

Xyne wrote:

I've written a trial script to convert PKGBUILDs to the new format. It can be found at the bottom of the info page. Feel free to let me know where it fails miserably. Also feel free to check that it correctly displays sources and optdepends ("correctly" meaning as specified on the page ).
It doesn't yet convert the architecture... I need to check a few PKGBUILDs which use if-then-else statements for that and see if I can find a way to convert architecture-specific data to the proposed format automatically.
Also, this script will probably be like pacpan... sometimes it will get the job done by itself, sometimes it will get most of it done and leave the user with a good starting point, and sometimes it will fail abysmally but hopefully only rarely.

We need a script that does the opposite
Anyhow, I'll take a look.

Last edited by Square (2009-08-01 01:17:50)

Xyne · 2009-08-01 02:19:56

Yeah, I know, but I got around to this one first.

louipc · 2009-08-01 03:02:03

Are you aware of PKGINFO by any chance?
Untar a built package and you'll get a file named '.PKGINFO'
Have a look at it. That's how libalpm parses package information without bash, or PKGBUILDs.
This XML, Json, Yaml stuff seems outright silly to me.

Xyne · 2009-08-01 03:27:45

louipc wrote:

Are you aware of PKGINFO by any chance?
Untar a built package and you'll get a file named '.PKGINFO'
Have a look at it. That's how libalpm parses package information without bash, or PKGBUILDs.

I don't think you've understood the point of this. I want a way to encapsulate the metadata required to build a package in a strict format that can be easily parsed from any language.

louipc wrote:

This XML, Json, Yaml stuff seems outright silly to me.

Seriously, wtf do so many people feel the need to just piss all over other people's ideas? Any time anyone ever suggests something different, there's always a stream of people ready to jump up and criticize it, especially in the open-source community. I'm really sick of all the "that's stupid" and "lol, what's the point" type of comments that I see all over the place. At least make an attempt to qualify your answer instead of assuming that there's no way there could be validity in other people's ideas just because you don't see the utility for yourself.

That attitude fucking annoys me, sorry.

Peasantoid · 2009-08-01 03:33:24

louipc wrote:

This XML, Json, Yaml stuff seems outright silly to me.

/me snorts loudly with conspicuous undertones of derision
Any particular reason for your statement? (I agree with Xyne as to the 'pissing on ideas' attitude, though I too have been guilty of it in the past.)

Addendum: If you'd left out JSON and YAML from that comment, I'd have agreed. XML must die.

Last edited by Peasantoid (2009-08-01 03:42:08)

louipc · 2009-08-03 11:19:31

Xyne wrote:

louipc wrote:
Are you aware of PKGINFO by any chance?
Untar a built package and you'll get a file named '.PKGINFO'
Have a look at it. That's how libalpm parses package information without
bash, or PKGBUILDs.
I don't think you've understood the point of this. I want a way to
encapsulate the metadata required to build a package in a strict
format that can be easily parsed from any language.

I completely understand the point of this.
I develop on the AUR, and we have had issues with parsing PKGBUILDs.
I've also realised that I don't really want to implement a parser that
takes all shell script and especially bashisms into account.

Xyne wrote:

Seriously, wtf do so many people feel the need to just piss all over
other people's ideas? Any time anyone ever suggests something different,
there's always a stream of people ready to jump up and criticize it,
especially in the open-source community. I'm really sick of all the
"that's stupid" and "lol, what's the point" type of comments that I see
all over the place. At least make an attempt to qualify your answer
instead of assuming that there's no way there could be validity in other
people's ideas just because you don't see the utility for yourself.
That attitude fucking annoys me, sorry.

Hah sorry, but you're asking for it when suggesting converting from
PKGBUILD to XML. One reason that I am not fond of this proposal is
because you haven't even considered adapting the existing system.
You're throwing out the baby with the bathwater.

OK. Here are some more specific critisms then.

- You're changing the data format.
- You're changing the variable names.
- You're changing how the data is structured.
- You've lost the inherent simplicity the PKGBUILD.

This is asking for way more than just easily parsable PKGBUILDs.
I'm not averse to change, and I definitely think that pacman, makepkg,
and the entire build system could use improvement, but I think we should
still keep the foundations we have in mind.

Here's what I would suggest:
- Keep simplicity in mind.
- Adopt formats that are already in use.
- Develop a format that is compatible with PKGBUILD, shell script.
- If you think that the formats are badly structured, suggest changes to
makepkg and libalpm to further improve the PKGBUILD/PKGINFO formats.

If solutions don't seem plausable within those measures, maybe then you
can think about forking the whole system.

makimaki · 2009-08-03 12:49:54

louipc wrote:

Hah sorry, but you're asking for it when suggesting converting from
PKGBUILD to XML.

I think you missed out on the latter sections of this thread. Xyne has very clearly stated that he's not suggesting XML, he was only using it as an example at the start but now his examples are completely different. On his first post, and his website he has changed the syntax http://xyne.archlinux.ca/ideas/pkgmeta
[quote=From Xyne's Website
Other Formats Which Have Been Considered
]XML was briefly considered because it is very good at encapsulating data in a way that's easy to parse programmatically and there are numerous tools and libraries already available to handle XML. XML was rejected though because many people find XML difficult to edit manually. It also adds a lot of characters to the final file.

My inclusion of an XML example in the original post on the forum nearly derailed the entire discussion... many bricks were shat.

JSON was also considered but due to perceived limitations (e.g. verbatim quoting such as XML CDATA), it was also rejected.

A custom format that relied on whitespace similar to Python and Haskell was also proposed along with an example, but it was pointed out that whitespace requires special quoting in forum posts and emails, which can lead to corrupt PKGBUILDs.

Xyne · 2009-08-03 15:42:20

louipc wrote:

Xyne wrote:
Seriously, wtf do so many people feel the need to just piss all over
other people's ideas? Any time anyone ever suggests something different,
there's always a stream of people ready to jump up and criticize it,
especially in the open-source community. I'm really sick of all the
"that's stupid" and "lol, what's the point" type of comments that I see
all over the place. At least make an attempt to qualify your answer
instead of assuming that there's no way there could be validity in other
people's ideas just because you don't see the utility for yourself.
That attitude fucking annoys me, sorry.
Hah sorry, but you're asking for it when suggesting converting from
PKGBUILD to XML. One reason that I am not fond of this proposal is
because you haven't even considered adapting the existing system.
You're throwing out the baby with the bathwater.
OK. Here are some more specific critisms then.
- You're changing the data format.
- You're changing the variable names.
- You're changing how the data is structured.
- You've lost the inherent simplicity the PKGBUILD.
This is asking for way more than just easily parsable PKGBUILDs.
I'm not averse to change, and I definitely think that pacman, makepkg,
and the entire build system could use improvement, but I think we should
still keep the foundations we have in mind.
Here's what I would suggest:
- Keep simplicity in mind.
- Adopt formats that are already in use.
- Develop a format that is compatible with PKGBUILD, shell script.
- If you think that the formats are badly structured, suggest changes to
makepkg and libalpm to further improve the PKGBUILD/PKGINFO formats.
If solutions don't seem plausable within those measures, maybe then you
can think about forking the whole system.

This is exactly why I ask people to glance at the page for updates to the idea before replying. XML was only ever given as an example of data encapsulation off the top of my head when I wrote the first post. It seems that all you did was skim the post, see XML, go into autorejection mode and lay some criticism on something that you hadn't fully considered. That's how it came across at least.

The latest working example is quite simple. There's still a hierarchy which doesn't translate directly into bash but I think the benefits such as clearly (from the editor's perspective, but also the programmer's) associating different types of checksums with each source, different metadata with each architecture and the possibilities for further extension such as different metadata for each distro (if they were ever added... think different post-install scripts, etc) outweighs the benefits of flat bash-like files. The names are just to make it clear what's what on a more abstract level. They're only for the working example and it's worth discussing the utility of keeping them exactly as they are versus using more universal names for future extensibility. Again though, you're criticizing something that I've directly said is only an idea and open for discussion, so it feels like you're not really focusing on the real discussion here.

I do appreciate your feedback but I would prefer it if you adopted a constructive attitude rather than a dismissive one. Take a look at the idea, recognize the motivations for things and make suggestions for improving the idea instead of condemning it in the future.

Also, as I said previously, I have very little expectation of this being adopted officially. Even if it were clearly a better way to do it, there are too many people who fight change tooth and nail because the current system is "good enough". I fully realize that I would probably have to fork/reduplicate the current tools to show the viability of this and I may just end up doing that once I feel comfortable with languages that would be appropriate for it.

XFire · 2009-08-03 16:27:13

The current idea on the website looks very much like a Lua, with a few abstractions. I dislike XML due to the amount of typing, and how messy it becomes when trying to read it, other than it's fine for me . Yes I do know that it has been removed from the idea.

I liked the RPM example (near the beginning) the Latex mark-up esque example _is_ readable, but for a new user it maybe a bit too complicated, or not, depends on how smart they are. I also liked the tabbed example, again for readability, it makes more sense to me than the current Bash example on your site. From that, would there be a way of working out the indentation if there was none, it may take more coding, but in the end I do think it would work out better that way.

The only problem with the current example is that it isn't too far from what is already there, whether that is intentional or not, I don't know. I think if you are going to push the boat out, it would make sense to develop something more out there, yet makes more sense. Again with the tabbed one, in it's design it appears to be far more KISS than the current one, but not in implentation perhaps.

I think it is a great idea, trying to renovate the package system, some may not agree though.

oopsicrappe · 2009-08-04 03:10:43

In my opinion, it is a bad idea to do this. It adds an unnecessary level of abstraction. A PKGBUILD is a bash scipt used by makepkg to build a package, and the current format is good for this use. However, for the purpose of the AUR, would it be useful if makepkg --source generated an xml or json or whatever file with metadata and included it in the source package? That way the metadata could be available in an easily parseable form on the AUR, and the PKGBUILD itself would remain unchanged.

XFire · 2009-08-04 07:26:36

oopsicrappe wrote:

In my opinion, it is a bad idea to do this. It adds an unnecessary level of abstraction. A PKGBUILD is a bash scipt used by makepkg to build a package, and the current format is good for this use. However, for the purpose of the AUR, would it be useful if makepkg --source generated an xml or json or whatever file with metadata and included it in the source package? That way the metadata could be available in an easily parseable form on the AUR, and the PKGBUILD itself would remain unchanged.

I think depending on how it swings, there maybe the option to use either. So someone could distribute the cleaner version, and it would be converted into bash, or just use the bash one. Therefore, people could use either, but it will make parsing the PKGBUILDs easier with the first instance.

brebs · 2009-08-06 00:09:37

I was sent an email about this by Xyne, so I'll respond here.

email wrote:

Universally Parsable PKGBUILDs is a project that seems like it could unite multiple distros by allowing universal build scripts to be used across the distros.

I was initially sceptical. So I thought about it a bit more. Then I became hugely sceptical. Might as well suggest "world peace" - sure it's a great idea, but there's a million complexities hidden under the surface.

I'm far too lazy to write a reasoned reply, so here's some points as to why this project is doomed to failure, in this universe:

* The doc doesn't even mention other distros.
* If parsing PKGBUILDs is difficult, then write a better parser, or change PKGBUILDs by the minimum necessary. Minimum change, means maximum chance of actually successfully happening.
* Motivating unpaid developers, even on the same project, is like herding cats. Motivating unpaid developers across *different distros*, to work together, will be roughly 99.9% impossible.
* PKGBUILDS, or .spec files, or ebuilds, or whatever the distro's packager uses, have to be a text file which is machine-readable, and (more importantly) human read/writeable. Humans will therefore push for them being as simple as possible, whilst satisying their own distro's needs, and not giving a damn about any other distro's needs.
* Users will moan like bleating sheep when this project *changes* anything, regardless of whether the change is an improvement. And if that change breaks something, then they will scream to high heaven, and may even utter the ultimate insult of threatening to change distro. Only 5% of *those* will actually make the required effort to change distro, though.
* Lots of other points to be inserted here about the complexities of how distros differ in their packaging, patching, release schedules, source-based vs binary, splitting packages into multiple sub-packages, mumble mumble

Square · 2009-08-06 01:13:37

The latest mockup doesn't seem to change much of anything. For me, it's no more or less readable than current PKGBUILDs.

Xyne · 2009-08-08 23:26:18

*sighs heavily at the blatant misunderstanding in this thread*

I'm no longer going to bother giving detailed replies to those who obviously made no effort to understand the point of this. Wasting my time is a habit I'm trying to get rid of.

@oopsicrappe
Your idea for generating xml, json, etc from a PKGBUILD with makepkg ignores the following:
*) makepkg needs to parse the PKGBUILD -> security risk with the AUR
*) it will depend on the architecture that parses it... my idea wouldn't\
*) it can still depend on the output of commands at runtime and thus change, which makes the output useless in some cases

@brebs
I don't even remember sending you an email, but I recognize that line and I know that it was towards the end. I also know that it was preceded by a paragraph for each main point and that the line you quoted was in the minor points, preceded by something like "maybe even in the long term" and succeeded by "I doubt that would ever actually happen though".

Nice way to take something completely out of context and then use it as a straw man argument, which I can only assume was your reason for replying here. *stop the reply here to avoid a flame war*

@Square
This isn't about trying to make it more human-readable. You obviously don't understand the point of this and I would be surprised if you actually read the page.

On a tangent, we need a facepalm smiley.

Last edited by Xyne (2009-08-08 23:26:30)

Diaz · 2009-08-09 00:05:13

I'm not a developer, but i see your point and understand the "problem" you are trying to solve.
I'm keeping my eyes on this thread and on this investigation, to see if there are any improvements that can be made to the actual model.

This is for developing ideas and i think you should keep on this, even it if it yelds results or not.

This is good for devoloping new ideas and thinking outside of the box, me likes it xD

brebs · 2009-08-09 01:06:18

Xyne wrote:

I also know that it was preceded by a paragraph for each main point and that the line you quoted was in the minor points, preceded by something like "maybe even in the long term" and succeeded by "I doubt that would ever actually happen though".

Xyne, your EXACT AND WHOLE email was:

Edit: Misfit138 has just informed me that it was Misfit138 and not Xyne who sent the email. This is what happens when people use pseudonyms, and assume we have a BIND-like name lookup between pseudonym and real name

Last edited by brebs (2009-08-12 23:15:01)

oopsicrappe · 2009-08-09 03:33:27

Xyne wrote:

@oopsicrappe
Your idea for generating xml, json, etc from a PKGBUILD with makepkg ignores the following:
*) makepkg needs to parse the PKGBUILD -> security risk with the AUR

I was thinking that since the metadata file would only need to be generated by the author of the PKGBUILD before uploading, the security risk would be minimal. But I admit my knowledge in the subject is lacking.

*) it will depend on the architecture that parses it... my idea wouldn't\
*) it can still depend on the output of commands at runtime and thus change, which makes the output useless in some cases

That's true. I just thought I'd throw the idea out there as a possible compromise, but it probably isn't good enough to be useful

For the purpose of uniting buildscripts for different distros, you might be better off starting from scratch than adapting the PKGBUILD. You would probably need a lot more flexibility than PKGBUILDs offer, for instance it would be useful to have something like Gentoo's USE flags so different distros could easily enable different compile time options globally. It would also be useful to be able to enable and disable different patches in the buildscript. Some sort of unified buildscript would be great, but it probably would be difficult to convince everyone to use it. Good luck!:)

Arch Linux

#26 2009-07-31 11:57:48

Re: A discussion about universally/programmatically parsable PKGBUILDs.

#27 2009-07-31 12:17:21

Re: A discussion about universally/programmatically parsable PKGBUILDs.

#28 2009-07-31 12:55:47

Re: A discussion about universally/programmatically parsable PKGBUILDs.

#29 2009-07-31 14:55:47

Re: A discussion about universally/programmatically parsable PKGBUILDs.

#30 2009-07-31 16:43:37

Re: A discussion about universally/programmatically parsable PKGBUILDs.

#31 2009-07-31 19:53:55

Re: A discussion about universally/programmatically parsable PKGBUILDs.

#32 2009-07-31 20:26:03

Re: A discussion about universally/programmatically parsable PKGBUILDs.

#33 2009-08-01 01:00:58

Re: A discussion about universally/programmatically parsable PKGBUILDs.

#34 2009-08-01 01:17:38

Re: A discussion about universally/programmatically parsable PKGBUILDs.

#35 2009-08-01 02:19:56

Re: A discussion about universally/programmatically parsable PKGBUILDs.

#36 2009-08-01 03:02:03

Re: A discussion about universally/programmatically parsable PKGBUILDs.

#37 2009-08-01 03:27:45

Re: A discussion about universally/programmatically parsable PKGBUILDs.

#38 2009-08-01 03:33:24

Re: A discussion about universally/programmatically parsable PKGBUILDs.

#39 2009-08-03 11:19:31

Re: A discussion about universally/programmatically parsable PKGBUILDs.

#40 2009-08-03 12:49:54

Re: A discussion about universally/programmatically parsable PKGBUILDs.

#41 2009-08-03 15:42:20

Re: A discussion about universally/programmatically parsable PKGBUILDs.

#42 2009-08-03 16:27:13

Re: A discussion about universally/programmatically parsable PKGBUILDs.

#43 2009-08-04 03:10:43

Re: A discussion about universally/programmatically parsable PKGBUILDs.

#44 2009-08-04 07:26:36

Re: A discussion about universally/programmatically parsable PKGBUILDs.

#45 2009-08-06 00:09:37

Re: A discussion about universally/programmatically parsable PKGBUILDs.

#46 2009-08-06 01:13:37

Re: A discussion about universally/programmatically parsable PKGBUILDs.

#47 2009-08-08 23:26:18

Re: A discussion about universally/programmatically parsable PKGBUILDs.

#48 2009-08-09 00:05:13

Re: A discussion about universally/programmatically parsable PKGBUILDs.

#49 2009-08-09 01:06:18

Re: A discussion about universally/programmatically parsable PKGBUILDs.

#50 2009-08-09 03:33:27

Re: A discussion about universally/programmatically parsable PKGBUILDs.

Board footer