(aif) tool to query/update structured plaintext data (yaml?) ?

Dieter@be · 2010-04-17 09:03:38

Hi,
for aif (http://projects.archlinux.org/aif.git/) I need to refactor the format used to store information about filesystems/blockdevices.
currently we use a plaintext format which is too simple and hard to parse (see http://bugs.archlinux.org/task/15640) (and even then, doesn't function correctly, because we cannot use whitespace in some fields, unless i add more separators or escaping)
See the BLOCKDATA variable at http://github.com/Dieterbe/aif/blob/mas … all-on-sda
I'm thinking of a format like this (i'm just brainstorming a bit):
(this is for a "lvm on top of dm_crypt" setup)

/dev/sda1
  raw
  no_label 
  filesystem:
    ext2
    yes
    /boot
    target
    no_opts
    no_label
    no_params

/dev/sda2
  raw
  no_label
  filesystem:
    dm_crypt
    yes
    no_mountpoint
    target
    no_opts
    sda2crypt
    -c aes-xts-plain -y -s 512

/dev/mapper/sda2crypt
  type: dm_crypt
  label:
  filesystem:
    type: lvm-pv
    create: yes
    mountpoint: 
    options:
    label:
    params:    

/dev/mapper/sda2crypt+
  type: lvm-pv
  filesystem:
    lvm-vg
    label: cryptpool
 
/dev/mapper/cryptpool
  type: lvm-vg
  label: cryptpool
  filesystems:
    1
      type: lvm-lv
      create: yes
      label: cryptswap
      size: 1G
      result:
        /dev/mapper/cryptpool-cryptswap
          filesystem:
            type: swap
    2
      type: lvm-lv
      label: cryptroot
      size: 5G
      result:
        /dev/mapper/cryptpool-cryptroot
        filesystem:
          type: xfs
          create: yes
          mountpoint: /

Right now I only made the lvm-lv's at the end recursive (so that the blockdevice created by putting an "lvm LV filesystem" on top of something gets nested below it, with the filesystem you put on that blockdevice beneath it), but probably it should do this everywhere, so that you only have /dev/sda1 and /dev/sda2 as root nodes.

Anyway: I want to do stuff like:
* what are the properties of all lvm-lv's which are on the lvm-vg with label $foo?
* which blockdevices are of type lvm-pv and have the lvm-vg with label $foo on them? (child notes can have multiple parents. think of lvm-vg's on multipe lvm-pv's, raid arrays, etc)
* which filesystems should be destroyed if i want to make /dev/sda2 free again, and in which order?
* which filesystems should be created in order to create the / filesystem?
* and obviously update properties of things, add/remove child nodes, etc.

maybe i should use yaml. are there commandline tools for this? I think with awk most of these things could be relatively easy but I'm not sure.

Last edited by Dieter@be (2010-04-17 09:04:37)

gradgrind · 2010-04-17 13:48:56

I'll probably get banned for life for saying this, but I'll do it anyway because I have quite a bit of practical experience. I know it's cool and quick to write things in bash, but there comes a point when a program reaches a certain level of complexity (and I would say that point is in fact very early in its development) when it can save a lot of pain if one simply accepts the fact that bash is very useful in short scripts but grotty for proper programs. My advice would be to use a programming language more suited to the task. I guess aif is getting quite big now and I think you might find a recode in (say) python would repay itself in future maintenance and development time.

Probably awk could do something like the task you mention, but then you are adding another difficult to read language to an already existing one. String handling in bash is a real pain, don't torture yourself unnecessarily.

Dieter@be · 2010-04-17 16:28:15

gradgrind wrote:

I'll probably get banned for life for saying this, but I'll do it anyway because I have quite a bit of practical experience. I know it's cool and quick to write things in bash, but there comes a point when a program reaches a certain level of complexity (and I would say that point is in fact very early in its development) when it can save a lot of pain if one simply accepts the fact that bash is very useful in short scripts but grotty for proper programs. My advice would be to use a programming language more suited to the task. I guess aif is getting quite big now and I think you might find a recode in (say) python would repay itself in future maintenance and development time.
Probably awk could do something like the task you mention, but then you are adding another difficult to read language to an already existing one. String handling in bash is a real pain, don't torture yourself unnecessarily.

I think you have a good point. I have in fact, regretted several times already that I started this in bash. Although the blockdevice/filesystem code is the main troublemaker because of the tree-ish structure it requires. If i can find a tool that i can "outsource" the querying/saving to, I think the biggest issues i have with bash are gone.

gradgrind · 2010-04-18 12:48:20

I have regretted starting something in bash several times. It's more than just a question of how difficult it is to do this, that or the other in bash, or of outsourcing snippets to awk or something. Consider the lifetime of the project, its maintenance and development costs (in hours, even if not in €) over the years, consider the fact that it may outlive your participation in its maintenance and development, that other people may have to figure out how to fix or improve it. I may be a pretty dud programmer - it's only a hobby - but I'm sure it's also true for better ones, that you can come back to a program you wrote some time ago and find at least bits that you can no longer understand. Using the wrong tool exacerbates this problem enormously. Things like bash/sed/awk are wonderful for quick hacks and short scripts, but terrible for proper programs.

Arch Linux

#1 2010-04-17 09:03:38

(aif) tool to query/update structured plaintext data (yaml?) ?

#2 2010-04-17 13:48:56

Re: (aif) tool to query/update structured plaintext data (yaml?) ?

#3 2010-04-17 16:28:15

Re: (aif) tool to query/update structured plaintext data (yaml?) ?

#4 2010-04-18 12:48:20

Re: (aif) tool to query/update structured plaintext data (yaml?) ?

Board footer