Tool to keep track of only manual systemfile edits

mesaprotector · 2024-10-03 04:28:18

Over the years I've edited a lot of stuff in /etc, /usr, and even other system directories, and in many cases I've forgotten why, or forgotten I changed anything at all. I know tools like etckeeper exist, but they have a different use case for two reasons: one, they make no distinction between files changed manually vs. automatically, and two, their focus is on keeping track of previous versions of those files, rather than keeping track of why those files exist in the first place. So I wanted a tool that, every time a file outside of /home was edited, created, or deleted directly by me, would prompt for a comment after the edit, and log that comment to a special file that could be referenced later. As a bonus, I figured it should also prompt for comments after installing something with pacman.

After exploring various approaches I decided the best way of implementing this was through using the entr tool to run a command on every change to the sudo logfile (sudo logs to syslog by default, but it can be changed). So - "echo /var/log/sudo.log | sudo entr -p prompt_script.sh".

The best concept of this script I currently have:
1. grabs the last line of the logfile
2. turns the command run through sudo into a variable
3. does nothing if the command does not match a set list (currently rm, rmdir, mkdir, touch, vim, cp, mv, any command with > or >>, and pacman with certain arguments)
4. if it does match, opens a new terminal window asking for a comment on the change
5. if the comment is blank the script simply exits
6. if the comment is not blank, it appends the command, date, and comment to the aforementioned file.

I've run into several issues so far. One is that any commands run through "sudo -s" or "sudo -i" do not get logged by sudo. I don't normally edit files when under sudo -s (I mainly use that when needing to read stuff within read-protected directories) but it'd be annoying to have to remember to edit the log manually, just in case I did.

The other issue, however, is that opening a new terminal window is just clunky and a poor choice for adding a comment that should be allowed to be as long as necessary. I made the script this way only because this is how my homebrew application launcher already works. Ideally the script should not prompt for a comment until after the edit has been made; how can you know exactly what changes you're going to make to a config file if you're just opening it in case you want to change the defaults?

I'd love input on the two issues mentioned, or any other suggestions. I don't have anything more specific to add now, but if anyone is interested in a tool like this, I'll continue to post updates, assuming it's okay to do so here. If not, I'm happy to reply to the sticky thread in this board.

Last edited by mesaprotector (2024-10-04 00:33:54)

dimich · 2024-10-03 05:24:02

# cd /etc
# git init .

?
Regarding other directories, I have no idea what might need to be edited outside of /etc/ system-wide.

mesaprotector · 2024-10-03 15:56:48

Git doesn't work well for this for the reasons I mentioned above. I'm aiming at a way of tracking which files have been manually edited, and why, not a versioning system.

I've added custom pacman hooks—that's /usr/share. When systemd decided it was okay to break suspend for Nvidia hybrid graphics laptops—that meant adding to /usr/lib. And of course any custom log files—those are edits to /var.

The only reliable way to track such changes is to run a post command on every edit, unless there's some way for a virtual filesystem to figure out that one file is being edited directly and should be noted and another is being edited by a pacman upgrade and shouldn't.

NOTE: figured out the first problem I mentioned above, as sudo somewhat recently added a "log_subcmds" option. It does also occasionally spam the log - there's a built-in option to disable it for package managers, but I'll see if I run into problems later if commands are calling rm, mkdir, etc. as subcommands.

Last edited by mesaprotector (2024-10-03 18:28:16)

seth · 2024-10-03 19:07:54

The only reliable way to track such changes is to run a post command on every edit

https://man.archlinux.org/man/inotifywait.1.en
However, there's ab·so·fuck·ing·lu·te·ly no way to track what files have been "manually edited" unless you provide a whitelist for what "manually" actually means, ie. you could wrap vim in a script that checks what file was opened, but that will fail if you open the file from within vim (unless you're utilizing vim scripts instead)

tracking which files have been manually edited, and why

You're describing a major feature of a version control system, the patch history is just a nice bonus for you.
I'd go w/ git and periodically check git diff and force you to comment on the changes.
You'll probably remember why you did that yesterday (leaving aside that you're free to manually commit changes whenever you feel happy with them)

mesaprotector · 2024-10-03 20:25:57

seth wrote:

The only reliable way to track such changes is to run a post command on every edit
https://man.archlinux.org/man/inotifywait.1.en
However, there's ab·so·fuck·ing·lu·te·ly no way to track what files have been "manually edited" unless you provide a whitelist for what "manually" actually means, ie. you could wrap vim in a script that checks what file was opened, but that will fail if you open the file from within vim (unless you're utilizing vim scripts instead)
tracking which files have been manually edited, and why
You're describing a major feature of a version control system, the patch history is just a nice bonus for you.
I'd go w/ git and periodically check git diff and force you to comment on the changes.
You'll probably remember why you did that yesterday (leaving aside that you're free to manually commit changes whenever you feel happy with them)

Tracking specifically "manual" edits is the whole point. Git doesn't do this - therefore I'm not interested in using it for this purpose. It's a useful tool, but it's a very poor fit here.

The other problems with it include:
- what files do you even include in the repo? It'd likely have to be / with a handful of exceptions (/home and /var/cache, for one), meaning you're tracking a bunch of things you have no need to track, but you can't be more selective because you have no idea what files you'll need to edit in the future.
- every single edit would still need to trigger a git commit, via entr or otherwise. if it's once a day like etckeeper's default (let alone "periodically"), it's useless - unless you plan on remembering every single command you ran for the day.

I happen to prefer entr's syntax to inotifywait's, but they seem equivalent. I'm happy to test both.

seth · 2024-10-03 20:55:41

Tracking specifically "manual" edits is the whole point. Git doesn't do this

Nothing does that. How do you define "manual" here?

After exploring various approaches I decided the best way of implementing this was through using the entr tool to run a command on every change to the sudo logfile

This will track invocations of sudo, not "manual edits".
You're tracking specific sudo commands which puts severe restrictions on how you get to "manually edit" files (as you figured)
If you're accepting those limitations anyway, shadow sudo w/ a shell function that parses the command before forwarding it to sudo and casewise asks you for annotation. Likewise have shell function shadows for the roots shell rc.
But this will not be able to deal with situations where you open vim as root (what you probably should™ not do anyway, but use sudoedit) and open the target file from within vim.
The closest you'll get here will be https://wiki.archlinux.org/title/Audit_ … ies_access but this will, like inotify or git quickly become a problem when you just want to audit "everything" (see below).

Also

opening a new terminal window is just clunky and a poor choice for adding a comment

leaving aside that you don't need a terminal to enter some comment (any dialog would do) you'd already have an open terminal (to run sudo)?

what files do you even include in the repo?

The ones you edit, you'd git add it to the repo to put it under control before making significant changes.

dimich wrote:

I have no idea what might need to be edited outside of /etc/ system-wide

Why do you think you'd ever edit anything in */lib ? Even /usr/share is wrong™ since there's a good chance it'll get nuked with the next udpate.

it's useless - unless you plan on remembering every single command you ran for the day.

No?
You get a "git diff" for the changes to annotate, you'll likely remember for some hours why you made that change.

mesaprotector · 2024-10-03 21:30:44

I give up. You're determined to attack me despite the couple of bits of good advice hidden in your posts.

Please close this thread, mods. I'll go elsewhere. If you want Arch to be a welcoming community maybe put a harness on your attack dog.

Last edited by mesaprotector (2024-10-03 21:33:52)

seth · 2024-10-03 21:37:13

Nobody's trying to attack you.
I'm trying to explain to you that what you want to do is either severely underspecified or systematically impossible (not technically, but you're looking for some magic blackbox that makes the system know whenever you want to annotate any change to any file) and also based on some questionable assumptions.

mesaprotector · 2024-10-04 00:33:20

I apologize for calling names, I should not have; however, checking this thread has only upset me. It is disheartening to ask for coding help and instead be hit repeatedly with being told I'm using my system wrong. Since FluxBB has no option to hide threads I've already posted in, I will just try to ignore this thread while checking the rest of the forum and it will probably die either naturally or otherwise.

seth · 2024-10-04 07:08:28

If you insist on tracking the sudo log I'd first and foremost just "tail -f" it

But you've actually not asked for help with a specific problem but rather presented an open question around contradictory desires and (while I tend to simply comment fringe configs where I'll likely forget wtf I did that directly in the configuration file, on the line before or the same line) everyone's best suggestion will likely remain git because you keep track of what you changed and only have to annotatate it and can git blame lines (it's basically like having the config file full of comments that are kept somewhere else and you can force-track your activity w/ git diff)

If your desire is to only comment after the change, you'll either have to wait for the sudo process to end, or you shadow the tools you're gonna use to make those edits and trail them w/ "Why did you to that?", though for tools like vim neither approach will not cover files you opened inside vim so you'd rather use sth. like https://serverfault.com/a/332413

And again: it's not that your doing anything wrong, it's that you're trying to implement a vague idea of a feature which is why you're running into those open problems (you're asking "how do I track edits" before actually answering "what edits do I want to track", but hide the latter question behind a generic "manual" - "manual" is actually a question mark in your task chain)

Feel free to revisit this whenever you want.

It's not like the rest of us has never been at this point.
Edit: as a matter of fact, the problem is so common that there's a meme

Last edited by seth (2024-10-04 07:15:40)

mesaprotector · 2024-10-12 20:45:14

It's worked perfectly for a couple days now.

I have a systemd service that monitors sudo.log, and when a matching command is detected it checks "ps ax" every 0.1 seconds until the relevant terminal has nothing running in the foreground (other than a shell).

Once the terminal is free, the service pushes the text "addcomment" into it with a newline at the end. That being a simple y/n prompt script that opens vim (with the relevant lines from sudo.log pre-included) on y, and appends the finished comment to a separate user-readable logfile.

I set sudo's log_subcmds option to only cover subcommands of bash, so in the rare case I'm editing things with "sudo -s" it will still prompt. If someone did use "sudo vim" directly I imagine they could just tell sudo to log subcommands of vim as well. I only call vim as root with sudoedit so that is not an issue.

Last edited by mesaprotector (2024-10-12 20:46:03)

seth · 2024-10-13 08:03:19

it checks "ps ax" every 0.1 seconds

https://man.archlinux.org/man/core/man-pages/wait.1p.en

Arch Linux

#1 2024-10-03 04:28:18

Tool to keep track of only manual systemfile edits

#2 2024-10-03 05:24:02

Re: Tool to keep track of only manual systemfile edits

#3 2024-10-03 15:56:48

Re: Tool to keep track of only manual systemfile edits

#4 2024-10-03 19:07:54

Re: Tool to keep track of only manual systemfile edits

#5 2024-10-03 20:25:57

Re: Tool to keep track of only manual systemfile edits

#6 2024-10-03 20:55:41

Re: Tool to keep track of only manual systemfile edits

#7 2024-10-03 21:30:44

Re: Tool to keep track of only manual systemfile edits

#8 2024-10-03 21:37:13

Re: Tool to keep track of only manual systemfile edits

#9 2024-10-04 00:33:20

Re: Tool to keep track of only manual systemfile edits

#10 2024-10-04 07:08:28

Re: Tool to keep track of only manual systemfile edits

#11 2024-10-12 20:45:14

Re: Tool to keep track of only manual systemfile edits

#12 2024-10-13 08:03:19

Re: Tool to keep track of only manual systemfile edits

Board footer