Run script before loggin starts

agon · 2011-02-24 22:06:27

Hi everyone

I'm using a tmpfs for my /var/log folder, and because I don't want to loose all logs I have modified /etc/rc.local.shutdown following this thread so everything gets backed up each time in a different folder named after the time at which I shutdown.
https://bbs.archlinux.org/viewtopic.php … 10#p528010

But I'd prefer to have a XYZ folder with all logs which gets loaded to my tmpfs /var/log, so everything works like normal, and finally before shutdown, everything is copied back to folder XYZ. This way my HDD is only accessed for loggin matters at startup and shutdown. So far it's not difficult, except I want to run my script which does just that, before any loggin starts.

How can I execute commands or run a script while booting just between my tmpfs is mounted (declraed in /etc/inittab) and loggin starts? If possible including the minilogger at arch boot

Thanks very much in advance

Last edited by agon (2011-02-24 22:07:02)

ewaller · 2011-02-24 22:08:42

/etc/rc.local

agon · 2011-02-24 22:29:31

Thank you. Will try right now

---
edit:

Done

I added "cp -Rp /var/log-sync/* /var/log" and everything gets coppied to RAM
In /etc/rc.local "cp -Rp /var/log/* /var/log-sync/" and everything is copied back to the HDD-folder

Thank you!!

---
edit:

but overwritting lots of logs. See reply #5

Last edited by agon (2011-02-25 08:41:30)

toad · 2011-02-24 23:52:37

Please mark the thread as solved.

Thanks.

agon · 2011-02-25 08:21:40

I didn't mark it as solved yet because I wanted to do some testing.

This morning I also aded a line in /etc/rc.local to copy to my desktop everything from /var/log/ (all new generated logs in RAM) before overwritting with the old ones (stored in the HDD). It seems I overwritte a lot of them

Can I run a script or command even earlier than using /etc/rc.local ? So the overwrittes are minimized but without needing to patch the kernel.

Last edited by agon (2011-02-25 08:29:55)

toad · 2011-02-25 09:09:28

The wiki has it all

https://wiki.archlinux.org/index.php/Arch_Boot_Process

agon · 2011-02-25 14:11:29

I already read that article, and also searched the net.

But everything I get is juts to tecnical. So I dont know where I can touch without compromissing stability and security. Is it a good idea 8and posiible) to include a "cp" command in /etc/rc.sysinit. And where would it be a good place?

SeanM · 2011-02-25 15:06:25

Alright... I know thus far you have been looking into moving the copying process earlier, but to be fair you can't get it soon enough to prevent anything from logging before it syncs (to be fair, you could, but the ram drive would have to be managed by a hypervisor and you would have virtualize the installation... FAR more complicated then you are looking for...)

May I suggest an alternative train of thought?

How about moving it later. Once everything is loaded, all the drives mapped, and everything is done heavily logging (mostly), then you could move the log files that were created to a ramdisk and continue from there. For instance, wait for the boot to complete, then MOVE everything in /var/log to /tmp/tmplog, mount your ramdrive overtop of /var/log, and move back the files created this boot (remembering to preserve all attributes both ways). Logging will be down for most applications for a while, but that shouldn't be too much of an issue. On system shutdown, you could then add the same logic in reverse, but moving from the ramdisk to the archive partition.

Just a thought... You will never be able to get early enough with copy or move commands. I would save yourself the headache and change solutions. Either aim for further back in the boot chain, OR...

My personal suggestion would be to avoid the ramdisk entirely and let the kernel handle it with named pipes, though I'm not sure of the implications of that with regards to memory performance... Generally named pipes are not intended to be left hanging with data for the entire life of a system boot, but really there shouldn't be any downside to it... Just like ramdisks are really supposed to be for 99% of what people use them for ;-)
Basically named pipes are like ramdisks for files (there are a lot more complications, but for this purpose that will do...)
You can remap the log files you want to avoid writing to disk as FIFO device nodes. The applications that log to them shouldn't care, because they behave like regular files to them (gotta love *nix, eh?). Then, make a small script that just dumps the contents of the pipe to disk whenever you find it convenient (shutdown, 5 am, dinner time, on command... Really this approach gives you more control over when you care to dump to disk).
Remember that not all your log files need to be remapped to avoid disk usage... Most of the time there may only be a half dozen logs that are frequently updated.
Good super basic intro:
http://linux.about.com/od/commands/a/bl … kfifox.htm

agon · 2011-02-25 15:34:56

Great. Thank you I didn't know I can do it that way!
I'm right now away from my laptop, but as soon as I can I'll give it a try. I'll also do some research on ramdisks and pipes.

I'll tell you latter if I manage it

Last edited by agon (2011-02-25 15:36:16)

agon · 2011-02-26 12:23:41

Here we are again

I'm trying it now the way you suggest. I've written two simple scripts (my first ones ) which do as following:

[ LOGs => RAM ]

I call to this script in /etc/rc.local. It saves all logs in ram and then links the so that they appear again in /var/log

note: my /tmp folder is a tmpfs

#!/bin/bash
#
#/usr/bin/log-ram.sh
#    
# Create /tmp/log: necessary for each boot because /tmp is volatile
# Copy /var/log to tmp/log: copy my complete logs to ram
# Rename /var/log to /var/log.backup: so I have a backup of them
# Create a new empty /var/log
#Link /var/log to /tmp/log: all logs (located in ram) appear in /var/log

echo "Copiando LOGs de HDD a RAM"
 
mkdir /tmp/log
cp -Rp /var/log/* /tmp/log/
mv /var/log /var/log.backup
mkdir /var/log
ln -sf /tmp/log/* /var/log/

Questions:
-Do you recommend calling this script latter or is it here ok (/etc/rc.local)? If it would be latter, where do you reccomend it? As you already noted, it has to be excecuted as root

[ LOGs => HDD ]home sweet home)

I call this script in /etc/rc.local.shutdown

All logs get copied back to /var/log and also a backup is made. Note that each time this script is ececuted a new backup is generated. That menas that for now I have to manually erase them when they get to many. Later (when I learn a bit more of scripting) I'll make the oldest get automatically erased.

#!/bin/bash
#
#/usr/bin/log-unram.sh
#
#Refresh the content of /var/backup so it's up to date
#Remove /var/log: so all symbolic links get erased
#Recreate /var/log: is there a faster way to do this and the last step?
#Copy from /tmp to /var: from RAM to HDD
#Rename and change directory of /var/log.backup with an unique identifier (time at which script is run).

echo "Copiando LOGs de RAM a HDD"
now=`date +"%Y%m%d_%Hh%M"`

cp -Rp /tmp/log/* /var/log.backup/
rm -r /var/log
mkdir /var/log
mv /tmp/log /var/log
mv /var/log.backup /var/log.old/$now/

Questions:
Is there a faster way to erase al symbolic links

****************

As said, this are my two first attempts to writte a bash script (excetp the hello world one lol). So I probably did a lot wrong.
For now it seems to work (when calling them from shell) but I still didnt tried them with real powerup & shutdown. (Ill do it after posting this).

Thanks in advance

---

EDIT:
These scripts are give me problems. I'm trying to make now a syngle script which copies from HDD -> RAM or RAM -> HDD depending on if logs are already loaded or not. This way If my system crashes I dont have problems while booting (like the firefox-sync script).

As soon as I have something, I'll post it here

Last edited by agon (2011-02-27 13:06:11)

agon · 2011-03-05 15:01:25

I was unable to get this running.

Has anybody attempted it before me?

begruntled · 2011-04-14 13:37:20

I did this by using an /etc/rc.d/ daemon, which can be executed very early in the boot up sequence, and 'mount --bind' which allows a directory to be in two places at once. The original discussion is here :

https://bbs.archlinux.org/viewtopic.php?id=52565

Basically :

0. Back up your /var/log directory. You shouldn't need to, but if you make a mistake somewhere you might accidentally delete it, and some programs will complain if their log files go missing.

1. Get /tmp mounting in ram with a line in /etc/fstab. Mine looks like this

tmpfs /tmp tmpfs defaults,noatime,mode=1777,size=128M 0 0

2. Install rsync (you probably already have)

sudo pacman -S rsync

3. As root, create a file /etc/rc.d/tmpvarlog with contents

#!/bin/bash

. /etc/rc.conf
. /etc/rc.d/functions

case "$1" in

start)
stat_busy "Setting Up /var/log"
mkdir /tmp/var_log
/usr/bin/rsync --archive --delete /var/log/ /tmp/var_log
/bin/mount --bind /tmp/var_log /var/log
add_daemon tmpvarlog
stat_done
;;
stop)
stat_busy "Saving /var/log To Disk"
/bin/umount /var/log
/usr/bin/rsync --archive --delete /tmp/var_log/ /var/log
rm -rf /tmp/var_log
rm_daemon tmpvarlog
stat_done
;;
restart)
$0 stop
sleep 1
$0 start
;;
*)
echo "usage: $0 {start|stop|restart}"

esac
exit 0

4. Make the tmpvarlog script executable

sudo chmod a+x /etc/rc.d/tmpvarlog

5. Add tmpvarlog to your DAEMONS list on the last line of /etc/rc.conf, BEFORE syslog-ng or whatever logger you use. No reason not to make it the first entry unless you are doing something strange. Mine looks like this;

DAEMONS=(tmpvarlog syslog-ng network netfs crond dbus)

6. Restart.

Because you are using mount --bind the mount point doesn't show up in df, but it is there and working Your /var/log will be copied into /tmp/var_log on startup and copied back on shutdown, and thanks to the magic of mount --bind apps writing to /var/log will write to the ram instead. If you want some security against crashes you can use cron to periodically back up /tmp/var_log (somewhere other than /var/log or it will get confused).

Hope that helps.

Last edited by begruntled (2011-04-14 18:43:59)

Arch Linux

#1 2011-02-24 22:06:27

Run script before loggin starts

#2 2011-02-24 22:08:42

Re: Run script before loggin starts

#3 2011-02-24 22:29:31

Re: Run script before loggin starts

#4 2011-02-24 23:52:37

Re: Run script before loggin starts

#5 2011-02-25 08:21:40

Re: Run script before loggin starts

#6 2011-02-25 09:09:28

Re: Run script before loggin starts

#7 2011-02-25 14:11:29

Re: Run script before loggin starts

#8 2011-02-25 15:06:25

Re: Run script before loggin starts

#9 2011-02-25 15:34:56

Re: Run script before loggin starts

#10 2011-02-26 12:23:41

Re: Run script before loggin starts

#11 2011-03-05 15:01:25

Re: Run script before loggin starts

#12 2011-04-14 13:37:20

Re: Run script before loggin starts

Board footer