You are not logged in.

#1 2021-06-04 07:07:14

Arciere
Member
From: Torino, Italy
Registered: 2020-10-07
Posts: 59

[Solved] Delete old backups from shell

Hello everybody,
I apologize in advance if this is not the correct section.
I roughly figured out how to append the date on a file. So I would like to know if by chance there is a way, from the terminal, to delete a certain file category.
In my specific case, I have a folder in which I periodically save a backup copy of a personal folder. Until now I deleted the previous save and wrote the new one (giving rm and then cp). However, from now on, I would like to append the dates to these backups that I'm going to do (thus keeping more than one save) and then automatically delete all previous backups except the last n in chronological order.
Is it possible to do this from the terminal, perhaps through an option of rm? Or maybe through a script? Or maybe via an rsync function?

Last edited by Arciere (2021-06-04 17:01:03)


Arciere

Offline

#2 2021-06-04 07:57:42

jasonwryan
Anarchist
From: .nz
Registered: 2009-05-09
Posts: 30,424
Website

Re: [Solved] Delete old backups from shell

Sure it's possible. But you'd need to more clearly articulate your data model. Provide examples of the file names, and date formats, and how you'd like to determine a file is redundant.

See, for example: https://hg.sr.ht/~jasonwryan/shiv/brows … an?rev=tip


Arch + dwm   •   Mercurial repos  •   Surfraw

Registered Linux User #482438

Offline

#3 2021-06-04 08:26:18

sabroad
Member
Registered: 2015-05-24
Posts: 242

Re: [Solved] Delete old backups from shell

Arciere wrote:

I would like to append the dates to these backups that I'm going to do (thus keeping more than one save) and then automatically delete all previous backups except the last n in chronological order.
Is it possible to do this from the terminal, perhaps through an option of rm? Or maybe through a script? Or maybe via an rsync function?

rsync can do differential backups, eg.:

rsync -a --delete --quiet --inplace --backup --backup-dir=/location/to/backup/incr/$DAY /path/to/backup/ /location/to/backup/full/

Last edited by sabroad (2021-06-04 08:27:56)


--
saint_abroad

Offline

#4 2021-06-04 08:55:14

Arciere
Member
From: Torino, Italy
Registered: 2020-10-07
Posts: 59

Re: [Solved] Delete old backups from shell

jasonwryan wrote:

Sure it's possible. But you'd need to more clearly articulate your data model. Provide examples of the file names, and date formats, and how you'd like to determine a file is redundant.

See, for example: https://hg.sr.ht/~jasonwryan/shiv/brows … an?rev=tip

Thanks a lot for the answer!
I built the following formatting to attribute to the files for my backups:

filename\($(date +%d-%m-%Y_%H:%M:%S)\)

The result is the following:
filename(31-12-2020_17:46:59)
Could this be fine or should I format them in US format, for example?

What do you mean by "How you'd like to determine a file is redundant"?
I took a look at the link you kindly posted to me, but unfortunately I am not at that level and therefore I admit that I understand very little.

Last edited by Arciere (2021-06-04 08:55:57)


Arciere

Offline

#5 2021-06-04 08:58:54

Arciere
Member
From: Torino, Italy
Registered: 2020-10-07
Posts: 59

Re: [Solved] Delete old backups from shell

sabroad wrote:

rsync can do differential backups, eg.:

rsync -a --delete --quiet --inplace --backup --backup-dir=/location/to/backup/incr/$DAY /path/to/backup/ /location/to/backup/full/

Thank you very much for your link and your example too.
I'd like to ask you a few newbie questions about the script of that example:

  1. What does the -e option after "if [" stand for?

  2. What does "fi" stand for?

  3. Could you kindly explain to me what rsync checks with that if first? I do not understand... sad

  4. How is that rsync set up, does it also delete previous backups? If so, how do it knows which ones?

Sorry for the stupid questions, but I'm not an expert.

Last edited by Arciere (2021-06-04 09:18:48)


Arciere

Offline

#6 2021-06-04 09:25:36

Arciere
Member
From: Torino, Italy
Registered: 2020-10-07
Posts: 59

Re: [Solved] Delete old backups from shell

I was thinking, since appending date and time with cp is not a problem, the quickest way would be to simply tell rm to remove n old files / folders. So I was wondering: is there a way to tell rm to do this?


Arciere

Offline

#7 2021-06-04 09:25:40

lambdarch
Member
Registered: 2021-01-10
Posts: 67

Re: [Solved] Delete old backups from shell

Arciere wrote:

I built the following formatting to attribute to the files for my backups:

filename\($(date +%d-%m-%Y_%H:%M:%S)\)

The result is the following:
filename(31-12-2020_17:46:59)
Could this be fine or should I format them in US format, for example?

You'd better prefix the filenames by a date in a naturally sortable format, like this:

$(date +%Y-%m-%d_%H:%M:%S)-filename

Then you work with an array:

files=(/backup/dir/*)
n_files=${#files[@]}
for ((n = 0; n < n_files - n_to_keep; n++)); do
  rm "${files[-1-n]}"
done

where n_to_keep is the number of files you want to keep, to define (not tested: prefix rm with echo before trying this smile ).

Offline

#8 2021-06-04 09:33:07

Arciere
Member
From: Torino, Italy
Registered: 2020-10-07
Posts: 59

Re: [Solved] Delete old backups from shell

lambdarch wrote:

You'd better prefix the filenames by a date in a naturally sortable format, like this:

$(date +%Y-%m-%d_%H:%M:%S)-filename

Then you work with an array:

files=(/backup/dir/*)
n_files=${#files[@]}
for ((n = 0; n < n_files - n_to_keep; n++)); do
  rm "${files[-1-n]}"
done

where n_to_keep is the number of files you want to keep, to define (not tested: prefix rm with echo before trying this smile ).

Thanks so much for your tip!
I have some questions for you, sorry:
What is an array and how do I set it up?
Let's try to see if I understand: are you suggesting, by entering the date first and only after the file name, to use the date and time as a single natural number and then determine if a file is "new" or "old" just by looking if the starting number is more or less large? Please, correct me if I have not understood anything!
If I understand correctly, you are brilliant, I hadn't thought about it!


Arciere

Offline

#9 2021-06-04 09:53:41

lambdarch
Member
Registered: 2021-01-10
Posts: 67

Re: [Solved] Delete old backups from shell

Yes, that's it, that's how we usually do it.
For your questions about arrays, see man bash wink
The line

files=(/backup/dir/*)

is a way to set up an array.

Offline

#10 2021-06-04 09:56:56

Arciere
Member
From: Torino, Italy
Registered: 2020-10-07
Posts: 59

Re: [Solved] Delete old backups from shell

lambdarch wrote:

Yes, that's it, that's how we usually do it.
For your questions about arrays, see man bash wink
The line

files=(/backup/dir/*)

is a way to set up an array.

Thank you very much, really!! smile
Forgive the idiotic question: what language is it? So I look for some online resources...
So if I wanted to tell the system: "Delete everything you find in a folder apart from the last n files in order of creation." is that the most practical way to do it?

EDIT: I asked before looking and I apologize: bash is just the name of the language. Forgive me.

Last edited by Arciere (2021-06-04 09:59:56)


Arciere

Offline

#11 2021-06-04 10:01:47

lambdarch
Member
Registered: 2021-01-10
Posts: 67

Re: [Solved] Delete old backups from shell

The language is bash ^^
"most practical" I don't know: this is a (bash-builtin) way to do it when files are named correctly.

Offline

#12 2021-06-04 13:10:48

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,447
Website

Re: [Solved] Delete old backups from shell

If you'd be happy deleting files older than a specific date or number of days old (rather than keeping 'n' files) just use find:

find $path_to_backups -mtime +$days

When you have replaced the path and days variables, understand what the command does, and are satisfied with the resulting list of deletion candidates, just run the command again appending the flag '-delete'.

I'd also suggest if you are going to continue using arch linux that you invest some time in learning about your system.  I'd strongly advise working through the "linux tutorial" and "bash scripting" guides here: https://ryanstutorials.net/

Last edited by Trilby (2021-06-04 13:12:14)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#13 2021-06-04 13:23:02

Slithery
Administrator
From: Norfolk, UK
Registered: 2013-12-01
Posts: 5,776

Re: [Solved] Delete old backups from shell

I just use rsnapshot.


No, it didn't "fix" anything. It just shifted the brokeness one space to the right. - jasonwryan
Closing -- for deletion; Banning -- for muppetry. - jasonwryan

aur - dotfiles

Offline

#14 2021-06-04 16:56:54

Arciere
Member
From: Torino, Italy
Registered: 2020-10-07
Posts: 59

Re: [Solved] Delete old backups from shell

I am writing to update you on my little problem.
After several searches I finally found a command that satisfies me. This explains how you can use the simple ls + rm commands to delete certain files.
So here the author suggests the following command:

ls -1t | tail -n +11 | xargs rm

Where +11 indicates that you are going to remove the old backup copies from the 11th onwards (going back in time), only if there are any though.
Although excellent, essential and clean, this command did not convince me precisely because it is based on the date, recorded by the system, of the last modification of this backup file / folder.
Knowing myself, however, I knew that in my case it would be fatal, because I always go to poke my nose in my folders, opening, closing, making copies, etc. without, however, making any actual real changes and thus involuntarily changing these dates of latest changes. Consequently, the system could give precedence to a file that it considers recent only because I have unintentionally manipulated it, thus "stealing the place" of a really more recent file.
So I made a small change.

Given the premise that the backup files are saved as the kind @lambdarch suggested (thus prefixing the name of the copy with its creation date and time), just replace the t with the r (and some other very small changes) in this way:

ls -1dr /backupFolder/* | tail +11 | xargs rm -rfv

In this way the system will then consider the real date and time because we wrote it to them.
I infinitely thank everyone for your availability and kindness!


Arciere

Offline

#15 2021-06-04 17:25:51

lambdarch
Member
Registered: 2021-01-10
Posts: 67

Re: [Solved] Delete old backups from shell

It's brittle to use ls: you don't control the output format, it's not made for that.
If you want to use an external command, use find instead:

find /backupFolder -print0 | sort -z | tail -z -n+11 | xargs -0 rm -rfv

Last edited by lambdarch (2021-06-04 17:36:21)

Offline

#16 2021-06-04 17:46:22

Arciere
Member
From: Torino, Italy
Registered: 2020-10-07
Posts: 59

Re: [Solved] Delete old backups from shell

lambdarch wrote:

It's brittle to use ls: you don't control the output format, it's not made for that.
If you want to use an external command, use find instead:

find /backupFolder -print0 | sort -z | tail -z -n+11 | xargs -0 rm -rfv

Why do you say this?
I did several tests and, if combined with -d + "/ *" at the end of the path, it gives the exact absolute path of the files or folders in question...
However, your command is also very interesting, I already like it smile
Tomorrow I will deepen it and try it, thank you very much!! smile


Arciere

Offline

#17 2021-06-04 17:53:19

lambdarch
Member
Registered: 2021-01-10
Posts: 67

Re: [Solved] Delete old backups from shell

The filenames can contain spaces, special characters and so on: that's why ls is a bad idea (this is a little mentioned in the link you quote by the way).

In fact there is no need for find here:

printf '%s\0' /backupFolder/* | tail -z -n+11 | xargs -0 rm -rfv

EDIT: I didn't see the -d in your ls: in this case, it's probably better to use find as above, adding -type d, and maybe -maxdepth 1.
To summarize, this would give:

find /backupFolder -maxdepth 1 -type d -print0 | sort -z | tail -z -n+11 | xargs -0 rm -rfv

Last edited by lambdarch (2021-06-04 18:05:18)

Offline

#18 2021-06-06 07:22:39

Arciere
Member
From: Torino, Italy
Registered: 2020-10-07
Posts: 59

Re: [Solved] Delete old backups from shell

lambdarch wrote:

The filenames can contain spaces, special characters and so on: that's why ls is a bad idea (this is a little mentioned in the link you quote by the way).

In fact there is no need for find here:

printf '%s\0' /backupFolder/* | tail -z -n+11 | xargs -0 rm -rfv

EDIT: I didn't see the -d in your ls: in this case, it's probably better to use find as above, adding -type d, and maybe -maxdepth 1.
To summarize, this would give:

find /backupFolder -maxdepth 1 -type d -print0 | sort -z | tail -z -n+11 | xargs -0 rm -rfv

So,
I've done a lot of testing, but I can't apply your command correctly. I am stubborn about wanting to use the last one you wrote because you say that ls could give problems in case of spacing or particular characters, but I can't get over it.
Trying it exactly as you wrote it not only also takes into account the same /backupFolder folder, but deletes on the contrary, ie the last backups made in order of time, and not the oldest.
I then tried to make some changes, but I admit I can't.
I tried like this:

find /backupFolder -maxdepth 1 -print0 | tail -n +2 | sort -r | sort -z | tail -z -n +11 | xargs -0 rm -rv

and also:

find /backupFolder -maxdepth 1 -print0 | tail -z -n +2 | sort -r | sort -z | tail -z -n +11 | xargs -0 rm -rv

First of all I deleted -type d because I realized that in that backup folder there could also be files and not just folders.
Furthermore I added tail -n +2 to tell him not to also consider /backupFolder and it works; before, to do the same thing, I also tried targeting /backupFolder/*, but in this way it stopped working -maxdepth 1, which is more important in my case.
The first works up to sort -r, but it doesn't delete anything.
In the second case I pasted the second part of your command right after sort -r, but it goes back to sorting the results as he wants and then it goes to delete the newest ones instead of the oldest ones.


To conclude: what is the correct change I need to make to tell the system to sort the results from Z to A (because they all start with date, so most recent date at the top), and to tell it not to also consider /backupFolder as a result?
Thank you for your patience!

P.S.: It does not appear that sort -rz or sort -zr work...

Last edited by Arciere (2021-06-06 07:27:44)


Arciere

Offline

#19 2021-06-06 07:58:14

Arciere
Member
From: Torino, Italy
Registered: 2020-10-07
Posts: 59

Re: [Solved] Delete old backups from shell

Trilby wrote:

If you'd be happy deleting files older than a specific date or number of days old (rather than keeping 'n' files) just use find:

find $path_to_backups -mtime +$days

When you have replaced the path and days variables, understand what the command does, and are satisfied with the resulting list of deletion candidates, just run the command again appending the flag '-delete'.

I'd also suggest if you are going to continue using arch linux that you invest some time in learning about your system.  I'd strongly advise working through the "linux tutorial" and "bash scripting" guides here: https://ryanstutorials.net/

Thanks so much for the link (and for your suggestion)!! This site is beautiful and I did not know it, now I look at it all well.


Arciere

Offline

#20 2021-06-06 10:40:51

lambdarch
Member
Registered: 2021-01-10
Posts: 67

Re: [Solved] Delete old backups from shell

Well, it depends on what you really want to do in the end: list directories or not, search depth, etc.
In any case, it is useless to concatenate commands like that: if you use find, filter as much as you can with its options, then sort the result (if needed and only once), select some lines (if needed and only once), and finally act on these lines.

As you describe your need in #18, it seems to me that find is unnecessary:

printf '%s\0' /backupFolder/* | tail -z -n+11 | xargs -0 rm -rfv

should do the job.

You can replace rm -rfv with printf '%s\n' to see what it gives.
On a small example:

$ ls -l ./
total 0
-rw-r--r-- 1 user user 0  6 juin  12:35 20210601-filemane
-rw-r--r-- 1 user user 0  6 juin  12:35 20210602-filemane
-rw-r--r-- 1 user user 0  6 juin  12:35 20210603-filemane
-rw-r--r-- 1 user user 0  6 juin  12:35 20210604-filemane
-rw-r--r-- 1 user user 0  6 juin  12:35 20210605-filemane
-rw-r--r-- 1 user user 0  6 juin  12:35 20210606-filemane
-rw-r--r-- 1 user user 0  6 juin  12:35 20210607-filemane
-rw-r--r-- 1 user user 0  6 juin  12:35 20210608-filemane
-rw-r--r-- 1 user user 0  6 juin  12:35 20210609-filemane
$ printf '%s\0' ./* | tail -z -n+5 | xargs -0 printf '%s\n'
./20210605-filemane
./20210606-filemane
./20210607-filemane
./20210608-filemane
./20210609-filemane
$ 

Offline

#21 2021-06-06 12:56:43

Arciere
Member
From: Torino, Italy
Registered: 2020-10-07
Posts: 59

Re: [Solved] Delete old backups from shell

lambdarch wrote:

Well, it depends on what you really want to do in the end: list directories or not, search depth, etc.
In any case, it is useless to concatenate commands like that: if you use find, filter as much as you can with its options, then sort the result (if needed and only once), select some lines (if needed and only once), and finally act on these lines.

As you describe your need in #18, it seems to me that find is unnecessary:

printf '%s\0' /backupFolder/* | tail -z -n+11 | xargs -0 rm -rfv

should do the job.

You can replace rm -rfv with printf '%s\n' to see what it gives.
On a small example:

$ ls -l ./
total 0
-rw-r--r-- 1 user user 0  6 juin  12:35 20210601-filemane
-rw-r--r-- 1 user user 0  6 juin  12:35 20210602-filemane
-rw-r--r-- 1 user user 0  6 juin  12:35 20210603-filemane
-rw-r--r-- 1 user user 0  6 juin  12:35 20210604-filemane
-rw-r--r-- 1 user user 0  6 juin  12:35 20210605-filemane
-rw-r--r-- 1 user user 0  6 juin  12:35 20210606-filemane
-rw-r--r-- 1 user user 0  6 juin  12:35 20210607-filemane
-rw-r--r-- 1 user user 0  6 juin  12:35 20210608-filemane
-rw-r--r-- 1 user user 0  6 juin  12:35 20210609-filemane
$ printf '%s\0' ./* | tail -z -n+5 | xargs -0 printf '%s\n'
./20210605-filemane
./20210606-filemane
./20210607-filemane
./20210608-filemane
./20210609-filemane
$ 

So, I would like to show files and folders, but not the contents of them.
What would be the best choice in this case: find or printf (or ls at this point)?
Your #20 first command also deletes the newer backups instead of the older ones... Where should I enter a sort -r? I tried putting it before tail but it doesn't seem to work...

Remove me a doubt please: I have read that the -0 option of xargs is used precisely to avoid errors in case of spacing and must be used in combination with -print0, so of these two I do not ask myself questions and I accept them as they are. But as for the -z option next to both sort and tail I'm not finding anything: what are they for? I guess they can't be omitted, can they?

Last edited by Arciere (2021-06-06 13:03:32)


Arciere

Offline

#22 2021-06-06 13:47:33

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,447
Website

Re: [Solved] Delete old backups from shell

Arciere wrote:

But as for the -z option next to both sort and tail I'm not finding anything: what are they for?

How are you not finding anything?  You're clearly not looking.  It is very clearly covered in the man pages for each of these tools.


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

#23 2021-06-06 13:49:33

Arciere
Member
From: Torino, Italy
Registered: 2020-10-07
Posts: 59

Re: [Solved] Delete old backups from shell

Trilby wrote:
Arciere wrote:

But as for the -z option next to both sort and tail I'm not finding anything: what are they for?

How are you not finding anything?  You're clearly not looking.  It is very clearly covered in the man pages for each of these tools.

You're right, I'm not used to looking in the man page, I still search the web when I don't know something. My mistake.


Arciere

Offline

#24 2021-06-06 13:51:53

Arciere
Member
From: Torino, Italy
Registered: 2020-10-07
Posts: 59

Re: [Solved] Delete old backups from shell

So it's not enough just to put -print0 after find and -0 to xargs? Do I always have to put -z in every command I give in this case as well?


Arciere

Offline

#25 2021-06-06 13:58:01

Trilby
Inspector Parrot
Registered: 2011-11-29
Posts: 29,447
Website

Re: [Solved] Delete old backups from shell

If the input you are `sort'ing or 'tail'ing is delineated by NULL rather than newline, then yes, that's precisely what that flag is for.  Otherwise `sort` will just receive one line of input, so there is nothing to sort and it will output exactly what it was given as input completely unchanged.

Of course, you could quite easily test this yourself rather than asking.

You are expected to read man pages, and make an effort to learn on your own.

Do not take on "help vampire" habits.

Last edited by Trilby (2021-06-06 13:59:15)


"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

Offline

Board footer

Powered by FluxBB