[SOLVED] How to create recursive subdirectories (mkdir -p)

MorningWood · 2020-07-25 15:39:54

I'm trying to create a number of subdirectories titled "Data" within a specific parent folder and I've tried mkdir -p *DIRECTORYNAME*/{Data} but the command winds up creating a literal, single interpretation titled {Data} within the parent directory and doesn't repeat into existing directories.
Is there any other way of going about this?
Thanks!

Last edited by MorningWood (2021-06-10 09:54:32)

graysky · 2020-07-25 15:48:59

Remove the brackets. Still confused about what you want to do.

If you want to make /foo/bar/baz/qux but you only have /foo/bar on the filesystem, this will work:

mkdir -p /foo/bar/baz/qux

Last edited by graysky (2020-07-25 15:50:47)

MorningWood · 2020-07-25 15:52:51

graysky wrote:

Remove the brackets. Still confused about what you want to do.

Trying to create recursive subdirectories within a single parent a la 1/2/Data 1/3/Data 1/4/Data and so on...

Removing the brackets acts exactly as the same argument, just sans brackets in the subdirectory title.

Last edited by MorningWood (2020-07-25 15:54:09)

graysky · 2020-07-25 16:09:55

Setup a test bed:

mkdir here
cd here
for i in {1..5}; do mkdir -p tit/$i; done

So now you have:

└── tit
    ├── 1
    ├── 2
    ├── 3
    ├── 4
    └── 5

Add a Data dir under each like this:

find tit -type d -print0 | xargs -0 -i mkdir -p {}/Data

Result:

└── tit
    ├── 1
    │   └── Data
    ├── 2
    │   └── Data
    ├── 3
    │   └── Data
    ├── 4
    │   └── Data
    ├── 5
    │   └── Data
    └── Data

It gave an extra Data which you can remove. Maybe there is a more elegant way.

Ropid · 2020-07-25 16:19:32

It's also possible without a 'for' loop, like this:

mkdir -p foo/{1..4}/Data

Here's an example showing how it behaves:

$ echo foo/{1..4}/Data
foo/1/Data foo/2/Data foo/3/Data foo/4/Data

$ mkdir -p foo/{1..4}/Data

$ tree
.
└── foo
    ├── 1
    │   └── Data
    ├── 2
    │   └── Data
    ├── 3
    │   └── Data
    └── 4
        └── Data

9 directories, 0 files

graysky · 2020-07-25 16:22:55

@Ropid - You're right about the loop, that is just there to give some dirs. If the OP's dirtree is different than that, he will need to use the find command which just puts a Data dir at the end of what it find. It will do it for all hit though as you saw in my case so might not be that neat and tidy.

eschwartz · 2020-07-26 03:16:09

MorningWood wrote:

I'm trying to create a number of subdirectories titled "Data" within a specific parent folder and I've tried mkdir -p *DIRECTORYNAME*/{Data} but the command winds up creating a literal, single interpretation titled {Data} within the parent directory and doesn't repeat into existing directories.
Is there any other way of going about this?
Thanks!

mkdir -p *DIRECTORYNAME*/{Data}

As written, this will try to use shell globbing rules to find an existing file/directory or files/directories named "{Data}" inside a parent directory containing the fragment "DIRECTORYNAME" with match-anything globbing before and after. Failing to find such an existing full path, it will fall back to treating the entire string as a literal. Finally, it will execute mkdir -p with each glob expansion as an argument.

If there were successful matches, mkdir will do nothing, since the directories already existed. If there were no successful matches, it will create a directory literally named "*DIRECTORYNAME*", and then create another directory inside it, literally named "{Data}".

Given DIRECTORYNAME is certainly not actually what you wanted, the first lesson here is "give the actual information you're dealing with".

For the second lesson here, I'll read between the lines and guess that you expected to match a bunch of existing directories with *DIRECTORYNAME*, and in each of those directories, create a new directory named {Data} (except not with actual brackets, unless your use case is a LOT odder than I thought). This will never ever work. You need to pregenerate matches for *existing* directories, and only after append the directory you want to create.

for d in *foobar*/; do
    mkdir -p "$d"/baz
done

We can continue to endlessly theorize about things vaguely similar to what you asked in the hope it helps somehow, or you can give the exact, full, unexpurgated case you're dealing with and we'll be empowered to immediately give you the correct answer.

Trilby · 2020-07-26 03:49:30

Still no need for a loop or multiple mkdirs if the above is the actual goal:

find -type d -name '*DIRECTORYNAME*' -exec mkdir {}/Data \+

@OP, I also highly doubt you want to create recursive subdirectories. You just want to create subdirectories under many targets (I think). These are very different goals. Perhaps you meant that you want to create a subdirectory under anything that matches where the search for the match descends recursively. But this is still not the same as creating recursively: you don't want infinitely many 'Data' directories under every 'Data' directory, right?

Last edited by Trilby (2020-07-26 03:52:28)

MorningWood · 2020-07-26 16:55:20

Sorry for not making it more clear, all. The *DIRECTORYNAME* was just a placeholder for the parent directory title. So, @Trilby your assumption was correct.
I have a media library that consists of directories for artists which house numerous existing album directories. I would like to match every album directory within each artist and create Artwork and Data directories within said album directory. Thus I would like to create a subdirectory where the search for the match descends recursively; not create subdirectories recursively.

So to reiterate I would like to initiate the following:

Create subdirectories for Data & Artwork within Artist 1>Album 1
Artist 1>Album 2
Artist 1>Album 3

Artist 2>Album 1
Artist 2>Album 2
Artist 2>Album 3

Although this does not need to be on a macro level, so it doesn't need to descend from artist to artist but would be preferred.

I have tried Trilby's solution but get the following output:

find: In ‘-exec ... {} +’ the ‘{}’ must appear by itself, but you specified ‘{}/Data’

Trilby · 2020-07-26 17:03:17

That's an odd find error that I can't replicate - but nevermind that, I (we) were completely wrong about what you were trying to do. You do not want create "Data" as a subdir to every child directory that matches a regex - rather you just want to create the subdir in all child directories as a specified and consistent depth. That is a much simpler problem. The following will get the list of directories you want to add a subdirectory too, right?

find /path/to/music -type d -maxdepth 2 -mindepth 2

If so, this will do it:

find /path/to/music -type d -maxdepth 2 -mindepth 2 -exec mkdir '{}/Data' \+

You might get a similar error as in your last post which we can work on debuging if it happens again, but I just created a test directory tree and this command worked as intended.

EDIT: standby, this does seem to fail with gnu find.

This should work ... but I'd love to hear from others why GNU find is choking on this when it should run just fine (busybox's find runs it without issue).

find /path/to/music -maxdepth 2 -mindepth 2 -type d | xargs -I DIR mkdir DIR/Data

Last edited by Trilby (2020-07-26 17:07:03)

Ropid · 2020-07-26 17:20:15

It seems that "{}/Data" argument only doesn't work when using the "-exec" version that ends with a "+". It works when using ";". I mean this here:

find /path/to/music -maxdepth 2 -mindepth 2 -type d -exec mkdir '{}/Data' ';'

There's also a strange warning message when you put "-type d" in the front. The warning goes away when it's put after the two depth arguments.

eschwartz · 2020-07-26 17:31:06

find -exec will replace the standalone string "{}" with filenames, per the POSIX spec. It's implementation-defined what the standalone string "{}/Data" does, and GNU find replaces it with one instance of the string+suffix per subprocess with \; as a special case, but for + cannot handle attempting to modify the paths which {} inserts.

"This should work" -- piping to xargs does exactly nothing useful, it does one mkdir per directory, GNU find -exec 'mkdir -p {}/Data' \; works just fine too if you want to run mkdir multiple times.

busybox find, oddly, seems to support -exec {} + where the entire string "{}/Data" is appended to the command repeatedly, each time with {} replaced by a resulting filename. It's not immediately obvious if this is a feature or a bug, and I suspect the reason busybox does it is as a side effect of treating \; and + interchangeably (you can disable support for the latter, in which case it acts as an alias of the former).

Trilby · 2020-07-26 17:42:51

eschwartz wrote:

"This should work" -- piping to xargs does exactly nothing useful, it does one mkdir per directory, GNU find -exec 'mkdir -p {}/Data' \; works just fine too if you want to run mkdir multiple times.

So it doesn't do "nothing useful" unless gnu-find also does nothing useful as it does the exact same thing.

eschwartz · 2020-07-26 18:36:42

Well, it avoids using a working builtin by adding clumsy reprocessing with the notoriously fragile xargs for the same inefficient processing.

(I don't like xargs.)

bash, executed outside of find thus not being protected from ARG_MAX overflow:

readarray -t -d '' dirs < <(find /path/to/music -maxdepth 2 -mindepth 2 -type d -print0)
mkdir -p "${dirs[@]/%/\/Data}"

createme=()
for i in */*/; do
    createme+=("$i/Data")
done
mkdir -p "${createme[@]}"

Working find | xargs implementation, requires GNU:

find /path/to/music -maxdepth 2 -mindepth 2 -type d -printf '%p/Data\0' | xargs -0 mkdir -p

The surprising busybox feature to interject {} with suffixes multiple times in combination + doesn't work on FreeBSD either, FreeBSD find returns this if you try:

find: -exec: no terminating ";" or "+"

I'm not sure why though.

Trilby · 2020-07-26 20:19:17

Apparently busybox's behavior is counter to the POSIX specification in this case as the specification does indeed say the + must be immediately after the {} and that these must be at the end of the command.

But damn if busybox's non-POSIX behavior isn't useful

eschwartz · 2020-07-26 20:52:08

That's actually a nuance which escaped me, and handily clarifies why FreeBSD find insists it cannot even detect a terminating +. GNU find's error message is a lot more obvious, though.

On rereading the spec, the unspecified handling of "{}" with prefix/suffix is limited to the \; mode, and the + mode defines one required standalone "{}" immediately before the +, other arguments containing "{}" either alone or with prefix/suffix as unspecified. It could use some rewording, I feel.

Raynman · 2020-07-27 09:38:52

Does the readarray with find have any advantages over the following?

dirs=(/path/to/music/*/*/)

zsh can do brace-like expansion with array variables:

mkdir -p ${^dirs}Data

Or instead a oneliner with glob qualifiers:

setopt histsubstpattern
mkdir -p /path/to/music/*/*/(:s/%/Data)

As another alternative, GNU parallel with -X also handles {}/Data as busybox find does.

eschwartz · 2020-07-27 12:36:29

Raynman wrote:

Does the readarray with find have any advantages over the following?
dirs=(/path/to/music/*/*/)

If the goal is merely to find subdirectories according to */*/, it seems you already know the answer to this perfectly well. But I've attempted to cover a variety of somewhat diverse methods rather than repeating the same thing with slight variations...

Raynman wrote:

zsh can do brace-like expansion with array variables:
mkdir -p ${^dirs}Data
Or instead a oneliner with glob qualifiers:
setopt histsubstpattern
mkdir -p /path/to/music/*/*/(:s/%/Data)

"Use this non-default shell with exotic settings turned on" doesn't make much sense. No one is going out of their way to install zsh just for this.

And the brace-like expansion doesn't seem to have any functional difference to bash "${dirs[@]/%/\/Data}" except for quibbling over syntax.

Raynman wrote:

As another alternative, GNU parallel with -X also handles {}/Data as busybox find does.

GNU parallel is like bringing out an elephant gun to hunt rabbits, except the elephant gun plays ads every time you fire it, load it, safety/unsafety it, read its manual, or walk 20 feet while carrying it. How are you supposed to hunt rabbits with all that noise?

Trilby · 2020-07-27 13:12:56

In fact no fancy shell bells and whistles are needed. A fully POSIX version:

set -- /path/to/music/*/*

mkdir -p $(printf "%s/Data " "$@")

Note that this will try to create a subdirectory under any regular files at the specified depth as well, but it will fail to do so and just emit an error while continuing with other paths. If needed, you can silence the error by redirecting stderr.

Last edited by Trilby (2020-07-27 13:15:55)

eschwartz · 2020-07-27 13:16:34

The fancy bells and whistles are needed to handle directories with whitespace.

And this is music stuff, so there are assuredly such directories!

MorningWood wrote:

Create subdirectories for Data & Artwork within Artist 1>Album 1
Artist 1>Album 2
Artist 1>Album 3
Artist 2>Album 1
Artist 2>Album 2
Artist 2>Album 3

Arch Linux

#1 2020-07-25 15:39:54

[SOLVED] How to create recursive subdirectories (mkdir -p)

#2 2020-07-25 15:48:59

Re: [SOLVED] How to create recursive subdirectories (mkdir -p)

#3 2020-07-25 15:52:51

Re: [SOLVED] How to create recursive subdirectories (mkdir -p)

#4 2020-07-25 16:09:55

Re: [SOLVED] How to create recursive subdirectories (mkdir -p)

#5 2020-07-25 16:19:32

Re: [SOLVED] How to create recursive subdirectories (mkdir -p)

#6 2020-07-25 16:22:55

Re: [SOLVED] How to create recursive subdirectories (mkdir -p)

#7 2020-07-26 03:16:09

Re: [SOLVED] How to create recursive subdirectories (mkdir -p)

#8 2020-07-26 03:49:30

Re: [SOLVED] How to create recursive subdirectories (mkdir -p)

#9 2020-07-26 16:55:20

Re: [SOLVED] How to create recursive subdirectories (mkdir -p)

#10 2020-07-26 17:03:17

Re: [SOLVED] How to create recursive subdirectories (mkdir -p)

#11 2020-07-26 17:20:15

Re: [SOLVED] How to create recursive subdirectories (mkdir -p)

#12 2020-07-26 17:31:06

Re: [SOLVED] How to create recursive subdirectories (mkdir -p)

#13 2020-07-26 17:42:51

Re: [SOLVED] How to create recursive subdirectories (mkdir -p)

#14 2020-07-26 18:36:42

Re: [SOLVED] How to create recursive subdirectories (mkdir -p)

#15 2020-07-26 20:19:17

Re: [SOLVED] How to create recursive subdirectories (mkdir -p)

#16 2020-07-26 20:52:08

Re: [SOLVED] How to create recursive subdirectories (mkdir -p)

#17 2020-07-27 09:38:52

Re: [SOLVED] How to create recursive subdirectories (mkdir -p)

#18 2020-07-27 12:36:29

Re: [SOLVED] How to create recursive subdirectories (mkdir -p)

#19 2020-07-27 13:12:56

Re: [SOLVED] How to create recursive subdirectories (mkdir -p)

#20 2020-07-27 13:16:34

Re: [SOLVED] How to create recursive subdirectories (mkdir -p)

Board footer