How to organize your files? [pdf... specially for scientists

bartuka · 2015-06-25 18:28:10

Hello guys,

I've been facing a lot of issues trying to organize my files in my computer. I have a science related major and want to organize my pdfs, books, docs, ppts and so on. The standard My Documents/Downloads/Desktop/Pictures organization tree simply doesnt work for me at all.

Right now I have my files organized in themes. For example, if I download a video about eletrodynamics I will place it in the same place as all other files related to that subject. [Physics -> Physics 4 -> Eletrodynamics] . When I see that the folder is growing large, I make another folder to narrow down the subjects even more.

It's good, but somehow I feel that must have better ways to do this.
I would like to hear from you about this.

Thx

Last edited by bartuka (2015-06-25 18:30:12)

drcouzelis · 2015-06-25 19:03:16

There are two general ways to organize files: You can organize them into a hierarchy using directories (like you did) or you can use special software to "tag" the files with descriptive information that allows you to browse them in different ways (like Gmail labels).

Also as you pointed out, there are different ways to organize the hierarchy, such as by filetype or by topic or other. I can't make a suggestion without more information about what and how much you're organizing, but I do have this tip: As soon as something is no longer in active use, create an "Archive" directory, toss the files in there, and forget about them. Focus on using and organizing what you actually need but without getting rid of the old stuff.

...As for organizing personal photos and music, I've come up with wonderful ways to organize them. Let me know if you're interested in that info.

bartuka wrote:

It's good, but somehow I feel that must have better ways to do this.

It sounds like you're doing fine. Is there a specific issue you're trying to address?

dockland · 2015-06-25 19:08:02

I have my folders (in the cloud) in a structure like:

Executables - With subfolders according to if it's Applications, ISO:s,
Images - With subfolders Mine (pictures that i've taken my self) and Wallpapers and so on.
Downloads
Documents - With subfolders according to what type it is e.g. Documents, Spreadsheets, Text-files and so on.

In each "main" folder i have a folder called "old" where i script files not accessed the last 6 months is automatically moved to in their own subfolder. Same folder hierarchy there so to speak.

I dont have any music or movies on my computers. I have them on a 3 TB NAS. But then, i'm not a collector, i delete a movie when i've seen it and stream my music. (:

E: My file and folder structure needs to be the same on every OS i use @ home as well as @ work. OSX, Win, Android, IOS and Linux. Therefore i can't trust "tags" and similar.

Last edited by dockland (2015-06-25 19:50:24)

Trilby · 2015-06-25 19:14:24

As an academic researcher, I have more pdfs stored than I'd care to count. Do what works for you - but for me, a directory hierarchy would not work at all. All research today is "interdisciplinary" (though I loathe that word). So where do you file the paper that uses new advances in physics to improve genome sequencing? Is that physics, is it biology? What if the technique were used to understand cancer, then is it medicine?

I just keep all my pdfs in a single folder ~/doc/pdfs/ Each pdf is named identical to the key in my reference management database - generally that would be the last name of the first author and the year followed by a letter when needed (e.g., Trilby2015, Trilby2015a, ...)

Then get a decent reference management software (like a bibtex database, and chose any number of frontends) and it will allow you to 'tag' or keep notes about different papers along with all the citation information. You can then search by author, by keyword, by your own notes or tags, etc. You can even come up with handy ways to open every pdf written by a certain author matching a certain keyword. For example, if I want to pull up the pdf of every paper I have by Melissa Bateson that matches a keyword "decision", I can just do the following:

lobster -s author=bateson+decision -o

(lobster used in example).

Malkymder · 2015-06-25 19:39:03

I would also use software to manage the documents, here's a short list of some i find usefull

Available using pacman
calibre
gnome-documents

Available on AUR
referencer
jabref

bartuka · 2015-06-25 19:42:53

Thx for the comments.

The problem I want to address is that I have many pdfs and many different subjects. For example, I research in two main subjects that is related to many small subjects in physics, computer science, matematics and geology. (plus mine obssession for computer in general.. hehehe go Arch). So, I want to be able to FIND a specific paper about a subject. I might remember the main topic of that paper, but rarely more than that. (after some time).

I've tried to use Evernote and it was quite good for a while, but the portability was awfull. Tag systems are way too complicated to deal with. I think I will stick with the method I described. Remember topics are usually easy. (I have almost 300GB of files to organizze... too boring). Really appreciate the tip about the Archive folder, but I will have one of that for each subfolder? =/

drcouzelis, I really wish to hear about your methods for music and photos.. these two folders are just a meeessss!! I have to pass through the whole thing to find something in there. God knows how I like ranger and the preview image for that.

bartuka · 2015-06-25 19:55:00

Trilby, that sounds really good option.
I've tried to use jabref a while ago, but I haven't tried enough to like it. I will definetly try it again. But, how do you deal with docs, pdfs and ppts that are, let's say, lecture notes or even some random presentation you found about something. Can I also put that on jabref for example?

Trilby · 2015-06-25 20:04:03

bartuka wrote:

But, how do you deal with docs, pdfs and ppts that are, let's say, lecture notes or even some random presentation you found about something. Can I also put that on jabref for example?

Absolutely.

Jabref itself by default has "columns" for pdfs and urls. But you can also associate *any* other filetype. Jabref also manages filetype associations. So if the "file" for a given entry in the database is a pdf, it will open it in your chosen pdf reader; if it is a doc, it will open it in libreoffice, or your chosen doc reader; if it is a url, it will open it in your web browser; etc.

I don't use jabref anymore, the interface was far to cumbersome for my tastes [1] - but all the cool things that jabref can do have very little to do with jabref - they are just part of the underlying bibtex database. And even though it's called a "database" it is just a plain text file that you can edit, grep, parse with scripts, etc. You can define your own fields for a Article in a bibtex database, so if "pdf" doesn't seem like the right field, you could use a different one, like File, or Document, whatever (but consistency will help searching).

Even better, with bibtex you can create different entry types. So you would use Article (or @Article in bibtex syntax) for regular publications. But you could also use @Paper for papers you wrote for classes, and @Paper would have different data fields than @Article. You could have another tpye @Website, etc.

But I'd encourage you to keep to as few entry types as possible, and keep the field names as consistent as possible. Parallelism in the data makes searching much easier. So, going back to the start of this post, jabref now just uses a generic FILE field. It doesn't matter if the FILE associated with an entry is a pdf, or a doc, or a url - the filetype is inferred from the name or content and it is displayed in the appropriate tool.

EDIT for note [1]: I would still highly recommend jabref. It is the best readily-available general-purpose user-friendly bibtex frontend available in my opinion. But I just didn't need user-friendly or general-purpose - so I made my own to do exactly what I want but nothing else.

bartuka · 2015-06-25 20:10:27

That sounds great. I had a problem with password storage some time ago and I solve it with a simple text file wheere I can encrypt and search the entries. I was considering to do the same for my pdfs, but it seems a lot of work. The bibtex sounds like the txt file process.

I just installed jabref to take a look at the interface and it really looks cumbersome... lool. What options do you suggest for me?.

Really thx for the tips guys. I will also look for more information about bibtex later on.

Edit for your edit note[1] : LOL, you just answered my question. Some delay here. rsrs.

Last edited by bartuka (2015-06-25 20:12:58)

Trilby · 2015-06-25 20:16:00

Well, my 'lobster' script linked above does everything I want. But it is just a simple CLI-only tool to manage a bibtex database. It depends on bibtool, which itself is a phenomonally powerful CLI tool for working with bibtex files. But for me bibtool's commands were just very odd - so 'lobster' is basically just a wrapper around bibtool.

bartuka · 2015-06-25 20:26:44

Malkymder had posted some softwares to manager files. I will see them. Thx.

And I will stick with the "user-friendly" options for the beginning of this task and then I try your script Trilby. I really appreciate your help on this matter. It is painfull to look at all these files searching for somethinf from two years ago that I know that the answer to my problem is in that file. =[

drcouzelis · 2015-06-25 20:29:57

bartuka wrote:

Really appreciate the tip about the Archive folder, but I will have one of that for each subfolder?

Sure! Other alternatives include using hidden ".archive" directories instead (so you don't have to always see them) or a "higher up" archive folder which duplicates your main hierarchy.

bartuka wrote:

drcouzelis, I really wish to hear about your methods for music and photos.. these two folders are just a meeessss!!

For photos: Under "~/Images", I have a directory for every year, followed by a directory for every event.

~/Images/2015/20150622 - Trip to Texas with Grandpa/*.jpg

Each directory is now a photo album dated in chronological order, and contains anywhere between 1 and a couple hundred photos. It's portable, easy to backup (I threw "2014" on a DVD on January 1), and is easy to search in (When did we take that trip to Texas? "ls -1d ~/Images/*/*Texas*").

For music: Under "~/Music", I have a directory hierarchy labelled with genres, albums, artists, and track names.

~/Music/Electronic/Maf/Maf - Mafland.mp3

Most of the time, the artist matches the album name, because when I want Beach Boys music I couldn't care less about the difference between "Pet Sounds" and "Smile", I just want a directory full of my favorite "The Beach Boys" music. If order matters, then it'll be something like:

~/Music/Game/Goldeneye (Rip, Atmospheric)/Goldeneye - 03 Mission Select.mp3"

Lastly, I wrote a dirty little script that takes an MP3 file, deletes any ID3 information, and recreates it based on it's directory hierarchy and filename: https://github.com/drcouzelis/scripts/b … c-add-tags

graysky · 2015-06-25 20:32:13

https://aur.archlinux.org/packages/mendeleydesktop/

???

bartuka · 2015-06-25 21:09:39

Haha! I will try your tips for music organization as soon as possible. The image folder is just unhopeful. rsrs. And graysky, I used Mendeley for a while but I didnt like it very much.

The package called referencer in AUR is really great and simple. I can even export everything as bibtex in the future. I like it. I will give it a try.

drcouzelis · 2015-06-26 14:13:26

bartuka wrote:

The image folder is just unhopeful.

Leave the photos you have now the way they are. They won't become any more disorganized if you just leave them. And start a new directory with a new organization scheme for any new photos.

The best time to start organizing your files is ten years ago. The second best time to start organizing your files is today!

chaonaut · 2015-06-26 14:49:44

i have two basic file hierarchies, „cosmos“ for stuff that has been put to order yet, and „chaos“ for… well, for chaotic stuff
i don't like to rely on any kind of cataloging software, so trying to name everything in such manner that allows nearly anything to be found by file/dir names. (for images, that is just a directory containing some image „category“, and then i have to look for certain file with an image viewer, since any image file naming scheme is completely useless.)
that's all.

oh, and i love those MacOS-style color labels.
i think, with mc² they will appear to the text console.

Last edited by chaonaut (2015-06-26 14:56:06)

bartuka · 2015-06-26 15:41:22

I have to tell, the tip Trilby suggested is really great for organizing papers/pdfs/docs and whatever. I just started to organize my "Book" folder and I found it to be very consistent. The good part about it, is that I can have several tags associated to the same file (I know, as few as possible is the idea) and the package called Referencer (AUR) is just incredibly fast for finding stuff. I loved the feature that enable me to search only inside the tag-related-files.

This is a screenshot of how my folder looks like right now and the Referencer screen: http://s24.postimg.org/7u4aw8x9h/files.png

However, I am thinking to still keep my hierarchy-folder-structure to save projects like Scripts... I usually have some programs in C/Fortran/Python that I'd like to keep it in a specific place.

Thx so much for every single suggestion untill here. About the image folder... you are right! I will start to organize it as soon as possible. I liked the "chaotic" folder idea. rsrs.

progandy · 2015-06-26 16:00:25

If you want to tag your normal files, why don't you add a dantalian tagspace? http://darkfeline.github.io/dantalian/

Trilby · 2015-06-26 16:11:09

bartuka wrote:

The good part about it, is that I can have several tags associated to the same file (I know, as few as possible is the idea)

Actually having many tags for an item can be useful. My arguement was that you should not let fields proliferate. There is some clash of terminology here depending on just which software/database is used. But what should be kept at a minimum is field types: so don't use "keywords" for some datase entries and "tags" for others and "notes" for yet others: try to keep entries as consistent as practical in their format: All items probably have a title - so that is one field, most will have an author (or authors) so that is a field. But some items will have a pdf, others will have a url, yet others will have a 'doc' or 'ppt' or ... If there is a distinct field for each of these, it gets harder to structure useful searches/queries and harder yet to automate opening of these things. But all these can go under a "File" field.

Within a field like keywords or tags, you should feel free to put as many keywords/tags as are relevant to that item as it will only help you construct more specific searches.

chaonaut · 2015-06-26 19:32:14

progandy wrote:

If you want to tag your normal files, why don't you add a dantalian tagspace? http://darkfeline.github.io/dantalian/

wow! cool thing. could help manage images, because now i hardlink them manually (can you believe?)

Arch Linux

#1 2015-06-25 18:28:10

How to organize your files? [pdf... specially for scientists

#2 2015-06-25 19:03:16

Re: How to organize your files? [pdf... specially for scientists

#3 2015-06-25 19:08:02

Re: How to organize your files? [pdf... specially for scientists

#4 2015-06-25 19:14:24

Re: How to organize your files? [pdf... specially for scientists

#5 2015-06-25 19:39:03

Re: How to organize your files? [pdf... specially for scientists

#6 2015-06-25 19:42:53

Re: How to organize your files? [pdf... specially for scientists

#7 2015-06-25 19:55:00

Re: How to organize your files? [pdf... specially for scientists

#8 2015-06-25 20:04:03

Re: How to organize your files? [pdf... specially for scientists

#9 2015-06-25 20:10:27

Re: How to organize your files? [pdf... specially for scientists

#10 2015-06-25 20:16:00

Re: How to organize your files? [pdf... specially for scientists

#11 2015-06-25 20:26:44

Re: How to organize your files? [pdf... specially for scientists

#12 2015-06-25 20:29:57

Re: How to organize your files? [pdf... specially for scientists

#13 2015-06-25 20:32:13

Re: How to organize your files? [pdf... specially for scientists

#14 2015-06-25 21:09:39

Re: How to organize your files? [pdf... specially for scientists

#15 2015-06-26 14:13:26

Re: How to organize your files? [pdf... specially for scientists

#16 2015-06-26 14:49:44

Re: How to organize your files? [pdf... specially for scientists

#17 2015-06-26 15:41:22

Re: How to organize your files? [pdf... specially for scientists

#18 2015-06-26 16:00:25

Re: How to organize your files? [pdf... specially for scientists

#19 2015-06-26 16:11:09

Re: How to organize your files? [pdf... specially for scientists

#20 2015-06-26 19:32:14

Re: How to organize your files? [pdf... specially for scientists

Board footer