How to create a repository(not just custom) using your hard drive

smitty · 2008-01-17 20:12:26

I don't know if many people know about this, so I am giving this a shot. There are three major articles on wiki.archlinux.org: Custom local repository,
Using a CD-ROM as a repository, and Offline Installation of Packages. These are available online through the WIKIs at archlinux.org.
I was first confused because when I was reading "Offline installation of packages", I didn't know what these ".db.tar.gz" files where. I came mainly from a Debian / Ubuntu
background (I actually tried many distros before this), so getting used to the way the repository works and no graphical install manager for it. However, I enjoyed a challenge and
I found out that these are database packages that contain descriptions and locations on where these files are located. The ones on the ftp server are already compiled. I don't know if,
however they are compiled with the most recent versions.
With all that said, I thought you had to have it all in one directory in order for this to work, but as it turns out, location is not really an issue. I decided to have a directory reside on the root.
I chose root because it's only for the install of my own packages. I could have done it as a seperate user account, such as "repos" in PCLinuxOS (another distro I tried). I didn't want to have a seperate account for this. Therefore, I created "/root/repository". Within this directory I created directories for all repository archives. I basically did a "cd /mnt/dvd" and migrated to the particular repository directories. I would copy all the "pkg.tar.gz" files into their respective directories with "cp * ~/repository/<name-of-dir>". For intance, I started with the "core" directory, because there was some things I didn't install in the core directory during installation and if the packages needed it, it was there. This follows for the rest of the directories, such as "community", "testing", and "unstable", etc.You can go to the ftp mirrors to find out what directories are available. The main point is that your files should be in the format ".pkg.tar.gz". These are package files that get converted into a sort of database format that as I mentioned, informs the system the description and where the files are located, and so on.
The command to perform this, is "tar -xvf /root/repository/core/core.db.tar.gz *.pkg.tar.gz". You can replace core with whatever repository you are adding. So, for example, "extra.db.tar.gz" would be in the "extra" directory. This information is located in the "Offline installation of packages". The command to create this database is called, "repo-add".
The format for this command is "repo-add /path/to/dir.db.tar.gz *.pkg.tar.gz". So, if it's the core packages you would "cd ~/repository/core" and "repo-add core.db.tar.gz *.pkg.tar.gz".
Then, you need to edit the "/etc/pacman.conf" configuration file for pacman. I basically would comment all out except for the repositories I need. So, for example "[core]" and "/etc/pacman.d/core" would tell where normally the servers are located for these files. This information is located int the "Custom local repository" article.using the "repo-add" command.
Furthermore, I edited each server file located in "/etc/pacman.d/<repository>" where repository is core, extra, etc. I would perform, "nano /etc/pacman.d/core" for example and comment out all servers. I then add a "local repository" by typing in "file:///root/repository/core", saved it, and then did a "pacman -Sy" to update the repository database. Now, I can do "pacman -S <package-name>" where package-name is whatever I wanted to install. Voila! Please let me know of any suggestions, questions, insights, or comments. I hope I'm not missing anything in this article. I do remember using "rm -rf * in the "/var/lib/pacman/<repository>"directories and using "tar xvf <repository>.db.tar.gz". I don't if that something to do with it, though. Be careful with the "rm -rf *" command, because you can erase your hard drive if you are not careful, for those who aren't informed.

P.S. Please note all these are done with the root user.

pressh · 2008-01-17 22:23:07

You tricked me!

I started reading this supposing that *at some point* a question would come....but it never came.
I really do appreciate your hard work in writing this howto but let me give you some hints if you do something similar in the future :
- Try to keep it short and clear
- Use some white lines that would scare people less

Takes me to the point that I'm not sure what extra information there is in your article which is not already in the wiki.

tomk · 2008-01-17 23:50:02

It certainly sounds like you've been working hard, and I hope others get some benefit from it. Just one observation - you have basically created a snapshopt of the repos as they are now. How do you plan to keep your local repos in sync with the official repos?

lilsirecho · 2008-01-18 00:17:15

For a snapshot, this creates a heavy burden in root.

I suggest in the interest of more speed in boot that the repo be placed in USB flash drives which are removeable.

You seem to like to separate the repos, so do so woth flash device storage of your downloads.

In order that same be upgradeable, use such a system as follows;

Core repo, upgradeable flash drive A; ...pacman --cachedir /media/disk/A -Syw (Core package name)

Extra repo, upgradeable flash drive B; ....pacman --cachedir /media/disk/B -Syw (extra package name)

etc.......................................

This places each repo on a separate flash drive and provides for dependencies and upgrades.

Any of the packages can be installed with:....pacman -Ud /media/disk/(X) (package.pkg.tar.gz).

Why you would separate the repos is not apparent since many dependencies cross over the repo boundaries!

Keep the repos out of the HDD!!!!!

smitty · 2008-01-18 04:24:15

As far as keeping in sync with the updated repositories, you can simply enable them by editing the "/etc/pacman.d/<repository>" and re-enabling the servers you want. This is the reason I had commented these out. You could technically delete these repository files yourself and create new ones, but you wouldn't be able to "re-enable" the servers, since you would have deleted them.

In relation to having it all on flash drive that depends on how much storage capacity you have for your flash drive. I didn't know the speed benchmarks of a flash drive vs hard drive, but yes, it does make sense to use that as a medium, as well. I don't think the "location" where the repository resides matters, as I mentioned in the article, from a technical point of view.

The WIKIs seem to point to certain methods of implementation, but not the method that I wanted and it had nothing to do with the way I eventually did it. One method, for example, mentioned that you could have it in all one directory. The problem with this approach, I see, is that if I wanted to track down a package, I wouldn't know where the repository orginated from. I hope this helps.

smitty · 2008-01-18 08:20:24

pressh, I understand and appreciate your point of view... well taken! Are you implying that I should have written in steps, such as 1, 2, and 3? Also, should I have got ridden of the redundant information if it is contained in the Wiki article and / or taken out the commands on how to apply them and left only with the explanation? Is this what you imply? Sorry if I seem redundant with these questions, but I'm curious so I can improve for the future. I am new to this and open to any suggestion and comments.

gradgrind · 2008-01-18 09:21:48

smitty wrote:

pressh, I understand and appreciate your point of view... well taken! Are you implying that I should have written in steps, such as 1, 2, and 3? Also, should I have got ridden of the redundant information if it is contained in the Wiki article and / or taken out the commands on how to apply them and left only with the explanation? Is this what you imply? Sorry if I seem redundant with these questions, but I'm curious so I can improve for the future. I am new to this and open to any suggestion and comments.

Maybe you could either edit the existing wiki pages where they were not clear to you, or else add a new wiki page, or both. Certainly give the whole a clearer (visual) structure, and (if they don't already exist) add links between the connected wiki pages.

tomk · 2008-01-18 09:57:39

smitty wrote:

As far as keeping in sync with the updated repositories, you can simply enable them by editing the "/etc/pacman.d/<repository>" and re-enabling the servers you want.

OK... but as soon as you do that, your on-disk repos become obsolete. I thought the idea was to create on-disk repos to use as a replacement for the official repos?

pressh · 2008-01-18 09:57:51

gradgrind wrote:

smitty wrote:
pressh, I understand and appreciate your point of view... well taken! Are you implying that I should have written in steps, such as 1, 2, and 3? Also, should I have got ridden of the redundant information if it is contained in the Wiki article and / or taken out the commands on how to apply them and left only with the explanation? Is this what you imply? Sorry if I seem redundant with these questions, but I'm curious so I can improve for the future. I am new to this and open to any suggestion and comments.
Maybe you could either edit the existing wiki pages where they were not clear to you, or else add a new wiki page, or both. Certainly give the whole a clearer (visual) structure, and (if they don't already exist) add links between the connected wiki pages.

Yes that is partly what I mean. Further you could get rid of the information that is not really needed to follow the guide (for example what the command 'repo-add' does. People could if they are interested look it up in the script itself, or you could add it here and link to it).
And yes a bit of structure would be nice. You don't have to nessesarily call it 1,2,3, as long as it has some kind of structure in it (the visual point is very important here). You could take a look at existing wiki pages on the web and see how most of them (not all of them are good of course) are structured.

smitty · 2008-01-18 19:13:56

gradgrind wrote:

smitty wrote:
pressh, I understand and appreciate your point of view... well taken! Are you implying that I should have written in steps, such as 1, 2, and 3? Also, should I have got ridden of the redundant information if it is contained in the Wiki article and / or taken out the commands on how to apply them and left only with the explanation? Is this what you imply? Sorry if I seem redundant with these questions, but I'm curious so I can improve for the future. I am new to this and open to any suggestion and comments.
Maybe you could either edit the existing wiki pages where they were not clear to you, or else add a new wiki page, or both. Certainly give the whole a clearer (visual) structure, and (if they don't already exist) add links between the connected wiki pages.

Once I create the article, what are the rest of the procedures?

smitty · 2008-01-18 19:19:35

tomk wrote:

smitty wrote:
As far as keeping in sync with the updated repositories, you can simply enable them by editing the "/etc/pacman.d/<repository>" and re-enabling the servers you want.
OK... but as soon as you do that, your on-disk repos become obsolete. I thought the idea was to create on-disk repos to use as a replacement for the official repos?

Good point. Yes, that's the idea, initially. So, from that standpoint, would the end-user want to know how to upgrade from there? This generally would be my assumption. I think the
more choices one has the better. Then, the question would be, would the end-user want a separate article on using non-ftp repository and regular ftp repositories?

smitty · 2008-01-18 19:23:57

pressh wrote:

gradgrind wrote:
smitty wrote:
pressh, I understand and appreciate your point of view... well taken! Are you implying that I should have written in steps, such as 1, 2, and 3? Also, should I have got ridden of the redundant information if it is contained in the Wiki article and / or taken out the commands on how to apply them and left only with the explanation? Is this what you imply? Sorry if I seem redundant with these questions, but I'm curious so I can improve for the future. I am new to this and open to any suggestion and comments.
Maybe you could either edit the existing wiki pages where they were not clear to you, or else add a new wiki page, or both. Certainly give the whole a clearer (visual) structure, and (if they don't already exist) add links between the connected wiki pages.
Yes that is partly what I mean. Further you could get rid of the information that is not really needed to follow the guide (for example what the command 'repo-add' does. People could if they are interested look it up in the script itself, or you could add it here and link to it).
And yes a bit of structure would be nice. You don't have to nessesarily call it 1,2,3, as long as it has some kind of structure in it (the visual point is very important here). You could take a look at existing wiki pages on the web and see how most of them (not all of them are good of course) are structured.

That's a good point, too. How do I found out what articles are more effective? I am doing research on this particular matter at the moment and came across articles that have tips on technical writing. Could this help in the long run? Or, is it better to get feedback from other users and improve that way? In other words, do first, and ask later, as one user point out?

gradgrind · 2008-01-19 09:50:41

smitty wrote:

... How do I found out what articles are more effective? I am doing research on this particular matter at the moment and came across articles that have tips on technical writing. Could this help in the long run? Or, is it better to get feedback from other users and improve that way? In other words, do first, and ask later, as one user point out?

It's probably a matter of personal taste, but learning by doing can be very effective and satisfying (perhaps a little embarrassing at times too, but so what?). We're talking about a wiki here, anyone can edit it, improve it, even - though hopefully not too often - put garbage in there. It's a terrific social exercise!

tomk · 2008-01-19 10:49:34

smitty wrote:

tomk wrote:
I thought the idea was to create on-disk repos to use as a replacement for the official repos?
Good point. Yes, that's the idea, initially.

If you're setting up local repos that will stay in sync with the official repos, you're effectively creating your own mirror, so maybe you should have a look at this too: http://wiki.archlinux.org/index.php/Local_Mirror

Arch Linux

#1 2008-01-17 20:12:26

How to create a repository(not just custom) using your hard drive

#2 2008-01-17 22:23:07

Re: How to create a repository(not just custom) using your hard drive

#3 2008-01-17 23:50:02

Re: How to create a repository(not just custom) using your hard drive

#4 2008-01-18 00:17:15

Re: How to create a repository(not just custom) using your hard drive

#5 2008-01-18 04:24:15

Re: How to create a repository(not just custom) using your hard drive

#6 2008-01-18 08:20:24

Re: How to create a repository(not just custom) using your hard drive

#7 2008-01-18 09:21:48

Re: How to create a repository(not just custom) using your hard drive

#8 2008-01-18 09:57:39

Re: How to create a repository(not just custom) using your hard drive

#9 2008-01-18 09:57:51

Re: How to create a repository(not just custom) using your hard drive

#10 2008-01-18 19:13:56

Re: How to create a repository(not just custom) using your hard drive

#11 2008-01-18 19:19:35

Re: How to create a repository(not just custom) using your hard drive

#12 2008-01-18 19:23:57

Re: How to create a repository(not just custom) using your hard drive

#13 2008-01-19 09:50:41

Re: How to create a repository(not just custom) using your hard drive

#14 2008-01-19 10:49:34

Re: How to create a repository(not just custom) using your hard drive

Board footer