Arch Linux

amish

Member

Registered: 2014-05-10

Posts: 470

[solved][hack] make pacman use repo.db.tar.gz file instead of repo.db?

I have a personal repo

/etc/pacman.conf

[repo]
Server =  https://foo.com/repo
SigLevel = Optional TrustAll

when i run pacman -Syu, it looks for https://foo.com/repo/repo.db

This has been working fine from about 6-8 months.

Now for some reason.. suddenly my web hosting provider started giving 403 for .db file. (any file with extension .db gets blocked)

error: failed retrieving file 'repo.db' from foo.com : The requested URL returned error: 403

I am following up with them but either they will say NO or its going to take them time to resolve.

So my question is - how to make pacman to look for file https://foo.com/repo/repo.db.tar.gz instead of https://foo.com/repo/repo.db

Thank you

Last edited by amish (2018-06-08 16:47:50)

eschwartz

Fellow

Registered: 2014-08-08

Posts: 4,097

Re: [solved][hack] make pacman use repo.db.tar.gz file instead of repo.db?

This is defined here: https://git.archlinux.org/pacman.git/tr … v5.1.0#n74

It's a constant, not a configurable option. The compression type is variable, and the extra copy with the tar.* extension only exists at all due to the way repo-add is designed.

---

Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

progandy

Member

Registered: 2012-05-17

Posts: 5,192

Re: [solved][hack] make pacman use repo.db.tar.gz file instead of repo.db?

You could redirect all your queries through a php script, that does 301 redirects to the real files.
Or maybe you can do it with an .htaccess and mod_rewrite, something like this

RewriteEngine On
RewriteCond %{QUERY_STRING} \.db$
RewriteRule ^repo$ %{QUERY_STRING}.tar.xz [QSD,PT,L]

RewriteRule ^repo$ %{QUERY_STRING} [QSD,PT,L]

The url should then be something like this:

repo.example.com/myrepo/repo?myrepo.db
repo.example.com/myrepo/repo?some.pkg.tar.xz

real paths:
repo.example.com/myrepo/myrepo.db.tar.xz
repo.example.com/myrepo/some.pkg.tar.xz

Last edited by progandy (2018-06-08 13:19:34)

---

| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

amish

Member

Registered: 2014-05-10

Posts: 470

Re: [solved][hack] make pacman use repo.db.tar.gz file instead of repo.db?

This is defined here: https://git.archlinux.org/pacman.git/tr … v5.1.0#n74

It's a constant, not a configurable option. The compression type is variable, and the extra copy with the tar.* extension only exists at all due to the way repo-add is designed.

Yes I had gone through code. Actually when using pacman -F .. it switches to .files extension using:

alpm_option_set_dbext(handle, ".files");

https://git.archlinux.org/pacman.git/tr … 5.1.0#n714

So there is internal support to change the extension. So we can probably have an option in pacman.conf like these:

[repo]
Server = https://foo.com/repo
SigLevel = Optional TrustAll
DbExt = .db.tar.gz

But it may also require some changes to "src/pacman/callback.c" and "src/pacman/sync.c" which has string constants ".db"

apg

Developer

Registered: 2012-11-10

Posts: 211

Re: [solved][hack] make pacman use repo.db.tar.gz file instead of repo.db?

pacsync --dbext=.db.tar.gz

eschwartz

Fellow

Registered: 2014-08-08

Posts: 4,097

Re: [solved][hack] make pacman use repo.db.tar.gz file instead of repo.db?

It's a constant, not a configurable option. The compression type is variable, and the extra copy with the tar.* extension only exists at all due to the way repo-add is designed.

Yes I had gone through code. Actually when using pacman -F .. it switches to .files extension using:

alpm_option_set_dbext(handle, ".files");

https://git.archlinux.org/pacman.git/tr … 5.1.0#n714

So there is internal support to change the extension. So we can probably have an option in pacman.conf like these:

[repo]
Server = https://foo.com/repo
SigLevel = Optional TrustAll
DbExt = .db.tar.gz

But it may also require some changes to "src/pacman/callback.c" and "src/pacman/sync.c" which has string constants ".db"

Yes, I'm aware that the libalpm functions take parameters, which are then used internally to support files databases with the same code.

I'll stand by what I said... this is hardcoded in pacman.

If your question is "can it be done" -- No, it cannot.
If your question is "can newer versions of pacman receive new added functionality", that's not a solution for today, and it also might not get accepted.

Of course, since libalpm supports it, you can write your own package manager frontend... or use apg's.

---

Managing AUR repos The Right Way -- aurpublish (now a standalone tool)

amish

Member

Registered: 2014-05-10

Posts: 470

Re: [solved][hack] make pacman use repo.db.tar.gz file instead of repo.db?

You could redirect all your queries through a php script, that does 301 redirects to the real files.
Or maybe you can do it with an .htaccess and mod_rewrite

Unfortunately It didn't work. Any URL ending with .db gets blocked at global level even before .htaccess.

I have no idea why web hosting provider is doing that. They upgraded from EasyApache 3 to EasyApache4 and then all the trouble started.

Keeping fingers crossed.

progandy

Member

Registered: 2012-05-17

Posts: 5,192

Re: [solved][hack] make pacman use repo.db.tar.gz file instead of repo.db?

You could redirect all your queries through a php script, that does 301 redirects to the real files.
Or maybe you can do it with an .htaccess and mod_rewrite

Unfortunately It didn't work. Any URL ending with .db gets blocked at global level even before .htaccess.

My suggestion is to put the package and database in the query string after "?" instead of the filename. The example htacces I added does that and should work if .db files are blocked. At least my test worked, I blocked db files like this and the rewrite rules allowed access.

<Files *.db>
Order Deny,Allow
Deny from All
</Files>

Last edited by progandy (2018-06-08 16:10:05)

---

| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

amish

Member

Registered: 2014-05-10

Posts: 470

Re: [solved][hack] make pacman use repo.db.tar.gz file instead of repo.db?

I could not understand what you mean by - "Put the package and database in the query string?"

It is pacman which picks up repourl and appends repo.db to it.

Do  you mean I put
Server = https://foo.com/repo?

with question mark at the end?

Ok let me try it. I cudnt understand your solution earlier.

Last edited by amish (2018-06-08 16:30:11)

amish

Member

Registered: 2014-05-10

Posts: 470

Re: [solved][hack] make pacman use repo.db.tar.gz file instead of repo.db?

Or maybe you can do it with an .htaccess and mod_rewrite, something like this

RewriteEngine On
RewriteCond %{QUERY_STRING} \.db$
RewriteRule ^repo$ %{QUERY_STRING}.tar.xz [QSD,PT,L]

RewriteRule ^repo$ %{QUERY_STRING} [QSD,PT,L]

Ok this work around worked perfectly.

Earlier I missed it that I need to add question mark at the end of "Server="

Thanks a lot @progandy

amish

Member

Registered: 2014-05-10

Posts: 470

Re: [solved][hack] make pacman use repo.db.tar.gz file instead of repo.db?

I made RewriteRules bit stricter like this:

First - /etc/pacman.conf

Note: question mark at the end of Server line

[repo]
Server =  https://foo.com/repo?
SigLevel = Optional TrustAll

Now .htaccess

RewriteEngine On
RewriteCond %{QUERY_STRING} ^/repo\.db$
RewriteRule ^repo$ repo.db.tar.gz [QSD,PT,L]

RewriteCond %{QUERY_STRING} ^/([-:\.\w]+)
RewriteRule ^repo$ %1 [QSD,PT,L]

First condition does not check for all .db but just repo.db because the directory is not supposed to contain any other DB file.

Second condition takes care of random paths in query string (especially / and \). And only alphanumeric (plus -, : and . allowed in query string)

Rules also remove leading / from query string

Last edited by amish (2018-06-08 17:51:38)

Lone_Wolf

Member

From: Netherlands, Europe

Registered: 2005-10-04

Posts: 11,919

Re: [solved][hack] make pacman use repo.db.tar.gz file instead of repo.db?

pacsync --dbext=.db.tar.gz

Somehow everyone overlooked that ?
see https://github.com/andrewgregory/pacutils for details

---

Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.

(A works at time B)  && (time C > time B ) ≠  (A works at time C)

amish

Member

Registered: 2014-05-10

Posts: 470

Re: [solved][hack] make pacman use repo.db.tar.gz file instead of repo.db?

I wanted a solution where I dont have to remember extra command and dont have to install additional package.

amish

Member

Registered: 2014-05-10

Posts: 470

Re: [solved][hack] make pacman use repo.db.tar.gz file instead of repo.db?

After following up with my web hosting provider - they refused to disable the (newly introduced) rule in name of security.

The web hosting provider has started using something called "modsecurity" which comes with cPanel (an extremely popular interface)

If more cPanel web hosting provider start using the below mentioned modsecurity rule - then many custom Arch repositories may start having problem due to ".db" extension getting blocked.

Just for the record, in case someone else also happens to have same issue.

Reference:
cPanel rule page: OWASP ModSecurity CRS
Upstream Source - OWASP ModSecurity Core Rule Set (CRS) Project (Official Repository)
Upstream Website - The OWASP ModSecurity Core Rule Set (CRS)

The rule that matched is in REQUEST-949-BLOCKING-EVALUATION.conf which calls REQUEST-920-PROTOCOL-ENFORCEMENT.conf which checks for tx.restricted_extensions which is defined in REQUEST-901-INITIALIZATION.conf.

The rule blocks following extensions: (as per log)

ModSecurity: Warning. String match within ".asa/ .asax/ .ascx/ .axd/ .backup/ .bak/ .bat/ .cdx/ .cer/ .cfg/ .cmd/ .com/ .config/ .conf/ .cs/ .csproj/ .csr/ .dat/ .db/ .dbf/ .dll/ .dos/ .htr/ .htw/ .ida/ .idc/ .idq/ .inc/ .ini/ .key/ .licx/ .lnk/ .log/ .mdb/ .old/ .pass/ .pdb/ .pol/ .printer/ .pwd/ .resources/ .resx/ .sql/ .sys/ .vb/ .vbs/ .vbproj/ .vsdisco/ .webinfo/ .xsd/ .xsx/" at TX:extension. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-920-PROTOCOL-ENFORCEMENT.conf"] [line "452"] [id "920440"] [rev "2"] [msg "URL file extension is restricted by policy"] [data ".db"] [severity "CRITICAL"] [ver "OWASP_CRS/3.0.0"] [maturity "9"] [accuracy "9"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-protocol"] [tag "OWASP_CRS/POLICY/EXT_RESTRICTED"] [tag "WASCTC/WASC-15"] [tag "OWASP_TOP_10/A7"] [tag "PCI/6.5.10"]

Additional log generated: (so Google search result can lead here)

ModSecurity: Access denied with code 403 (phase 2). Operator GE matched 5 at TX:anomaly_score. [file "/etc/apache2/conf.d/modsec_vendor_configs/OWASP3/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "30"] [id "949110"] [msg "Inbound Anomaly Score Exceeded (Total Score: 5)"] [severity "CRITICAL"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"]

Looks like time to change web hosting provider soon.

Last edited by amish (2018-06-12 06:54:07)

Alad

Wiki Admin/IRC Op

From: Bagelstan

Registered: 2014-05-04

Posts: 2,412

Website

Re: [solved][hack] make pacman use repo.db.tar.gz file instead of repo.db?

Or just get a VPS for a few euros per month so you can do whatever you like?

Last edited by Alad (2018-06-12 10:31:36)

---

Mods are just community members who have the occasionally necessary option to move threads around and edit posts. -- Trilby

Trilby

Inspector Parrot

Registered: 2011-11-29

Posts: 29,525

Website

Re: [solved][hack] make pacman use repo.db.tar.gz file instead of repo.db?

Looks like time to change web hosting provider soon.

You may want a VPS.  I'm honestly amazed that shared hosting still exists as you can get a VPS for the same price (or even cheaper) from linode or OVH (and possibly others).

You can get linode for $5/mo, or if you really must keep a minimal shared-hosting budget, OVH's entry level VPS is $3.35/mo (which is cheaper than most entry-level shared hosting services).

Last edited by Trilby (2018-06-12 10:35:40)

---

"UNIX is simple and coherent..." - Dennis Ritchie, "GNU's Not UNIX" -  Richard Stallman

progandy

Member

Registered: 2012-05-17

Posts: 5,192

Re: [solved][hack] make pacman use repo.db.tar.gz file instead of repo.db?

Looks like time to change web hosting provider soon.

You may want a VPS.  I'm honestly amazed that shared hosting still exists as you can get a VPS for the same price (or even cheaper) from linode or OVH (and possibly others).

You don't have to worry about maintaining a secure linux installation. It can be an advantage if you are only responsible for the php and other cgi scripts.

---

| alias CUTF='LANG=en_XX.UTF-8@POSIX ' |

amish

Member

Registered: 2014-05-10

Posts: 470

Re: [solved][hack] make pacman use repo.db.tar.gz file instead of repo.db?

Exactly. Shared hosting has its advantage if all you have is few MB of data. For 2.5$/mo I get 20GB space - 100 GB transfer and 5 email accounts.

Security / DoS etc becomes web hosting providers headache. Plus you get variety of popular packages which can be installed in single click.

Ofcourse on long run VPS wins.

Anyway workaround by progandy has been working fine for me from few days. So its not so urgent anymore.