pkg to db

rasat · 2005-03-31 08:36:59

kpiche wrote:

# sed -n "/%DESC%/,$ { /%DESC%/ d; /^$/ Q; p }" /var/lib/pacman/local/mjpegtools-1.6.2-1/desc

Works fine.
To apply is getting interesting but I need a small help. How to add the name of package before description (in DB, not desc file). Below codes display desc only. The /tmp/pkg.tmp is list of all packages in DB.

sqlite3 /etc/pacmenu/pacmenu.db "SELECT name FROM pkgs" > /tmp/pkg.tmp

cat /tmp/pkg.tmp | while read LINE; do
sed -n "/%DESC%/,$ { /%DESC%/ d; /^$/ Q; p }" /var/lib/pacman/*/$LINE*/desc
done

Result will be inserted in table "desc":
name (primary key same as in table "pkgs")
description

EDIT:
Where do I find the packages' group name?

cmp · 2005-03-31 09:23:53

some packages have a %GROUPS% field in their desc file, have a look at the xfce4 tools for example
in /var/lib/pacman/extra/xfce4-mixer-4.2.1-1/desc theres a field

%GROUPS%
xfce4

kpiche · 2005-03-31 13:07:52

rasat wrote:

sqlite3 /etc/pacmenu/pacmenu.db "SELECT name FROM pkgs" > /tmp/pkg.tmp

cat /tmp/pkg.tmp | while read LINE; do
sed -n "/%DESC%/,$ { /%DESC%/ d; /^$/ Q; p }" /var/lib/pacman/*/$LINE*/desc
done

Do you mean something like:

cat /tmp/pkg.tmp | while read LINE; do
    desc=`sed -n "/%DESC%/,$ { /%DESC%/ d; /^$/ Q; p }" /var/lib/pacman/*/$LINE*/desc`
    sqlite3 "INSERT $LINE, $desc INTO some table"
done

Whatever the sqlite insert syntax is. You might be better off writing a conversion script in Python or Perl.
Can a package be in multiple groups?

rasat · 2005-03-31 20:30:00

Thanks for the hint. Got it to work.

sqlite3 /etc/pacmenu/pacmenu.db "SELECT name FROM pkgs" > /tmp/pkg.tmp

cat /tmp/pkg.tmp | while read LINE; do
desc=`sed -n "/%DESC%/,$ { /%DESC%/ d; /^$/ Q; p }" /var/lib/pacman/*/$LINE*/desc`
sqlite3 /etc/pacmenu/pacmenu.db "INSERT INTO desc (name,description) VALUES ('$LINE', '$desc');"
done

rasat · 2005-04-01 10:05:22

I want to simplify the description search. Grep can search an assigned number of lines above and under matched pattern.

# grep -1 "%DESC%" /var/lib/pacman/current/abiword-2.2.4-1/desc
(shows one line above and under)

How to get one line under only without showing the matched pattern (%DESC%)?

cactus · 2005-04-01 10:27:19

grep -1 "%DESC%" /var/lib/pacman/current/abiword-2.2.4-1/desc | tail -n 1

rasat · 2005-04-01 10:40:02

Thanks.
I should have posted my question immediately instead of browsing Internet for one hour.

phrakture · 2005-04-01 16:16:09

hmmm, the only thing I'm not sure of... what happens if the description has embedded newlines - I don't know if it *can* happen, but I know the pacman source reads until it finds a blank line... so I guess it's possible

kpiche · 2005-04-01 16:27:31

phrakture wrote:

hmmm, the only thing I'm not sure of... what happens if the description has embedded newlines - I don't know if it *can* happen, but I know the pacman source reads until it finds a blank line... so I guess it's possible

That's why I wrote the sed script: it quits (Q) when it sees a blank line thus it can handle a multiline desc.

rasat · 2005-04-01 19:09:12

I am not getting a clean insert of description in DB, so there is an experiment going on with different syntax. The problem seems to be in some of the packages themselves. Moreover, processing the descriptions take too long time (40-45s). I may just make one dump file and will be updated through download.

rasat · 2005-04-02 09:06:13

This is simple and fast:

# Current
ls /var/lib/pacman/current/ >/etc/pacmenu/package.txt
cat /etc/pacmenu/package.txt | sed 's|/| |g' | sed 's|-([^-]*-[^-]*)$| 1|g' >/tmp/desc.tmp

echo -e "${YELLOW}Inserting current descriptions in database....${NORMAL}"

awk '{ print $1 }' /tmp/desc.tmp | while read LINE; do
descp=`sed -n "/%DESC%/,$ { /%DESC%/ d; /^$/ Q; p }" /var/lib/pacman/current/$LINE-*/desc | awk '{print $1, $2, $3, $4, $5, $6}`
sqlite /etc/pacmenu/pacmenu.db "INSERT INTO desc (name,description) VALUES ('$LINE', '$descp');"
done

I reduced the inserted descriptions to 6 columns. Many of the desc are too long when six words will do.

The following errors are not caused by sed synatx but sqlite when inserting the descriptions from /var/lib/pacman/. What may cause or how to hide the error messages?

Inserting current descriptions in database....
SQL error: near "s": syntax error
SQL error: near "d": syntax error
SQL error: near "s": syntax error
SQL error: near "s": syntax error
SQL error: near "s": syntax error
SQL error: near "Itri": syntax error
Package datbase is ready.
Press enter to continue

cactus · 2005-04-02 09:26:43

It appears that some of the package descriptions have apostrophes in them...

(eliott@hermes ~)$ awk '{ print $1 }' desc.txt | while read LINE; do grep "'" /var/lib/pacman/current/$LINE-*/desc; done

Dillon's Cron Daemon
LGPL'd re-implementation of Motif
A front-end for CORBA 2.2 IDL and Netscape's XPIDL
Roaring Penguin's Point-to-Point Protocol over Ethernet client
Tools to access a server's filespace and printers via SMB
The whois client by Marco d'Itri

cactus · 2005-04-02 09:43:40

this should replace the 's with 's if that is what you want to do...

awk '{ print $1 }' desc.txt | while read LINE; do descp=`sed -n "/%DESC%/,$ { /%DESC%/ d; /^$/ Q; s/'/\\'/; p}" /var/lib/pacman/current/$LINE-*/desc`; 
sqlite /etc/pacmenu/pacmenu.db "INSERT INTO desc (name,description) VALUES ('$LINE', '$descp');"
done

ie..

Dillon's Cron Daemon
LGPL'd re-implementation of Motif
A front-end for CORBA 2.2 IDL and Netscape's XPIDL
Roaring Penguin's Point-to-Point Protocol over Ethernet client
Tools to access a server's filespace and printers via SMB
The whois client by Marco d'Itri

rasat · 2005-04-02 10:39:31

Syntax "s/'/\\'/;" didn't work, but works if " ' " removed "s/'//;".

Now current has no error, but extra still has these:

Inserting extra descriptions in database....
SQL error: near "that": syntax error
SQL error: near "documents": syntax error
SQL error: near "'less'');": syntax error
SQL error: near "Game": syntax error
SQL error: near "for": syntax error
SQL error: near "game": syntax error
SQL error: near "an": syntax error
SQL error: near "s": syntax error
SQL error: near "'Vectoroids is based on the X-Window PDA game Agendaroids,'');": syntax error
SQL error: near "files": syntax error
SQL error: near "programs": syntax error
Package datbase is ready.
Press enter to continue

EDIT:
Solved by repeating the syntax ( s/'//; s/'//; s/'//; s/'//; s/'//; ).

phrakture · 2005-04-04 15:11:48

rasat wrote:

EDIT:
Solved by repeating the syntax ( s/'//; s/'//; s/'//; s/'//; s/'//; ).

you can do "s/'//g" to replace all occurances.... that's what you want... though I'd do "s/'/\'/g" so the single quotes remain...

rasat · 2005-04-05 09:15:49

Thanks guys for all the help. Pacmenu-2.0 is ready.
http://bbs.archlinux.org/viewtopic.php?t=11433

Arch Linux

#26 2005-03-31 08:36:59

Re: pkg to db

#27 2005-03-31 09:23:53

Re: pkg to db

#28 2005-03-31 13:07:52

Re: pkg to db

#29 2005-03-31 20:30:00

Re: pkg to db

#30 2005-04-01 10:05:22

Re: pkg to db

#31 2005-04-01 10:27:19

Re: pkg to db

#32 2005-04-01 10:40:02

Re: pkg to db

#33 2005-04-01 16:16:09

Re: pkg to db

#34 2005-04-01 16:27:31

Re: pkg to db

#35 2005-04-01 19:09:12

Re: pkg to db

#36 2005-04-02 09:06:13

Re: pkg to db

#37 2005-04-02 09:26:43

Re: pkg to db

#38 2005-04-02 09:43:40

Re: pkg to db

#39 2005-04-02 10:39:31

Re: pkg to db

#40 2005-04-04 15:11:48

Re: pkg to db

#41 2005-04-05 09:15:49

Re: pkg to db

Board footer