You are not logged in.

#26 2005-03-31 08:36:59

rasat
Forum Fellow
From: Finland, working in Romania
Registered: 2002-12-27
Posts: 2,293
Website

Re: pkg to db

kpiche wrote:

# sed -n "/%DESC%/,$ { /%DESC%/ d; /^$/ Q; p }" /var/lib/pacman/local/mjpegtools-1.6.2-1/desc

Works fine.
To apply is getting interesting but I need a small help. How to add the name of package before description (in DB, not desc file). Below codes display desc only. The /tmp/pkg.tmp is list of all packages in DB.

sqlite3 /etc/pacmenu/pacmenu.db "SELECT name FROM pkgs" > /tmp/pkg.tmp

cat /tmp/pkg.tmp | while read LINE; do
sed -n "/%DESC%/,$ { /%DESC%/ d; /^$/ Q; p }" /var/lib/pacman/*/$LINE*/desc
done

Result will be inserted in table "desc":
name (primary key same as in table "pkgs")
description

EDIT:
Where do I find the packages' group name?


Markku

Offline

#27 2005-03-31 09:23:53

cmp
Member
Registered: 2005-01-03
Posts: 350

Re: pkg to db

some packages have a %GROUPS% field in their desc file, have a look at the xfce4 tools for example
in /var/lib/pacman/extra/xfce4-mixer-4.2.1-1/desc theres a field

%GROUPS%
xfce4

Offline

#28 2005-03-31 13:07:52

kpiche
Forum Fellow
From: Ottawa, ON, Canada
Registered: 2004-03-30
Posts: 246
Website

Re: pkg to db

rasat wrote:
sqlite3 /etc/pacmenu/pacmenu.db "SELECT name FROM pkgs" > /tmp/pkg.tmp

cat /tmp/pkg.tmp | while read LINE; do
sed -n "/%DESC%/,$ { /%DESC%/ d; /^$/ Q; p }" /var/lib/pacman/*/$LINE*/desc
done

Do you mean something like:

cat /tmp/pkg.tmp | while read LINE; do
    desc=`sed -n "/%DESC%/,$ { /%DESC%/ d; /^$/ Q; p }" /var/lib/pacman/*/$LINE*/desc`
    sqlite3 "INSERT $LINE, $desc INTO some table"
done

Whatever the sqlite insert syntax is.  You might be better off writing a conversion script in Python or Perl.
Can a package be in multiple groups?

Offline

#29 2005-03-31 20:30:00

rasat
Forum Fellow
From: Finland, working in Romania
Registered: 2002-12-27
Posts: 2,293
Website

Re: pkg to db

Thanks for the hint. Got it to work.

sqlite3 /etc/pacmenu/pacmenu.db "SELECT name FROM pkgs" > /tmp/pkg.tmp

cat /tmp/pkg.tmp | while read LINE; do
desc=`sed -n "/%DESC%/,$ { /%DESC%/ d; /^$/ Q; p }" /var/lib/pacman/*/$LINE*/desc`
sqlite3 /etc/pacmenu/pacmenu.db "INSERT INTO desc (name,description) VALUES ('$LINE', '$desc');"
done

Markku

Offline

#30 2005-04-01 10:05:22

rasat
Forum Fellow
From: Finland, working in Romania
Registered: 2002-12-27
Posts: 2,293
Website

Re: pkg to db

I want to simplify the description search. Grep can search an assigned number of lines above and under matched pattern.

# grep -1 "%DESC%" /var/lib/pacman/current/abiword-2.2.4-1/desc
(shows one line above and under)

How to get one line under only without showing the matched pattern (%DESC%)?


Markku

Offline

#31 2005-04-01 10:27:19

cactus
Taco Eater
From: t͈̫̹ͨa͖͕͎̱͈ͨ͆ć̥̖̝o̫̫̼s͈̭̱̞͍̃!̰
Registered: 2004-05-25
Posts: 4,622
Website

Re: pkg to db

grep -1 "%DESC%" /var/lib/pacman/current/abiword-2.2.4-1/desc | tail -n 1


"Be conservative in what you send; be liberal in what you accept." -- Postel's Law
"tacos" -- Cactus' Law
"t̥͍͎̪̪͗a̴̻̩͈͚ͨc̠o̩̙͈ͫͅs͙͎̙͊ ͔͇̫̜t͎̳̀a̜̞̗ͩc̗͍͚o̲̯̿s̖̣̤̙͌ ̖̜̈ț̰̫͓ạ̪͖̳c̲͎͕̰̯̃̈o͉ͅs̪ͪ ̜̻̖̜͕" -- -̖͚̫̙̓-̺̠͇ͤ̃ ̜̪̜ͯZ͔̗̭̞ͪA̝͈̙͖̩L͉̠̺͓G̙̞̦͖O̳̗͍

Offline

#32 2005-04-01 10:40:02

rasat
Forum Fellow
From: Finland, working in Romania
Registered: 2002-12-27
Posts: 2,293
Website

Re: pkg to db

Thanks.
I should have posted my question immediately instead of browsing Internet for one hour. smile


Markku

Offline

#33 2005-04-01 16:16:09

phrakture
Arch Overlord
From: behind you
Registered: 2003-10-29
Posts: 7,879
Website

Re: pkg to db

hmmm, the only thing I'm not sure of... what happens if the description has embedded newlines - I don't know if it *can* happen, but I know the pacman source reads until it finds a blank line... so I guess it's possible

Offline

#34 2005-04-01 16:27:31

kpiche
Forum Fellow
From: Ottawa, ON, Canada
Registered: 2004-03-30
Posts: 246
Website

Re: pkg to db

phrakture wrote:

hmmm, the only thing I'm not sure of... what happens if the description has embedded newlines - I don't know if it *can* happen, but I know the pacman source reads until it finds a blank line... so I guess it's possible

That's why I wrote the sed script:  it quits (Q) when it sees a blank line thus it can handle a multiline desc.

Offline

#35 2005-04-01 19:09:12

rasat
Forum Fellow
From: Finland, working in Romania
Registered: 2002-12-27
Posts: 2,293
Website

Re: pkg to db

I am not getting a clean insert of description in DB, so there is an experiment going on with different syntax. The problem seems to be in some of the packages themselves. Moreover, processing the descriptions take too long time (40-45s). I may just make one dump file and will be updated through download.


Markku

Offline

#36 2005-04-02 09:06:13

rasat
Forum Fellow
From: Finland, working in Romania
Registered: 2002-12-27
Posts: 2,293
Website

Re: pkg to db

This is simple and fast:

# Current
ls /var/lib/pacman/current/ >/etc/pacmenu/package.txt
cat /etc/pacmenu/package.txt | sed 's|/| |g' | sed 's|-([^-]*-[^-]*)$| 1|g' >/tmp/desc.tmp

echo -e "${YELLOW}Inserting current descriptions in database....${NORMAL}"

awk '{ print $1 }' /tmp/desc.tmp | while read LINE; do
descp=`sed -n "/%DESC%/,$ { /%DESC%/ d; /^$/ Q; p }" /var/lib/pacman/current/$LINE-*/desc | awk '{print $1, $2, $3, $4, $5, $6}`
sqlite /etc/pacmenu/pacmenu.db "INSERT INTO desc (name,description) VALUES ('$LINE', '$descp');"
done

I reduced the inserted descriptions to 6 columns. Many of the desc are too long when six words will do.

The following errors are not caused by sed synatx but sqlite when inserting the descriptions from /var/lib/pacman/. What may cause or how to hide the error messages?

Inserting current descriptions in database....
SQL error: near "s": syntax error
SQL error: near "d": syntax error
SQL error: near "s": syntax error
SQL error: near "s": syntax error
SQL error: near "s": syntax error
SQL error: near "Itri": syntax error

Package datbase is ready.
Press enter to continue


Markku

Offline

#37 2005-04-02 09:26:43

cactus
Taco Eater
From: t͈̫̹ͨa͖͕͎̱͈ͨ͆ć̥̖̝o̫̫̼s͈̭̱̞͍̃!̰
Registered: 2004-05-25
Posts: 4,622
Website

Re: pkg to db

It appears that some of the package descriptions have apostrophes in them...

(eliott@hermes ~)$ awk '{ print $1 }' desc.txt | while read LINE; do grep "'" /var/lib/pacman/current/$LINE-*/desc; done

Dillon's Cron Daemon
LGPL'd re-implementation of Motif
A front-end for CORBA 2.2 IDL and Netscape's XPIDL
Roaring Penguin's Point-to-Point Protocol over Ethernet client
Tools to access a server's filespace and printers via SMB
The whois client by Marco d'Itri

"Be conservative in what you send; be liberal in what you accept." -- Postel's Law
"tacos" -- Cactus' Law
"t̥͍͎̪̪͗a̴̻̩͈͚ͨc̠o̩̙͈ͫͅs͙͎̙͊ ͔͇̫̜t͎̳̀a̜̞̗ͩc̗͍͚o̲̯̿s̖̣̤̙͌ ̖̜̈ț̰̫͓ạ̪͖̳c̲͎͕̰̯̃̈o͉ͅs̪ͪ ̜̻̖̜͕" -- -̖͚̫̙̓-̺̠͇ͤ̃ ̜̪̜ͯZ͔̗̭̞ͪA̝͈̙͖̩L͉̠̺͓G̙̞̦͖O̳̗͍

Offline

#38 2005-04-02 09:43:40

cactus
Taco Eater
From: t͈̫̹ͨa͖͕͎̱͈ͨ͆ć̥̖̝o̫̫̼s͈̭̱̞͍̃!̰
Registered: 2004-05-25
Posts: 4,622
Website

Re: pkg to db

this should replace the 's with 's if that is what you want to do...

awk '{ print $1 }' desc.txt | while read LINE; do descp=`sed -n "/%DESC%/,$ { /%DESC%/ d; /^$/ Q; s/'/\\'/; p}" /var/lib/pacman/current/$LINE-*/desc`; 
sqlite /etc/pacmenu/pacmenu.db "INSERT INTO desc (name,description) VALUES ('$LINE', '$descp');"
done

ie..

Dillon's Cron Daemon
LGPL'd re-implementation of Motif
A front-end for CORBA 2.2 IDL and Netscape's XPIDL
Roaring Penguin's Point-to-Point Protocol over Ethernet client
Tools to access a server's filespace and printers via SMB
The whois client by Marco d'Itri

"Be conservative in what you send; be liberal in what you accept." -- Postel's Law
"tacos" -- Cactus' Law
"t̥͍͎̪̪͗a̴̻̩͈͚ͨc̠o̩̙͈ͫͅs͙͎̙͊ ͔͇̫̜t͎̳̀a̜̞̗ͩc̗͍͚o̲̯̿s̖̣̤̙͌ ̖̜̈ț̰̫͓ạ̪͖̳c̲͎͕̰̯̃̈o͉ͅs̪ͪ ̜̻̖̜͕" -- -̖͚̫̙̓-̺̠͇ͤ̃ ̜̪̜ͯZ͔̗̭̞ͪA̝͈̙͖̩L͉̠̺͓G̙̞̦͖O̳̗͍

Offline

#39 2005-04-02 10:39:31

rasat
Forum Fellow
From: Finland, working in Romania
Registered: 2002-12-27
Posts: 2,293
Website

Re: pkg to db

Syntax "s/'/\\'/;" didn't work, but works if " ' " removed "s/'//;".

Now current has no error, but extra still has these:

Inserting extra descriptions in database....
SQL error: near "that": syntax error
SQL error: near "documents": syntax error
SQL error: near "'less'');": syntax error
SQL error: near "Game": syntax error
SQL error: near "for": syntax error
SQL error: near "game": syntax error
SQL error: near "an": syntax error
SQL error: near "s": syntax error
SQL error: near "'Vectoroids is based on the X-Window PDA game Agendaroids,'');": syntax error
SQL error: near "files": syntax error
SQL error: near "programs": syntax error

Package datbase is ready.
Press enter to continue

EDIT:
Solved by repeating the syntax ( s/'//; s/'//; s/'//; s/'//; s/'//; ).


Markku

Offline

#40 2005-04-04 15:11:48

phrakture
Arch Overlord
From: behind you
Registered: 2003-10-29
Posts: 7,879
Website

Re: pkg to db

rasat wrote:

EDIT:
Solved by repeating the syntax ( s/'//; s/'//; s/'//; s/'//; s/'//; ).

you can do "s/'//g" to replace all occurances.... that's what you want... though I'd do "s/'/\'/g" so the single quotes remain...

Offline

#41 2005-04-05 09:15:49

rasat
Forum Fellow
From: Finland, working in Romania
Registered: 2002-12-27
Posts: 2,293
Website

Re: pkg to db

Thanks guys for all the help. Pacmenu-2.0 is ready.
http://bbs.archlinux.org/viewtopic.php?t=11433


Markku

Offline

Board footer

Powered by FluxBB