Arch Linux

lseubert

Member

From: Maryland, USA

Registered: 2009-05-18

Posts: 141

In search of bookmark file cleaning software...

Good day all,

I have a bloated bookmark file (in Opera format, but I can convert to Firefox) in desperate need of cleaning. It has a lot of deadlinks to 404 sites, and a lot of duplicate bookmarks.

I need some software to go through and clean out the deadlinks and duplicates, before I manually deal with the rest of the bookmarks.

Any suggesion for good software to do this? A variety of google searches turned up plenty of programs, but they run on Windows. Thanks.

---

"To the question whether I am a pessimist or an optimist, I answer that my knowledge is pessimistic, but my willing and hoping are optimistic."
    -- Albert Schweitzer

deej

Member

Registered: 2008-02-08

Posts: 395

Re: In search of bookmark file cleaning software...

You'd probably spend longer checking that the software wasn't deleting 'good' sites than
doing it manually ! I've just cleaned my Opera Bookmarks out, manually - OK, only about
500 in total, but it didn't take long: duplicates first [ Opera lists them alphabetically anyway, so
no great hardship ], then attack the rest. It's also a nice refresher as to what sites one has
bookmarked but not visited for years...

Deej

lseubert

Member

From: Maryland, USA

Registered: 2009-05-18

Posts: 141

Re: In search of bookmark file cleaning software...

Well, I took a look at the alphabetic sort, as suggested. Yeah, it will help me find duplicates.... manually.... slowly. Very slowly.

Please understand that I have around 4,000 links bookmarked. This is why I need some good bookmark manager software to automatically remove dead links and duplicate links, of which I am sure there are many.

Only once those are removed, do I want to wade into the muck of cleaning up and reorganizing the rest of my bookmark file by hand.

Any other suggestions folks? There are lots of Windows based utilities that do this, but I continue to search for equivalent Linux software in vain

---

"To the question whether I am a pessimist or an optimist, I answer that my knowledge is pessimistic, but my willing and hoping are optimistic."
    -- Albert Schweitzer

deej

Member

Registered: 2008-02-08

Posts: 395

Re: In search of bookmark file cleaning software...

4,000 ? Jeez...

Well, here's the kernel of an idea:

1> Export bookmarks to a text file / html file

2> Run a 'uniq' script on the file to remove duplicates: [ Search Term: 'linux remove duplicates in html' brings up the goods ]
     You may need to run 'html2txt' on the file if it's in html format, before you run a script from the above search.

3> Run 'txt2html' to regain your html bookmarks. The next stage is to test the links... more on that when I get the time - or
     you could google for 'linux test links' or some such thing.

Sorry I couldn't be more precise, I've a lot on at the moment. I'll keep an eye on the thread for any problems you may have.

Deej

lseubert

Member

From: Maryland, USA

Registered: 2009-05-18

Posts: 141

Re: In search of bookmark file cleaning software...

Thanks for the tip deej.

Running sort, uniq, and wc on the exported opera6.html bookmark file produces some good news, and some bad news. The good news is that I only have 209 duplicates. The bad news is that those 209 duplicates are just a shade over 3.5% of my total bookmark file, which has some 6,200 entries. Egad! Why did I procrastinate so much over cleaning up my bookmark file all these years? No wonder Firefox was so pig slow.

A few problems. Opera exports the bookmarks into an html file, but not in the order I have set up. It sorts alphabetically and then exports. I'll have to see if this default behavior can be changed. And I am going to thoroughly search for a native Opera tool that can clean up bookmarks. No luck so far.

Also, while I found and installed the html2text package, there is no txt2html or text2html package. There are various scripts though, so I can always use those. pacpan HTML::FromText takes care of this.

Thanks, and I'll keep updating this thread, hopefully marking it with a [SOLVED] someday.

---

"To the question whether I am a pessimist or an optimist, I answer that my knowledge is pessimistic, but my willing and hoping are optimistic."
    -- Albert Schweitzer

Procyon

Member

Registered: 2008-05-07

Posts: 1,819

Re: In search of bookmark file cleaning software...

Can you post an example?

deej

Member

Registered: 2008-02-08

Posts: 395

Re: In search of bookmark file cleaning software...

Looks promising:

http://widgets.opera.com/widget/6667/

...unfortunately you'll need to get v.3 from the author !

The search goes on...

Deej

[EDIT]

And another:

http://widgets.opera.com/search/?order=name&q=mass

Neither work for me - just grey windows - but you can try them for
youself.

Last edited by deej (2009-06-10 13:18:34)

lseubert

Member

From: Maryland, USA

Registered: 2009-05-18

Posts: 141

Re: In search of bookmark file cleaning software...

Can you post an example?

What would you like an example of? I'm not sure what you are referring to from your brief post.

---

"To the question whether I am a pessimist or an optimist, I answer that my knowledge is pessimistic, but my willing and hoping are optimistic."
    -- Albert Schweitzer

lseubert

Member

From: Maryland, USA

Registered: 2009-05-18

Posts: 141

Re: In search of bookmark file cleaning software...

Looks promising:

http://widgets.opera.com/widget/6667/

http://widgets.opera.com/search/?order=name&q=mass

I checked them both out deej. The first is really set up to hunt for bad links on a website, while the second helps with W3C HTML validations. Neither one is designed to remove duplicate bookmarks, and clean out dead links. Thanks for looking into this though.

I contacted Opera tech support, and they didn't have too much Linux specific advice, except to cobble together some piped bash commands as you had suggested.

They did suggest a freeware Windows program called AM-Deadlink. I installed Wine and then AM-Deadlink. While it runs slowly on an old PC with huge bookmark files, it does indeed run. So for now, I'll use it to fix things up.

I guess this problem is sort of solved, and I'll mark this thread solved by and by, but I would appreciate a free software Linux solution to this problem. Maybe this will inspire me to finally getting around to learning Python and hacking up a program to do this sort of job.

---

"To the question whether I am a pessimist or an optimist, I answer that my knowledge is pessimistic, but my willing and hoping are optimistic."
    -- Albert Schweitzer

Procyon

Member

Registered: 2008-05-07

Posts: 1,819

Re: In search of bookmark file cleaning software...

I mean what structure/format does the exported html file have. If all the data of a bookmark can be defined on 1 line, (which would be pretty hard with folders and such), then you cannot only sort it, but with a simple script you can also run wget on every URL to see if it doesn't 404.

sveri80

Member

Registered: 2008-11-12

Posts: 38

Re: In search of bookmark file cleaning software...

Well, i wrote a bookmark management software some years ago.
It featured an import from opera and netscape/firefox html.
And it featured a search for duplicate bookmarks and checked for
valid bookmarks.

However, nobody ever made use of that software so i stopped
developing it and i cannot tell anymore what did work and what not.

However, you still can download it there:
http://sveri.net/yajbm/yajbm.zip
and try it yourself.
Its written in Java and only depends on JRE 1.4 IIRC.

So if you want to you can try it out, but dont nail me on the
features, maybe it doesnt work anymore.
For instance the bookmark import from opera was written
for version 9 i believe, maybe they changed something.

deej

Member

Registered: 2008-02-08

Posts: 395

Re: In search of bookmark file cleaning software...

From 'man wget' :

wget --spider --force-html -i bookmarks.html

Haven't tried it myself yet, but it can't hurt to give it a go.

Good ol' Linux

Deej

[EDIT]

Well.  I tried the command and here provide a slightly improved version:

wget --spider --force-html --tries 2 -nv -i bookmarks.html -o bad.txt

This will only log the bad links, to bad.txt, whereas the original logged everything.
I've added 'tries 2' because the default was 20 - a little too many tries for me !

Rough-as-a-butchers-dog, but it works !

Deej

Last edited by deej (2009-06-10 21:35:11)

DevoidOfWindows

Member

Registered: 2009-05-24

Posts: 133

Re: In search of bookmark file cleaning software...

https://addons.mozilla.org/en-US/firefox/addon/10897

lseubert

Member

From: Maryland, USA

Registered: 2009-05-18

Posts: 141

Re: In search of bookmark file cleaning software...

I mean what structure/format does the exported html file have. If all the data of a bookmark can be defined on 1 line, (which would be pretty hard with folders and such), then you cannot only sort it, but with a simple script you can also run wget on every URL to see if it doesn't 404.

Ah, I see what you mean. For the record, the format goes like so:

#URL
    ID=222
    NAME=Arch Linux
    URL=http://www.archlinux.org/
    CREATED=1243025806
    VISITED=1244753802
    DESCRIPTION=
    UNIQUEID=185073D0471311DE9243AFB0C45D1451

And ultimately, you are right - best to write a script. That will be a project for another day though.

---

"To the question whether I am a pessimist or an optimist, I answer that my knowledge is pessimistic, but my willing and hoping are optimistic."
    -- Albert Schweitzer

lseubert

Member

From: Maryland, USA

Registered: 2009-05-18

Posts: 141

Re: In search of bookmark file cleaning software...

Well, i wrote a bookmark management software some years ago.
It featured an import from opera and netscape/firefox html.
And it featured a search for duplicate bookmarks and checked for
valid bookmarks.

Thanks sveri80. I'll tinker around with the script this weekend and see how it goes.

---

"To the question whether I am a pessimist or an optimist, I answer that my knowledge is pessimistic, but my willing and hoping are optimistic."
    -- Albert Schweitzer

lseubert

Member

From: Maryland, USA

Registered: 2009-05-18

Posts: 141

Re: In search of bookmark file cleaning software...

From 'man wget' :

wget --spider --force-html --tries 2 -nv -i bookmarks.html -o bad.txt

This will only log the bad links, to bad.txt, whereas the original logged everything.
I've added 'tries 2' because the default was 20 - a little too many tries for me !

Rough-as-a-butchers-dog, but it works !

Deej

Yup, that it would. I'll convert my file over the html and try it out. I am a bit worried about Opera's seeming flakey export capability though. My imports back into Firefox from Opera-exported 'bookmarks.html' haven't gone well. It might be FireFox choking on a huge bookmark file though.

Right now, it looks like I'll have to cull down this bookmark file from within Opera, and then export it. If I do roughly 100 links a day, I'll be finished in around seven weeks. That should be just in time for FireFox 3.5. However, if FF is still a slow pig (which is why I switched to Opera in the first place), I'll explore the various WebKit based browsers. Opera is a goner because there are too many websites that it can't render properly anymore.

Anyway, thanks for the wget code. Wget never ceases to amaze. I have managed some surprisingily intricate downloads of entire websites with that program.

---

"To the question whether I am a pessimist or an optimist, I answer that my knowledge is pessimistic, but my willing and hoping are optimistic."
    -- Albert Schweitzer

lseubert

Member

From: Maryland, USA

Registered: 2009-05-18

Posts: 141

Re: In search of bookmark file cleaning software...

https://addons.mozilla.org/en-US/firefox/addon/10897

Many thanks for that tip. I downloaded the extension and tried it out. It seems to work fine. Problem is, my Opera-exported bookmark file was screwed up - cut very short. So while it worked OK on a truncated file, I can't get the proper file into Firefox. (For the time being.) So while I have some software now to deal with the bookmark file, various browsers are still choking on file importation and leaving me with a screwed up file.

S'OK - I almost always backup

Kids. Let this be a lesson to you. Don't ever go years without cleaning up your bookmark file. Do it once a month, just like you floss once a day.

Now get off my lawn!

---

"To the question whether I am a pessimist or an optimist, I answer that my knowledge is pessimistic, but my willing and hoping are optimistic."
    -- Albert Schweitzer

rwd

Member

Registered: 2009-02-08

Posts: 671

Re: In search of bookmark file cleaning software...

Broken links are hard to detect in cases where you get a redirect instead of a 404 response. But 4000 links..you might be better off by just starting over and changing your bookmarking policy .