mimeo: open files by mimetype or file name

bashnev · 2016-01-22 02:35:46

Apologies for the poorly expressed bug report. I was being rushed at the time. Thanks for the fix and to Eschwartz for adding clarity.

eschwartz · 2016-01-22 03:25:57

Happy to clarify the issue.

As I noted in the other thread, I've seen the problem manifest before, so I knew what to look for.

Treferwynd · 2016-01-25 13:21:50

Using mimeo 2016.1.23-1, I have the same issue as xduugu, it starts two programs if there's a match in associations.txt:

$ cat ~/.config/mimeo/associations.txt
/usr/bin/firefox %U
  ^https?://example

$ mimeo -c http://example
/usr/bin/firefox http://example
firefox http://example

$ mimeo -c http://notexample
firefox http://notexample

And is terribly slow...

$ time mimeo -c http://example
/usr/bin/firefox http://example
firefox http://example

real	0m2.777s
user	0m0.130s
sys	0m0.020s

EDIT: Actually not quite the same as xduugu, the mimetype detection seems fine

$ mimeo -m http://example
http://example
  x-scheme-handler/http

Last edited by Treferwynd (2016-01-25 13:41:33)

xduugu · 2016-01-25 16:29:13

I found an issue with URLs. For some reason, the path of an URL is first treated as local path. I guess it's because of the latest file://-URI changes.

Everything fine:

$ mimeo -m 'http://127.0.0.1'
http://127.0.0.1
  x-scheme-handler/http

mimeo opens "/" in a file manager, because the local path "/" exists:

$ mimeo -m 'http://127.0.0.1/'
http://127.0.0.1/
  inode/directory

"/root" also exists

$ mimeo -m 'http://127.0.0.1/root'
http://127.0.0.1/root
  inode/directory

"/foo" does not exist and the URL is correctly recognized:

$ mimeo -m 'http://127.0.0.1/foo'
http://127.0.0.1/foo
  x-scheme-handler/http

Treferwynd · 2016-01-25 18:41:28

Interestingly enough, I don't have that issue

$ mimeo -m 'http://127.0.0.1/'
http://127.0.0.1/
  x-scheme-handler/http

Xyne · 2016-01-25 22:45:58

The double-command with custom associations has been fixed.
Only "file" URLs that point to localhost will be treated as files.

@Treferwynd
Is it slow on repeated runs? If so it's probably just the delay from reading the initially uncached (on disk) files. I have added more debugging commands to track file reading and writing. Use the --debug flag to see what's taking so long and I'll see what I can do to speed it up.

It's also strange that the behavior for 'http://127.0.0.1/' was different on your system. I may have made some erroneous assumptions in the code that checks for the localhost in the URL. Please save the following script as "localhost_test" and post the output of

./localhost_test 'http://localhost/' 'http://127.0.0.1/'

localhost_test

#!/usr/bin/env python3

import socket
import sys
import urllib.parse

def test(arg):
  parsed_url = urllib.parse.urlparse(arg)
  localhost = socket.getfqdn(socket.gethostname())
  hostname = parsed_url.hostname if parsed_url.hostname else 'localhost'
  remotehost = socket.getfqdn(hostname)
  print('{}\n  remote host: {}\n  local host:  {}\n  equal:       {}'.format(
    arg,
    remotehost,
    localhost,
    remotehost == localhost
  ))

if __name__ == '__main__':
  try:
    for arg in sys.argv[1:]:
      test(arg)
  except (KeyboardInterrupt, BrokenPipeError):
    pass

If those URLs are not detected as localhost, do you have localhost lines in /etc/hosts?

xduugu · 2016-01-25 23:52:36

Everything fine now. Thanks again.

Treferwynd · 2016-01-26 19:19:55

I've added my hostname to /etc/hosts (never had any use for that) and now the script returns true.
But I don't get why is there a problem with 'http://127.0.0.1' being detected as 'x-scheme-handler/http'

$ ./localhost_test 'http://localhost/' 'http://127.0.0.1/'
http://localhost/
  remote host: localhost.localdomain
  local host:  localhost.localdomain
  equal:       True
http://127.0.0.1/
  remote host: localhost.localdomain
  local host:  localhost.localdomain
  equal:       True

$ mimeo -m 'http://127.0.0.1/'
http://127.0.0.1/
  x-scheme-handler/http

For the speed issue, I'm gonna check tomorrow, right now I'm pretty fried right now, but yes, it is kinda slow every time I invoke it

$ mimeo -c "http://example.wot" --debug
DEBUG: loading arguments from /home/tref/.config/mimeo/default_arguments.txt
DEBUG: opening /home/tref/.config/mimeo/associations.txt
/usr/bin/firefox http://example.wot

$ mimeo -c "http://notexample.wot" --debug
DEBUG: loading arguments from /home/tref/.config/mimeo/default_arguments.txt
DEBUG: opening /home/tref/.config/mimeo/associations.txt
DEBUG: loading /home/tref/.config/mimeapps.list
DEBUG: parsing /usr/share/applications/firefox.desktop
firefox http://notexample.wot

It hangs a bit after the "opening ..." and "parsing ..." lines. Sometimes it's fast, sometimes it's slow. Tomorrow I'm gonna look into the source

Xyne · 2016-01-26 20:19:02

Treferwynd wrote:

But I don't get why is there a problem with 'http://127.0.0.1' being detected as 'x-scheme-handler/http'

The localhost detection and the MIME-type for URLs were independent issues. I realized that Mimeo was failing to recognize 127.0.0.1 as localhost on your system because of your previous post:

Treferwynd wrote:

Interestingly enough, I don't have that issue
$ mimeo -m 'http://127.0.0.1/'
http://127.0.0.1/
  x-scheme-handler/http

Before the update, you should have seen the same output as xduugu. After the last update that issue would only appear for 'file://127.0.0.1/path/to/local/file'. I'll see if I can find a way to reliably detect the localhost regardless of the system hosts file.

Let me know if you figure what's slowing down mimeo.

Treferwynd · 2016-01-27 11:08:00

So I thought 2016.1.25.2 also solved the speed issue, so I wrote a little (ugly) script to measure things, but 2016.1.25.2 is noticeably slower than 2016.1.18.2

Running the same mimeo command 100 times:

$ pacman -Q mimeo
mimeo 2016.1.18.2-1
$ runtime 100 mimeo -c "http://example.wot"
Average:   0.102 s
Max:       0.135 s
Min:       0.074 s
Total:    10.191 s

$ pacman -Q mimeo
mimeo 2016.1.25.2-1
$ runtime 100 mimeo -c "http://example.wot"
Average:   0.184 s
Max:       1.034 s
Min:       0.103 s
Total:    18.442 s

The new version, while also being a tad slower, sometimes just hangs for a second or so.

I'll see if I can find the cause.
But... am I the only one with this issue? Did I do something stupid? :S

Treferwynd · 2016-01-27 16:18:57

Found the culprit, it's the socket.getfqdn() call, sometimes it takes an unreasonable time.
So mimeo is fine, but I have no idea why this happens

E.g.

$ time python -c 'import socket; socket.getfqdn("google")'
real	0m0.954s

Xyne · 2016-01-27 21:11:07

Thanks for tracking down the delay. I would not have been able to reproduce it on my system. I have no idea of how to speed up the call itself within Mimeo and I suspect that it depends on some system configuration. I have, however, restructured the code slightly to avoid that call if the URL is not a "file" URL.

Btw, where can I find "runtime" from your previous post? I wasn't able to find it in any official package with "pkgfile" and I see no obvious candidate in the AUR. It looks quite useful.

Treferwynd · 2016-01-27 23:56:37

No problem! Actually I don't think you can do something, I think it's my system that it's borked, I mean, it works fine 95 times out of 100, I've really no idea what's happeninng
The script is just a super ugly bash function that I wrote just to test the speed issue, I wouldn't recommend it, but here it is anyway: https://gist.github.com/Treferwynd/b32515f9d5ca6c564335

EDIT: I updated mimeo and behold, the issue is gone!

$ pacman -Q mimeo
mimeo 2016.1.27-1
$ runtime -q 100 mimeo -c "http://example.wot"
Average:      0.104 s
Max:          0.142 s
Min:          0.076 s
Total:       10.367 s

Thanks for the quick fixes, and well, you know, mimeo in general!

Last edited by Treferwynd (2016-01-28 00:08:35)

Xyne · 2016-01-28 19:43:33

Thanks for posting the script. I started playing around with the idea in Python after discovering the "timeit" module and ended up with a little script that I have named timestats.

xduugu · 2016-02-10 16:53:58

I found a bug

Apparently, the associations are only considered for the first file argument.

associations.txt:

pymol %F
  \.pdb$

$ mimeo -c a.pdb 
pymol a.pdb
$ mimeo -c b.pdb 
pymol b.pdb
$ mimeo -c a.pdb b.pdb 
WARNING: failed to determine command precursor for b.pdb
pymol a.pdb
WARNING: failed to determine command(s) for b.pdb

By the way, the behavior I try to archive is:

$ mimeo -c a.pdb b.pdb 
pymol a.pdb b.pdb

But I suppose that mimeo cannot combine arguments by the command when using the %F place holder.

Xyne · 2016-02-10 20:30:35

It handles %F, %U and the other placeholders. The problem was a broken loop condition which has been fixed. It behaves as expected now.

xduugu · 2016-02-10 20:39:18

Works. Thanks again.

Xyne · 2016-02-10 20:41:27

xduugu wrote:

Works. Thanks again.

Whoa , that was a fast upgrade.

rtfupi · 2016-04-11 20:32:24

Xyne wrote:

NB! To use files in the now-deprecated $XDG_DATA_HOME/applications directory (~/.local/share/applications by default), you must use the --deprecated flag. See the changelog for further information.

In which directory mimeo will look for user defined desktop files?

ewaller · 2016-04-11 20:35:43

rtfupi, Welcome to Arch Linux.

In case you are wondering where your post went, you hit the 'report' link, not the 'reply' link.

Xyne · 2016-04-14 23:31:03

rtfupi wrote:

Xyne wrote:
NB! To use files in the now-deprecated $XDG_DATA_HOME/applications directory (~/.local/share/applications by default), you must use the --deprecated flag. See the changelog for further information.
In which directory mimeo will look for user defined desktop files?

$ mimeo --help
...
  --create <filename> <Name> <Exec> <MIME-type matcher or ""> <"term" or "">
                        Create a minimal desktop file. Edit the created file
                        if necessary. If an empty string is passed instead of
                        a MIME-type matcher, the file will not specify any
                        MIME-type associations. The fifth argument indicates
                        if "Terminal" should be set to "true" in the created
                        file. See "--mimeman-help" for more information. The
                        created files are saved in $XDG_DATA_HOME/applications
                        (~/.local/share/applications by default). This
                        directory is only checked when the --deprecated flag
                        is used.

As the help message indicates, desktop files created directly with mimeo are saved in the deprecated applications directory, for legacy reasons and because the current XDG standard does not support the files in any user-controlled directory. If you add your own files elsewhere, you must add them to one of the locations specified by the standard. Just check where desktop files are stored on your computer or read the latest XDG specification.

rtfupi · 2016-04-15 10:01:09

Xyne wrote:

As the help message indicates, desktop files created directly with mimeo are saved in the deprecated applications directory, for legacy reasons and because the current XDG standard does not support the files in any user-controlled directory. If you add your own files elsewhere, you must add them to one of the locations specified by the standard. Just check where desktop files are stored on your computer or read the latest XDG specification.

Hm... I understood from the standard that files ($XDG_DATA_HOME/applications/$desktop-mimeapps.list and $XDG_DATA_HOME/applications/mimeapps.list) are deprecated but not the directory ($XDG_DATA_HOME/applications/).

Last edited by rtfupi (2016-04-15 10:05:32)

Xyne · 2016-04-16 22:52:42

rtfupi wrote:

Hm... I understood from the standard that files ($XDG_DATA_HOME/applications/$desktop-mimeapps.list and $XDG_DATA_HOME/applications/mimeapps.list) are deprecated but not the directory ($XDG_DATA_HOME/applications/).

In the "Adding/removing associations" section of the specification, the author wrote:

(in the context of reading mimeapps.list files)
The adding and removal of associations only applies to desktop files in the current directory, or a later one (in precedence order).
...
The suggested algorithm for listing (in preference order) the applications associated to a given mimetype is:
create an empty list for the results, and a temporary empty "blacklist"
visit each "mimeapps.list" file, in turn; a missing file is equivalent to an empty file
add to the results list any "Added Associations" in the mimeapps.list, excluding items on the blacklist
add to the blacklist any "Removed Associations" in mimeapps.list
add to the results list any .desktop file found in the same directory as the mimeapps.list which lists the given type in its MimeType= line, excluding any desktop files already in the blacklist. For directories based on XDG_CONFIG_HOME and XDG_CONFIG_DIRS, there are (by definition) no desktop files in the same directory.
add to the blacklist the names of any desktop files found in the same directory as the mimeapps.list file (which for directories based on XDG_CONFIG_HOME and XDG_CONFIG_DIRS, is none)
repeat the last four steps for each subsequent directory

See the step in bold in particular. The specification determines the directory order for reading mimeapps.list files. Only desktop files in those directories are considered. Desktop files in the user configuration directories are explicitly prohibited as is the mimeapps.list file in the user applications directory, which thus prevents desktop files in that directory from being considered.

Basically, the current specification has deprecated user desktop files. I have no idea why they would remove such a useful functionality, but they did.

rtfupi · 2016-04-17 19:05:50

Xyne wrote:

See the step in bold in particular. The specification determines the directory order for reading mimeapps.list files. Only desktop files in those directories are considered. Desktop files in the user configuration directories are explicitly prohibited as is the mimeapps.list file in the user applications directory, which thus prevents desktop files in that directory from being considered.
Basically, the current specification has deprecated user desktop files. I have no idea why they would remove such a useful functionality, but they did.

Thank you for the clarification. I'll try to ask about this in freedesktop@lists.freedesktop.org

Xyne · 2016-04-18 00:54:32

rtfupi wrote:

Thank you for the clarification. I'll try to ask about this in freedesktop@lists.freedesktop.org

If it turns out that I have misunderstood the specification or if they decide to change it, let me know and I will update mimeo.

Arch Linux

#126 2016-01-22 02:35:46

Re: mimeo: open files by mimetype or file name

#127 2016-01-22 03:25:57

Re: mimeo: open files by mimetype or file name

#128 2016-01-25 13:21:50

Re: mimeo: open files by mimetype or file name

#129 2016-01-25 16:29:13

Re: mimeo: open files by mimetype or file name

#130 2016-01-25 18:41:28

Re: mimeo: open files by mimetype or file name

#131 2016-01-25 22:45:58

Re: mimeo: open files by mimetype or file name

#132 2016-01-25 23:52:36

Re: mimeo: open files by mimetype or file name

#133 2016-01-26 19:19:55

Re: mimeo: open files by mimetype or file name

#134 2016-01-26 20:19:02

Re: mimeo: open files by mimetype or file name

#135 2016-01-27 11:08:00

Re: mimeo: open files by mimetype or file name

#136 2016-01-27 16:18:57

Re: mimeo: open files by mimetype or file name

#137 2016-01-27 21:11:07

Re: mimeo: open files by mimetype or file name

#138 2016-01-27 23:56:37

Re: mimeo: open files by mimetype or file name

#139 2016-01-28 19:43:33

Re: mimeo: open files by mimetype or file name

#140 2016-02-10 16:53:58

Re: mimeo: open files by mimetype or file name

#141 2016-02-10 20:30:35

Re: mimeo: open files by mimetype or file name

#142 2016-02-10 20:39:18

Re: mimeo: open files by mimetype or file name

#143 2016-02-10 20:41:27

Re: mimeo: open files by mimetype or file name

#144 2016-04-11 20:32:24

Re: mimeo: open files by mimetype or file name

#145 2016-04-11 20:35:43

Re: mimeo: open files by mimetype or file name

#146 2016-04-14 23:31:03

Re: mimeo: open files by mimetype or file name

#147 2016-04-15 10:01:09

Re: mimeo: open files by mimetype or file name

#148 2016-04-16 22:52:42

Re: mimeo: open files by mimetype or file name

#149 2016-04-17 19:05:50

Re: mimeo: open files by mimetype or file name

#150 2016-04-18 00:54:32

Re: mimeo: open files by mimetype or file name

Board footer