filename in non-english are not saved correctly

rezad · 2015-04-18 18:08:07

hi
I recently (last week) installed archlinux.
I come from manjaro so I am to some extent familiar with archlinux.
now I am trying to solve some issues that have poped up in arch that there were not in manjaro( mostly because manjaro was pre-configured).

when I use jdownloader to save youtube videos the files names are displayed correctly in jdownloader but not in konsole or dolphin file manager. (they are shown as "????" characters).
this happens for Persian and Russian characters.

I thought this was jdownloader issue but this doesn't happen on my pc that has a manjaro installed on it.
and another thing that makes it weird is that if i create files in dolphin with say Persian language they are shown and treated correctly. but the ones that are created by jdownloader have this issue.

and another related symptome is that some files that I download from subtitle websites that are in zip format and have some rare European characters in them (like the French and German ones) I cant open them or extract them in dolphin file manager.
I had a similar issue like this in manajro but in manajro I used file-roller to open the files and that app would work with this files correctly.
but in archlinux even file-roller has this issue.

can somebody help me with this?

Knute · 2015-04-18 18:09:43

Did you generate the appropriate locales for the languages that you use?

cedeel · 2015-04-18 18:19:24

Are you using an UTF-8 locale?

rezad · 2015-04-18 20:33:24

cedeel wrote:

Are you using an UTF-8 locale?

I dont know how to answer that cause but I can provide info if you tell me how to do so. I have used wiki for Locale instruction.

[me@mypc ~]$ cat /etc/locale.conf
LANG=en_US.UTF-8
[me@mypc ~]$ cat /etc/locale.gen
en_US.UTF-8 UTF-8

anything else I should provide?

btw at first I had some problem (that I think were update related) that when I executed some command I would get errors about default lang. after some looking around I added a " en_US ISO-8859-1" line to /etc/locale.gen then ran the locale-gen command and then removed that line and ran it again.( It seemed to have missed the first time configuration of "en_US.UTF-8 UTF-8" at first and just would give me this :

[me@mypc ~]$ sudo locale-gen
[sudo] password for me:
Generating locales...
Generation complete.

but after adding that former mentioned line and re-running it and then removing that line and re-running it for the 3rd time i would get

[me@mypc ~]$ sudo locale-gen
[sudo] password for me:
Generating locales...
en_US.UTF-8... done
Generation complete.

so maybe it was missing before.

but even after this I still have this issue.

can you help me with some instruction that resets any bad or incorrect config that may have caused this?
or at least some info about how to fix this?

BTW why do i get this?
"[me@mypc ~]$ history |grep locale
Binary file (standard input) matches"
this didn't happen with manjaro

is my archlinux install broken?
should I re-install?

Last edited by rezad (2015-04-18 21:03:57)

cedeel · 2015-04-18 22:29:08

Do me a favour and paste the output of localectl

rezad · 2015-04-19 05:22:10

[me@mypc ~]$ localectl
System Locale: LANG=en_US.UTF-8
VC Keymap: n/a
X11 Layout: n/a
[me@mypc ~]$

runical · 2015-04-19 08:26:15

Please post output in code tags. Makes it way more readable.

As for your problem: Is your font capable of displaying the languages you named earlier?

rezad · 2015-04-19 09:09:20

do you mean font used in dolphin file manager (kde setting?)
I haven't changed anything from default and in font setting in kde setting it has a list of fonts for different part and all of them are serif or sans-serif. Are you sure this is a font issue?
it happens in kde and konsole and in gnome apps.
if it was font shouldn't it be correct in one of them at least?
and if it was font should the issue happen for files that I create from new file option in dolphin too?

as i said it is not happening for all the files. I can manually create a filename in Persian by changing keyboard layout and it has no issue. But jdownloader has this filename issue on archlinux that doesnt have on manajro .
and the subtitle zip file that i mentioned had this issue on manjaro kde but not when opened with file-roller( a gnome app). but they have this issue in file-roller in archlinux.

I havea sample zip file that when opened the filename inside zip file has this issue.
it shows this way in ark "De.Kus.2004 subt. ingls.srt" and give error that it doesnt exist when trying to open it"Could not open /tmp/kde-me/arkHIZ6CC/De.Kus.2004 subt. ingls.srt"

using unzip from konsole extracts to this "De.Kus.2004 subt. ingls.srt" in konsole display and to this in dolphin"De.Kus.2004 subt. ingl?s.srt" but at list the file is extracted with correct size. but i have to rename it to be able to open it.

what i think is this is invalid encoding issue. because in firefox "open file" window it says "invalid encoding" in front of that extracted srt file.

if you can help me will fixing the issue with that file then i can ignore jdownloader issue as a jdownloader issue and not a archlinux issue.

ty

runical · 2015-04-19 09:17:27

No, I'm not sure it is a font issue and given the new information I do not think it is. What happens if you download the file without jdownloader? Is the encoding still mangeled. Also, what happens if you download other files with these encodings?

And yes, I meant the fonts used in KDE. Some fonts don't have mulit language support, that is why I asked.

Last edited by runical (2015-04-19 09:18:35)

rezad · 2015-04-19 09:30:59

sorry I don't know if you see the zip file example names i provided or not , but it shows the same in my firefox.I don't know why.it was supposed to show them differently.

so how do i fix this encoding issue?

as I said I can create the file from file manager and even copy paste the name from youtube page in firefox onto filename in dolphin. but the issue of the subtitle zipfile is more important to me than the jdownloader one.

I had this issue before on linux but if I am not wrong is showed itself when i mounted usb portable drives that where ntfs formated(or fat32 though i dont think it was fat because it was 2Tb drive) but that was fixed with a config change
( i dont remember what it was but it had to do with mount options) .

it nearly made me delete some movies too cause a software that check for empty and duplicate files thought they where empty files because it couldn't open them . but remounting with fixed setting and they where shown correctly.

EDIT:
I think my issue is this.
https://ubuntuforums.org/showthread.php?t=1236012"
but it is more serious because I cant use gui apps to extract files from zip file so then I dont get the original file so I cant rename them.

Last edited by rezad (2015-04-19 09:35:22)

rezad · 2015-04-19 09:49:03

I think this is file name encoding issue. doesn't linux have a list to go through for situations like this?(like firefox had) it does know that the encoding is invalid , cant it use the correct encoding automatically?

but another thing that I found out was that in playonlinux packages the name of some of its files have this issue too. they are Japanese i think.

ls ~/.PlayOnLinux/configurations/icones/Hanahira*
/home/me/.PlayOnLinux/configurations/icones/Hanahira! (?%81??%81??%81??%82%89?%81???%81)

should the package maintainer use utf-8 encoding for archlinux packages content file names?

Last edited by rezad (2015-04-19 09:51:10)

Lone_Wolf · 2015-04-19 12:09:00

I had a similar issue with wine / japanese filenames, solved it by uncommenting ja_JP.UTF-8 UTF-8 in /etc/locale.gen , then re-running locale-gen .

I'm not sure, but to me it seems 2 things are needed :
- a font that supports the necessary symbols
- enabling of the locale for those symbols

shulamy · 2015-04-19 14:26:48

where is your language in /etc/locale.conf ?

ezik

rezad · 2015-04-19 16:20:11

Lone_Wolf wrote:

I had a similar issue with wine / japanese filenames, solved it by uncommenting ja_JP.UTF-8 UTF-8 in /etc/locale.gen , then re-running locale-gen .

so I can uncomment the needed encodings (more that one) and it covers that encodings? so It doesn't choose one , but covers all of them?

Lone_Wolf · 2015-04-20 11:06:22

rezad,
it was a long time ago i had that problem and locale handling is like a black box to me.

It's easy to reverse if it doesn't help though, why don't you try it ?

rezad · 2015-04-20 14:20:11

Lone_Wolf wrote:

rezad,
it was a long time ago i had that problem and locale handling is like a black box to me.
It's easy to reverse if it doesn't help though, why don't you try it ?

I have tried it.
adding other encoding before the famous en-US,UTF-8 one gives warning for default lang and falling back.
I see these error when running apps from command line.
adding other encodings after that encoding doesn't show this warning but i have a suspicion that it mean that other encoding are suppressed,

i have to try it with a file with a wired but known encoding and then adding that encoding to config and testing again.

my problem is how do i find the filename encoding?(not find content encoding)

this is the file in question. can somebody tell me what is the filename encoding?
https://www.sendspace.com/file/fwmxpc

EDIT: my findings on this matter.

even adding all the langs to /etc/locale.gen and then using locale-gen command doesnt fix this. cause this are different issues.

this problem as a whole doesn't have a solution, cause linux unlike windows doesn't have a default inbuilt filename encoding and it all depends on your environment.

linux just treat the filename in the encoding you tell it. so it cant detect anything cause there is nothing to detect.
from its perspective filename are just binary data ready to be interpreted as wished by the user.

widows on the other hand uses UTF-16.

plz if anybody have another contradicting point ,then reply but plz dont say I have to use correct encoding. the only way to fix it is using convmv util and that requires finding the correct original encoding.

thanks for all the replys.

Last edited by rezad (2015-04-20 14:45:14)

Lone_Wolf · 2015-04-20 20:55:41

I looked into that file and found a workaround :

in Krusader , select the zip file , press enter.
(this opens the zip file using a protocol called krarc )

It will list the file as De.Kus.2004 subt. inglés.srt
Copy that file to a temp location, it will create the filename with the correct encoding.

run file -bi De.Kus.2004 subt. inglés.srt , it will output

text/plain; charset=iso-8859-1

I've then run unzip on the file to get the filename with the question mark in it, then used

convmv -f iso-8859-1 -t UTF8 De.Kus.2004\ subt.\ ingls.srt

This did show the correct filename.

rezad · 2015-04-20 21:34:40

ty for your reply and i will use krusader if it works for other files too but as i said these are workaround not solutions.

and another thing. file -bi doesnt show encoding of filename .it shows the encoding of file content.

if you dont believe me read the file man or even better test it manually by creating a text file with ,say, persian name and then use text editor like kwrite and put an english word in it. using file -bi on it will say us-ASCII even though ascii doesnt cover persian. then open the text file again with kwrite and write a persian word there. then save and use file -bi again. it will now say utf8 .

in my file case the gilename encoding and file content encoding happen to both be iso8859 but it is this file only and these two have nothing to do with each other.

again Ty for your reply and if krusader somehow helps i will try that on other files too .thanks.

Lone_Wolf · 2015-04-20 22:36:18

I've peeked a bit in the Krusader code, the parts dealing with krArc are not very big.

They use the QTextCodec class that seems to be designed to deal with conversions like you want to do.

http://doc.qt.io/qt-4.8/qtextcodec.html#details
http://doc.qt.io/qt-4.8/qtextcodec.html

The krArc code looks rather simple, and could be used as an example to write general code for detecting which codec a string uses and convert it.
How are your programming skills ?

rezad · 2015-04-21 04:14:14

Lone_Wolf wrote:

I've peeked a bit in the Krusader code, the parts dealing with krArc are not very big.
They use the QTextCodec class that seems to be designed to deal with conversions like you want to do.
http://doc.qt.io/qt-4.8/qtextcodec.html#details
http://doc.qt.io/qt-4.8/qtextcodec.html
The krArc code looks rather simple, and could be used as an example to write general code for detecting which codec a string uses and convert it.
How are your programming skills ?

wow. so you are one of THOSE guys.

sorry to report I am just starting to learn JS .

Lone_Wolf · 2015-04-21 11:13:32

I used to be a programmer (about 2 decades ago), but apart from some not-web-related javascript programming, i barely code anymore.

However, i do know how to read/interpret code.

Keep in mind that QT is a toolkit , not a language.
There may be ways to access QT classes & functions from within js.

Added

The qt5-declarative package description (Classes for QML and JavaScript languages ) suggests like it would be worth looking into.

You could try asking for help in arch "programming & scripting" section .

Last edited by Lone_Wolf (2015-04-21 11:23:15)

Arch Linux

#1 2015-04-18 18:08:07

filename in non-english are not saved correctly

#2 2015-04-18 18:09:43

Re: filename in non-english are not saved correctly

#3 2015-04-18 18:19:24

Re: filename in non-english are not saved correctly

#4 2015-04-18 20:33:24

Re: filename in non-english are not saved correctly

#5 2015-04-18 22:29:08

Re: filename in non-english are not saved correctly

#6 2015-04-19 05:22:10

Re: filename in non-english are not saved correctly

#7 2015-04-19 08:26:15

Re: filename in non-english are not saved correctly

#8 2015-04-19 09:09:20

Re: filename in non-english are not saved correctly

#9 2015-04-19 09:17:27

Re: filename in non-english are not saved correctly

#10 2015-04-19 09:30:59

Re: filename in non-english are not saved correctly

#11 2015-04-19 09:49:03

Re: filename in non-english are not saved correctly

#12 2015-04-19 12:09:00

Re: filename in non-english are not saved correctly

#13 2015-04-19 14:26:48

Re: filename in non-english are not saved correctly

#14 2015-04-19 16:20:11

Re: filename in non-english are not saved correctly

#15 2015-04-20 11:06:22

Re: filename in non-english are not saved correctly

#16 2015-04-20 14:20:11

Re: filename in non-english are not saved correctly

#17 2015-04-20 20:55:41

Re: filename in non-english are not saved correctly

#18 2015-04-20 21:34:40

Re: filename in non-english are not saved correctly

#19 2015-04-20 22:36:18

Re: filename in non-english are not saved correctly

#20 2015-04-21 04:14:14

Re: filename in non-english are not saved correctly

#21 2015-04-21 11:13:32

Re: filename in non-english are not saved correctly

Board footer