[SOLVED] Suddenly FTP server shows as empty

walteweiss · 2021-02-12 18:24:36

Hello everyone,

Suddenly I cannot access my FTP server, I can connect, but it shows an empty directory with no content.

That happened a couple of days ago, after my entire house lost power due to the snowstorm. Nothing was updating at that moment and the server had been serving content to a client at the moment of the outage. Before the outage I had updated linux kernel from 5.10.11 to 5.10.14, but forgot to reboot to apply the new kernel.

Also I updated some extra irrelevant (I think) dependencies. The full list, in case someone will notice something:

gcc-libs (10.2.0-4 -> 10.2.0-6)
autoconf (2.70-1 -> 2.71-1)
automake (1.16.2-3 -> 1.16.3-1)
findutils (4.7.0-2 -> 4.8.0-1)
e2fsprogs (1.45.7-1 -> 1.46.0-1)
libldap (2.4.56-1 -> 2.4.57-1)
systemd-libs (247.2-1 -> 247.3-1)
curl (7.74.0-1 -> 7.75.0-1)
binutils (2.35.1-1 -> 2.36-3)
bison (3.7.2-1 -> 3.7.3-1)
fuse-common (3.10.1-1 -> 3.10.2-1)
gcc (10.2.0-4 -> 10.2.0-6)
glib2 (2.66.4-2 -> 2.66.6-1)
groff (1.22.4-3 -> 1.22.4-4)
libunwind (1.3.1-2 -> 1.4.0-1)
libwebp (1.1.0-1 -> 1.2.0-1)
systemd (247.2-1 -> 247.3-1)
linux (5.10.11.arch1-1 -> 5.10.14.arch1-1)
llvm-libs (11.0.1-1 -> 11.0.1-2)
lmdb (0.9.27-1 -> 0.9.28-1)
pacman-mirrorlist (20210110-1 -> 20210206-1)
pacman (5.2.2-1 -> 5.2.2-2)
php (8.0.1-1 -> 8.0.2-1)
php-fpm (8.0.1-1 -> 8.0.2-1)
php-sqlite (8.0.1-1 -> 8.0.2-1)
python (3.9.1-1 -> 3.9.1-2)
python-packaging (20.8-2 -> 20.9-1)
python-setuptools (1:52.0.0-1 -> 1:53.0.0-1)
systemd-sysvcompat (247.2-1 -> 247.3-1)
youtube-dl (2021.01.24.1-1 -> 2021.02.04.1-2)

That's the only thing I did before the outage happened.

For FTP server I use vsftpd v. 3.0.3-7, which was upgraded half a year ago (2020-07-12).

What I've tried up to this moment:

- checked my hardware (LAN cables, router, disks)
- tried to connect to my server via ssh and UPNP, both work
- reinstalled vsftpd and double-checked that the config is all the same
- downgraded kernel (back to 5.10.11) and updated (to 5.10.15), it made no difference
- confirmed that systemd service works and is active
- netcat the ports, seems fine
- explicitly added to my ufw allow rules ports 20, 21, 22, although I think that's unnecessary
- read vsftpd log, it tells OK, but I see an empty directory with no content, which is not true
- I even upgraded my router firmware (wasn't on latest), just in case, changed nothing
- rebooted the server and clients multiple times, of course

FTP server serves /mnt/Media/ as read-only to anonymous clients in my local network, which is specified in my /etc/fstab as OverlayFS Mounts

overlay /mnt/Media/Movies overlay noauto,x-systemd.automount,lowerdir=/mnt/400/Movies/:/mnt/500/Movies:/home/Media/Movies/ 0 0 overlay
overlay /mnt/Media/Series overlay noauto,x-systemd.automount,lowerdir=/mnt/400/Series/:/mnt/500/Series:/home/Media/Series/ 0 0 overlay
overlay /mnt/Media/Music overlay noauto,x-systemd.automount,lowerdir=/mnt/400/Music/:/mnt/500/Music:/home/Media/Music/ 0 0 overlay
overlay /mnt/Media/4K overlay noauto,x-systemd.automount,lowerdir=/mnt/400/4K/:/mnt/500/4K:/home/Media/4K/ 0 0 overlay
overlay /mnt/Media/Downloads overlay noauto,x-systemd.automount,lowerdir=/mnt/400/Downloads/:/mnt/500/Downloads:/home/Media/Downloads/ 0 0 overlay

/mnt/400, /mnt/500 and /home/Media are three separate disks (old and rusty, but they seemed to work, according to SMART checks and, well, they all accessible via UPNP or rsync, so it's not them).

At this point I have no idea what causes the problem and why FTP server is unaccessible, when I did nothing to change anything, but updated the packages the same day, a couple of hours prior to the outage. As far as I can tell no hardware is damaged, and I believe if it was damaged then it won't work via other protocols, all of which are seem to work just fine as before.

I don't think that matters, but my hardware is an old Intel Atom machine, not a Raspberry Pi.

- Arch Linux x86_64
- Intel Atom 230 (2) @ 1.595GHz [5.0°C] (ha, it's pretty cold right now!)

I tried to connect to the server with different machines:

- MacBook Pro via Wi-Fi
- iMac via Ethernet
- Raspberry Pi 2B via Ethernet

None of that works when it comes to FTP. MacBook and iMac play content via UPnP just fine, and access SSH as well.

I'm eager to ivestigate the issue, but I have no ideas what to try next.

Last edited by walteweiss (2021-02-18 08:17:54)

seth · 2021-02-13 16:09:45

If you ssh into the system, check/ensure that all overlayfs are mounted and simply restart the vsftp service, does it now list the expected files?
Can you download a file via its direct ftp url?
To be sure, post the vsftp.conf

walteweiss · 2021-02-15 18:50:11

seth wrote:

If you ssh into the system, check/ensure that all overlayfs are mounted

I’m not sure it’s the best way, but I go to the directory and it lists all the files. Also I have a minidlna server that serves that very same overlayfs directory and it works correctly.

seth wrote:

and simply restart the vsftp service, does it now list the expected files?

No, it’s not.

I restarted the service with

sudo systemctl reload-or-restart vsftpd.service

and

sudo systemctl restart vsftpd.service

with no difference, it shows that it works, as

sudo systemctl status vsftpd.service

produces this output, which looks good to me:

     Loaded: loaded (/usr/lib/systemd/system/vsftpd.service; enabled; vendor preset: disabled)
     Active: active (running) since Fri 2021-02-12 20:54:34 EET; 2 days ago
   Main PID: 336 (vsftpd)
      Tasks: 1 (limit: 1135)
     Memory: 508.0K
     CGroup: /system.slice/vsftpd.service
             └─336 /usr/bin/vsftpd

Feb 12 20:54:34 arch-atom-230-jbod-01 systemd[1]: Started vsftpd daemon.

seth wrote:

Can you download a file via its direct ftp url?

No, it doesn’t work. Although I can download any file with rsync.

seth wrote:

To be sure, post the vsftp.conf

Here's my /etc/vsftpd.conf.

I commented the last lines after the incident:

#virtual_use_local_privs=YES
#local_root=/home/Media/$USER
#user_sub_token=$USER

and added

isolate_network=NO

but regardless of this change it makes no difference to how it works (it doesn’t).

Last edited by walteweiss (2021-02-15 19:14:06)

seth · 2021-02-15 21:27:16

So it's not just a listing issue.
Can you serve files out of other directories? (eg. /srv/ftp - you could copy some files from /etc there for a test)

fukawi2 · 2021-02-16 02:23:49

What are the logs for vsftp during your connection attempt? `journalctl -bu vsftpd`

walteweiss · 2021-02-16 19:32:22

seth wrote:

So it's not just a listing issue.
Can you serve files out of other directories? (eg. /srv/ftp - you could copy some files from /etc there for a test)

I changed the directory in the config, created 5 files with `touch` command and changed ownership of all the files for the guest user, to be sure it's not an ownership problem, and still I see an empty directory, as it was before. It behaves the same way it did before, with OverlayFS share.

fukawi2 wrote:

What are the logs for vsftp during your connection attempt? `journalctl -bu vsftpd`

It returns this error:

Failed to parse boot descriptor 'u

(maybe it doesn't recognize `-bu` and wants `-b -u`, although afaik it should be similar)

I tried

sudo journalctl -b -u vsftpd

instead and it showed this message:

-- Journal begins at Thu 2019-11-28 20:52:51 EET, ends at Tue 2021-02-16 21:20:>
Feb 15 21:23:22 arch-atom-230-jbod-01 systemd[1]: Started vsftpd daemon.

Also

sudo less /var/log/vsftpd.log

returns this: (many lines with similar strings)

Mon Feb 15 21:28:54 2021 [pid 2] CONNECT: Client "192.168.0.3"
Mon Feb 15 21:28:54 2021 [pid 1] [ftp] OK LOGIN: Client "192.168.0.3", anon password "<no_password>"

In addition to all the information I provided, I see a clear delay while connecting to the server: when it worked connection established quickly, a second or two. After the incident happened it takes a noticeable amount of seconds before the connection happens, 10 seconds maybe.

seth · 2021-02-16 21:42:17

Have you seen https://unix.stackexchange.com/question … -directory ?

walteweiss · 2021-02-17 17:53:32

seth wrote:

Have you seen https://unix.stackexchange.com/question … -directory ?

Sorry, I forgot to mention in my initial post that I did some googling and tried a few solutions I saw as relevant, but I haven't noted what I tried that wasn't helpful. I don't remember finding this link you provided, and I believe it's exactly my problem described, so thank you!

I tried some advices from there and none of them worked, unfortunately:

- tried to disable write permissions for the root directory of the ftp share `/mnt/Media/` in my case.
- And after that `hide file=NO` as in the first answer.
- tried `sudo setsebool -P allow_ftpd_full_access 1` as in the second answer and the third answer, with this error: `setsebool: command not found`
- tried `pasv_enable=YES` and also `pasv_enable=NO` as in the next answer or another one

So far no fruitful cure for my itch from stackexchange, I'll keep investigating. Thank you all guys for your input, if you have any other ideas, please let me know. I'll let you know if I'll find anything helpful too.

Last edited by walteweiss (2021-02-17 17:55:32)

loqs · 2021-02-17 18:58:54

Just to rule it out have you tried seccomp_sandbox=NO in vsftpd.conf and restart the service.

walteweiss · 2021-02-18 08:16:47

loqs wrote:

Just to rule it out have you tried seccomp_sandbox=NO in vsftpd.conf and restart the service.

Oh, wow! That helped!
As far as I remember, I tried all the methods from the wiki page, but seems like either I did something wrong (e.g. forgot to reload the service) or just missed this issue, since I don't use FileZilla, I don't remember. Re-reading the wiki page was the very first thing I tried the next day.

Well, thank you! I'm sorry we all did more research that it needs to be.

Solution

I added

seccomp_sandbox=NO

to my vsftpd.conf and it solved the issue.

You can do that with this one string:

echo 'seccomp_sandbox=NO' >> /etc/vsftpd/vsftpd.conf

Causes

Now I'm just curios to understand how could that happened if I changed nothing myself and what this string is about. The wiki page for vsftpd links to Bugzilla at Red Hat from 2012 which explains the issue, but I wasn’t able to understand it. As I get it, the reason could be the kernel upgrade with this setting having a different default so we have to set this explicitly, or what. If I'll get to understand it better, I'll edit this message.

If someone can explain this issue with simpler terms, please do.

Last edited by walteweiss (2021-02-18 08:20:49)

Arch Linux

#1 2021-02-12 18:24:36