The answer, after trying dozens of approaches, turned out to be that the permissions on the /media/pi directory weren't set right. Finding this answer was made tricky because I was comparing the directory permissions to those on another pi that didn't have the problem, and the directory permissions matched. I learned however that the other system had an Access Control List set on the directory that gave additional permissions to the pi user. Setting up the original pi to have an access control list to match the working one fixed the problem of the thumb drive not being mounted correctly when plugged in, but then for some reason whenever an eject was done from the file browser the eject widget on the task bar would show an error that the drive had been removed without being ejected. That problem was fixed by updating the lxpanel package on the old pi; it turns out that the eject widget is part of the lxpanel and not a stand-alone app.
Troubleshooting notes taken during the course of figuring all this out follow:
This result from googling the error message is somebody who eventually figured out that his system was missing usbmount. I could try installing usbmount, but it doesn't seem like this would be my problem because USB drives used to mount on my system (unless something somehow got messed up when I cloned the SD card): http://unix.stackexchange.com/questions/325962/the-specified-directory-media-fa-faf9-c03-is-not-valid-error-when-trying-to
By following these instructions, I am able to mount the drive into a new location, access it, and unmount it. I found that the strange serial number in the error message appeared in the list of devices by id, and that a /dev/sda1 device had been created, so all of the assumptions in these instructions about what I would find were true: http://www.raspberrypi-spy.co.uk/2014/05/how-to-mount-a-usb-flash-disk-on-the-raspberry-pi/
Here's another set of instructions on mounting a thumb drive that look good but I haven't really looked at them yet because the first one worked so well: https://www.raspberrypi.org/forums/viewtopic.php?f=29&t=35695
Another tutorial about USB drive mounting. Doesn't cover the automounter but has an amazing assortment of tips and tricks that I haven't seen elsewhere:
https://pimylifeup.com/raspberry-pi-mount-usb-drive/
Trying to enable the wifi:
"It should just work" meaning there's a graphical tool from the menu bar (which I can't find): https://www.raspberrypi.org/forums/viewtopic.php?f=28&t=138137
Gorgeous step by step instructions on enabling the wifi from the command line, which worked right up to the point of configuring the wifi password in a plain text file. The problem is that our guest network requires authentication via a web page with a form. But I was able to scan for local networks at least using these steps:
https://www.raspberrypi.org/documentation/configuration/wireless/wireless-cli.md
http://www.makeuseof.com/tag/setup-wi-fi-bluetooth-raspberry-pi-3/
So trying to login this type of network is basically impossible in Pi: https://www.raspberrypi.org/forums/viewtopic.php?f=91&t=152447
The problem is so bad that all of the serious suggestions on this page are ridiculous: raspberrypi.stackexchange.com/questions/34008/connect-to-public-wifi-hotspot
Here's somebody who actually finds an answer, but later on somebody points out that I can just use the portable hot spot from my phone: unix.stackexchange.com/questions/118289/how-do-i-authenticate-to-a-wireless-providers-open-network-without-using-a-br
Just turning on the hotspot on my phone, the hotspot wasn't found when scanning. This set of instructions involves tethering the phone to the Pi over USB before making the connection to the hotspot. Why? I don't know! But it didn't work; the Pi never found the phone, possibly due to whatever problem is occuring with the USB in the first place.
http://www.daveconroy.com/how-to-tether-your-raspberry-pi-with-your-iphone-5/
Returning to working on why automount is failing, looking in the message logs, there are strange error messages such as: "error opening /media/pi/9005-048F/BDMV/index.bdmv"
Somebody reports that these are file types for blu-ray disks? https://www.raspberrypi.org/forums/viewtopic.php?f=91&t=154836
This thread's resolution seems to suggest that it's a power starvation issue (if so this is bad news for my system): https://www.raspberrypi.org/forums/viewtopic.php?f=28&t=124532
About the 'action 17' errors also seen in the log (apparently unrelated stuff):
https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=155803
https://www.raspberrypi.org/forums/viewtopic.php?f=91&t=152405
So, I downloaded usbmount, and a thumb drive is now mounted and opens in file browser when plugged in.
But it can't be ejected without a dialog asking for the password for the user account (there isn't one, which means that this dialog can't be answered! A couple of search results for the dialog came up with suggestions to change some policy settings:
askubuntu.com/questions/552503/stop-asking-for-authentication-to-mount-usb-stick
Now the eject buttons for the drive in the file browser seem to unmount the drive, but a message appears from tbe eject app saying that the drive was removed wirhout being ejected.
(along the way, found this interesting thread about somebody having problems booting from a usb hd, solved by adding a config parameter for a wait before first access to an external drive:
https://www.raspberrypi.org/forums/viewtopic.php?f=28&t=109589 )
Things that are different between the working and nonworking images:
File permissions for working system:
/media: drwxr-xr-x root:root
/media/pi: drwxr-x---+ root:root
/media/pi/SETTINGS: drwxr-xr-x root:root
File permissions for the nonworking system:
/media drwxr-xr-x root:root
/media/pi: drwxr-x--- root:root
ls /media/pi: Permission Denied
The SETTINGS directory is apparently the NOOBS config files.
http://raspberrypi.stackexchange.com/questions/60609/settings-drive-accidentally-renamed/60683
Looking at all files under /etc, SETTINGS is mentioned in the mtab file, which is the list of all presently mounted file systems.
the (+) on the file permissions for the working system means that it's an access control list?
https://ubuntuforums.org/showthread.php?t=2206434
https://wiki.archlinux.org/index.php/Access_Control_Lists
Again, in the following example /media/pi has a (+). His drive doesn't automount though and he never gets an answer to his question
https://www.raspberrypi.org/forums/viewtopic.php?f=66&t=135178B
Results from getfacl /media/pi in the working system:
#file: media/pi
#owner: root
#group: root
user::rwx
user:pi:r-x
group::--
mask::r-x
other:---
Added an ACL to /media/pi using the example in this page (this page is filled with extremely easy-to-follow instructions BTW):
http://www.htpcguides.com/properly-mount-usb-storage-raspberry-pi/
Now I can access /media/pi as user pi, and when I plug the thumb drive in, it comes up in the browser without error!!! Also, the drive is not greyed out in the eject widget!!!
But there's still something not right that works using the new NOOBS configuration! When I eject it using the eject button in the browser, the eject widget still says "...was removed without ejecting Please use menu to eject before removal" This does not happen in the working system.
Apparently the button in the file manager is separate from the button in the eject widget. There are many, many posts online from people who have problems using the button in the file manager but they are almost all people who don't have the "eject" package installed. I triple checked and it is installed on both of my systems.
Here's are some of those people who simply had to run sudo apt-get install eject. How jealous these links make me!
http://raspberrypi.stackexchange.com/questions/14843/how-to-eject-usb-device-on-raspberry-pi-not-just-unmount
http://kj-til.blogspot.com/2013/11/eject-usb-drive-error-and-fix.html
https://www.raspberrypi.org/forums/viewtopic.php?t=18364
Here's a guy who wants to have the eject buttons in the file browser be removed or disabled. It seems like this would be pertinent to helping me find out how to change their behavior, but sadly nobody answers this forum question! Also this would be pertinent because the file browser is including eject buttons for the main SD card which makes no sense and probably would be bad if somebody hits them: http://askubuntu.com/questions/862592/remove-or-disable-eject-button-in-files-file-manager-for-second-internal-hdd
I did a bunch of googling of how to configure the file browser.
I learned that it's called "File Manager" and it has a menu option for "Preferences" which has settings for how to respond to a USB volume being plugged in!!!!! Strangely there doesn't seem to be settings in preferences for how to respond to requesting and eject? I don't understand why that wouldn't be there. However, I discovered that the File Manager program has a name; it's named PCManFM. Amazingly it was developed in Taiwan, released under GNU public license, and is the default file browser in LDXE which Raspbian is a flavor of.
This wiki article on pcmanfm goes out of its way to mention that pcmanfm handles thumb drives (the keyword is "volumes") but does not point to any specifics about how it is configured. Also the original page seems to have been hacked so this link is to an archived version: https://webcache.googleusercontent.com/search?q=cache:2Sk4YDcqHgEJ:https://wiki.lxde.org/en/PCManFM+&cd=2&hl=en&ct=clnk&gl=us
Here is an overview of what pcmanfm is, with a hint that it handles automounting but it doesn't say anything here about unmounting. It mentions that there is a daemon mode for detecting new removable media, and specifically mentions my USB automounting problem with a link to a different article: https://wiki.archlinux.org/index.php/PCManFM
The link from the pcmanfm link about USB drive authorization problems. This actually is about the policy settings that were discussed in a different answer that I found during my disastrous experiment with installing usbmount; this wiki page is for something called "polkit" which seems to be a tool for modifying the policies that I was manually editing to get usbmount to allow an unmount without a password: https://wiki.archlinux.org/index.php/Polkit#Mounting_storage_without_password
This github README mentions that Volume management is done with glib, which relies on gvfs. Also that there is an --enable-udisks configure option, but doesn't go into detail about how those are setup.https://github.com/lxde/libfm
This blog post from a pcmanfm developer in 2010 gives the hint that the configuration file is named libfm.conf, and there are also relevant configuration parameters in ~/.config/libfm/libfm.conf. Worryingly, a commenter asks downthread if there will be an "unmount safely option," so maybe one didn't get implemented in 2010? https://webcache.googleusercontent.com/search?q=cache:5ufMWNwOmn0J:https://blog.lxde.org/2010/07/04/direct-udisks-support-for-libfm-and-pcmanfm-is-finished/+&cd=2&hl=en&ct=clnk&gl=us
Searching my working unit, I find a virtual pile of applicable .conf files under /etc! These are not necessarily in the locations hinted at by the 2010 blog post, so some tweaking has apparently occured
/etc/xdg/libfm/libfm.conf
/etc/xdg/pcmanfm/default/pcmanfm.conf
/etc/xdg/pcmanfm/LDXE/pcmanfm.conf
/etc/xdg/pcmanfm/LDXE-pi/pcmanfm.conf
Of those, libfm.conf doesn't have anything particularly interesting except for a flag for "use_trash" which means that if I ever need to turn that on or off I now know where to do it. The contents of the other three .conf files vary with respect to each other, with the default and LDXE-pi pcmanfm.conf containing three scant configuration lines in a [volume] block of parameters, none of which seem to include anything about unmounting or what code gets run for mounting! The LDXE pcmanfm.conf oddly doesn't contain a [volume] section. All three have parameters different from each other that seem mostly to do with the look and feel.
This git repository lists the --enable-udisks option as a parameter for building it (sad) but says that it is defaulted to yes (happy)!!! https://github.com/raspberrypi/libfm-1.2.3/blob/master/configure
This man page for udisks indicates that detach and eject are both options, with parameters that are not listed in this page: https://udisks.freedesktop.org/docs/1.0.5/udisks.1.html
This github code indicates that there is a utility named udisks-eject; perhaps this is what is called by pcmanfm on my working system (Update, it's unlikely). If so I wonder how to find the option that specifies it? https://github.com/patrikf/udisks-eject/blob/master/udisks-eject
Unrelatedly, here is an interesting developer's discussion on what context menus to put on the eject options for Nautilus, a similar file browser to pcmanfm. They're talking about the same thing so there's probably interesting stuff to learn here, but it's mostly argument about GUI philosophies.https://bugs.launchpad.net/ubuntu/+source/nautilus/+bug/28835
Having learned all that, I keep finding hints that the configuration for pcmanfm is in .desktop files, which I don't know much about but have to learn now I guess.
This guy researched how to customize the pcmanfm option. This is getting soooo close! He's trying to set it up to use udisksctl which I've also seen in other links. He is advised to put something in ~/.local/share/file-manager/actions/umount.desktop http://askubuntu.com/questions/633027/eject-usb-drive-action-for-pcmanfm
Sadly, I don't seem to have ~/.local/share/file-manager/actions/umount.desktop. Actually my ~/.local/share directory doesn't have a file-manager directory, just applications, desktop-directories, gvfs-metadata, and Trash, none of which even contain anything interesting.
This page, which I actually found a while ago, mentions config files in ~/.config! It mentions a .config/pcmanfm directory, which I have! I have a LXDE-pi directory instead of the lubuntu directory mentioned in the post but that all makes sense! And there's pcmanfm.conf! Its contents are a little different from the one in /etc/xdg/pcmanfm/LXDE-pi though. There's some wind size, then interestingly the local one has a parameter close_on_unmount=1! This parameter isn't in any of the three pcmanfm.conf files under /etc/xdg. I wonder where it came from? https://ubuntuforums.org/showthread.php?t=2214272
OMG this link referred to from the previous link has so much relevant information! https://ubuntuforums.org/showthread.php?t=2214389&p=12973843#post12973843
But wait! This post under the pi forums has somebody adding a behavior to pcmanfm by creating a ~/.local/share/file-manager/actions out of thin air and then putting stuff into it. Well, since my working system doesn't have a ~/.local/share/file-manager directory, I know that this can't be how it is doing it, but this still seems like a really potentially useful trick: https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=146711
This is all very well and good, but I checked /var/log/syslog on both the working and now-only-slightly-not-working system and they show nearly identical log messages when a drive is ejected from pcmanfm. Both systems have the following messages from udisksd:
udisksd[task number]: Cleaning up mount point /media/pi/(device 8:1 is not mounted)
udisksd[task number]: Unmounted /dev/sda1 on behalf of uid 1000
Also, that is the message on both systems whether the drive is ejected from pcmanfm or the ejector widget.
So, perhaps the problem that's causing the ejector desktop widget to throw an error message when a drive is ejected from pcmanfm isn't in pcmanfm after all, but in the ejector widget instead.
Could this problem be solvable by adjusting the ejector's configuration? The first step is to find the ascii config files that govern the configuration.
This page doesn't talk about the configuration files, it's about an exercise in setting up the configuration of a generic LXDE desktop to look like the Raspbian desktop, but it's delightfully geeky. www.zdnet.com/article/how-to-customise-your-linux-desktop-lxde/
This page however does go into great specifics on where to find the setup files under /etc/xdg/lxsession/LXDE-pi and ~/.config/lxsession/LXDE-pi.www.raspberrypi-spy.co.uk/2014/05/how-to-autostart-apps-in-rasbian-lxde-desktop/
Interestingly, the actual setup seems to be under ~/.config/lxpanel/LXDE-pi/panels. The file which establishes the ejector is ~/.config/lxpanel/LXDE-pi/panels/panel, and at the end there is just a section as follows:
Plugin {type=ejecter}
Config {
}
So, on the working system at least, the Config for "ejecter" is empty. It is interesting that it is spelled "ejecter" though. Googling for more information about it reveals github repositories for ejecter.c. This seems to indicate that, bad news, this is a compiled part of the kernel. Interestingly though the checkin notes in these repositories seem to show that the file might have been updated recently to specifically address the error messages that I'm seeing. It could be that thanks to the horrible old NOOBS that we were shipped that the problem might be just that the problem kernel is old enough not to have recent updates to ejecter.c!
https://github.com/raspberrypi/lxpanel/commits/master/plugins/ejecter
https://github.com/raspberrypi/lxpanel/blob/master/plugins/ejecter/ejecter.c
Reading through the code on the above repositories, I can see that it contains the text from the dialog messages that show up whenever something is ejected either "wrong" or "right" on my troublemaking distro. But there just don't seem to be arguments. It is an internally complicated, but seemingly stand-alone and toasterlike tool. There are calls in the code to a DEBUG routine which probably does stuff if the appropriate compile flags are set, this isn't going to help me. All I've got to pin my hopes to now is that there might be a way to upgrade the app on my crappy system with the version of the app on my good system. But how?
This thread is just some neckbeards helping a noob bang on an only slightly related problem, but lots of minor clues like ways to figure out version numbers of installed packages (debsums? dpkg's -S flag? apt-file?) and the locations of the desktop setup files: https://www.raspberrypi.org/forums/viewtopic.php?f=66&t=148101&sid=7830622a975fc16cd8e9db8ec742aceb
Here's a github repo that shows ejecter.c being under the lxpanel app: https://github.com/raspberrypi/lxpanel/blob/master/plugins/ejecter/ejecter.c
Some background on lxpanel and its configuration files, but nothing that specifically mentions ejecter: https://wiki.lxde.org/en/LXPanel#Main_Config_File
A pretty nice introductory tutorial about lxpanel which describes how the applets work (again without specifically mentioning ejecter). It does say something to the effect of "The configuration is stored in ascii files, but it's not a good idea to muck around with them directly." Which is probably a good idea but I can't stop myself. http://pclosmag.com/html/Issues/201010/page07.html
Another description of lxpanel that mentions "volume control" with no further details like if it's built-in or separate. Maybe I have to update all of lxpanel to update ejecter? This was from a search of "lxpanel" on this site, where a search of "ejector" produced no results: https://packages.debian.org/jessie/lxpanel
Alright, now we're getting close: Here's a guy who had a problem in lubuntu running ejecter specifically, and to troubleshoot it he was able to run it from a command line! His problem was that it crashed though, and it's not clear if this was due to a bug or because you can't run it from a command line. The forum thread ends with the discovery that there is a bug report out on ejecter and it is recommended that it not be used! Not sure why there wouldn't be better google hits if this actually happened. https://forum.lxde.org/viewtopic.php?f=8&t=31726
Ok this is interesting: A guy asks if there's an app in ubuntu that will unmount USB drives, and somebody describes in perfect detail how to add ejecter as an "indicator" that can be started at login by adding a config file to ~/.config/autostart. This is the clearest implication that ejecter is a complete and stand-alone executable, and furthermore the responder also show it being downloaded using apt-get! (Note: somebody else replied with a link to a list of file browsers that can be run under ubuntu. It was inapplicable to the original question but actually still interesting): http://askubuntu.com/questions/145087/lubuntu-panel-applet-or-indicator-to-display-mounted-volumes
So now I need to ask, if ejecter is a stand-alone package that can be downloaded using apt-get, is there a way for me to update one of my copies of ejecter? Can I find out what version is running on each of my systems first?
This link has many answers! Apparently I can use sudo apt-get --only-upgrade install [package name]. Some say that just sudo apt-get install is necessary to update. The Synaptic Package Manager has a "mark for upgrade" option but it is not clear if this will upgrade a single package or the entire distro. Apparently sudo apt-cache policy [package name] will get the list of all version numbers, and then it's possible to specify a particular version number when doing sudo apt-get install!!! Someone reported that the syntax for only-upgrade is apt-get --only-upgrade true install [package]. And then there are examples of using dpkg -s to find if something is installed, complete with a gloriously terse one-liner. http://askubuntu.com/questions/44122/how-to-upgrade-a-single-package-using-apt-get
Wow! Hints for how to list available versions! There's a package called rmadison that does it, and apparently the "policy" argumement might apply to apt-cache not apt-get: http://askubuntu.com/questions/447/how-can-i-see-all-versions-of-a-package-that-are-available-in-the-archive
More of the above hints. Also the question is asked how to tell what version is installed but the answer is that unless a -v argument was coded in there's no way to tell: http://askubuntu.com/questions/428772/how-to-install-specific-version-of-some-package/428778
This link, however seems to indicate that aptitude or dpkg will work just fine to get a list of the version numbers of installed packages: http://askubuntu.com/questions/17823/how-to-list-all-installed-packages
Not directly pertinent but still very interesting, this forum thread contains a discussion of how to use the list of version numbers of installed packages to make an exact clone of an installation: http://askubuntu.com/questions/9135/how-to-backup-settings-and-list-of-installed-packages
An unrelated other example of using apt-cache policy to get version numbers: https://www.raspberrypi.org/forums/viewtopic.php?t=136412&p=907548
Based on all of the above, I was able to itereate through getting the installed versions of the lxpanel on both the old and the new pi, then update lxpanel on the old pi, and the error message from the eject plug-in no longer appears when the thumb drive is ejected from the browser!!!
On the new system:
$ lxpanel -v
lxpanel 0.7.2
$apt list --installed | grep lxpanel
lxpanel/stable.now 0.7.2-1+rpi12 armhf [installed]
lxpanel-data/stable.now 0.7.2-1+rpi12 all [installed.automatic]
$apt-cache madison lxpanel
lxpanel | 0.7.2-1+rpi12 | http://archive.raspberrypi.org/debian/ jessie/ui armhf Packages
lxpanel | 0.7.2-1 | http://mirrordirector.raspbian.org/raspian/ jessie/main armhf Packages
$apt-cache policy lxpanel
lxpanel:
Installed: 0.7.2-1+rpi12
Candidate: 0.7.2-1+rpi12
Version table:
*** 0.7.2-1+rpi12 0
500 http://archive.raspberrypi.org/debian/ jessie/ui armhf Packages
100 /var/lib/dpkg/status
0.7.2-1 0
500 http://mirrordirector.raspbian.org/raspian/ jessie/main armhf Packages
On the old system:
$ lxpanel -v
lxpanel 0.7.2
$apt-cache madison lxpanel
lxpanel | 0.7.2-1+rpi16 | http://archive.raspberrypi.org/debian/ jessie/ui armhf Packages
lxpanel | 0.7.2-1 | http://mirrordirector.raspbian.org/raspian/ jessie/main armhf Packages
$apt list --installed | grep lxpanel
lxpanel/now 0.7.2-1+rpi4 armhf [installed,upgradable to: 0.7.2-1+rpi16]
lxpanel-data/now 0.7.2-1+rpi4 all [installed,upgradable to: 0.7.2-1+rpi16]
The above results show that the old system has an older version of lxpanel (0.7.2-1+rpi4 versus 0.7.2-1+rpi12) and furthermore apt-list shows that there is an upgrade waiting (0.7.2-1+rpi16)!
Performed the upgrade using:
sudo apt-get --only-upgrade install lxpanel
checked for any signs of errors in the update, and rebooted. The system came back up, and lxpanel didn't look any different, but now I am able to insert the thumb drive, open it in a file browser, then eject it without getting any alerts from the eject widget in lxpanel. Checking the installed version using apt list --installed shows that the new version is 0.7.2-1+rpi16. Mission accomplished!
Now that that's done, here are a few links to unrelated but interesting learning that I encountered along the way:
Here's how to add a command to an lxpanel menu: https://forum.lxde.org/viewtopic.php?t=31110
Here's how to add a CD eject button (not even remotely close to what I am trying to do, but interesting) to a desktop widget:https://forum.lxde.org/viewtopic.php?f=8&t=36444
Here's another one on adding commands to an lxpanel menu: http://unix.stackexchange.com/questions/177386/how-can-i-add-applications-to-the-lxpanel-application-launch-bar-via-cli
How to write C code to build a new plugin for lxpanel! https://wiki.lxde.org/en/How_to_write_plugins_for_LXPanel:_pre-0.7_version
The wiki for lxpanel itself: https://wiki.lxde.org/en/LXPanel
Some discussions of commands for safely ejecting and unmounting a USB drive. Interesting stuff about the lsblk command and udisksctrl: http://unix.stackexchange.com/questions/178638/eject-safely-remove-vs-umount
Some man pages for the the udisks command line tool: http://manpages.ubuntu.com/manpages/trusty/en/man1/udisks.1.html
http://linuxreviews.org/man/udisks/
https://udisks.freedesktop.org/docs/1.0.5/udisks.1.html
Some C code for something called udisks-eject. Is this somebody's wet dream or actually a standard feature? https://github.com/patrikf/udisks-eject/blob/master/udisks-eject
