FreeNASs2a mounted as rw after upgrade

[jgreco]

I'm not /quite/ sure what happened. Upgraded from a recent nightly to 8.0.1-R. The system tried to reboot itself, hung after syncing, and after 10 minutes I killed it. It rebooted fine, upgraded its database, rebooted. Upon reboot, I had some unrelated problems (vlan/network issue, see recent ticket) so went to shell. Noticed it was /very/ slow, which is usually a hallmark of the root partition being mounted rw. I think the microSD I use may be sluggish or something.

So there are three things here.

1) The root filesystem was NOT mounted read-only, which is a majorish bug. (It did mount ro after another reboot though.) Users using the web management system to upgrade their systems might well leave the system in this state, writable base system not good.

2) The root filesystem should also be mounted "noatime". Probably always. This seems to help substantially with the lag slag.

3) It's unclear why the system is laggy when the root filesystem is mounted rw. This suggests to me that some unintended writing may be going on to the root filesystem. May be wise to investigate.

[gcooper]

1. This is curious. It sounds like an upgrade failed from the sounds of it.. can you please describe what version you upgraded from and what version you upgraded to, etc? BTW, what you're doing by upgrading from a nightly to 8.0.1 RELEASE will not work -- guaranteed. That being said, I think you might have run into an unusual atomicity corner case.
2. If stuff's mounted ro, then noatime is a noop (there's nothing to update).
3. Err... ok. More info like your boot logs would help a lot because I'm not seeing this after I updated from an image that I built on trunk to another one I built last night.

FWIW Flash will always be slow compared to hard drives or memory disks.

[jgreco]

N36L dmesg.boot

[jgreco]

1) 8r8022 to 8.0.1-RELEASE. Unfortunately it's on a HP N36L and I don't have a serial console for it, so all I can really say is that it was after "syncing disks" and before "rebooting", so it's not in FreeNAS itself. The upgrade doesn't seem to have failed, as the unit's now booting off partition 2, it updated its config upon reboot, and claims to be running RELEASE-amd64 (8081).

2) If stuff's not mounted ro, then noatime is not a noop (whatever happened to belt and suspenders). In general, atime updates are useless in many environments, but on flash in particular, it's sensitive to write cycles. In any environment where the wasted IOPS are detrimental, noatime is a good idea. I created the original FreeBSD noatime implementation for a reason.

3) I'm glad you're not seeing it, but I'm seeing it now and then. I had originally been writing it off as some side effect of USB1.1-based flash, but the N36L is USB2, reads flash at 10MB/sec, writes at 1.5MB/sec, and still I see it happen. I've been thinking it might be a ZFS thing of some sort though, as I've also seen a lot of ZFS-related lag in these systems. I'm still kind of looking into it.

ZFS is particularly laggy on a single-core system especially if you start doing things like compression and snapshots.

FWIW some of my early FreeBSD work was the inspiration for PicoBSD and NanoBSD; I've got well more than a decade of experience with embedded FreeBSD systems, so when I say "slow", please interpret that as "unexpectedly slow compared to other similar systems," not "slow compared to hard drives."

[gcooper]

This shouldn't be an issue anymore (post the work that was done via 8.0.4-RELEASE) or at the very least mitigated in such a way that this occurring is unlikely. If it does occur again, please reopen this bug.