Add configuration management #17

benhylau · 2018-01-10T00:46:29Z

See: #6 and https://gist.github.com/hamishcoleman/75de3d6c5040efccf392f984306ea2c2

benhylau · 2018-01-10T00:47:42Z

debian/packages.d/systemd.customise.add/init.d/20confd.sh

+
+try_partition() {
+    # Create file system node from partition
+    #mknod /dev/$1 b $2 $3 2>/dev/null


@hamishcoleman Works without this. Do we need?

strange - I'm confused as to what is creating the device nodes in that case - this is running on a system with an empty /dev and before udev has hotplugged anything

I was running the script manually so it wasn't tested at the correct state of OS startup

benhylau · 2018-01-10T00:47:59Z

debian/packages.d/systemd.customise.add/init.d/20confd.sh

+}
+
+# Mount proc if not already mounted
+#mount -t proc proc /proc


@hamishcoleman Works without this. Do we need?

strange, I dont know what would be mounting proc in this case - this is running before systemd has mounted any filesystems

Same reason. I am running this after all the systemd stuff

You know, we could add a blockdevice to the qemu test environment and allow that to be used to simply test persistance

benhylau · 2018-01-10T00:50:13Z

debian/scripts/usr/local/bin/config_save

+    # Find conf.d in root of partition and save current configurations in /etc
+    if [ -d "/mnt/$CONFDIR" ]; then
+        echo Saving configurations to /dev/$1: "/mnt/$CONFDIR/50-node-config-save.tar.gz"
+        # TODO Handle symlinks properly (i.e. excluding them from the tar)


Right now it tars up everything in /etc and when extracting the tar you end up with a bunch of symlink errors like this:

tar: ./network/if-post-down.d/hostapd: Cannot create symlink to '../../hostapd/ifupdown.sh': Operation not permitted

I guess we should exclude symlinks in the tar?

@hamishcoleman this is the biggest problem though

cannot exclude symlinks - they are vital to be stored as part of the config.

Dont know why you get untar errors - tar quite happily handles creating symlinks

Not sure, but I was also runnings the script not in the expected place

benhylau · 2018-01-10T00:52:05Z

debian/scripts/usr/local/bin/config_save

+    # Check each partition matching sd*[0-9] (e.g. sda1) or mmcblk*p* (e.g. mmcblk0p1)
+    case $name in
+        sd*[0-9]|mmcblk*p*)
+            echo Checking for configuration directory on $name


Right now we only write to partitions with a conf.d directory. We should include an empty conf.d in the fat16 we create alongside boot.

What is the appropriate place to do this? Is there an obvious common Makefile?

benhylau · 2018-01-10T00:52:34Z

debian/scripts/usr/local/bin/config_save

@@ -0,0 +1,36 @@
+# Save current configurations to partition containing conf.d


@hamishcoleman Where shall I place this file?

There isnt really a well named place for non-debian customisations in the repo (I had some ideas, but it didnt come up enough to fix yet) - I think you found the debian/packages.d/systemd.customise.add/usr/local/sbin/ dir which should work for now.

It wouldn't automatically run this right? And bin or sbin in this case? System never runs it, it's the administrator of the nodes that may use it.

sbin is for tools that the administrator runs - so it is definitely sbin

Local is a bit of a toss up for me. By definition it sorta applies sorta does not.

The /usr/local hierarchy is for use by the system administrator when installing software locally. It needs to be safe from being overwritten when the system software is updated. It may be used for programs and data that are shareable among a group of hosts, but not found in /usr.

Since were not "upgrading" we don't really need local?

benhylau · 2018-01-10T00:54:00Z

debian/packages.d/systemd.customise.add/init.d/20confd.sh

+# Mount proc if not already mounted
+#mount -t proc proc /proc
+
+cat /proc/partitions | while read major minor size name; do


This reads conf.d from all partitions. I guess it's okay for now? It'll be a little weird if multiple partitions have configurations because tar archive order rule doesn't apply here cross partitions.

we could exit on first found - which has issues if the files are hard to overwrite. we could also collate all the tar files from all the partitions - but that has an unwanted increase in complexity.

by checking all partitions, we try to avoid painting ourselves into an unexpected corner

Neither first found or collect sort seem better than current. They are just different but just as non-ideal. I suggest keeping it as is and discourage people from having two partitions with conf.d at root, and state current behaviour.

I suggested the collect-and-sort method as that is how a lot of things (like systemd) handle multiple sources of config overrides - its an annoying amount of additional work, but could be done if this ever becomes an issue

On WINDOWS machines, only the FIRST partition is ever recognized OS on usb devices.

Unless you have some reason to put config on partition >2 you could just stick with partition 1 to make it most compatible.

The issue still exists when there are multiple physical disks connected to the board

benhylau · 2018-01-10T00:54:54Z

debian/scripts/usr/local/bin/config_save

+# Mount proc if not already mounted
+#mount -t proc proc /proc
+
+cat /proc/partitions | while read major minor size name; do


This writes to multiple partitions if they all have conf.d. We don't expect that to happen and it's probably not a big deal.

I dunno - I'd be wanting it to only save to one of them - perhaps exit after the first found. Perhaps allowing the device name to be given as an optional parameter

I added breaking after first found

hamishcoleman · 2018-01-10T01:17:15Z

debian/packages.d/systemd.customise.add/init.d/20confd.sh

+    if [ -d "/mnt/$CONFDIR" ]; then
+        confs=$(find /mnt/$CONFDIR -name *.tar.gz)
+        for conf in $confs; do
+            echo Applying configurations from /dev/$1: $conf


Why not just use "for conf in /mnt/$CONFDIR/*.tar.gz" ?

Using find will traverse subdirs and is not known to be sorted

hamishcoleman · 2018-01-10T01:18:22Z

debian/packages.d/systemd.customise.add/init.d/20confd.sh

+        confs=$(find /mnt/$CONFDIR -name *.tar.gz)
+        for conf in $confs; do
+            echo Applying configurations from /dev/$1: $conf
+            tar --extract --gzip -f $conf -C /etc


tar will automatically detect the compression type - you can leave off "--gzip"

hamishcoleman · 2018-01-10T01:25:47Z

debian/scripts/usr/local/bin/config_save

+
+try_partition() {
+    # Create file system node from partition
+    #mknod /dev/$1 b $2 $3 2>/dev/null


as this is run from a booted system, it should not make nodes - udev will do that for us

hamishcoleman · 2018-01-10T01:27:10Z

debian/scripts/usr/local/bin/config_save

+    # Find conf.d in root of partition and save current configurations in /etc
+    if [ -d "/mnt/$CONFDIR" ]; then
+        echo Saving configurations to /dev/$1: "/mnt/$CONFDIR/50-node-config-save.tar.gz"
+        tar --create --gzip -f 50-node-config-save.tar.gz -C /etc .


In the future, we probably want both the tar filename and the source files to be optional parameters - but dont complicate things right now

hamishcoleman · 2018-01-10T01:27:51Z

debian/scripts/usr/local/bin/config_save

+}
+
+# Mount proc if not already mounted
+#mount -t proc proc /proc


as this is running in a booted system, we should not be mounting proc

benhylau · 2018-01-11T23:45:02Z

debian/packages.d/systemd.customise.add/usr/local/sbin/config_save

+            try_partition $name $major $minor
+            S=$?
+
+            # Save only to the first partition found with conf.d


@hamishcoleman Break like this after first successful save?

seems reasonable

benhylau · 2018-01-11T23:49:23Z

@hamishcoleman I haven't tested this yet, and the following two issues remain:

The tar extract fails with those symlink errors, but when I run this properly they may go away
Create empty conf.d directory as part of base image

benhylau · 2018-01-12T00:21:43Z

Okay so the extract part works. I have:

/conf.d/00-base.tar.gz -> some.config (empty file)
/conf.d/01-more.tar.gz -> some.config ("some stuff" text, this should replace the previous)
                       -> test/ (new directory)
                       -> test/other.config (empty file)

After first boot:

root@ramdisk-2c5d7020:/etc# cat some.config
some stuff
root@ramdisk-2c5d7020:/etc# ls test/other.config
test/other.config

Then I tested config_save:

root@ramdisk-2c5d7020:/etc# echo "new stuff" >> test/other.config
root@ramdisk-2c5d7020:/etc# cat test/other.config
new stuff
root@ramdisk-2c5d7020:/etc# config_save
Checking for configuration directory on mmcblk0p1
Saving configurations to /dev/mmcblk0p1: /mnt/conf.d/50-node-config-save.tar.gz
Configurations saved to /dev/mmcblk0p1: /mnt/conf.d/50-node-config-save.tar.gz

// Power cycle

root@ramdisk-2c5d7020:/etc# cat test/other.config
new stuff

So then I tried to manually extract the tar to make sure there isn't symlink errors:

root@ramdisk-2c5d7020:/# cat /proc/partitions
major minor  #blocks  name

   1        0       4096 ram0
   1        1       4096 ram1
   1        2       4096 ram2
   1        3       4096 ram3
 179        0   15454208 mmcblk0
 179        1    1024000 mmcblk0p1
root@ramdisk-2c5d7020:/# mount /dev/mmcblk0p1 /mnt
root@ramdisk-2c5d7020:/# tar --extract -f /mnt/conf.d/50-node-config-save.tar.gz -C /etc

No errors. So, everything seems to work at least for the case when I have only a single FAT partition connected to the system.

darkdrgn2k · 2018-01-12T00:56:50Z

To identify the config partition possibly look for the boot directory that identifies the partition created by the mesh-orange instead of conf.d.

That way the conf.d files are always internal vs some other random partition that is plugged in (ie usb drive). It would also allow for pulling the sd card out and plugging it into a computer to edit the config files externally (since its the first vfat partition)

On the other hand mesh-orange image will overwrite the image and as such the config.

darkdrgn2k · 2018-01-12T01:41:14Z

During the boot sequence the tar tries to extract the files with a timestamps causing allot of warnings about future files (since there is no RTC on the board)

Possibly adding -m to the tar extraction command line could be a work around

     -m, --touch
           don't extract file modified time

Example of issue (seen on serial console)

Starting Ramdisk System

[   49.172575] mmc1: new high speed SDIO card at address 0001
Checking for configuration files on mmcblk0p1
Applying configurations from /dev/mmcblk0p1: /mnt/conf.d/50-node-config-save.tar.gz
tar: ./machine-id: time stamp 2016-11-03 17:16:42 is 1478193352.681373228 s in the future
tar: ./pam.conf: time stamp 2017-05-27 15:44:02 is 1495899792.675860228 s in the future
tar: ./dhcp/dhclient-exit-hooks.d/timesyncd: time stamp 2017-07-05 20:31:25 is 1499286635.675276894 s in the future
tar: ./dhcp/dhclient-exit-hooks.d: time stamp 2017-07-05 20:31:25 is 1499286635.674987019 s in the future
tar: ./dhcp: time stamp 2017-07-05 20:31:25 is 1499286635.674800228 s in the future
tar: ./init.d/dropbear: time stamp 2017-05-19 21:41:21 is 1495230031.674368519 s in the future


.... LOTS MORE LINES ....


tar: ./systemd/system/getty.target.wants: time stamp 2018-01-12 00:17:36 is 1515716203.932523726 s in the future
tar: ./systemd/system/sockets.target.wants: time stamp 2016-11-03 17:16:42 is 1478193349.932343685 s in the future
tar: ./systemd/system/network-online.target.wants: time stamp 2016-11-03 17:16:42 is 1478193349.93215006 s in the future
tar: ./systemd/system: time stamp 2016-11-03 17:16:42 is 1478193349.931961935 s in the future
tar: ./systemd: time stamp 2017-07-05 20:31:25 is 1499286632.931794601 s in the future
tar: ./rc3.d: time stamp 2018-01-12 00:17:20 is 1515716187.931630476 s in the future
tar: .: time stamp 2016-11-03 17:16:44 is 1478193351.931470976 s in the future
[   52.455463] ip_tables: (C) 2000-2006 Netfilter Core Team

benhylau · 2018-01-12T07:29:03Z

To identify the config partition possibly look for the boot directory that identifies the partition created by the mesh-orange instead of conf.d.

I also thought about checking for boot directory, but felt that it was an unnecessary restriction. Perhaps we have a board that won't use a file named boot, or I want my conf.d on a USB drive that I can plug into arbitrary nodes and see same behaviour.

So then I thought about making an additional identifier like .mesh-orange-use-me as a sibling to conf.d that I can search for, but then I realized conf.d is that identifier. If it's too generic, then we can rename conf.d to something like mesh.orange.conf.d then there wouldn't be unintended collisions. If two drives actually both contain mesh.orange.conf.d that probably means someone stuck in two SD cards both flashed with the mesh-orange img, in which case it is actually impossible to tell which one they intend to boot. So... we are back to conf.d, although we can debate whether the name is not unique enough.

darkdrgn2k · 2018-01-12T16:22:19Z

I also thought about checking for boot directory, but felt that it was an unnecessary restriction. Perhaps we have a board that won't use a file named boot, or I want my conf.d on a USB drive that I can plug into arbitrary nodes and see same behaviour.

Are we building a production node here or prototype node. The act of plugging a USB with config files into a device sounds like something you DONT want to do in a production model due to being a easy way to "hack" in.

Just a thought.
Maybe have a production and prototype mode?

benhylau · 2018-01-12T17:45:38Z

If you have physical possession of the device, it is no different to plug a key USB or swap an SD card. I don't see being able to pick up configurations from more than one storage device being a security risk.

benhylau · 2018-01-12T20:31:57Z

Does this look right for the FAT partition of a Raspberry Pi 2 image? I stuck a conf.d in there, but the partition doesn't have a boot directory like all the other boards. This is because of the Raspberry Pi bootloader right?

$ ls -l /mnt/
total 56752
-rwxr-xr-x 1 root root    16523 Jan 12 10:27 bcm2709-rpi-2-b.dtb
-rwxr-xr-x 1 root root    17624 Jan 12 10:27 bcm2710-rpi-3-b.dtb
-rwxr-xr-x 1 root root    16380 Jan 12 10:27 bcm2710-rpi-cm3.dtb
-rwxr-xr-x 1 root root    50248 Jan 12 10:27 bootcode.bin
-rwxr-xr-x 1 root root       75 Jan 12 10:27 cmdline.txt
drwxr-xr-x 2 root root    16384 Jan 12 10:27 conf.d
-rwxr-xr-x 1 root root       66 Jan 12 10:27 config.txt
-rwxr-xr-x 1 root root    18693 Jan 12 10:27 COPYING.linux
-rwxr-xr-x 1 root root     2583 Jan 12 10:27 fixup_cd.dat
-rwxr-xr-x 1 root root 52579140 Jan 12 10:27 initrd
-rwxr-xr-x 1 root root  4579248 Jan 12 10:27 kernel7.img
-rwxr-xr-x 1 root root     1494 Jan 12 10:27 LICENCE.broadcom
-rwxr-xr-x 1 root root   659492 Jan 12 10:27 start_cd.elf

hamishcoleman · 2018-01-13T03:18:58Z

boards/common-uboot.mk

+# $1 is the mtools config file
+# $2 is the mtools drive letter
+define uboot_confdir
+    MTOOLSRC=$1 mmd $2conf.d


If you just add this line to the uboot_bootdir macro above, then you dont have to edit every single board config to make it happen. That macro is already described as the place that all needed boot partition actions are done.

I wanted to make conf.d creation as an explicit step because it is not required as part of boot. I know the alternative works, but this was done intentionally to not mix two things, even though they all end up in the same boot partition.

In that case, create a super macro ("uboot_doitall"?) that calls both uboot_bootdir and uboot_confdir - they both take the same params and reducing the amount of lines in each board file makes it easier for others to see and port the important parts.

benhylau · 2018-01-15T01:02:44Z

@hamishcoleman Thoughts re: timestamp warnings? #17 (comment)

benhylau added 2 commits January 9, 2018 19:31

Add configuration management

996494b

Fix config_save script

40e590e

benhylau requested review from hamishcoleman and darkdrgn2k January 10, 2018 00:46

benhylau commented Jan 10, 2018

View reviewed changes

hamishcoleman reviewed Jan 10, 2018

View reviewed changes

Fixes for configurations management

a5b7835

benhylau commented Jan 11, 2018

View reviewed changes

Create empty conf.d in boot partition

4005cf3

hamishcoleman reviewed Jan 13, 2018

View reviewed changes

Group boot partition directory creations

771280b

hamishcoleman merged commit 771280b into master Jan 15, 2018

benhylau deleted the confd branch January 17, 2018 10:02

		@@ -0,0 +1,36 @@
		# Save current configurations to partition containing conf.d

Add configuration management #17

Add configuration management #17

Conversation

benhylau commented Jan 10, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benhylau commented Jan 11, 2018

benhylau commented Jan 12, 2018

darkdrgn2k commented Jan 12, 2018

darkdrgn2k commented Jan 12, 2018

benhylau commented Jan 12, 2018

darkdrgn2k commented Jan 12, 2018

benhylau commented Jan 12, 2018

benhylau commented Jan 12, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

benhylau commented Jan 15, 2018

benhylau commented Jan 12, 2018 •

edited

Loading