HomeHome ArchiveArchive

The state of QGIS Globe

The Region of Umbria, Italy, sponsored 4 days of work to update QGIS Globe for current QGIS versions. Most of the functionality is working again and the globe is now compatible with osgEarth 1.0 up to 1.3. The bad news is, that the globe plugin is not working on Windows with OSGeo4W. It seems that one of the OSGeo4W libraries (GDAL?) is compiled with an incompatible MS compiler version. Christmas holidays are coming…

At least it gives Linux users the possibility to play with the globe using the current development version and do exciting stuff like Oslandia does:

PostGIS 3D demo PostGIS 3D demo

Imagine someone would sponsor four weeks of QGIS Globe work!

New configuration options in QGIS server

In QGIS server, it is now possible to selectively exclude layers from WMS publication. These layers will be available only on the desktop and hidden from WMS clients. Similarly, print layouts can be excluded from WMS publication. Of course, these settings are conveniently accessible from the project properties dialog of QGIS (but you need to have a nightly build or a recent compile):

screenshot

Additionally, attributes per layer can be excluded from WMS or WFS publication in the vector properties dialog:

screenshot

There is also a new request type called ‘GetProjectSettings’. The output of this request is similar to the GetCapabilities output, but with more details and more specific to QGIS:

  • Initial visibility of layers
  • Information about vector attributes and their edit types
  • Information about layer order and drawing order
  • List of layers published in WFS

These new features have been developed in collaboration and with funding from the city of Uster. More details are available on the QGIS server / webclient wiki page

Creating png8 images with QGIS server

For providing maps via WMS over the internet, it is important to generate image files with a small size. Because normally, most of the perceived WMS delay comes from transfering large images files over the internet (and not from map rendering itself). Therefore, QGIS server supports the conversion of png24 and png32 images into png8, therefore generating a file with only 1/3 resp. 1/4 of the original size (but with lower quality). However, until recently, QGIS server was using the default algorithm from the Qt library for conversion to png8, resulting in very ugly png8 images if used with a transparent background and sometimes with shifted colors. Therefore, I decided to implement a better conversion algorithm: the median cut algorithm, which was first described by Heckbert in 1982 ( Paul S. Heckbert, Color Image Quantization for Frame Buffer Display. ACM SIGGRAPH ‘82 Proceedings). This algorithm creates color boxes in 4-dimensional colour space and subdivides at the mean of the largest dimension. It stops if the number of boxes equals the requested number of colors (256 in the case of png8 conversion). First tests show the quality of the converted images are quite good for vector maps. So a performant configuration for a webmap means probably to fetch background rasters as jpg and to overlay vectors as png8 with transparent background.

Let’s see with an example what the conversion from png24 to png8 means in terms of image quality. The first file is a png24 image, 590 KB size: screenshot

And this one is the conversion to png8, only 190 KB size. screenshot

To test the png8 conversion, you need to have the latest QGIS version from git (or a nightly build tomorrow). Then simply replace the FORMAT paramter in the url with ‘&FORMAT=image/png; mode=8bit’.

Howto shrink a remote root ext3 filesystem on Debian wheezy

This howto describes how to resize a root ext3 filesystem on a remote Debian wheezy server.

This howto is not an original work but only an update of an older howto by Stefan @ https://thunked.org/. This version is specifically adapted to a server running Debian wheezy.

DISCLAIMER:

THERE IS A DECENT CHANCE THAT IF YOU FUCK THIS UP YOUR
REMOTE SYSTEM SIMPLY WONT BOOT AT ALL.  I URGE YOU TO TEST
THIS LOCALLY BEFORE USING THIS METHOD ON A PRODUCTION
SYSTEM. ESPECIALLY IF YOUR SYSTEM IS NOT DEBIAN WHEEZY,
SINCE THAT'S THE ONLY ONE I HAVE TESTED.

THE QUICK WAY:

If you don't want to read the whole thing you can only
execute the commands I run and probably skip the
explanations.

I’ve only done this on Debian Wheezy. If you’re using another distro the initrd layout and init scripts may be a bit different. However, I suspect it looks very similar on almost every distro out there. On Debian my root partition is an ext3 partition.

The general idea is pretty simple: you can’t shrink a mounted partition and it’s impossible to unmount or replace your root partition in a live system, so we have to resize the partition before it is mounted. What we’ll do to accomplish this is change the initrd image to make the init scripts resize the root partition before mounting it. This is by far the most flexible and easy method to resize your root partition I could think of. Most suggestions I found on google required you to create separate OS on a new root partition and boot into that, but I did not have any space to create a new root partition on my remote machine.

Unpacking the initrd image is fairly straight forward:

$ mkdir ~/initrd; cd ~/initrd
$ gunzip -c /boot/initrd.img-3.2.0-3-amd64 | cpio -i --make-directories
62631 blocks
$ ls -l
total 40
drwxr-xr-x 2 root root 4096 Sep 11 20:23 bin
drwxr-xr-x 3 root root 4096 Sep 11 20:23 conf
drwxr-xr-x 6 root root 4096 Sep 11 20:23 etc
-rwxr-xr-x 1 root root 6797 Sep 11 20:23 init
drwxr-xr-x 7 root root 4096 Sep 11 20:23 lib
drwxr-xr-x 2 root root 4096 Sep 11 20:23 lib64
drwxr-xr-x 2 root root 4096 Sep 11 20:23 run
drwxr-xr-x 2 root root 4096 Sep 11 20:23 sbin
drwxr-xr-x 6 root root 4096 Sep 11 20:23 scripts

First, we have to copy all the programs we need to resize our partition onto the initial ram disk. For my ext3 file system I need e2fsck and resize2fs. The programs are depending on a few libraries, so you’ll need to copy those to the new initrd image too. Libraries can also depend on other libraries, make sure you recursively check dependencies until you don’t have any missing dependencies anymore.

$ ldd /sbin/e2fsck
    linux-vdso.so.1 =>  (0x00007fff3594c000)
    libext2fs.so.2 => /lib/x86_64-linux-gnu/libext2fs.so.2 (0x00007f7cdddd5000)
    libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f7cddbd1000)
    libblkid.so.1 => /lib/x86_64-linux-gnu/libblkid.so.1 (0x00007f7cdd9a9000)
    libuuid.so.1 => /lib/x86_64-linux-gnu/libuuid.so.1 (0x00007f7cdd7a4000)
    libe2p.so.2 => /lib/x86_64-linux-gnu/libe2p.so.2 (0x00007f7cdd59c000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7cdd214000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f7cdcff8000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f7cde01e000)
$ ldd /sbin/resize2fs
    linux-vdso.so.1 =>  (0x00007fffa216a000)
    libe2p.so.2 => /lib/x86_64-linux-gnu/libe2p.so.2 (0x00007fa01f7a2000)
    libext2fs.so.2 => /lib/x86_64-linux-gnu/libext2fs.so.2 (0x00007fa01f55f000)
    libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007fa01f35a000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa01efd3000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fa01edb7000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fa01f9b0000)

We only copy the libraries that are not yet present in the initramfs. Fortunately in our case there are no recursive dependencies of those libraries.

$ for i in libext2fs.so.2.4 libcom_err.so.2.1 libe2p.so.2.3; do 
     cp -i /lib/x86_64-linux-gnu/$i lib/x86_64-linux-gnu/; 
  done
$ cd lib/x86_64-linux-gnu/
$ ln -s libcom_err.so.2.1 libcom_err.so.2
$ ln -s libext2fs.so.2.4 libext2fs.so.2
$ ln -s libe2p.so.2.3 libe2p.so.2
$ cd ~/initrd/

$ cp /sbin/e2fsck ~/initrd/bin/
$ cp /sbin/resize2fs ~/initrd/bin/

Next, we need to edit the init script. Debian uses busybox in its initrd image so the init script will be interpreted by a bourne shell. If you look through the init script file you’ll find the moment where the script mounts the root file system:

$ cat scripts/local
...
    # FIXME This has no error checking
    # Mount root
    if [ "${FSTYPE}" != "unknown" ]; then
            mount ${roflag} -t ${FSTYPE} ${ROOTFLAGS} ${ROOT} ${rootmnt}
    else
            mount ${roflag} ${ROOTFLAGS} ${ROOT} ${rootmnt}
    fi
...

Simply add in the commands to resize the file system right before the the file system is mounted. Resize2fs in Debian does not want to resize the file system before it is forcefully checked. It may be wise to add the -p or -y flag to e2fsck. -y will answer yes to all questions, this could prevent a hung system but may cause more damage to your files or filesystem. The -p flag will only automatically answer yes to safe operations. resize2fs takes two parameters, the first is the block device that has the ext2 or ext3 file system and the second is the new size you want to give it. By default the size is in blocks, but you can append a unit to change that. ‘K’ for kilobytes, ‘M’ for megabytes, ‘G’ for gigabytes and ‘T’ for terabytes. If you don’t specify a size, it will enlarge the file system to the total size of the partition or logical volume. After adding the commands the init script will look something like this:

    #RESIZEROOTFS MODIFIED!!! DONT RUN MORE THAN ONCE
    _log_msg "Starting e2fsck"
    /bin/e2fsck -p -f -C 0 /dev/sda4 || true
    _log_msg "Starting resize2fs"
    /bin/resize2fs /dev/sda4 100G || true

    # FIXME This has no error checking
    # Mount root
    if [ "${FSTYPE}" != "unknown" ]; then
            mount ${roflag} -t ${FSTYPE} ${ROOTFLAGS} ${ROOT} ${rootmnt}
    else
            mount ${roflag} ${ROOTFLAGS} ${ROOT} ${rootmnt}
    fi

If you have access to the system’s console, you might want to add a “-C 0” to e2fsck’s parameters. That will show you the progress of the fscheck.

There’s one last thing we have to do before re-packing the initrd image. e2fsck and resize2fs will fail if there is no /etc/mtab file available so we’ll have to make sure /etc/mtab exists.

$ touch ~/initrd/etc/mtab
$ cd ~/initrd/
$ find ./ | cpio -H newc -o > /tmp/initrd.cpio
64097 blocks
$ gzip -c /tmp/initrd.cpio > /boot/initrd-resize.img

And lastly, we need to add a new default boot option in grub. In Debian grub’s configuration file is constructed from various bits under /etc/grub.d. The resulting total configuration file is put under /boot/grub/grub.cfg. Open up the grub.cfg file in your favorite text editor and look for the default entry. It’s usually the first one, but may vary. Add a copy of the original entry to /etc/grub.d/40_custom and change the initrd image to the one we just created.

$ cat /etc/grub.d/40_custom
#!/bin/sh
exec tail -n +3 $0
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.

menuentry 'resize' --class debian --class gnu-linux --class gnu --class os {
        insmod gzio
        insmod part_gpt
        insmod ext2
        set root='(hd0,gpt1)'
        search --no-floppy --fs-uuid --set=root 0aa8bc27-17e3-4ae2-a9cf-497ab444970b
        echo    'Loading Linux 3.2.0-3-amd64 ...'
        linux   /vmlinuz-3.2.0-3-amd64 root=UUID=660f79dc-c152-4e15-ad61-7075b42de609 ro  quiet
        echo    'Loading initial ramdisk ...'
        initrd  /initrd-resize.img
}

Now make sure that it’s this entry that will be booted into by default. Set GRUB_DEFAULT in /etc/default/grub to the name of the entry you’ve just created:

$ cat /etc/default/grub
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
# info -f grub -n 'Simple configuration'

#GRUB_DEFAULT=0
GRUB_DEFAULT="resize"
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX=""
...

Finally, you can reboot the system. When it comes back online (if it comes back grin) your file system will be resized. Be aware that e2fsck and resizefs take a long time on big disks. On my system the two took 2 hours for a 1.5T filesystem. Thus don’t prematurely reboot your system if it doesn’t come up again quickly.

Don’t forget to remove the new grub entry, so your file system doesn’t get resized every time you boot.

Original howto at https://thunked.org/general/howto-shrink-a-remote-root-ext3-filesystem-t96.html

Original howto Written by Stefan @ https://thunked.org/

This version for Debian wheezy by Tomáš Pospíšek