In my last post about training on multispectral images I was able to improve my classification accuracy on the EuroSat dataset from 75% to 90%, this using a very simple model. While 90% much better there are likelly further improvments possible by improving the model.

When seeking to improve the model there are two overall strategies one might attempt. Either one might try to analyze the problem and try to find a model that nicly adapts to that understanding, this is teoretically nice but the prommise of machine learning is that the computer can try to do some of this work for us. The other approach is to try different combination and see what works. Doing this manually is rather tedious so we can let the computer do this for us. This approach of letting the computer explore different ways of setting up our model is called hyperparameter optimization, where the hyperparameter part is reffereing to things like number of layers, choice of activation functions and the number of nodes per layer, this is to differentiate it from the ordinary parameters which are the weights of the model. The methods of analysis and hyperparameter optimization can of cource be combined, it can be very powerfull to alternate them and using analysis and understanding of the problem to create the broad strokes of the model and then use hyperparameter optimization to find the exact parameter values of that model.

In this article I will extend my earlier code for classifying the EuroSat dataset and show how you can use hyperparameter optimization to help find a better model. I will use a library called Talos to do this but it can be rather easily done without any dependecies as well if there is a reason to.

This article is intended to demonstrate how to write code for hyperparameter optimization, it is not intended to atempt to create an optimal solution to the classification of the EuroSat images, for much better results on that you can read the article EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification by Patrick Helber, Benjamin Bischke, Andreas Dengel and Damian Borth.

To implement hyperparameter optimization we need to define our parameter space and then tell our code to explore it, in our case using the talos library.

In this example we define the parameters with the following python code

And then pass it to talos like this

Note the call to the create_model function. This is where we have moved our keras model definition and it is defined like this

While this is a bit more complex than our model definition was previously note that half the code is just to allow us to vary the number of of convolutional layers, something that talos at the time of writing this does not automate.

As this is running the training over and over again this code will take quite a while to run (for me it took overnight) I think it is prudent to save the results of the run to a file rather than printing them directly. Doing this allows us to analyze the output in new ways without having to rerun the code. To do this I have defined some helper functions to select the parts of talos output (stored in the tt object) to store and write that to disk. The call looks like this

And the helper functions are implemented like this

To analyze the output I made another script I called example_inspect.py with the following code

Running this gets us (among other things) the following table (sorted on val acc):

As can be seen from the above table our best result of this run is 91% validation accuracy, only a small improvement over the 90% we got earlier. This can be either because I was lucky with my initial guess of parameters or because we are changing the wrong parameters in the model in our experiment. As this is meant to demonstrate the technique I will not delve deeper into this in this article. Despite this I hope this gives you an overview of how hyperparameter optimization can be realized.

Our new source now looks like this (example.py) and the inspect code like this (example_inspect.py). The generators code is unchanged.

In my recent post about using Keras generators I was able to achive 75% classification accuracy on the EuroSat dataset using a very simple model. While there is a lot to do in regards to improving the model there is a simple change that can be made without the need for the analysis work needed for an improved model.

In my generators post I elected to use the JPEG variant of the dataset, for reasons of not introducing to many new concepts into that post. Alternatively what can be done is to use the multispectral TIFF images from the dataset, thus gaining access to much more information for the machine learning to base its conclusions on.

This turned out to be a relatively simple thing to do which surprised me as very little informations on this was available online, I mostly found blog posts of people asking how to get it working.

Starting with the code in my post on generators (generators.py, example.py) we can make a simply replace the read_image function and the code will be able to process multispectral images. Code below

What this code does is that it stops using the Keras load_img function and instead uses the Rasterio library to directly read images to numpy arrays. This function will return a 3D array with a depth equal to the number of bands in the image.

Making that change and running the same test as in the generators post we get the following results

As you can see from the program output we are getting a much better result of approximately 90%. We can also see that from about epoch 100 we are mostly oscillating around this value, this tells us that we are likely at the limit for how good our simple model can become, necessitating a more thought out one for better results (potentially more and or better training data might also be needed). We could keep running for more epochs but that would most likely lead to over training.

Our new source now looks like this (example.py). The generators code is unchanged.

I recently wanted to use Keras, a deep learning framework, to solve an image classification problem and ran into an issue. Keras built-in image load functions assumes that my training data is organized in a single folder with a subfolder for each class of images. This is then replicated for the validation data unless Keras automatic validation split is used. In my case the data where spread out over several folders (an artifact from how the data was sourced) and it would be impractical to copy the data which were already taking up a significant part of the total disk space in the development system.

The solution to this is to use Keras generators. There are two kinds of generators in Keras, either a simple python generator using yield or a class inheriting from keras.utils.Sequence. The later one is the more flexible one and what this post focuses on.

My initial attempt did work but was rather messy to use and when I needed to extend it to handle splitting the data into three parts (test,validation and training) doing that in the original design would have been very messy. So I took a step back and figured that I wanted the following operations.

• create empty generator
• add a directory with files to the generator
  this could be extended to add data from other sources or directory structures
• shuffle the data
• split the generator into new generators using a list of split-points (real number between 0 and 1)
• a way to get the class names of the generator
• a way to get the filename of images yielded by the generator

Of these, the key operations are the splitting and mapping of generated images to filenames. The splitting is important as it allows us to control how many and how large sets we are splitting our data into, allowing for training, validation and test sets or more. The mapping of images back to filenames are important as it allows us to use the generators for prediction as well as allowing us to generate lists of images which the network gets wrong for manual analysis of the networks behaviour.

In addition to this we have some additional operations included later as their need became apparent.

• A function to set constructor properties after the fact, such as verbosity
• A function to preload the images into a cache
• Controls for the batch size used
• Controls for restricting the maximum number of images per class each epoch

While not central to the functioning of the generator these functionalities proved needed in practical application.

To create a generator based on keras.utils.Sequence we are required to provide a few methods to get it to work.

Adding our methods we arrive at

When we have these methods we are starting to be able to write useful code. If we adopt the convention that all methods except split, get_filenames and the methods from keras.utils.Sequence will return self we can now do.

Once we have this in place we will not add any more external methods, we will however define some useful properties that the generator will have defined that a user of the generator can access. The primary ones are:

• filenames - a list of all filenames known to the generator
• classes - a corresponding list of class numbers for each filename
• classnames - a list where class names can be looked up from class numbers

These are the ones most useful to access. Some further properties we will define, mostly to configure the behaviour of the generator (using __init__ or the set method) are:

• batch_size - the number of images returned on each call of __getitem__
• verbose - to spam or not to spam stdout
• max_per_class_and_epoch - a limit on how many images of each class to return
• auto_shuffle - if the generator should be shuffled between epochs
• scale - a number to scale all pixel values in an image with
• image_load_function - a function that can load an image into a numpy array
• image_cache - a cache object that can be passed to the image load function

I think most of these are rather obvious, the one I want to comment on are the max_per_class_and_epoch. I added that one after I got problems with the training, turned out that I had many more examples of one of my classes so the training got stuck in a local maxima where it always predicted that class. This option solved that by ensuring that in each epoch the generator will always produce the same number of each class as long as its value is set lower than the number of images in the smallest class in the training set.

I will not go through the implementation in detail, if you are interested you can look at the source yourself. I will however show some examples of how to use the code.

To use the generator some steps are needed and other are probably recommended. The following example shows how to read images from a folder in the same manner as Keras built in image data generator and then split that dataset in a consistent way. I will be using the EuroSAT dataset available at https://github.com/phelber/eurosat in this example.

As can be seen from the code we start by creating the image generator and passing it an image load function (to be defined later) and a scale factor (here used to scale pixels into the range 0-1). We then add a directory with data to the generator by passing a reader function (to be defined) as well as a path to a directory of images. At this point we have a generator capable of being used in training etc.

In the next step we shuffle the generator to avoid the risk of all the images of some classes ending up in the same part of the data when we split into test, training and validation sets. We follow the shuffle by splitting the data placing the data in the range of 0%-20% into the first set, 20%-40% into the next set and 40%-100% into the last set. We then disables verbosity for all sets and store the sets as test, validation and training.

The final step is preloading the images in the validation and training set to avoid slowdowns caused by disk access during training.

To make this work we need to define the functions for reading a directory and for reading the individual image files. We do that using the following code.

The first of these functions reads a single image using Keras load_img function and caches it as well as applying any supplied rescaling.

The second function uses the Keras ImageDataGenerator to get filenames and classes for them from a directory. If the data is stored in some other organisation than the one handled by Keras ImageDataGenerator we only need to supply a function of this type that can read that format to add_dir and we can keep using the rest of the code unchanged and without needing to reorganize data on disk. Also as were the original motivation we are not restricted to one call to add_dir but can add many directories if we have several datasets we want to combine.

Having read the data we can then define a simple model and train a network using this code. (full source here example.py)

Running this produces output as follows and as we can see we are with this basic model still reaching validation accuracies of 75%.

That's all for this post, I hope to write more about machine learning in the feature if I do you should be able to find them using the tags on this post.

All source code for this post

generator: generators.py
example: example.py

I have spent some time coding in python and ran accross the problem of parsing' comand line parameteras and validating them. The python argparse library proved to be great at parsing but at first glance did not provide obvious means for validating the parameters.

It turns out however that there is a feature in the argparse library we can exploit to easily add that.

Consider the following example from the argparse documentation:

Consider especially the parameter type=int to the first argument.

On the surface what this does is passing a function to convert the argument to the desired type which was not what I was trying to do. But considering that this is an arbitrary function call this is a great place for injecting my own validation code.

The naive approach would write a validation function for each parameter and then passing it to the type argument and then do the checks there. This is quite good but it can be improved.

Consider this function

which is then used like this

In this example the situation is actually not improving, we are checking if the argument can be converted to string and if a false value is returned we are trowing an exception. This is a bit redundant as the int function already throws.

But lets consider another example

This is more useful, now we can use standard function to check if input files exists and any other function that already exists and can return a boolean based on a string like a pathname (os.path.isdir comes to mind).

We are still not done thou.

What if we want to check more than one thing about a parameter, or we want to check that the file does not exists.

Enter the following functions

These functions allows us to combine several things to check regarding the same parameter or to negate the value of a checking function.

The code above first checks that the output file does not exists and then for the input parameter checks that the file exists and is a valid zip file (assuming the valid_zip function exists).

That's all for this post, hope you find it useful

Source code for this: create_checked.py

Consider Java code that is based on the rather common pattern of factories, producing instances of some interface you want to use. In the simple case this works fine but what do you do when you want to modify a method in the returned instance? If there where no factory involved you could just inherit from the class anonymously and override that method. But given that the object is created via a factory that option is closed to us.

Luckily there is a solution for us in the Java API, the java.lang.reflect.Proxy classes newProxyInstance method. This however have a rather clunky interface where we are expected to handle Method objects and InvocationHandler instances.

We can do better than that.

After a bit of experimenting I came up with the following API

public class Decorator { @Retention(RetentionPolicy.RUNTIME) @Target(ElementType.METHOD) public static @interface Override {} public static <I, T extends I,D> I decorate(T proxyBase, Class<I> asInterface, D decorations) throws NoSuchMethodException { // ... Implementation } }

Which is used like this

source code: Decorator.java
test case: Test.java

Enjoy