People tracking and other innovative uses for image features found through reinforcement learning

Our recent paper proposes DISK: a new, powerful approach to finding features which can be reliably matched across different images. Thanks to using principles of reinforcement learning, it is very flexible - the scientist merely needs to define a reward for each proposed feature match and the algorithm learns to maximize it. Local features are already widely used in 3d geometry estimation (see the paper); in this project we will ask you to apply this framework to a problem of your choice.

As you can see in a visualization, even though our algorithm was trained on static images of buildings, it can find and track points on humans and other moving objects. One of our suggestions is that the student finds a way to reward it for correctly matching points on the same person across different video frames to obtain features which reliably describe and allow for tracking persons. This project is very flexible: if you want a well defined objective, using DISK to implement a human tracker is one. If you have a different application idea, we’re also happy to look into that. It should be noted that we do not require familiarity with reinforcement learning, but if the student wishes, it is one possible area of work. Depending on the scope of work, this can range anywhere between a BSc semester project to a MSc thesis.

Familiarity with Python is necessary and some experience with machine learning will be helpful.