from dataclasses import dataclass

Quantification

The quantification of behavior is a complex topic, and many algorithms and approaches can be taken to extract different types of information from the data.

Sleep amount

Single point modelling

idtracker.ai returns the X, Y coordinate of the center of each fly over time, keeping track of their identities. We can use this, together with some heuristics that define sleep (lack of locomotor activity for more than 5 minutes) to quantify sleep in the fly hostel.

Analysis should be done using the XXXXXX.npy trajectory files produced by copying the idtracker.ai output. See idtrackerai

Annotation

We use several GUIs to annotate behavior

  • Plain mpv with custom scripts to quickly pin down specific points in space (XY plane) and time on which a behavior of interest occurs
  • SLEAP to annotate the exact position in space of several body parts of the fly
  • deepethogram to annotate what behaviors the fly is performing at each frame of a video

MPV workflow

The required mpv scripts are stored here

  • One set of scripts with entrypoint in make_behavior_dataset.lua is used to annotate behaviors upon right click on any pixel of the video. The script saves the x,y coordinates of the mouse click, as well as the frame number and the video name (experiment date+chunk). This is enough to extract the blob from idtrackerai output. The script also prompts the user for behavior label
  • show_metadata.lua can be used to display temperature, humidity, light reading and zt as computed from the start time of the experiment and the saved reference_hour on the video top left corner. The toggling works by pressing TAB.

mpv should be launched as follows:

# cd to the imgstore folder. It is very important to run mpv on this folder, so different experiments are not mixed up
cd /Users/FlySleepLab_Dropbox/Data/flyhostel_data/videos/2022-04-02_16-09-56/

# launch mpv
mpv --script=~/.config/mpv/scripts/user-input.lua   --window-scale=0.45 000100.mp4

For each behavior the user labels, a new csv is created under experiment_folder/mpv_annotation. These csv have the following data

  • x: int
  • y: int
  • t: int
  • behavior: str
  • filename: str. Example: (2022-04-11_10-24-06_000100)

Once the user has labeled several behaviors, mini videos of the animal of interest performing the behavior are generated by reading the csv and looking up the blobs in the idtrackerai. This is done with https://github.com/shaliulab/behavior_extractor and the following script

from behavior_extractor.main import document_behaviors, read_metadata
EXPERIMENT="YYYY-MM-DD_HH-MM-SS"
data = read_metadata(experiment=EXPERIMENT)
document_behaviors(experiment=EXPERIMENT, data=data, interval=1000, framerate=40, n_jobs=1)