AI-Powered Species Detection and Measurement in the New England Groundfish Fishery
Client: NOAA
The National Fish and Wildlife Foundation, along with NOAA, Teem.Fish and Lynker Analytics have worked together to produce machine learning models for the purpose of detecting, speciating and length measurement of various groundfish species in the New England fishery. With the core goal being able to more efficiently perform electronic monitoring on fishing vessels to gain greater understanding of the current distributions of species caught and their age brackets.
Presently, all electronic monitoring for the purpose of speciating and aging fish is done manually by marine biologists watching long video feeds of CCTV imagery captured on fishing vessels part of the EM (electronic monitoring) program. This is very labour intensive and time consuming which limits the amount of video data that can be analysed.
Species and age data is essential for marine biologists to monitor the health of fish stocks within fisheries. Because of this Lynker-Analytics teamed up with Teem.Fish with research funding from NFWF (National Fish and Wildlife Foundation) in order to utilize Teem.Fish’s catalogue of existing on board analysed CCTV footage to build machine learning models to automate this process. Hoping that by automating a large part of the video analysis done scientists currently would allow for expansion of the EM program to more vessels and to significantly reduce the cost of the current operation.
As all data used in this project is private this is an AI generated image that shows a mock inference on the type of data that was worked with.
In order to facilitate the development of computer vision machine learning models for this task Teem.Fish provided marine scientists to act as annotators on frames of data extracted from the catalogue of on vessel footage. Lynker-Analytics provided a training pipeline based on the Tator annotation tool to allow these scientists to annotate data and the models to be trained. Ultimately around 13,000 image frames were annotated with bounding boxes, keypoints and species. On this a variety of Yolov7 Keypoints and segmentation models were trained.
Confusion matrix of all trained-on species, normalized by true predictions.
The results of this research were promising. With a pipeline for detecting, speciating and determining the length of fish being built with a model for segmenting the measurement ruler, a model for keypoints, detection and speciation of fish and a tracking algorithm called SORT being used. This pipeline allowed for the fish to be detected, classified and tracked across video frames. While the ruler detection allowing calibration of the pixel keypoint length of the fish to be turned into a centimetre measurement.
There were many learnings from this research, particularly that although our pipeline managed to achieve a 91% F1-score across its tasks and a mean absolute length estimation error of around 6% this could not be guaranteed when used on new vessels. The sheer variability in quality of input video data meant that in many cases our model performed well (well the AI generated example image) but the camera placement, image quality, weather conditions and many other variables created extensive challenges where the pipeline would fail.
This is caveated by the fact that the data used is from general CCTV footage on vessels where cameras were not specifically setup for the monitoring of fish, merely picked as the best view available. As such the results obtained are very promising for the integration of AI in EM in future.
