Vision System Monitors Fishing Vessels at Sea

June 21, 2021

Story

Commercial fishing vessels headed out to sea often bring a human at-sea monitor to record data and create an observer report about the harvest. Sending a person out to sea to perform this task involves a fair amount of downtime due to the ratio of time spent sorting the catch and hauling.

(Figure 1)

Video monitoring systems (Figure 1), however, provide an effective and economical method for enabling the same services without having to physically send an extra person onto a fishing boat.

Verifying the Catch

Prior to going out to sea, commercial fishing vessels must report their upcoming trip and plans to the US government, which contracts out with fishery observer companies. These companies have the option to do at-sea monitoring or electronic monitoring on randomly assigned fishing vessels. Fishery observers collect data from commercial fishing and processing vessels, as well as from shore-side processing plants. The monitors, who are typically professionally trained scientists, gather first-hand data on what is caught and thrown back — which must be verified against the captain’s report. According to the National Oceanic and Atmospheric Administration (NOAA), the data helps with fishery monitoring, fish population assessment, fishing quotas, and fishing and safety regulations, supporting science, conservation, and management activities.

A human observer will spend as much as 70% of their time onboard without much to observe, however, since long periods are spent in transit and with the fishing gear being underwater. Video monitoring systems have become increasingly popular for fishery observation for this reason.

(Figure 2)

While the cameras cannot do all the tasks — including taking physical fish samples — plenty of video monitoring benefits exist. Cameras, for instance, can monitor for 24 hours and see in the dark with infrared imaging technology (Figure 2). And while observation companies must still pay trained experts to review video, onshore, after-the-fact video monitoring typically costs about one-third to half of an at-sea monitoring program.

System Components

(Figure 3)

Integrated Monitoring provides a full technology stack for electronic monitoring by contracting with satellite and cellular companies to specify and integrate the technology onto a vessel. Mobile satellite operator, Inmarsat, provides Fleet Broadband and Fleet Xpress satellite connectivity, with satellite antennas from Cobham Satcomm. The company typically sources video technology from FLIR, Axis Communications, and Hanwha Techwin. Typical models provide 4K resolution at 15 fps recording (Figure 3).

The systems also include wireless temperature, pressure, and movement monitoring sensors from Maretron, and a cellular antenna. These items, in addition to the cameras, all connect to a secure communication server supplied by CoastIPC, which acts as the heart of the system. 

Cloud Connectivity, Capable Computers

Once the company fits a vessel with the appropriate package — which depends on size, type, and fishing method — the system connects the vessel to cloud-based services to transmit captured video. Other video monitoring systems on the market may use digital video recorder units, which involves recording video and taking a disc off the vessel and sending it in for observation. Using cloud services allows a vessel to pull into port and transmit that data wirelessly. So instead of going out to sea and filling up a disc over a week or two, sending it in via FedEx for review, and waiting two or three weeks to review captured video, cloud systems transmit that data within a few hours of docking.

(Figure 4)

The server also enables additional capabilities for the system, including controlling the pan-tilt-zoom cameras remotely to view the whole deck and hone in on active areas of the boat to get a much better idea of the type of fish being caught or handled (Figure 4). Furthermore, machine vision algorithms running on the server also identify critical tracking events, such as catch being brought onboard, and stores that optimized video and sensor data in the cloud.

Edge AI, Crew Conveniences

(Figure 5)

Integrated Monitoring designs AI capabilities, which are typically based on the PyTorch or TensorFlow open-source machine learning libraries, into its systems, making the PC (Figure 5) that much more important. For example, the company trains models on a vessel for scene change detection, such as the product being sorted or people moving on the deck. Additional AI tools that may be deployed into the systems include box-model detection of objects such as fish or people, as well as pose estimation and tracking. Pose estimation helps distinguish human activity, including when crew members lie down to work with fishing gear. Between the location of the person, and the position of the lens and camera, pose estimation helps provide a good understanding of what is going on in a particular scene.

Certain tasks may require hardware beyond what comes in a traditional PC, but CoastIPC offers servers/computers from leading companies, such as Neousys Technology, that can expand and adapt to new capabilities, such as utilizing graphics processing units (GPUs), vision processing units (VPUs), and tensor processing units (TPUs). This makes Integrated Monitoring’s system ready to expand into new AI capabilities, should the need arise.

The server also provides the crew with modern conveniences, which can be a real swing factor when it comes to deploying a system such as the one provided by Integrated Monitoring versus a standalone video monitoring system, which may only provide phone and SMS messaging today. The company equips its PCs with a satellite and 4G cellular gateway and managed firewall, as well as dual Wi-Fi zones on deck for captain and crew. While at sea, a 2G speed connection allows owners or regulators to pull up screenshots and check out the situation onboard to ensure compliance. The 2G connection also provides communication back and forth to the company and access to NOAA weather buoy data on the vessel, auction prices, fish price data, as well as social apps and email.

A Safe, Efficient Option

Video monitoring has become more popular in the United States, for example, because deploying at-sea observers on enough fishing vessels to obtain accurate data on catches and discards has become expensive. Many regional councils have developed plans to move forward with video monitoring to improve the accuracy of at-sea data and provide cost-effective coverage on fishing vessels. This way, as long as the video system is running on the boat, regulators only need to look at 10 to 20% of random samples to determine representative data for a fleet.

(Figure 6)

In New England, Amendment 23 of the Northeast Multispecies Fishery Management Plan went into effect in 2021. This amendment requires 100% observer coverage on all trips but allows electronic monitoring as an approved option to human observers, which makes video-based systems even more important going forward. The process helps identify important details, including fishing location, type of fishing gear, and mesh net sizes used, while also preventing vessels from bringing in illegal species and going beyond the vessel’s annual allowable quota –— which helps with conservation (Figure 6). Ultimately, video monitoring has not replaced at-sea monitoring but instead offers an efficient method to expand observation to cover entire fishing fleets, which previously would have been cost-prohibitive.

Integrated Monitoring has more than 1,000 vessels under contract all over the world, including in North America with projects funded by NOAA, the National Fish and Wildlife Foundation, the United States Coast Guard, and the state of Massachusetts. The company also provides services to both government and corporate fishing industry clients in several international markets, including, Chile, Peru, Mexico, the Maldives, Thailand, and Indonesia.