Since their inception, field-programmable gate arrays (FPGAs) were often targeted at the largest market segment – the communications industry. Although the major FPGA developers are still focused on communication applications, they are increasingly more focused on the storage and server markets.

But what about the broad industrial market?

Often, the broad industrial market’s requirements are not as performance-oriented or complex as what is required in storage, server, or communications applications. For all the hype around the industrial Internet of Things (IoT) market, it is not clear how engineers can leverage available technology to make its potential a reality. An example of one technology that is seeing growing adoption in the industrial market is audio processing. By pairing the capabilities of an audio processor with the flexibility of an FPGA, many innovative applications can be supported.

An audio processor is just as the name implies – a processor that is optimized to process sound. It often leverages an ARM- based or RTOS-friendly processor architecture, has hard blocks like digital to analog converters (DACs), multiple digital microphone inputs, hardware accelerators optimized for the audible spectrum, and an I2S or SPI interface. An audio processor is normally bundled with software or firmware designed to perform certain echo cancellation or noise reduction functions.

An FPGA uses a gate-based architecture that is ideal for processing signals in a parallel fashion. It also has internal memory, hard multipliers and accumulators, and ample I/O flexibility. Some FPGAs are considered SoCs because they have quad-core, A-class ARM processors, but this level of horsepower is not necessary when the FPGA is paired with an audio processor. The ideal pairing for an audio processor is with a generic, flexible FPGA, or an FPGA that incorporates a microcontroller such as an ARM Cortex-M3. Combining an audio processor with an FPGA of this type, with or without a Cortex-M3, creates an ideal division of labor for many tasks in unique industrial communications and control applications.

Audio listening for smart cameras

One interesting function that an audio processor can perform is audio detection when using two microphones. For example, with the appropriate firmware in an audio processor, the device can determine degree information associated with sound location. If voices or sounds are heard in front of the device (for example, in the due north position), this would be processed as “90 degrees.” If sound was sourced to the right at the northeast location, the audio processor would output “45 degrees.” In addition, the firmware can create a beam in front of the two microphones to attenuate noise source outside of the beam. The beam can be steered in the direction of the interested sound source by leveraging a FPGA.

This type of surveillance camera includes the following major components, also displayed in Figure 1:

1. An image sensor to capture the picture
2. An image signal processor (ISP) to handle the video data
3. An audio processor to clean up the sound path and determine location of the audio source
4. An FPGA to connect to the audio processor and implement a motor control algorithm to turn the camera towards the sound

This type of smart camera could focus more closely on the image where the sound is originating. The FPGA could also be used to bridge to a high-speed Wi-Fi module based on PCIe to stream the image, or it could trigger some warning or alarm function. This approach could also be used to steer a camera and a listening microphone to optimize the performance of a videoconferencing unit. In this application, an audio processor would be used to detect where the sound originates by using the beamformer to listen to who is speaking. Instead of pointing the camera at the sound, the beamformer, under FPGA control, would also be directed to the person speaking.

Industrial IoT sound detection applications

Another unique industrial IoT application enabled by an FPGA paired with an audio processor is maintenance, diagnostics, and failure prevention (Figure 2). Imagine that you had sensitive enough hearing to know when a motor or other moving component was getting weak and about to fail. Relevant examples in the industrial IoT include an elevator motor or an earth-drilling bit. By knowing the sound profile of a weakening motor or drill and monitoring for this audio signature with an audio processor and FPGA, product failures and down time can be prevented.

To implement this type of solution, sound profile firmware is inserted in the audio processor to enable monitoring of the audio signature associated with impending failure. For a motor application, this signature might be the whining sound of a bearing starting to break down or a drill whose audio profile changes to a higher level pitch as it becomes duller and has to work harder. With the sound profile residing in the audio processor, the solution then listens and continually matches the sound against the stored failure profile. Meanwhile, the FPGA talks to the audio processor and communicates to a network or some other peripheral to relay the status. If the FPGA includes an ARM Cortex-M3, it can run a lightweight TCP/IP stack and send information over Ethernet or a wireless standard. Of course, other unique communications capabilities could be leveraged such as CAN bus, USB, or a proprietary protocol.

When the audio processor detects the sound profile characteristic of a weakening condition, it signals to the FPGA and then immediately communicates this information over the network. By catching the failing condition early, the FPGA can also be programmed to trigger a response in the form of a system override. With the example of an elevator, the FPGA could wait until it secures confirmation that the elevator has reached the ground floor and everyone is out, and then communicate to the central control system that the elevator is now out of service. Additional examples where this could be used include automotive electric motors, fluid pipelines, and other industrial IoT applications. If the sound profile is well understood, this approach could even be used to trigger a call to maintenance so downtime would be significantly minimized.

Local audio storage and playback

Solutions pairing an audio processor and FPGA are also useful in applications where it is most effective to implement audio storage and retrieval locally. This includes home automation applications or to secure encrypted audio.

In these examples, the audio processor receives the voice or sound and passes it to the FPGA via an I2S bus. The FPGA then formats the data for storage in an SPI flash or other non-volatile memory. This design would also allow playback from SPI flash through the FPGA to the audio processor. Other options for this type of design include encrypting and decrypting audio for security applications. Alternatively, the FPGA could facilitate communications so the audio would be available remotely.

Recently, Arrow Electronics created a hardware kit to demonstrate the flexibility that an audio processor and an ARM Cortex-M3 FPGA can provide (Figure 3). The Microsemi SmartFusion2+ (SF2+) evaluation kit features a Microsemi Timberwolf audio processor and the SmartFusion2 SoC FPGA. The kit has on-board flash and DDR memory, as well as USB and Ethernet interfaces. In addition, a number of peripheral options can be added by leveraging the Arduino shield connector set and the PMOD interfaces.

Arrow has created a complete HDL and C code reference design for the kit that allows up to four different audio recordings to be stored, and supports playback control. The firmware for both the Timberwolf audio processor and the HDL and C code for the FPGA provide a starting point for exploring solutions that combine an audio processor and a FPGA.

There are a number of unique and compelling applications that can be ideally implemented using the combination of an audio processor and an FPGA. The audio processor performs the task of detecting or listening for an event while the FPGA is used to provide custom responses. Of course, the additional FPGA logic also allows for custom function or other logic requirements such as bridging, hardware acceleration, or protocol communication, all of which can be explored with the availability of a hardware solution, reference design, and audio software.

Ted Marena is Director of SoC/FPGA Products at Microsemi.

Microsemi

www.microsemi.com

@Microsemi_SoC

LinkedIn: www.linkedin.com/company/microsemi

YouTube: www.youtube.com/user/MicrosemiCorp