As part of our featured section on the evolution of instrumentation, here’s a perspective on chip instrumentation integrated directly into circuitry. Determining if a PCI Express (PCIe link comes up correctly and quickly diagnosing system problems are now much easier tasks with integrated instrumentation in PCIe switch silicon and supporting software.

Anyone who has been in the electronic systems business for an appreciable amount of time has seen several paradigm shifts in the way systems are designed, interconnected, and debugged. One notable change is in parallel buses disappearing and being replaced by serialized, virtualized, and switched connections.

Decades ago, almost every node in a circuit could easily be probed with an oscilloscope. Only five years ago, one could count on the availability of a parallel bus that could be monitored with a logic or protocol analyzer. Today, systems larger than a single System-on-Chip (SoC) consist of multiple SoC-like components interconnected by high-speed serial buses, perhaps through a packet switch. The few visible interconnect traces not hidden under surface-mount technology components or buried in internal PCB layers can’t be touched for fear of Heisenberg.

The case for integrated instrumentation

If complex system problems are to be debugged at all or root cause for simpler problems reached in a matter of minutes to hours (instead of days to weeks), system interconnect switches must now also function as logic and protocol analyzers. Switches also should be capable of evaluating signal integrity at the physical layer, exercising external links at full wire speed, and injecting errors of all kinds to evaluate the system’s ability to detect and recover from such errors. These measures are necessitated by the nearly insurmountable challenges of hooking up external test equipment for these purposes, given the high degree of integration and confines of today’s system packaging.

The 5 Gbps speed of PCIe Gen 2 (see Figure 1) presents additional challenges to component developers and their customers, challenges that will be intensified by the Gen 3 specification under development. The higher frequencies reduce timing and signaling margins overall, making the physical layer more susceptible to the nonidealities of real-world transmission line environments and working against even minimally invasive probing.

To overcome physical layer impairments, SERDES designers utilize the concept of receive equalization. Using equalization, the signal integrity eye opening is meaningfully observable only at the output of the equalizer, buried inside the component. This can only be done with the aid of integrated instrumentation.

Accelerating product bring-up

Experienced and successful design teams have long implemented features that allow them to debug problems that arise during the course of product development. With transistor densities available today, designers can integrate logic analysis capabilities alongside critical mission functions, often without affecting cost. The ability to look at even a tiny subset of the logic nodes in a multimillion gate ASIC can be of inestimable value in quickly reaching the root cause of a component- or system-level bug.

While once reserved for factory use only, these advanced capabilities are now being opened up to customers through software that provides a high-level interface. This interface frees the customer from the need to have detailed knowledge about circuit implementation in order to make sense of the results, while protecting the manufacturer’s intellectual property. At the same time, the scope of integrated analysis tools is expanding from a single component focus to system-wide issues.

While integrated instrumentation is useful throughout the development cycle, its value can be appreciated most readily when considering the issues involved in initial board bring-up. When a system or subsystem consists of several SoCs surrounding a central PCIe packet switch, the switch is the focus of attention after that first power-on. Have the links come up? Is signal integrity as predicted and required? Did enumeration and subsequent configuration succeed? Are the packet flow patterns nominal? The sooner these questions are answered, the sooner the team can zero in on what Murphy has thrown at them and move toward a successful product release. With integrated instrumentation backed up by vendor-provided support software, these answers are readily available.

Using the internal logic analyzer

The first order of business when powering on a new board is to determine if the PCIe links have come up. PCIe switches commonly provide a link status output for each port that can control an LED for visual indication. If a link doesn’t come up, it is extremely helpful to trace the transitions of the Link Training and Status State Machine (LTSSM) defined in the PCIe specification.

A switch with even a simple integrated logic analyzer can be configured to capture and time-stamp each transition of a port’s LTSSM into internal trace memory during a link training cycle. Afterwards, the trace can be read out via I2C or PCIe and analyzed with the aid of supplied software. Such a trace can often quickly discriminate between an interoperability issue, manufacturing problem, and signal integrity issue.

If tracing the LTSSM doesn’t identify the answer, an internal logic analyzer can be pointed first at the output of the SERDES’ serial-to-parallel converter, elastic buffer, descrambler, and so forth in an attempt to pinpoint the malfunction.

While an integrated logic analyzer may provide the ability to probe deep into the guts of an ASIC, only those registers or state machines either close to an external interface or architected as called for in an industry standard will be meaningful to an end user.

Optimizing link performance

Once each link is up, it’s necessary to estimate the signal integrity eye opening and optimize it. Advanced SERDES allow the width of the eye opening to be measured with the help of Built-In Self-Test (BIST) features. This is done using sometimes patented techniques that offset the sampling point from the center of the eye and then determine if this results in bit errors. Links are put in loopback mode and bit error rates tested using internal bit error rate test logic. The debug processor can step through settings of the SERDES equalizers and drive strength options, reporting the ones that work best and even burning them into an optional serial Electrically Erased Programmable Read-Only Memory (EEPROM) for automatic loading at the next power-on cycle.

Software that provides a degree of automation to this measurement and parameter optimization process can be provided (see Figure 2).

With a debug processor connected to the switch’s I2C bus, an engineer has sideband access to all the switch’s internal registers. The developer can peek and poke manually while the application is running to check its progress or cause a script to be executed that compares the actual switch configuration state to the expected one.

The debug processor can configure available performance monitors and then display the collected statistics in real time while an application is running to give a real-time indication of its health. Purely as a matter of convenience, similar access can be provided through the debug port to downstream devices’ control and status registers. When used in this manner, the debug processor acts like the management or service processor used in servers and, indeed, their functions may be combined.

After functional debugging, stress testing and error injection are two necessary parts of product development. A switch’s normal data path is easily modified to function as an exerciser. As a first step, the switch sets aside a portion of its buffer memory, allows software to build packets therein, and then transmits the packets on a selected link. More advanced implementations include counting, looping, and branching mechanisms to support more complex packet streams and add a pseudorandom element to the error injection.

When operating some portion of a switch as an exerciser, the ports involved no longer act as switches. Therefore, this capability is not as unobtrusive as other development acceleration features found in PCIe switches. Nevertheless, the simple availability of a component that can perform error injection and generate a full wire-speed packet stream on up to an x16 Gen 2 link is a huge boon to the product validation community.

Implementing embedded instrumentation

At PLX Technology, integrating these capabilities into PCIe silicon evolved out of a selfish desire to reduce manufacturing costs and facilitate internal debug and validation efforts. It quickly became evident that these capabilities were perhaps even more valuable to customers for the accelerative effect on their product development process. Because technology is advancing to ever-higher interconnect speeds, PLX believes that integrated instrumentation is fast becoming essential, especially for PCIe switches.

Implementing the SERDES eye measurement capability requires close cooperation with both SERDES and SERDES BIST intellectual property vendors, then diligent work by design teams to exploit the capabilities thus provided.

Implementing on-chip logic analysis is enabled by a design methodology that requires Register Transfer Level (RTL) engineers to provide selectable probe output of critical states from each major module in much the same way that they are required to insert verification assertions. This internal standardization allows script development to partially automate the addition of the probe buses that feed into the trigger logic and trace memory into the RTL hierarchy. Standardization also reduces the software effort required to harness this data and shape at least some of it into a form that is useful to an end user.

Integrated instrumentation and supporting software tools are showing excellent potential and becoming more user-friendly as the technology develops. This represents a new and promising direction for PCIe switch vendors and customers – one that can remove major sources of pain and delay from the system development process.

Jack Regula is chief scientist at PLX Technology, based in Sunnyvale, California. An inventor of early switch fabric technology, he has delivered several technical presentations at industry conferences and authored multiple technical articles in leading trade publications. Jack received a BSEE (cum laude) and an MEE from Rensselaer Polytechnic University, an education funded by National Merit and New York State Regents scholarships, as well as the National Science Foundation.

PLX Technology
408-774-9060
[email protected]
www.plxtech.com