Polygon blockchain nodes briefly went out of sync, affecting Explorer, so confusion

DIRECTION (February 23, 4:42 UTC): Correcting the story to say that polygon nodes briefly went out of sync on Wednesday. An earlier version of this story stated that speculation about a possible blockchain failure was unfounded.

The Polygon blockchain ran into trouble on Wednesday after some of its nodes fell out of sync, causing a cascade of problems that hampered performance and knocked popular blockchain explorer PolygonScan offline.

The outage of PolygonScan created the impression that no transactions were taking place, so confusion, and eventually prompted Polygon’s head, Sandeep Nailwal, to tweet that there were “issues” with the blockchain explorer.

“At approximately 8:26 UTC, a few nodes went out of sync. This caused a reaction where some nodes were unable to validate blocks for a very short period of time,” a spokesperson said, referring to the process by which decentralized networks such as Polygon add information to their permanent registration.

Dapp developers who rely on affected nodes struggled to get services back online, according to the founder of an infrastructure company, Rivet.

“While I would hesitate to call it a disaster, it’s likely that more people were affected than they were talking about,” Rivet’s Greg Lang said of the incident.

“There could have been a temporary degradation in network performance,” the spokesperson said, explaining the consequences of some nodes falling behind. All nodes have come back in sync, the spokesperson said, and Polygon is investigating what caused the problem.

Polygon’s problems appear to stem from an “unusually large” block reorganization that occurred two minutes before Polygon said nodes went out of sync, according to Lang. Rivet is a provider of node infrastructure at Polygon.

When operators of a network publish two blocks at once, it creates a “fork” in the chain where there are dueling accounts for the chain’s permanent record. A block reorganization is how the network resolves the discrepancy: it excludes one of the records in favor of what the network considers to be the correct, “canonical” record.

Small block reorganizations are common on Polygon and other networks based on the Ethereum Virtual Machine (EVM), Lang said, and are not necessarily cause for concern. What was different on Wednesday was the size of the conversion: 157 blocks.

“The reorganization was definitely out of the norm in size” for Polygon reorganizations, Lang said. “From Rivet’s point of view, it exceeded our ‘no-touch’ reorg correction mechanisms by at least a couple of dozen blocks.”

Infrastructure providers handle reorganizations differently, Lang said, but Rivet can automatically handle shuffles of about 125 or less. Beyond that, Lang said Rivet must manually restore chain history from backups. He said it probably happened on Wednesday.

In general, Lang said that node operators can be thrown out by blockchain reorganizations and can even “serve stale data” (data from excluded blocks) if they don’t get back on track quickly.

“We were absolutely in ‘all hands on deck’ mode,” Lang said of Rivet’s efforts to fix the problem.

Perhaps the most notable publicly-turned result of the blockchain’s period of instability was an outage at the popular blockchain explorer PolygonScan. The Polygon spokesperson said the network issues “caused Polygonscan to get stuck” because the block explorer uses nodes that had fallen out of sync.

Polygon’s Nailwal tweeted that PolygonScan was having problems and that the OKLink explorer was available as an alternative.