The potentially perilous promise of food allergen sensors – The Verge

Nima, the parent company for a line of portable food sensors intended for people with allergies and sensitivities, says its culture is based on five core values, the first of which focuses on transparency. “We are more open than private,” Nima writes on its website. “We believe transparency builds trust.”

There’s just one problem: the unanswered questions about how well Nima’s products — a gluten sensor and a peanut sensor — actually work. Nima released its products, which share the same name as the company, without first publishing peer-reviewed, third-party test results. Instead, Nima cited internal data for the accuracy of its gluten sensor, and promised — but did not publish for almost three years — validation testing from an independent, third-party source.

A summary of that third-party validation information didn’t arrive until early September of 2018, more than a year after the gluten sensor was released in January 2017. The internal and third-party data the company did put out doesn’t match its original accuracy claims. The same week the gluten data was made public, Nima launched its peanut sensor, which also hasn’t been publicly peer-reviewed for accuracy. Critics assert that Nima’s missteps with the gluten sensor, along with the absence of peer-reviewed data on the peanut sensor’s effectiveness, raise questions about how reliable the peanut sensor will be.

There’s definitely a market for these products: around 15 million people in the US have a food allergy, and an unquantifiable number of people avoid certain foods for their own health reasons. About 3 million people in the US have celiac disease, according to the FDA; 18 million have a gluten sensitivity, according to Beyond Celiac, an advocacy group. Those numbers are enough to make investors take notice.

Although Nima’s sensors are used by people with medical conditions, they don’t require FDA approval. Nor is the device’s cost — almost $300 for the gluten starter kit, plus $5 for every additional one-time-use capsule — typically covered by insurance. Nima is the industry leader and has few serious challengers. It’s reportedly raised $14 million in funding, including $9.2 million during its series A round in 2016, with plans to develop tests for all eight major food allergies. Nima doesn’t publicly reveal sales data, but claims 100 percent yearly growth in users.

What makes the scientific community members who spoke to The Verge skeptical of Nima’s value for allergic individuals, though, is its practice of releasing its internal tests first and validation data later.

A more rigorous process involves writing up an article based on independent, third-party validation data from testing, and then submitting it to a group of scientists — they’re the “peers” of the peer review — who determine whether the work is sufficiently good enough to be published. It’s not a requirement that every device on the market submit to peer review, nor is it a perfect solution, but it is the norm among pharmaceutical companies and medical device makers. These papers give doctors and patients some confidence about the products they’re using. As molecular biologist Frank Gannon once wrote for the European Molecular Biology Organization’s EMBO Reports journal, “Most scientists will not consider a scientific pronouncement as valid unless it has been approved by this … process, known as peer review. Without such an external seal of approval,” Gannon added, “they would consider any results presented as preliminary, potentially flawed and generally of the same self-serving status as a press release.”

Nima’s critics in the scientific community would readily support the device if Nima sought peer-reviewed, third-party validation first and then marketed second, says Donald Duerksen, Professional Advisory Council chair of the Canadian Celiac Association. But Nima didn’t go that route. Instead, like other companies with products that do not require FDA approval, Nima has opted for a straight-to-market strategy, while seeking third-party validation data on its own timeline.

That practice — in addition to food sampling issues and the seriousness of allergic reactions — makes Tricia Thompson, who runs a website called Gluten-Free Watchdog, especially nervous about the peanut sensor. In an email to The Verge about a month before the sensor’s release, she wrote, “The peanut sensor terrifies me.”

Nima’s co-founders, Shireen Yates and Scott Sundvor, met at MIT in 2013, where Yates was going for her MBA and Sundvor was a mechanical engineering undergrad. Both have dietary restrictions, and Yates says she came up with the idea for a gluten sensor after struggling to find food she could safely eat at a wedding.

Originally, the startup was called GlutenTech, then 6SensorLabs, and finally Nima. After workshopping prototypes, Yates and Sundvor, joined by then-lead scientist and fellow MIT attendee Jingqing Zhang, moved to San Francisco to prepare for the product launch. Even before it was released, the gluten sensor was named one of Time’s “25 Best Inventions of 2015.” By that point, Yates and Sundvor were publicly discussing a move beyond gluten into the food allergy realm.

The concept behind Nima is simple: users stick a bit of food in the one-time-use capsule that goes into the device, then wait a few minutes for either a smiley face, or, depending on which sensor you’re using, a gluten or peanut found result.

The company cautions that the proprietary antibodies that make Nima work ultimately only amount to a “screen,” and “you should always use your best judgment in choosing what to eat and what to avoid.” Because Nima instructs users to put a pea-sized piece of food in the device, any large meal, or a sampling of a dish with multiple parts — like a burger with fries — is impossible to analyze in full, which might explain the “best judgment” disclaimer. That’s not how foods are typically assessed in a lab setting — much bigger samplings are ground up and analyzed with validated testing kits.

The sampling issue is a key sticking point for many of Nima’s critics, including David Stukus, a pediatric allergist at the Nationwide Children’s Hospital in Columbus, Ohio. It’s “kind of silly to me to take the time to test one tiny piece of food you’re about to eat, which likely doesn’t contain any of the allergen anyway, and then make the false assumption that the rest of the food is going to be fine,” Stukus says.

To simplify matters for consumers, Yates and Sundvor say they initially designed their product to be 99.5 percent accurate at 20 parts per million of gluten, the FDA-approved threshold at or above which a food cannot be labeled gluten-free. Food manufacturers seeking a “gluten-free” label for a product will send the product to an accredited laboratory, which will test whether it’s below the 20 parts per million threshold.

Nima is supposed to be a tool that will tell you with near certainty if something is at or above the 20 parts per million threshold (a gluten found result) or below it (a smiley face). In a January 2016 blog post, Nima cited internal testing on prototypes to show it was on track to hit its goal of 99.5 percent accuracy at 20 parts per million. The bottom of the blog post read that 2016 would bring “validation conducted by a third party,” which is the type of corroboration that would help to confirm Nima’s internal testing results. The front page of Nima’s website at the time went even further, announcing that third-party validation was already underway.

In September 2018, Nima told The Verge that the third-party data it was promising in January 2016 was in coordination with the Food Allergy Research and Resource Program (FARRP). FARRP and Nima did eventually work together; FARRP released its findings in 2018. But the origins of that partnership are unclear: Nima was unable to provide a succinct timeline of when they began entertaining the idea of third-party validation with FARRP, as well as who pitched the idea for third-party validation in the first place.

According to emails reviewed by The Verge, in October 2015, Zhang asked Steve Taylor, co-director at FARRP, for help conducting an internal study — not a third-party study — in late 2015. Taylor told The Verge he encouraged Zhang that “instead of testing in [Nima’s] own labs, they should contract with an external lab to do an independent assessment of the device.” (Third-party testing is generally considered more reliable than internal testing.)

Nima says it didn’t seek out an independent evaluation from FARRP until June 2016. According to Nima, it “proactively” went to Taylor at that time. Taylor provided The Verge with an email confirming that a formal research proposal was made in late June of 2016.

That seemingly contradicts what Nima said in its January 2016 blog post, though — third-party testing was not yet underway. Nima says the discrepancy was due to “delayed development timelines, which is a normal occurrence in any development cycle.” The company also said it was their intention to do validation testing that year. As for the apparent contradiction between the statement in the January blog post and the gaps in email exchanges, Nima said in part:

When we talked about 3rd party validation, that means we were evaluating every step of our development process with a 3rd party … When we referenced 3rd party testing in early 2016, our plan was to also do a full system validation with a 3rd party once we had fully designed the product and it was ready for distribution to customers … The start of this process was underway way before we finalized the Nima design. That’s all part of third party testing. FARRP was one of the partners we were considering.

The gluten sensor was actually released to the general public — without peer-reviewed, third-party validation data, which Taylor says he had not yet submitted to Nima — in January 2017, nearly a year and a half after Nima began taking preorders. Consumer reactions were mixed. “My early impressions about the device weren’t positive,” says Thompson, who runs Gluten-Free Watchdog. “I wrote in an email to my husband, ‘I don’t like the device. At most it will test whatever tiny bit is inserted. There is no real ability to homogenize a plate of food in a restaurant.’” Others, however, were enthused at first. “I pre-ordered Nima the second it went on sale,” recalls Jessica Hanson, a gluten-free blogger with celiac disease.

The original device gave three readings — “no gluten,” “low gluten,” and “high gluten.” This system confused consumers, so Nima abandoned it after a few months in favor of the binary result of a smiley face or “gluten found.” By the time the revision was made, Taylor had already been testing the sensor; he’d started in summer 2016 and finished in November 2016. But Taylor says the revision rendered his data useless and he had to start again with the new binary tester. That testing wasn’t finished until August 2017. According to the Wayback Machine, Nima removed the paragraph promising third-party validation from its January 2016 blog post, though the rest of the post was prominently featured on its website until September 2018.

Nima also spent 2016 and 2017 working on its own internal study, which was published online in the Food Chemistry journal in September 2018. The study, Nima’s first publicly accessible, peer-reviewed report, shows a 90.2 percent accuracy rating after factoring in testing errors. But Nima regularly gives “gluten found” readings at the three to 19 parts per million levels, the study found. Nima concluded this was ultimately for the benefit of consumers, who “would be happy for the device to report any gluten.”

Nima is implying that “gluten found” readings between three and 19 parts per million aren’t significant. But since those results fall below the FDA threshold of 20 parts per million, they could be interpreted as false positives. In the study, Nima only defines false positives as positive results below two parts per million. Critics worry that fearful users are at risk of becoming overly reliant on the sensor while avoiding foods containing between three and 19 parts per million of gluten, which is below the FDA’s 20 parts per million gluten-free threshold. To that point, pro-Nima consumer Valerie Cesari told The Verge she’ll use five capsules on a single plate of food — essentially an extra $25 and 15 minutes of her time — to confirm a dish is safe.

“Consumers, particularly in the gluten free space and the food allergy space, are desperate for anything to give them security about the safety of the food that they’re eating, and rightly so,” says Jules Shepard, who runs a gluten-free informational website called GF Jules. “If someone was to think that this was a security blanket of sorts, then they would let down their guard about looking at the full picture of what they were eating and the environment in which it was prepared because they knew they could just test it with the Nima.”

As part of the Food Chemistry internal study methodology, Nima utilized foods with specific levels of gluten, a common validation practice. Here’s the catch: a number of the foods that were tested contained between three and 19 parts per million of gluten, and as long as Nima didn’t register an error message for those foods, it logged the ensuing results as successful data points. To put Nima’s methodology in layman’s terms: Nima could test two identical pieces of bread, each with 10 parts per million of gluten (below the FDA’s gluten-free 20 parts per million threshold). If Nima got a “gluten found” result with the first piece of bread, great — that was considered a true positive, and a successful data point, since the bread does technically contain gluten. But if Nima registered a smiley face — a “gluten-free” result — with the identically glutinous second piece of bread, great — that was considered a true negative, since the bread is below the FDA standard of the test.

Either way, Nima won. It got to show off flashy accuracy statistics on its website that weren’t as significant as implied (and still didn’t meet Nima’s original claims), while its admitted lack of accuracy below the 20 parts per million FDA threshold was buried within the internal paper. In practical terms, a consumer who didn’t thoroughly examine Nima’s testing methodology would have only seen a “gluten found” or a smiley face, without any other context.

Nima also syncs to a smartphone app of the same name that lets users rate restaurants and products, including by marking if something is supposed to be gluten-free, but came back as gluten found. As of February 2019, the app had 4.7 stars out of five in the App Store with 519 ratings. (The website Sensor Tower estimates there have been fewer than 5,000 downloads.) The trouble with crowdsourcing this information, says Adrian Rogers, senior research scientist at a diagnostics competitor called Romer Labs, is “you’ve got brands where if this keeps happening, they’ll drop their gluten-free claims or drop their allergy-free claims and just go to their old ‘may contain’ issue, and you’ve got restaurants that could possibly go out of business because they pride themselves on their gluten-free stuff.”

In March of 2018, the Canadian Celiac Association issued a statement saying it didn’t recommend Nima to people with celiac or a gluten sensitivity. The CCA wanted to see peer-reviewed, third-party data. By then, Nima had submitted its now-published internal study for peer review, and possessed FARRP’s data, which was awaiting publication in a peer-reviewed journal.

Nima showed a PowerPoint summary of FARRP’s findings that March at a Columbia University celiac conference, stamping “confidential” and “proprietary” on slides it presented. What attendees saw was concerning enough for some to start quietly passing around photographs of the results. Among the findings published online in late September: For one out of every three tests where gluten is five parts per million of a food sampling, Nima registers “gluten found.” That sample is below the FDA’s gluten-free threshold. At 10 parts per million, Nima gives a “gluten found” result half the time; that sample is also below the FDA’s gluten-free threshold.

It gets worse. Remember the 20 parts per million threshold, the one Nima had claimed in 2016 would show a 99.5 percent accuracy rating? FARRP found that Nima’s accuracy was instead 79.5 percent, meaning one in five tests at the crucial 20 parts per million level didn’t detect gluten. Nima says that finding was skewed by the device repeatedly failing to detect one type of pasta among a host of other foods in the study; had that pasta not been included, Nima’s accuracy at 20 parts per million would’ve been 87 percent. But the device is supposed to work on all sorts of foods; saying a result shouldn’t count because it’s the wrong kind of pasta undercuts Nima’s claims just as much as the finding does.

There was some good news in the FARRP study: Nima is substantially more accurate at 40 parts per million and above, with an accuracy rate of 98.7 percent; so if a sample contains large amounts of gluten, Nima should be able to alert users with a “gluten found” result. Though that certainly has value, it’s not what Nima originally promised.

After reviewing the FARRP findings, Hanson, the formerly enthused buyer, is “horrified at how often foods at 10, 5, and 2 parts per million test positive for gluten with the Nima sensor,” she says. She vows she’ll never use hers again. That isn’t how all consumers feel, though. Cesari, who was diagnosed with celiac disease in late 2014, says, “The peace of mind — that I can go on a date with my husband, or I can go to a business lunch or dinner and not sit there riddled with anxiety — the benefits outweigh the costs for me.”

Jenny Powell, whose seven-year-old daughter was recently diagnosed with celiac and experiences severe symptoms, including numbness in her legs after ingesting gluten, agrees. “For what I’m expecting it to do, it’s done. It’s performed,” Powell says. “I don’t really need somebody else to test it and tell me it works the way they say it’s supposed to work.” Powell’s daughter hasn’t had a reaction since they purchased Nima.

The medical community has similarly mixed feelings. At Columbia University’s Celiac Disease Center, two leaders in the gluten field, Peter Green and Benjamin Lebwohl, sit on Nima’s scientific advisory board. They recently completed a study, accepted for publication in Clinical Gastroenterology and Hepatology, to see how Nima affected the quality of life and anxiety levels of 15 adults and 15 teenagers. The study was limited in scope, but the results indicated that adults perceived an improved quality of life after three months of varied usage. Teenagers, however, “exhibited no changes,” and 43 percent of teenagers said Nima made them anxious. Teenagers also struggled with the capsules, which was “embarrassing around friends and generally created a lack of enthusiasm.” Almost half of all the participants didn’t “recall the device’s testing limitations” — the gluten sensor can’t detect gluten in fermented items including soy sauce and barley malt — and “this knowledge deficit… would likely be even greater among the general population.”

Anne Lee, a dietician at Columbia who helped oversee the study (and who also has celiac), expressed her personal opinion about the device. “We don’t need it,” Lee says. “There are tools that we can use to go out and enjoy a meal. There are questions we can ask. The device is a machine someone’s making money on. It’s not FDA approved, it’s not 100 percent accurate. I don’t think it’s going to make my life any better.”

In contrast, Green, the Nima advisor, isn’t worried by Nima’s accuracy at low levels of gluten. “I think that there are people who are anti-many things,” he says. “They make an income out of being anti-everything. There’s a lot of this in celiac disease … You people shouldn’t go dampening the development of such devices. All techniques improve, and this is just the start of it. I think it’s innovative that people thought to develop such devices, and hopefully there’ll be more that can fulfill everyone’s needs.”

So far, the peanut sensor’s launch has been similar to that of the gluten sensor: its initial accuracy promises haven’t been verified yet through independent, peer-reviewed publications.

Yates says the company learned its lesson about being more upfront with validation data, so it released results from two outside labs alongside Nima’s own internal peanut sensor study. One lab, Microbac, said the sensor was 99.2 percent accurate. The other, BioFront, said it was 98.7 percent accurate.

But, as BioFront notes at the bottom of its own study, it isn’t an accredited testing lab. And Microbac used food samples formulated by Nima — which calls into question the independence of the analysis. It’s not clear when users can expect substantial, peer-reviewed, independent third-party data. “We’re always looking to get peer-reviewed literature for studies that we do along the way,” says Yates, though she didn’t offer specifics as to when such a step would take place. Reached in a follow-up email, Nima declined to comment on when such a study might be published in a peer-reviewed journal.

Critics have pointed out other potential issues with the sensor. The peanut sensor manual contains a “Use Caution” list of foods and a “Do Not Test” list of foods, both of which, the company notes, “may change as we continue to develop Nima and make updates to the system.” The user manual adds that the sensor may “incorrectly show a ‘Peanut Found’ message when testing foods containing sesame, tamarind, tomato paste/sauce, eggplant, cayenne, paprika, and the NOS energy drink.”

Most of those items, particularly the spices, show up in all sorts of dishes. Nima also admitted in the manual that it can’t accurately test solid chocolate, like chocolate bars. If users try anyway, they may receive a false negative — which, in the case of a peanut allergy, can be deadly. Unlike celiac disease or a gluten intolerance, where reactions are painful but not immediately life-threatening, for some individuals with peanut allergies, allergic reactions can occur within minutes of ingesting even a tiny amount of a food containing peanuts. The symptoms require urgent medical attention, including a shot of epinephrine, or individuals run the risk of their airways tightening, which can lead to difficulty breathing and even death.

There’s a bigger conceptual problem with the Nima peanut device, though: the FDA doesn’t have an established safe threshold for peanuts (or any allergen, for that matter). That’s because allergic reactions — up to and including anaphylactic shock and death — are fairly individual, says James Baker, director of the Mary H. Weiser Food Allergy Center at the University of Michigan. “I think it’s much harder to set limits with allergens than it is with gluten,” he says.

The FDA agrees with Baker. “At this time the FDA has not found sufficient scientific evidence or arguments for establishing allergen threshold levels,” spokesperson Megan McSeveney said in a statement. Even Nima acknowledges this: “Across regulatory agencies there is currently no recommended threshold for peanut parts per million levels in foods,” Nima wrote in a white paper.

According to its website and its white paper, Nima’s peanut sensor is set to detect the allergen at 10 parts per million, which the company says is “the lowest level that is believed to cause” a reaction. That threshold is based on findings presented by the FDA in 2008, which Nima says in its white paper “supported a target detection sensitivity for the device of 10 parts per million peanut in foods.”

McSeveney says the FDA does not concur with Nima’s reading of their work. “That is not accurate,” she wrote in an email response to The Verge, adding that the purpose of the 2008 report was “to describe approaches that could be used to establish thresholds” and that “the allergen doses presented in this report were for illustrative purposes and not for targeting or establishing parts per million threshold concentrations.”

Those in favor of thresholds for allergies argue that the thresholds would apply to the vast majority of allergic individuals. That would mean fewer “may contain” labels on foods, which are a constant source of anxiety for consumers and also cost manufacturers lost revenue. Nima’s former lead scientist, Zhang, vehemently disagrees with the FDA’s aversion to allergy thresholds. “With peanuts, no one has the guts to even put together a threshold,” says Zhang.

Taylor, the only person to date to do peer-reviewed, third-party testing on any type of Nima device, is also a vocal proponent of thresholds for allergens. And at the conclusion of his FARRP study, he was reasonably satisfied with the gluten sensor’s performance. He isn’t overly concerned with Nima’s so-so track record testing foods containing less than 20 parts per million of gluten, and says if consumers use the gluten sensor according to Nima’s instructions, “you should get sufficiently reliable results that a consumer will not get sick.”

But he tested Nima in a lab; consumers in the real world are likely to behave differently. Taylor acknowledged he can’t predict how real-world users handle Nima — and risks from misuse could be especially large with the peanut sensor. “I think [Nima’s] being pretty brave. They’re accepting a fair bit of corporate liability,” he says. “That’s going to be extremely challenging for them. It even more so has to be foolproof.”

As it turns out, Nima isn’t aiming for foolproof, which it also notes via disclaimers on its website. “When I had the idea for [Nima], I knew right away it was never going to be 100 percent foolproof guaranteed,” Yates told The Verge. “But I knew it was going to be a data point that millions of people need in order to make more informed decisions about what they’re going to eat.”

Yates and Nima have often cautioned that the device is just a data point. But Laurel Francoeur, an attorney who focuses on clients with food allergies, believes Nima could be liable in state and federal courts for any accidents or allergic reactions users might experience. “Because Nima does not have premarket approval from the FDA, it cannot benefit from the legal protections offered for products that have received that approval,” says Francoeur.

Despite the controversy around its two existing products, Nima plans to release a milk sensor in 2019. Nima declined to say whether it was pursuing peer-reviewed, third-party validation for the milk sensor, or whether peer-reviewed data would be released before the sensor hits the market.

Nima’s sensors are supposed to help people with serious medical conditions, but the sensors aren’t under the FDA’s purview, and their validation data wasn’t published in a peer-reviewed scientific journal before the sensors went to market. This is the core issue with devices like Nima: there’s no regulatory body or oversight required. That’s a shame, because Nima does appear to have some value for allergic individuals. Just how much, however, has been left to an already fearful group of consumers to decide for themselves.