Inside the R4D Project Selected for GENIAC: Automating Mercari’s Listing and Inspection Tasks with AI Robots

R4D is pleased to share that one of our research projects has been selected for the Generative AI Accelerator Challenge (GENIAC), a program spearheaded by the Ministry of Economy, Trade and Industry (METI) and the New Energy and Industrial Technology Development Organization (NEDO) to strengthen the development capabilities and social implementation of generative AI in Japan.

In this article, we’d like to delve into the research and look at where it’s headed.

Using AI robots to automate listing and inspection for Mercari’s crossborder transactions

At its core, this project aims to automate the listing and inspection tasks behind Mercari’s crossborder transactions.

Mercari is working to expand crossborder transactions so that goods of all kinds can circulate across borders. As part of these efforts, in September 2025, Mercari launched the Mercari Global App, a single worldwide app designed to allow overseas buyers to more easily, safely, and securely buy from Mercari users in Japan. The app is currently available in Taiwan, Hong Kong, and the US, and we aim to expand it to more than 100 countries and regions over the medium to long term.

Press release: Mercari Launches the Mercari Global App in the United States

One of the key pieces of infrastructure supporting this crossborder growth is Mercari’s inspection and shipping facility. To facilitate safe and secure crossborder transactions, items purchased from overseas buyers are first routed to Mercari’s facility in Japan, inspected there, and then shipped abroad.

Right now, all of this work is done by hand. This project aims to automate the work using physical AI, making our logistics operations more efficient.

The difficulty of handling a vast variety of items

Let’s look at what this inspection and listing work actually involves, and why it’s difficult to automate with robots.

In the inspection and listing process, items that arrive at the facility are taken out of their boxes and photographed from multiple angles. Inspectors check each item for scratches, stains, and other damage, then compare them against the information on the original listing page to confirm there are no discrepancies between their actual condition and the description stated on the listing page. Finally, each item is packed into a box suited to its size and shape.

As an example, here’s what the process looks like to inspect a pair of shoes:

1. Open the box
2. Unfold the tissue paper
3. Take out one shoe
4. Check the shoe from every angle (take photos)
5. Put the shoe back
6. Repeat steps 3-5 for the other shoe
7. Fold the tissue paper back over
8. Close the box

While these may seem like simple tasks for a person, engineers run into a wall when automating them with robots because the work is hard to turn into a fixed set of instructions.

Mercari’s inspection and shipping facility handles items across a wide range of categories, so the items received vary widely in shape and condition. Sometimes the work involves soft goods like fabrics and bags; other times it requires inspecting buttons, zippers, and similar parts.

This kind of non-standard work (handling a wide variety of irregularly shaped objects, especially soft ones) is extremely difficult to automate with conventional model-based robot control, where engineers program a set of predetermined movements and the robot follows them exactly.

In recent years, however, a new paradigm in robot control has emerged: Vision-Language-Action (VLA) models. A VLA model is an AI model that interprets visual input from a camera together with natural-language instructions from a person in order to generate a robot’s movements in physical space. This has dramatically widened the range of tasks that can be automated with robots.

That said, when we consider applying current VLA models to our inspection and listing tasks, we run into the following three problems.

• The robot’s actions rely on visual input from cameras alone, so it can’t finely modulate force based on how an item feels in the hand. For example, it’s difficult to know exactly how much force is needed to open and close a bag clasp with just visual input.
• If the live camera angle differs even slightly from the images that the model was trained on, the trained robot can no longer perform the task properly.
• Collecting data by teleoperating the robot is inefficient.

Three approaches to overcome the limits of current VLA models

To tackle each of the three problems mentioned above, we’re working on the following three technical approaches. The goal is to move past the limits of current VLA models and build a physical AI foundation that can automate inspection and listing tasks at Mercari. ⁠⁠

(1) Extending VLA into a multimodal model with tactile and force sensing
By using a robot arm fitted with force and tactile sensors, we can train the model on contact states and force levels that visual input alone struggles to capture. Pairing that data with visual and language input, we aim to build a physical AI model capable of tasks that require fine force control. ⁠⁠⁠⁠

(2) Developing camera-aware VLA that is robust to changes in viewpoint
Current VLA models depend heavily on viewpoint: reposition the camera and performance drops. Our approach is to use AI to take whatever view the camera has at inference time and transform it into a synthesized image that looks like the view the model was trained on. By making inference-time images resemble their training-time appearance, we address VLA’s vulnerability to changes in camera placement.

(3) Building a data collection platform that captures how the work feels to a human
Using dedicated interfaces such as smart glasses and universal manipulation interfaces (UMI), we’ll build a platform that collects first-person video synchronized with tactile and force data. This will allow the model to learn skills that teleoperation alone can’t provide. Our goal is a foundation for building physical AI models that can generalize across embodiment gaps. ⁠⁠

To try out these three approaches, we will begin field trials with item categories that have less variation, such as shoes and trading cards, and then expand over time to categories involving deformable items, including tops, bottoms, and bags.

How would AI robot listing and inspection benefit Mercari?

(1) Fully automating inspection and listing tasks and establishing a next-generation physical AI foundation
Inspecting and listing non-standard items is essential across logistics, manufacturing, retail, and other industries, yet it has been among the slowest work processes to automate. Fully automating it would cut labor hours and raise productivity. Beyond that, we believe the multimodal VLA and camera-aware VLA models developed through this project (models that integrate force and tactile sensing) could become a physical AI foundation applicable to physical tasks in any industry, helping strengthen the international competitiveness of Japanese industry as a whole.

(2) Expanding crossborder transactions and contributing to economic growth
Anime, comics, games, and other parts of Japanese hobby and entertainment culture are in high demand worldwide, and Mercari’s crossborder gross merchandise value (GMV) has grown 19 times over the past four years to roughly 110 billion yen. Making inspection and listing fully autonomous and more efficient through physical AI would give that growth a powerful engine.

(3) Accelerating resource circulation
By accelerating crossborder transactions, we will help expand resource circulation across national borders. Looking a little further ahead, if household robots become more common, the physical AI models behind these inspection and listing tasks could enable listing to be automated in the home. We believe that would put more of the resources sitting idle in people’s houses back into circulation, bringing us closer to a society that throws less away.

(4) Advancing decarbonization and reducing environmental impact
Through the real-world adoption of AI robots, we can automate and optimize tasks inside inspection and shipping facilities, using space more efficiently and shortening working hours. We believe this will help reduce the amount of gas and electricity consumed per unit of floor area for indoor climate control.

This concludes our overview of the R4D research selected for GENIAC: “Research and Development on Leveraging AI Robots for Listing and Inspection Tasks.”We’ll keep sharing more about this project and R4D’s other work, so stay tuned!