Mercari, Inc.’s research and development organization Mercari R4D (“R4D”) and its team dedicated to generative AI/LLM have published a set of guidelines for speedy product development using LLMs (Large Language Models). The guidelines are based on R4D’s joint research with Osaka University’s Research Center on Ethical, Legal, and Social Issues (“Osaka University ELSI Center”) regarding ethical, legal, and social issues (ELSI).
The guidelines compile information on highly generic use cases as well as associated risk countermeasures and other procedures written specifically for developers engaged in such tasks as applying existing LLM models to products. The information in the guidelines can be utilized as a reference when using LLMs in a company environment or when formulating guidelines.
Summary of the guidelines
You can download a PDF copy of the guidelines here.
Guidelines for Speedy Product Development Using LLMs (available only in Japanese)
Note: These guidelines assume the user has access to ChatGPT (used with an API), provided by OpenAI.
Preview image
Table of contents:
- Precautions
- Risk countermeasures and procedures by use case
- LLM use where the end user is an employee (for providing LLM-generated content externally)
- LLM user where the end user is a customer (for providing an LLM as a service)
- Information that can be included in a prompt
- Terms of Service confirmation and compliance
- Types of risk based on use cases
- Risk countermeasures and procedures by use case
- Details of risk countermeasures
- Related risks
- Details of risk countermeasures
Background of the guidelines
R4D and Osaka University’s ELSI Center have been collaborating on ELSI since 2021 and have advanced the sophistication of R&D ethics reviews and conducted technology assessments for new technologies at Mercari.
Although the number of services that use generative AI and LLMs has increased recently, there is a potential that using these emerging technologies could give rise to risks that service providers do not intend. Therefore service providers are expected to make careful risk assessments in order for customers to be able to use their services safely and securely. To that end, R4D has formulated the “Guidelines for Speedy Product Development Using LLMs” in order to facilitate a speedy product development process. In formulating these guidelines, we compiled a list of risks associated with the use cases of products using LLMs, and over multiple rounds of discussions, we looked frequently at these risks from an ELSI perspective.
Building development processes using the guidelines
Moreover, in order to design safe and secure services, it’s not enough to simply rely on the guidelines alone. R4D believes it is crucial to control risk throughout the entire development process.
Example application:
In the following process, we have built a means of updating the guidelines by making risk assessments while referencing the guidelines that, based on the assessment results of actual cases, determine whether the guidelines require updating.
Comment from the project head (Makoto Tago—R4D Director)
These guidelines were formulated over multiple rounds of discussions as part of our joint research with the Osaka University ELSI Center. Beyond the creation of these guidelines, the Osaka University ELSI Center is an organization that takes a broad approach to its co-innovative activities to resolve future social issues with product development by examining projects from the issue discovery phase and forming research and development teams as needed, fostering ties across research fields.
Going forward, R4D will build relationships across industry, academia, and government to connect communities, expand its alliances, and use the power of science and technology to address complex social issues.
Future development
These guidelines focus on the usage of existing models and do not support points involving the usage of fine-tuning or the creation of new models. In addition, as the usage of LLMs becomes more and more popular across the globe, risks that have not yet emerged will begin to appear. R4D’s support will not end with the formulation of this document, and we intend to reiterate the guidelines through periodical updates as circumstances dictate.
R4D’s ELSI research
See our webpage for more information about our work on ELSI: https://r4d.mercari.com/en/elsi/