RCS API

Overview

RCS (Rich Communication Services) is a messaging channel that lets your agent send and receive rich, interactive content (images, videos, graphs and location pins) over the same phone number used for SMS. PolyAI supports RCS through Twilio, alongside standard SMS. If a customer’s device or carrier doesn’t support RCS, messages automatically fall back to SMS, so no conversation is ever lost.

Learn more about RCS from Twilio directly: RCS overview and RCS onboarding.

How it works

Starting and holding a conversation

Your customers can reach your agent over RCS in a few ways:

Deep link or QR code. A customer clicks a link or scans a QR code that opens their device’s messaging app with a chat to your agent already started. See Twilio’s guide and Google’s deep link guide.
Existing contact. If a customer already has your agent’s RCS contact saved, they can start a conversation at any time.
SMS fallback. If a customer texts your fallback number over plain SMS, that conversation stays on SMS for its entire duration; it won’t switch to RCS mid-conversation.

After the initial message, the conversation is carried between the user and the agent over the same channel.

Handoff

Handoff is supported on RCS, in the same way as SMS. See Chat handoff integrations.

Media transfer from the user to the live agent will soon be enabled alongside text communication.

Sending messages to customers (outbound)

Outbound RCS messages work the same way as outbound SMS, with two differences:

You must pass the Twilio Messaging Service SID for your RCS sender in the messaging_service_sid field.
The recipient’s number in user_number must be prefixed with rcs:.

If the recipient’s device doesn’t support RCS, the message automatically falls back to SMS. No extra handling is needed on your end.

curl -X POST https://api.us-1.platform.polyai.app/v1/outbound-sms \
  -H "X-PolyAi-Auth-Token: YOUR_API_KEY" \
  -H "X-TOKEN-ID: YOUR_CONNECTOR_ID" \
  -H "Content-Type: application/json" \
  -d '{
    "message": "Hi Jane, your appointment is confirmed for tomorrow at 2pm. Reply to this message if you need to reschedule.",
    "user_number": "rcs:+14155551234",
    "messaging_service_sid": "YOUR_TWILIO_MESSAGING_SERVICE_SID"
  }'

Getting started

SMS must be set up on your agent, and via Twilio before RCS can be enabled. RCS uses the same phone number as SMS and relies on SMS as its automatic fallback channel.

Before RCS can be enabled for your agent, you’ll need:

An agent number managed by PolyAI’s Twilio account. This number is used as the SMS fallback whenever RCS isn’t available.
A registered Twilio RCS sender. This must be submitted to Google for RCS registration. While registration is pending, the sender can be tested using a developer’s test number.
A defined list of languages. These are used to provision the message templates that render components correctly in each language. Currently supported: en, fr, es, de.

Speak to your PolyAI representative. They’ll coordinate the Twilio setup on your behalf. Once complete, you’ll receive a Twilio Messaging Service ID to use for outbound messaging, and your agent’s Advanced Configuration in Agent Studio will be updated with the RCS settings automatically.

Sending and receiving rich content in Agent Studio

Sending attachments to users

Your agent can send rich media to customers, including images, graphs, links, and hosted videos, using the same attachments mechanism as other channels. If the customer’s device doesn’t support RCS, the content falls back to plain text automatically. Attachments sent to the user are not visible in Conversation Review.

More advanced UI components (for example, interactive cards, or richer layouts) are possible but incur an additional cost from Twilio. Speak to your PolyAI representative if this is required.

Receiving attachments from users

Customers can send your agent images, videos, voice notes, location pins, and button taps. Attachments reach the agent in various shapes (see below). The agent can access them and handle them accordingly. When media is shared, the user selects the media and, at the moment of sending, the carrier stores it in a bucket and returns a URL pointer. That URL is what the agent sees and can relay upstream or open. These attachments aren’t shown in Conversation Review directly. Instead, a marker is inserted into the conversation to tell the LLM that attachments were sent:

<metadata>list of metadata type tags attached</metadata>

The marker can contain any combination of:

<media/> — the user sent an image, video, or audio file.
<location/> — the user sent a location pin (latitude/longitude).
<button/> — the user tapped a call-to-action button on a rich card or carousel.

The underlying data is available on the Conversation object at conv.integration_attributes.get("metadata"), keyed by metadata type.

Examples

<media/>

{
  "media": [
    { "content_type": "image/jpeg", "url": "https://api.twilio.com/.../Media/ME123" },
    { "content_type": "application/pdf", "url": "https://api.twilio.com/.../Media/ME456" }
  ]
}

<location/>

{
  "location": {
    "latitude": "51.5074",
    "longitude": "-0.1278",
    "address": "10 Downing St, London",
    "label": "Home"
  }
}

<button/>

{
  "button": {
    "payload": "order_123_confirm",
    "text": "Confirm order",
    "type": "postback"
  }
}

A metadata tag like <metadata><media/><location/><button/></metadata> would map to:

{
  "metadata": {
    "button": { },
    "location": { },
    "media": [ ]
  }
}

Tool example: `parse_metadata`

LLM description: Parse user message when a metadata tag is passed. Request parameters:

Name	Context description	Type
`message`	The full message comprising of any `<metadata>` tag and of any extra text.	String

import json

METADATA_TAG = "<metadata>"

def parse_metadata(conv: Conversation, message: str):
    if METADATA_TAG not in message:
        return {
            "utterance": f"No metadata signature found in message: {message}"
        }

    if not conv.integration_attributes:
        return {
            "utterance": "No integration_attributes found"
        }

    metadata = conv.integration_attributes.get("metadata")
    if metadata is None:
        return {
            "utterance": "No metadata found"
        }

    return {
        "utterance": f"{message}\n{json.dumps(metadata)}"
    }

Opt-in and opt-out

Users can opt in or out of RCS communication. For RCS, PolyAI enables Twilio’s advanced configuration, which allows the agent to receive notifications when the user sends a STOP or START keyword. By adding specific knowledge or tools, you can customise the agent’s behaviour for these cases. The agent always receives the strings STOP and START, regardless of the actual word used by the user (which must be defined in Twilio, along with any localisation). The possible cases are:

Opt-out. The user receives the default info message defined in Twilio. The agent receives STOP, and any reply from the agent is not forwarded to the user, but is recorded in Conversation Review.
Opt-in. The user receives the default info message defined in Twilio. The agent receives START, and any reply from the agent is forwarded to the user and recorded in Conversation Review. You may want to trigger a tool that drops the utterance so the user does not receive a duplicate message.
Between opt-out and opt-in. Any message sent by the user does not reach either the agent or Conversation Review.
Help. When the user asks for HELP, they receive the default info message defined in Twilio. This message does not reach either the agent or Conversation Review.

API reference

Build & deploy

Runtime & data

Monitoring

Legacy

Overview

How it works

Starting and holding a conversation

Handoff

Sending messages to customers (outbound)

Getting started

Sending and receiving rich content in Agent Studio

Sending attachments to users

Receiving attachments from users

Examples

Tool example: `parse_metadata`

Opt-in and opt-out

​Overview

​How it works

​Starting and holding a conversation

​Handoff

​Sending messages to customers (outbound)

​Getting started

​Sending and receiving rich content in Agent Studio

​Sending attachments to users

​Receiving attachments from users

​Examples

​Tool example: parse_metadata

​Opt-in and opt-out

​Related

Overview

How it works

Starting and holding a conversation

Handoff

Sending messages to customers (outbound)

Getting started

Sending and receiving rich content in Agent Studio

Sending attachments to users

Receiving attachments from users

Examples

Tool example: `parse_metadata`

Opt-in and opt-out

Related