Text2Traj2Text: Learning-by-Synthesis Framework for
Contextual Captioning of Human Movement Trajectories

The movement trajectory is a sequence of $T$ locations, i.e., $X=(x_{1},\dots,x_{T})$, where $x_{t}\in\mathbb{R}^{2}$ corresponds to a 2-D location where the customer stayed at each time step $t$. Items in contact are the items closest to the customer at each time step, i.e., ${I}=({i}_{1},\dots i_{T})$. Purchased items are the items that the customer purchased, which form a subset of the items in contact, i.e., $\mathcal{P}\subset I$. Technically, it is feasible to collect those data from consenting customers via wireless indoor localization technologies Zafari et al. (2019) used in combination with point-of-sales (POS) systems. Nevertheless, such data collection is hard to scale in practice, as it is difficult to intervene in a retail store currently operating and obtain approval from each customer.

The contextual caption is a sequence of tokens, i.e., $S=(s_{1},s_{2},\dots)$, where $s$ is a token. We assume that each caption is concise, typically spanning a few sentences, and describes various aspects of the customer’s shopping behavior such as their preferences for price versus quality, required quantity, and other characteristics related to item choices (e.g., ready-to-eat, health-conscious).

First, we give a prompt (Fig. 4 in Appendix A) to an LLM for producing contextual captions on three types of information: (i) individual customer’s product preferences (e.g., “loves apple”), (ii) category-level interests (e.g., “interested in fruits”), and (iii) decision-making tendencies (e.g., “have a list of items to purchase”). The LLM’s output also includes the number of items planned to purchase (i.e., purchase quantity) and the person’s purchase consideration. Higher purchase consideration suggests more comparison of products before purchasing, while a lower one indicates a tendency to have pre-determined shopping plan.

Given a prompt (Fig. 5 in Appendix A) that contains the outputs from Step 1 (i.e., a contextual caption and purchase quantity) and item categories in a store, the LLM generates an action plan, a list of pairs of item categories and their expected purchase quantity, e.g., {“fruit”: 4, “meat”: 0, “alcohol”: 1}.

Given a prompt (Fig. 6 in Appendix A), the LLM converts each item category determined in Step 2 into more specific item information, i.e., (i) a purchase list consisting of the name of items planned to purchase, and (ii) an interest list of items that the individual is likely to show interest in. The interest item will contain more items as the purchase consideration is set higher. Also, the number of items in the purchase list may not always match the planned purchase quantity generated in the previous step, as the number of actual purchases can change based on other factors, such as the availability of suitable items in the store.

Finally, based on the purchase and interest lists generated in Step 3, we invoke a trajectory planner to instantiate concrete human movement trajectories on a store map. We first assign ranks to each item category stochastically for each trajectory generation, with the rank reflecting the category’s relative position in the store layout. The rank tendency is predefined based on the store’s layout, where categories located closer to the entrance typically receive a higher rank.

The purchase consideration is again considered here; if it is set high, ranks have higher variances, resulting in more exploratory behaviors. Starting from a fixed starting location $x_{0}\in\Omega$ (e.g., the entrance), the planner generates a feasible trajectory traversing items in the purchase and interest lists according to the category ranks in a store map like the one shown in Fig. 1.

As the input to the captioning model, we translate movement trajectory $X=(x_{1},\dots,x_{T})$, items in contact $I=(i_{1},\dots,i_{T})$, and purchased items $\mathcal{P}$, into textual representations. Importantly, movement trajectories can become lengthy as customers take more time for shopping, and can also contain many mundane moments. Here, we adopt a simple yet effective filtering technique to focus on important events in the trajectories. First, we calculate the displacement between consecutive locations, i.e., $\|x_{t}-x_{t-1}\|$, and extract moments when the individual stopped based on if the displacements are below a predetermined threshold. Then, items in contact at the stopping moments as well as those in the purchase list are simply concatenated: “Trajectory is fruit</s>vegetable</s> ...\n Customer purchase item list is [’Carrots’, ’Beef’...] \n Output:.”

The diversity of training data is crucial for the high generalization capability of learned models. While synthesized trajectories can sufficiently be diversified based on randomized ranks of item categories (in Step 4 of Sec. 3.1), the variety of contextual captions may still be limited due to the expressiveness of the used LLM. To ensure high diversity for the captions, we introduce data augmentation by paraphrasing; for each annotated trajectory, we let the LLM to produce alternative expressions of the caption with similar meanings, and relabel the trajectory accordingly.

Following Sec. 3.1, we synthesized 80 pairs of contextual captions and the corresponding movement trajectories using GPT-4 (OpenAI, 2023b), while assuming a scenario of shopping at a supermarket. See Appendix A for concrete prompts and Tab. 6 for the store map we used. We adopted a classical hierarchical planning framework for trajectory generation; a global planner (probabilistic roadmaps proposed by Kavraki et al. (1996)) first determines a sequence of sub-goals from the current item to the next one, and a local planner (dynamic window approach proposed by Fox et al. (1997)) then produces a collision-free trajectory between the sub-goals. The synthesized data were divided into 64 training and 16 validation samples and augmented by paraphrasing with GPT-3.5 OpenAI (2023a), where the number of added captions from a single original caption was 2, 4 or 8. For example, in the case of adding 8 paraphrases, the total number of training samples becomes $64\times 9$ (where $1$ is the original caption and $8$ is its paraphrased captions).²²2Synthesizing captions is more complex than paraphrasing them, where we adopted GPT-4 for the former task and GPT-3.5 for the latter to consider cost-effectiveness.

On the synthesized data, we fine-tuned the T5-Base model (Raffel et al., 2020) available on HuggingFace³³3https://huggingface.co/t5-base, as its encoder-decoder structure was demonstrated effective for multimodal generation tasks (Xu et al., 2023). All fine-tuning was conducted on a single Tesla T4 GPU using AdamW optimizer with a learning rate of $5.6\times 10^{-5}$, where the batch size and the number of epochs were set to 8 and 5, respectively. The model checkpoint with the BERT Precision score (Zhang* et al., 2020) highest for the validation data was used for evaluation.

We compared our captioning model against the following baselines: (a) T5-Small and T5-Base (Raffel et al., 2020) fine-tuned without paraphrasing-based data augmentation; (2) GPT-3.5 (OpenAI, 2023a), GPT-4 (OpenAI, 2023b), and the open-source benchmark Llama-2-7b-chat-hf (referred to as Llama2) (GenAI, Meta, 2023)⁴⁴4https://huggingface.co/meta-llama/Llama-2-7b-chat-hf. GPT-3.5, GPT-4, and Llama2 were used via in-context learning; following Maynez et al. (2023), a few (1, 2, or 4) samples randomly selected from the training data were given as examples, and contextual captions were generated for the given movement trajectory.

We employed ROUGE (R-1, R-2, R-L) (Lin, 2004) and BERT Score (BS-precision, recall, f1 score) (Zhang* et al., 2020) as evaluation metrics. ROUGE score captures lexical overlap by comparing n-grams and word sequences between generated and reference texts, while BERT Score, which utilizes BERT embeddings, measures semantic similarity.

Tab. 1 presents the quantitative results on 20 synthesized trajectories created in the same way as training/validation data. Overall, our model achieved the best performance even with an order-of-magnitude fewer parameters (223M) compared to the GPT family and Llama2 (over billions). We observe a monotonic improvement in nearly all metrics as the number of paraphrases increases, indicating the effectiveness of our data augmentation strategy. In contrast, T5-Small and T5-Base with vanilla fine-tuning demonstrated quite limited performances. The number of examples presented to GPT-3.5, GPT-4, and Llama2 was critical for their in-context learning ability, but this comes with increased inference costs and limits practical scalability.

Additionally, we investigate how each of the movement trajectories (with the list of nearby items) and the purchased items can contribute to the final performances using the validation dataset. In Tab. 2, we evaluated the following degraded variants: w/o Traj (resp. w/o Item) that removed trajectories (resp. purchased items) from the input; w/ Shuffle Traj (resp. w/ Shuffle Item) that replaced trajectories (resp. items) with those of other samples dataset according to the permutation feature importance method (Breiman, 2001; Fisher et al., 2019). These degraded versions all demonstrated quite limited performances, indicating the necessity of combining trajectories and purchased items for inferring contexts reliably. We also evaluate a more challenging case when the trajectory data are partially perturbed, possibly due to the inaccuracy of indoor localization systems. Our model is robust to such noises, as shown in the table (w/ 5% noise).

We recruited eight participants to collect real human data for our study. The entire experiment consisted of two phases with different tasks. In the first phase, two participants were instructed to create four plausible contextual captions about supermarket shoppers. Before they began, we provided them with three example captions to ensure appropriateness for our task. In the second phase, six participants were asked to create trajectories using a visual interface (Fig. 3) based on 10 randomly selected captions — half synthetic and half created by the participants in the first phase. On the visual interface, the current position of a participant was marked by a green rectangle, with details about the item adjacent to their current location shown in the top right corner and items currently added to their cart displayed in the bottom right. Participants were allowed to navigate in the store map and add or remove adjacent items from their cart using keyboard input. Each session began from a fixed location and ended when participant reached the cashier register, tracking whole movement trajectories and final purchases.

Two distinct store maps were adopted in the experiment to validate the generalization ability of trained models: one used for training data and another as a completely new environment. Participants first completed two pilot rounds on one map to familiarize themselves with the interface and layout, followed by five main rounds on this map for data collection. They then repeated the same process on the other map. The set and order of captions, as well as store maps, were randomized across participants. Each experiment lasted about one hour. Through this experimental procedure, we collected 60 sufficiently diverse trajectory data points from real humans, as summarized in Tab. 3.

Tab. 4 shows the quantitative results for human trajectories data, compared between when ground-truth captions are synthesized or created by human participants. Here we evaluated T5-Small and T5-Base, GPT-3.5/GPT-4/Llama2 each with 4 examples for in-context learning, and our captioning model with 8 paraphrases based on the previous result. Overall, our captioning model generalized well to those human-created data, with acceptably slight degradation of performances. Again, our model demonstrates comparable performance to GPT-3.5/4 and Llama2 despite its much smaller number of parameters. It is inevitably difficult to match generated captions with human-created ground truths exactly at word/phrase levels, as indicated by degraded ROUGE scores. Nevertheless, the semantic consistency measured by BERT Scores remains as high as that for synthesized captions, indicating the practical usability.

Tab. 5 compares the performance between when store maps are seen (i.e., identical to those for training data) and unseen. For all models, we confirmed negligible performance degradation. This is practically beneficial, as major retailers often operate multiple stores that can have different layouts and item availability, where captioning systems should be easy to deploy.