论文标题
通过联合自我监督的时间领域适应的行动细分
Action Segmentation with Joint Self-Supervised Temporal Domain Adaptation
论文作者
论文摘要
尽管完全监督的动作分割技术最近取得了进展，但性能仍然并不完全令人满意。一个主要的挑战是时空变化的问题（例如，不同的人可能以各种方式执行相同的活动）。因此，我们利用未标记的视频来通过将动作分割任务重新设计为由时空变化引起的域差异的跨域问题来解决此问题。为了减少差异，我们提出了自我监督的时间域适应（SSTDA），其中包含两个自我监督的辅助任务（二进制和顺序域预测），以共同使跨对齐的跨域特征空间与本地和全球时间动态嵌入，从而比其他Dimain Adains Adaptation（DA）进行更好的性能。在三个具有挑战性的基准数据集（GTEA，50 salads和早餐）上，SSTDA的表现优于当前的最新方法（例如，对于F1@25分数，早餐的59.6％至69.1％，从73.4％到81.5％，仅在50SALADS上，以及83.6％至83.6％的gtea and and and and and and and to 83.6％，占GT的83.6％和83.6％）。培训数据以相当的性能，证明了适应跨变体的未标记目标视频的有用性。源代码可在https://github.com/cmhungsteve/sstda上找到。
Despite the recent progress of fully-supervised action segmentation techniques, the performance is still not fully satisfactory. One main challenge is the problem of spatiotemporal variations (e.g. different people may perform the same activity in various ways). Therefore, we exploit unlabeled videos to address this problem by reformulating the action segmentation task as a cross-domain problem with domain discrepancy caused by spatio-temporal variations. To reduce the discrepancy, we propose Self-Supervised Temporal Domain Adaptation (SSTDA), which contains two self-supervised auxiliary tasks (binary and sequential domain prediction) to jointly align cross-domain feature spaces embedded with local and global temporal dynamics, achieving better performance than other Domain Adaptation (DA) approaches. On three challenging benchmark datasets (GTEA, 50Salads, and Breakfast), SSTDA outperforms the current state-of-the-art method by large margins (e.g. for the F1@25 score, from 59.6% to 69.1% on Breakfast, from 73.4% to 81.5% on 50Salads, and from 83.6% to 89.1% on GTEA), and requires only 65% of the labeled training data for comparable performance, demonstrating the usefulness of adapting to unlabeled target videos across variations. The source code is available at https://github.com/cmhungsteve/SSTDA.
