Triller claims some 65 million active users worldwide and is weighing a plan to list its shares publicly.
TikTok, which has at least 100 million in the United States and is one of the fastest-growing social platforms, is battling a Trump administration effort to ban the app because of its ties to China or put it in American hands.
The TikTok suit filed Wednesday seeks to move the case from Texas to California, where Triller is based. The company is asking the court to rule that its app does not infringe Triller's patents.
"A judicial declaration is necessary to resolve the real, immediate and justiciable controversy concerning these issues and to determine the respective rights of the parties regarding the... patent," TikTok's lawyers wrote in the complaint.
Triller's earlier lawsuit alleges that TikTok improperly used a patented system for synchronizing music videos with an audio track.