Abstract:
Videoquant (VQ), a startup utilizing a patent-pending statistical-GPT approach, aims to transform the advertising and social video
industries by predicting performant TV commercial and social video (i.e. YouTube) content early in production. In a pilot study with a well-known multinational company having a market cap in excess of $60 billion and marketing investments of roughly $2 billion annually, VQ’s effectiveness in identifying video content likely to perform well was assessed. The pilot involved VQ making outcome predictions for 22 videos based solely on their content, without any data provided by the brand. These predictions were later compared to the brand’s internal performance actuals to evaluate the accuracy of VQ’s predictions.
The test was performed as a blind experiment to ensure accuracy and reliability. The results showed a statistically significant correlation between VQ’s nominal predictions and actual real-world video performance (p-value = 0.039). Videos that VQ predicted would perform better achieved a mean of 339% (4.39x) higher downstream performance with 90% confidence bounds of (1.37x, 7.40x). This outcome suggests that implementing VQ’s technology could result in a cost savings of approximately $2.4M per $10M spent on TV and video marketing, an estimate that may be conservative given the narrow scope of the pilot.
Introduction:
Video production and testing is expensive compared to other marketing channels. High failure rates in TV commercials and social videos (e.g. YouTube) result from speculative decisions and outdated methods like surveys, panels, and focus groups that don’t reflect real-world behavior, leading to costly, ineffective content discovered only after significant capital investments.
Videoquant (VQ), a startup with a patent-pending statistical-GPT approach to this problem, aspires to transform the advertising and social video industries by predicting video content failure and success early in production.
The brand, a renowned multinational company with a market cap exceeding $60 billion, dedicates approximately $2 billion annually to marketing across more than 70 countries, with a significant portion allocated to video content. The brand licensed a pilot of Videoquant’s innovative video content technology to evaluate its ability to predict brand lift and performance metrics for various video content throughout its global marketing efforts.
What Is Videoquant:
Videoquant, a cutting-edge statistical-GPT technique and startup with four pending patent apps, mitigates early-stage mistakes in TV commercials, social videos, and other audiovisual content. Utilizing real-world behavior data from a proprietary database of over 3 million TV ads and videos, Videoquant predicts video content success during early production and generates concepts and materials for new video content with a higher probability of strong performance, helping prevent costly video mistakes.
Pilot Objective and Details:
The primary objective of the pilot was to assess Videoquant’s effectiveness in predicting the performance, including brand lift measures such as ad recall, on the brand’s video content. The pilot involved Videoquant making predictions on 22 videos it had never seen before. The client provided the content for the videos, and VQ analyzed them, producing a score for each one, ranging from 0 to 1 (with 1 indicating a high likelihood of viral success). The mean VQ score for videos in this sector was 0.22. Videos were labeled as high/low predictions based on whether they scored above or below this mean score.
To ensure the accuracy and reliability of the predictions, the test was performed as a blind experiment. Videoquant was kept unaware of the brand’s internal measures and benchmarks until after delivering predictions. This approach eliminated any potential biases and ensured that the predictions were based solely on Videoquant’s proprietary algorithms and ML models.
Quantitative Results:
Videoquant provided predictions in an immutable PDF, ensuring the integrity and transparency of the process. The rationale behind VQ’s classification of a video as high/low performing was not immediately apparent upon manual inspection. However, a deeper investigation of the VQ algorithm, utilizing the computation of Shapley values, provided clarity on the factors influencing these predictions.
Actual downstream results were shared with Videoquant only after the predictions were sent. Results are summarized in the accompanying plot.
The plot demonstrates that videos VQ predicted would perform poorly did indeed perform poorly, with a statistically significant p-value of 0.039.
The below table shows a comparison of the actual downstream performance of the videos across the Videoquant prediction levels.
One video had an actual outlier in VQ’s favor (high actual, high VQ prediction). Therefore, a non-parametric Wilcoxon rank sum was prioritized for this analysis to assess statistical significance due to the test’s robustness against outliers. However, a Welch 2-sample t-test is provided for completeness with the expectation that its p-value is inflated due to this outlier.
The test results suggest statistically significant prediction accuracy was achieved in this pilot. Video content that was predicted to perform better demonstrated significantly higher real-world media performance in downstream actuals, with those predicted to be high performers by VQ exhibiting a mean improvement of 339% (4.39x) compared to those predicted by VQ to be low performers. A confidence interval of (1.37x, 7.40x) was calculated with 90% confidence. It’s worth noting that this level of significance is particularly remarkable given the sample size of 22 videos.
Implications:
It’s estimated that this test result translates to a cost savings of roughly $2.4M per $10M in TV & video spend; this may be an underestimate. This value is based on calculations made from the test results and presumed shifting of video production and media efforts from the predicted low to the predicted high bucket.
However, we believe this impact estimate is likely conservative, and possibly even a floor, for specific reasons as follows:
- In this pilot study, we excluded development of highly innovative video concepts generated by the VQ system, which had higher predicted success probabilities than those in the “high” bucket. Videos with the VQ “high” label had an average score of 0.469 on a scale from 0 to 1 (higher = better). Although this score is significantly better than the average VQ score of 0.22, there is untapped potential in developing VQ-generated concepts with scores between 0.5 and 1.0.
- The pilot concentrated on recognizing patterns from auditory voiceovers within videos and excluded visual and auditory video attributes critically relevant to the business goal. We expect that incorporating more variables and data in future iterations of the technology will enhance performance.
- The VQ system’s performance objective is slightly different from the brand’s video goal, but we hypothesized that the two would correlate. This hypothesis was confirmed during the pilot.
- The downstream actuals were noisy, each video having its own actual margin of error as a result of a sampling scheme used by the brand to quantify the actuals.
The above pilot test results were consistent with Videoquant validation testing before the experiment, which demonstrated statistically significant power on a holdout sample of over 172,000 videos that the Videoquant technology hadn’t seen before. The plot below shows performance of the predictive model evaluated by comparing actual mean outcomes versus prediction means across deciles. Deciles were computed by first sorting the holdout sample of videos from the highest to the lowest based on VQ prediction. Then, videos were bucketed into groups, each representing 10% of the data, in descending order of predicted success. The mean prediction and mean actual values were compared within each bucket to assess the model’s accuracy. A model with strong predictive power exhibits higher actual means in the higher deciles, indicating that it is effectively able to distinguish between high and low-performing videos as we observed here.
The success of this pilot has had significant implications for both the brand and Videoquant. the brand is in talks to expand its footprint, incorporating VQ’s predictions into more of its video marketing strategy over the coming year. Additionally, the brand plans to collaborate with Videoquant for their 2024 Super Bowl TV commercial, a high-stakes advertising opportunity that underscores the company’s confidence in Videoquant’s capabilities.
Future Developments and Expansion:
The success of this pilot represents just the tip of the iceberg for Videoquant. During this pilot, VQ’s predictions included slightly over 1,300 video features across more than 2 million videos. While this may seem like a lot, founder Tim D’Auria believes that the most predictive features, related to visual and auditory characteristics of video, were not included in this pilot. “We believe that the performance we see in this pilot sets the lower bar for where we are heading,” says Mr. D’Auria. The company plans to increase data volume substantially over the coming year, further enhancing its predictive capabilities and solidifying its position as a market mover.
Videoquant currently has four patent applications pending, a testament to its innovative approach and commitment to continuous improvement. As Videoquant continues to refine its technology and expand its dataset, the potential applications for its predictions will grow, benefiting not only the brand but also other organizations seeking to optimize their video marketing strategies.
Broader Implications:
The success of Videoquant in predicting internal video performance measures for the brand without any prior data from the company demonstrates its potential for broader applications in the advertising industry. Its ability to accurately infer closely guarded internal metrics for any brand highlights a powerful and innovative approach to understanding and benefitting from competitor strategies. This capability opens up new possibilities for competitor-funded research and development, enabling companies to gain valuable insights for each dollar spent by rivals’ video marketing. This powerful tool has the potential to reshape the advertising landscape, driving innovation and fostering a more data-driven and competitive environment.