Property Finder rebuilds AI benchmark, reports 86% lower inference cost

Property Finder says 96.8% of the labels in its vision-AI evaluation set came from the incumbent model’s own predictions. It rebuilt the benchmark, then used it to select Meta’s Llama 4 Maverick 17B Instruct for production on Amazon Bedrock. The company and AWS say the switch improved measured accuracy and lowered estimated production inference cost, though the results come from a joint vendor-customer case study rather than an independent audit.

The joint account says the new setup costs an estimated $2.35 per 1,000 images, 86% less than Property Finder’s previous setup. That figure applies to production inference, not to rebuilding or running the evaluation. Property Finder says its system assesses 11 quality attributes across tens of thousands of property images each day in the Middle East and North Africa.

That inheritance made the test circular: alternatives were scored against answers produced by the model they were meant to challenge. The co-authors also describe character errors and hallucinated brand names in watermark labels, single labels assigned to images containing multiple room types, and human-added categories outside the original specification in the remaining 3.2% of samples.

To rebuild the benchmark, one Property Finder strategy and analytics expert annotated 356 UAE images without access to the incumbent model’s predictions. AWS conducted clarification reviews. The first 256 images became the development set for prompt tuning, with another 100 added later. The team also used a separate 230-image Egypt set for a cross-regional check, but the post does not disclose those results.

The team used Micro F1 as its main metric. It also tracked Macro F1, per-label precision and recall, per-image intersection over union, and exact-match accuracy for nine single-label tasks. Two tasks moved to multi-label output, while prompt iterations continued through version 27 using error analysis, AI-assisted revisions and, for photo classification, MIPROv2.

On the full 356-image UAE set, the co-authors report a Micro F1 score of 0.80 for Llama 4 Maverick with the final prompt, compared with 0.56 for the unnamed proprietary model A. They also say Llama matched or exceeded the incumbent on all 11 attributes and reached at least 90% exact-match accuracy on six of nine single-label attributes. The post cautions that both prompts were tuned on the 256 development images included in the final set, making the scores optimistic. The underlying images, labels, predictions and scoring code were not published with the case study.

The cost comparison has a different limitation. Its 86% estimate uses the previous system’s v21 prompt as the baseline, while the accuracy comparison uses a tuned v22 prompt whose cost may differ. The calculation uses December 2025 US East (Ohio) prices and assumes 9,000 input tokens and 200 output tokens per image. AWS calls $2.35 an upper estimate because the production prompt uses about 7,500 to 8,000 input tokens.

Property Finder says it now runs the model in production across all 11 attributes and has observed 40% to 60% lower inference latency in its own monitoring. AWS’s current model card identifies Llama 4 Maverick as a legacy model and lists March 30, 2027, as its end-of-life date.