Deloitte caught out using AI in $440,000 report | 7.30

The Discovery of Fabricated Citations [0:03]

• A forensic eye was needed to spot errors in a report.
• A law lecturer meticulously reviewed a federal government report commissioned by a consultancy firm.
• Over 20 mistakes were found, including numbering errors in footnotes.
• Many references were recognized as colleagues' names, but some cited books were attributed to colleagues the reviewer had never heard of.
• The reviewer concluded that some references were fabricated.

The discovery of numerous errors, including fabricated citations and misattributed references, was made by a law lecturer scrutinizing a federal government report. This meticulous review revealed a pattern of inaccuracies, leading to the conclusion that some information within the report was entirely fictitious.

It takes a forensic eye to pick up on errors and so it was Chris Rudge who was reviewing a report by the federal government in August.

Context of the Report and AI Involvement [0:51]

• The report was commissioned by the government to review the welfare compliance system following a previous scandal.
• The report had a hefty price tag of $440,000.
• It was revealed that the consultancy firm used AI to produce the report, which caused the mistakes.
• The use of AI was seen as a breach of integrity and trust.

This significant report, commissioned after a major scandal and costing $440,000, was found to have been generated using AI. This revelation sparked concerns about the integrity of the report and the trust placed in such a consultancy firm.

This was no ordinary report. After the robo debt scandal in which the former coalition government unlawfully pursued welfare recipients, the Albanesei government commissioned Deote to review the welfare compliance system. And it came at a hefty price tag of $440,000. But it turns out Deote used AI to produce the report causing the mistakes.

Specific Examples of Errors [1:20]

• The report wrongly referred to a key federal court case and misquoted the judge, with no such paragraphs existing.
• A quote of four or five lines was found to be completely fictitious.
• Another judge's name was incorrect; a speech attributed to Justice Natalie Kuis Perry was actually attributed to Justice Melissa Perry, and the speech itself did not exist.
• Citations were made up, including a book attributed to a law professor that she had never written.

The report contained concrete examples of fabricated content, such as non-existent court case references, entirely fictitious quotes, and incorrect attributions of speeches and academic works, highlighting the extent of the AI's generated inaccuracies.

No such paragraphs exist. And then there's a quote here of four or five lines which is completely fictitious.

AI Hallucination and its Implications [2:29]

• This incident is described as a classic example of an AI hallucination.
• AI hallucination occurs when a model produces output that is incorrect, incomplete, or unexpected.
• In this case, the AI produced citations that do not exist and are not real documents.

The situation is a prime illustration of AI hallucination, a phenomenon where artificial intelligence generates output that deviates from reality, in this instance, by creating non-existent citations and references.

This is a really classic example of an AI hallucinating. So, a hallucination is when a model produces output that is potentially incorrect, incomplete, or not what you would expect.

Financial Scale and Government Oversight Concerns [2:51]

• The consultancy firm reported significant revenue and has secured numerous government contracts.
• The errors in the report raised questions about why governments use private consultants over public services.
• There is an expectation of significant human oversight for reports relied upon for policy formulation.
• The concern is that the secretary or minister might accept the report at face value without detecting errors.

The scale of the consultancy firm's financial success and its numerous government contracts juxtaposed with the report's errors prompt scrutiny over the reliance on external firms and the adequacy of human oversight in government policy development.

The worry is that um the secretary or the minister who are who would be guided by the report doesn't detect the errors and kind of takes it at face value.

Report Correction and AI Declaration [03:36]

• After the errors were made public, the government asked the consultancy firm to correct the report.
• In the reissued version, the firm revealed it had used AI from Microsoft's Azure platform.
• The government has stated that consultants should declare their use of AI and maintain quality assurance.

Following the public disclosure of the report's flaws, the consultancy firm was prompted to issue a revised version, which disclosed its use of AI. This has led to government calls for transparency regarding AI usage by consultants and stringent quality control measures.

In Senate estimates today, the government slammed Deote and said it would move to ensure consultants declare their use of AI and maintain quality assurance.

Broader Societal Impact of AI Errors [04:44]

• The case raises an existential question about how to ascertain truth in the age of "AI slop."
• The distinction between real and fake content is becoming increasingly blurred.
• This is observed in academic writing, paper reviews, and the legal industry, where false quotes and citations are being produced.
• There is a risk of making decisions based on incorrect, hallucinated information, potentially without being able to discern the difference.
• It is difficult to determine if a document has been AI-generated, and tools claiming to do so are highly debated.

This incident extends beyond a single report, sparking a broader conversation about the increasing difficulty of distinguishing genuine information from AI-generated fabrications, a phenomenon termed "AI slop," with significant implications for various professional fields and societal decision-making.

Right now, we're living in the age of what some refer to as AI slop. And unless AI becomes smarter, the distinction between what's real and what's fake could get, well, sloppier.

Company Response and Refund [05:41]

• The consultancy firm declined to specify the percentage of the report generated by AI or comment on the implications for its research credibility.
• The company stated the matter was resolved directly with the client.
• The firm agreed to partially refund the government $97,000.
• There is public questioning as to why a full refund was not issued given the poor quality of the work.
• The firm's justification for the partial refund was that substantial work was done and interrogated, but the final report's quality was unsatisfactory.

The company involved remained largely reticent about the extent of AI usage and its impact on their credibility, while agreeing to a partial refund of $97,000, a decision that has drawn public skepticism regarding the full extent of accountability for the substandard work.

I think plenty of Australians out there are are wondering why there hasn't been a full refund in view of this um very poor quality work.