The two key elements to defamation are—
Public: this doesn’t mean mass media or put out there for the whole world; even making the statement to one other person can be “publically” making it, so a bad employment reference can easily meet this particular criteria.
False statement of fact: If it’s not a false statement of fact, it’s not actionable. The “false” part of this requirement is fairly straightforward—either the statement is true or not. If it is true, its truth can raised as defense to a charge or claim of defamation. So if something is true, it does not matter how negative it is.
However, it’s often less clear when something is a statement of fact (potentially defamation) or an opinion (not defamation). For example, take the statement, “Jane Doe is the worst employee I ever had.” That sounds like it’s a factual statement, but it’s actually not; it’s an opinion. It’s an opinion because whether or not Jane Doe is a bad employee or indeed the worst one ever is subjective. One manager might think this, another might disagree, and both statements are equally right or valid.
So getting back to the situation you describe: the first question to ask yourself is, was the statement made truly a fact or could it be considered a statement of opinion? If the latter, then it would almost certainly not be something you could sue over. The second question, assuming that you believe it’s a factual statement, is whether they might be able to prove you did the thing you are accused of? The standard of proof in this context is not “beyond a reasonable doubt,” but rather a showing that it’s more likely than not that you did the thing. If you believe they could do that—especially if motivated by a lawsuit—then they would have a good defense.
If the answer to both questions is “no,” and the statement was negative, was made to a third party, and caused you some loss or damage, you may have a cause of action; you should consult with an attorney.