I can't remember where I read it, but I know that Brahms once said himself that he was a joke and the only reason people loved his music is because they hadn't listened to anyone with talent who had come before him.
I think he specifically mentions Mozart.
I might have read it in the collected letters of Franz Liszt, so if that's the case, it's probably bullshit.
Brahms would often jokingly say one thing about his music when the opposite was the case. He referred the the massive and dramatic 10 minute second movement of his second piano concerto as a "little wisp of a scherzo." He also told the publisher of his second symphony, one of his happiest works, that the music should be engraved in funereal black because it was so depressing.
Also, somebody once told Brahms, "You know that you reference Beethoven's 9th Symphony in the first movement of your first symphony?" And Brahms' response was, "Any ass can see that!"